CentOS/RHEL 6u5 with Two-Factor Authentication Google Authenticator and SELinux

As already announced on several Social Media Platforms, I got Google Authenticator PAM-module enabled on a Centos(/RHEL) 6u5 box.

This implementation includes:

  • SELinux can run in enforcing mode;
  • Certain IP-ranges or users in certain groups can be excluded from Two-Factor Authentication;

Please note that it does not require the Google Authentication Service, the Google Authenticator is just a PAM module that enables HMAC-Based One-time Password (HOTP) algorithm specified in RFC 4226 and the Time-based One-time Password (TOTP) algorithm specified in RFC 6238, which is license under the Apache License 2.0.

Installing Google Authenticator PAM Module

First step is to install the (additional) required packages; just to be sure they’re there…

# yum -y install git pam-devel

Download and compile the code:

# cd $HOME

# git clone https://code.google.com/p/google-authenticator/

# cd $HOME/google-authenticator/libpam

# make

# make install

Modify /etc/ssh/sshd_config so the following setting is active:

ChallengeResponseAuthentication yes

UsePAM yes

But also disable pubkey authentication, to avoid bypassing 2FA (one of the nice caveats I run into)

PubkeyAuthentication no

Restart the SSH Daemon

# service sshd restart

Change the /etc/pam.d/sshd so it has the following contents (please note that the secret is stored in the $HOME/.ssh folder, this has the correct SELinux Context):

auth [success=1 default=ignore] pam_access.so accessfile=/etc/security/group-2fa.conf
auth required pam_google_authenticator.so secret=${HOME}/.ssh/google_authenticator
auth required pam_sepermit.so
auth include password-auth
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session optional pam_keyinit.so force revoke
session include password-auth

Now configure the /etc/security/group-2fa.conf, this file controls who/what are exempted from 2FA:

# This file controls which exemptions are made for
# disabling two factor authentication
# Users that are member of the usergroup no2fa are
# exempted of the requirement providing 2FA
+ : (no2fa) : ALL

# And we also trust the systems from the Subnet
# This subnet also contains hosts that are very secure
+ : ALL :
# Keep this line, to enforce non matching entries
# to enforce 2FA
– : ALL : ALL

Setting up the users

I created two users, one who is member of the no2fa group and one not.

# id pieter
uid=500(pieter) gid=500(pieter) groups=500(pieter)
# id testuser
uid=502(testuser) gid=502(testuser) groups=502(testuser),503(no2fa)

$ ssh [email protected]
Password: ********
Password: ********
Password: ********

As you can see, without luck… /var/log/secure gives you the following errors:

Jun 18 10:11:41 centos-testvm sshd[2896]: error: PAM: Cannot make/remove an entry for the specified session for pieter from workstation.example.com
Jun 18 10:11:41 centos-testvm sshd[2906]: pam_access(sshd:auth): access denied for user `pieter’ from `workstation.example.com’
Jun 18 10:11:41 centos-testvm sshd(pam_google_authenticator)[2906]: Failed to read “/home/pieter/.ssh/google_authenticator”
Jun 18 10:11:41 centos-testvm sshd[2898]: Postponed keyboard-interactive for pieter from port 16055 ssh2
Jun 18 10:11:42 centos-testvm sshd[2898]: Connection closed by

This is caused because google-authenticator is not configured yet…

The user testuser can login (which is in the no2fa group): 

$ ssh [email protected]
Password: ******
Last login: Wed Jun 18 09:51:05 2014 from workstation.example.com
[testuser@centos-testvm ~]$

So now we have to configure google-authenticator for the user pieter.

# sudo su - pieter
$ google-authenticator --label=${USER}@example.com --time-based --disallow-reuse --force
--window-size=6 --rate-limit=3 --rate-time=30 --secret=${HOME}/.ssh/google_authenticator
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/[email protected]%3Fsecret%3DABHNLFFWEOJPPL5QQ
Your new secret key is: DABHNLFFWEOJPPL5QQ
Your verification code is 408011
Your emergency scratch codes are:

Now the user pieter can log in using 2FA:

$ ssh [email protected]
Verification code: [Token]
Password: ********
Last login: Fri Jun 20 11:30:37 2014 from workstation.example.com
[pieter@centos-testvm ~]$

Customizing CoreOS images

For quite a while I’m impressed by the Docker and CoreOS projects and it has been quite a while on my todo list to look into it…

Since I’ve access to some playground with some old Workstations… I decided to start playing around with it, using PXE boot (this was already set up on that environment).

So I followed the instructions for PXE as described on the CoreOS PXE Boot page, although it kept complaining about an “invalid or corrupt kernel image“, while the checksums (MD5/SHA1) were OK. Since the TFT server is running on RHEL5, I did had an old version of pxelinux, so after the downloading the latest syslinux binary from kernel.org the system booted.

After I had the system booted the system ‘alerted’ me on the fact that the test-environment is using an MTU of 9000 (JumboFrames)…

This could not be fixed by the cloud-config configuration over HTTP method as far as I consider… because the cloud-config is loaded by the OS and therefore it requires up-and-running network-interface (with a correct MTU set);

So I  had to modify the CoreOS initrd, to update the MTU in /usr/lib/systemd/network/99-default.link:


So the we need to unpack the initial ramdisk.

Unpacking the CoreOS Ramdisk

Step 1) Create a temporary location in /tmp:

# mkdir -p /tmp/coreos/{squashfs,initrd,original,custom}

Step 2) Download or copy the ramdisk /tmp/coreos/original:

# cd /tmp/coreos/original/
# wget http://storage.core-os.net/coreos/amd64-usr/alpha/coreos_production_pxe_image.cpio.gz

Step 3) unzip the ramdisk:

# gunzip coreos_production_pxe_image.cpio.gz
# cd ../initrd
# cpio -id < ../original/

Step 4) Unsquash the squash filesystem and move the original-container:

# cd ../squashfs/
# unsquashfs ../initrd/usr.squashfs
# mv ../initrd/usr.squashfs ../usr.squashfs-original

Please note… that you need at least the squashFS 4.0 tools… but you can download the source and compile the binaries (at least it works on RHEL5).

And now you can access the unpacked image via /tmp/coreos/squashfs/squashfs-root and perform modifications, but please use the path minus the usr-prefix and relative to /tmp/coreos/squashfs/squashfs-root. So summarized:

/usr/lib/systemd/network/99-default.link can be found in:

So hack around and apply modification where needed.

Packing the CoreOS Customized Ramdisk

Now we have to repack the ramdisk, so we can load it…

Step 1) Repack the squashfs 

# cd /tmp/coreos/squashfs
# mksquashfs  squashfs-root/ ../initrd/usr.squashfs -noappend -always-use-fragments

Please ensure you use squashfs tools 4.0!

Step 2) Make it all a cpio archive and zip it

# cd /tmp/coreos/initrd
# find . | cpio -o -H newc | gzip > ../custom/coreos_CUSTOM_pxe_image.cpio.gz

Now boot it and use the custom image as initrd. 


Investigate disk usage

Recently I had to investigate the usage of a very big volume… with a lot of data-files, owned by several users.

I started with “agedu”, but somehow I was not able to get the information I needed.. so I started using find with stat and put everything into MySQL.

So the first step was to do the following find command:

# find /nfs/bigfiler -exec stat –format=”%F:%n:%s:%U:%u:%G:%g:%X:%Y:%Z” {} ; > /scratch/big-filer.info

Create a table in MySQL:

CREATE TABLE `pieter.bigfiler_content` (
`fileid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`file_type` char(32) NOT NULL,
`filename` varchar(512) NOT NULL,
`size` int(11) NOT NULL,
`user` char(16) DEFAULT NULL,
`uid` char(16) DEFAULT NULL,
`groupname` char(16) DEFAULT NULL,
`gid` char(16) DEFAULT NULL,
`time_access` datetime DEFAULT NULL,
`time_mod` datetime DEFAULT NULL,
`time_change` datetime DEFAULT NULL,
PRIMARY KEY (`fileid`),
KEY `idx_file_type` (`file_type`) USING BTREE,
KEY `idx_user` (`user`) USING BTREE

Load the data into MySQL:

LOAD DATA INFILE '/scratch/big-filer.info'
INTO TABLE pieter.bigfiler_content
(file_type, filename, size, user, uid, groupname, gid, @time_access, @time_mod, @time_changed)
SET time_access = FROM_UNIXTIME(@time_access),
time_mod = FROM_UNIXTIME(@time_mod),
time_change = FROM_UNIXTIME(@time_change);

And now you can run nice queries to analyse the data 😀 

YUM repo fails with koji/mock when base URL is used

Recently I started playing around with Koji for package building.

Everything was set-up pretty fast… but the first attempts to build a build-root failed…

After some troubleshooting I found the cause…

If you use the repo-data as available on an installation ISO (served via a webserver) and use mergerepos the location of an RPM will look like:

<location xml:base="CentOS" href="pam-"/>

The entry as available on the ISO is:

<location xml:base="media://1330913492.861127#1" href="CentOS/pam-"/>

The way to workaround this issue, is be recreated the repodata using createrepo:

# createrepo -u ${WEBSERVERPATH}/new-repo/ /media/cd/

And then the entry will look like:

<location xml:base=”” href=”pam-”/>

The bug can be found in bugzilla.

SSL Chain

I’ve ordered via http://www.cheapssls.com/ a simple SSL-certificate signed by Comodo for use with apache… although a lot of browsers (Firefox on Mac OS X, all browsers in Linux) didn’t accepted it (CA was not know…)
After some discussion with …

Block mail from certain countries with sendmail

If you have your own MTA running… you are probably known with the spam-problems… Once you’ve tuned the filters, you have to do it again… because a new spam-run comes in. I also blocked whole /8 subnets in different countries (India/China/…)… but that is not a “real” solution… aka I want to block the whole country…

 The “DNSBL” countries.nerd.dk  allows you to do so… the map ip-adresses to countries based on whois-information… so on my MTAs I added the following lines to the mc sendmail file:

FEATURE(dnsbl,`br.countries.nerd.dk', `554 - Rejected - SPAM from Brazil:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`in.countries.nerd.dk', `554 - Rejected - SPAM from India:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`kr.countries.nerd.dk', `554 - Rejected - SPAM from Korea:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`cn.countries.nerd.dk', `554 - Rejected - SPAM from China:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`ro.countries.nerd.dk', `554 - Rejected - SPAM from Romenia:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`co.countries.nerd.dk', `554 - Rejected - SPAM from Colombia:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`mk.countries.nerd.dk', `554 - Rejected - SPAM from Macedonia:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`vn.countries.nerd.dk', `554 - Rejected - SPAM from Vietnam:$&{client_addr} rejected')dnl
FEATURE(dnsbl,`ru.countries.nerd.dk', `554 - Rejected - SPAM from Russia:$&{client_addr} rejected')dnl

And within a few hours the first are already blocked… I hope this will reduce the amount of incomming spam at the “front door”. Because simply… I don’t know people in these countries…

How to update Python bindings to subversion.

Recently I run into the problem that a team had a requirement for subversion 1.6.6 (while CentOS 5u3 was not supporting this… but the vendor didn’t provide a newer release). This team also had a requirement to have TRAC… TRAC is depended on Python… but I was not allowed to update the subversion bindings for python by updating the it on the whole system… so… this is what I did:

  • Installed a number of devel packages:
       # yum install apr-devel neon{,-devel} apr-util-devel

  • Compiled sqlite version 3.6.13 and installed it on NFS:
      $ ./configure --prefix=/nfs/apps/webservices/trac-parent/sqlite/3.6.13
    $ make ; make install

  • Compiled subversion 1.6.6 and installed it on NFS:
    $ make clean; ./configure 

    $ make -j8 ; make install ; make swig-py ; make install-swig-py

  • Added the following line to /etc/sysconfig/httpd:
    export LD_LIBRARY_PATH=/nfs/apps/webservices/trac-parent/sqlite/3.6.13/lib/

  • Modified /etc/httpd/conf.d/trac.conf by adding a ‘PythonPath’ to the location-directive:
    <Location /projects>
    PythonPath "['/nfs/apps/webservices/trac-parent/subversion/1.6.6/lib/svn-python'] + sys.path"

  • Restart the trac daemon:
    # service httpd stop
    # service httpd start

  • Now you’ve to resync the trac-instance with Subversion (the
    repository_dir value in the trac.ini of the instance).. but make sure
    you use the correct bindings in Python:

    # export LD_LIBRARY_PATH=/nfs/apps/webservices/trac-parent/sqlite/3.6.13/lib/
    # export PYTHONPATH=/nfs/apps/webservices/trac-parent/subversion/1.6.6/lib/svn-python
    # trac-admin ${TRAC_INSTANCE_PATH} repository resync "*"

CentOS 5 enabling Two-factor SSH authentication via Google

Today I noticed a very nice article about enabling Google’s two-factor authentication for Linux SSH.

After reading it… I found some time to play with it… so I enabled it within 10 minutes on my CentOS 5 64bit play-ground server… but there are some small ‘caveats’.

hg – Command

To checkout the code, you must make install the mercurial RPM… this one is available via the EPEL repositories.

So after having the EPEL repositories enabled, run as root:

yum -y install mercurial

Compiling the PAM module

When you checked out the code.

hg clone https://google-authenticator.googlecode.com/hg/ google-authenticator/

You cannot compile directly the module… therefor you must apply a small change to the Makefile.

Change where /usr/lib/libdl.so is stated to /usr/lib64/libdl.so (3 occurrences)

$ make
$ sudo make install

Now you’ve to update the /etc/pam.d/sshd so it contains:

auth       required     pam_google_authenticator.so
auth       include      system-auth
account    required     pam_nologin.so
account    include      system-auth
password   include      system-auth
session    optional     pam_keyinit.so force revoke
session    include      system-auth
session    required     pam_loginuid.so

Configure SSH

You also have to make sure that in /etc/ssh/sshd_config the following settings are set on yes:

ChallengeResponseAuthentication yes
UsePAM yes

And restart the SSH-daemon

Set up your smartphone/credentials on the system

$ google-authenticator
Your new secret key is: SAEP64T5VZAVWAFB
Your verification code is 376046
Your emergency scratch codes are:
Do you want me to update your “~/.google_authenticator” file (y/n) y
Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y
By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) n
If the computer that you are logging into isn’t hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

And you’re done :-D

Give it a try to SSH to that box…

 TIP: Make sure you’ve an SSH session still open… or you might lock yourself out of the system…