At my current employer, I’ve setup FreeIPA to deal with the whole DNS/LDAP/Kerberos/PKI kerfuffle. At previous firms I’ve done a DIY setup for this: CentOS 5, tied into OpenLDAP, MIT Kerberos, and Cobbler—which required I backport OpenLDAP 2.4 and the version of MIT Kerberos that’s capable of using LDAP as a backend database to CentOS 5. On the SSL side, I let Puppet manage it’s own PKI and just used gnoMint for service certificates.
It worked pretty well, but I never got around to writing a web GUI that was supposed to sit in front of all of it—mainly because FreeIPA already existed and I secretly wished it would use OpenLDAP rather than FedoraDS (since renamed to “389”). Alas, three years later and it still hasn’t happened, and now I’m at another new financial industry startup. I could screw around for a few months getting OpenLDAP, Kerberos, Puppet, SSSD, and Foreman all talking together, or I could setup FreeIPA.
Aside from not using my LDAP directory of choice, it also doesn’t do DHCP or Kickstart, and it doesn’t have any sort of built-in support for Puppet, so I’ve got to hack it to work together. The first step is to setup Puppet to use the FreeIPA-integrated PKI with Passenger. Because I’m trying to do things the FreeIPA way, I’m also going to use mod_nss.
So the first question is always the same: why do I want to do all this? There are really only two reasons: the first is ease of scaling, the second is ease of management. If I’m using the builtin PKI that ships with puppet, I’ve got a scalability problem because the CA cert is tied to the hostname of the first puppetmaster. There are newer features in puppet that let you work around this, some of which we’ll use to ensure we’re using FreeIPA’s PKI properly, but I haven’t played with them.
The second issue is ease-of-management. FreeIPA is already maintaining a secure certificate distribution channel via certmonger, which allows users to pull down per-service certificates tied to Kerberos service principals. Is it really worth having a second PKI tree that puppet is managing itself when FreeIPA is already doing this for you out of the box? Only if you’re lazy about it.
On the downside, there is the issue of security. FreeIPA out of the box only supports a single toplevel CA, which means that all your certificates (IPA host certs, puppet certs, Website certs, etc.) are all in a single security domain—there’s no built-in way to restrict this access to puppet. Users can’t invent certs, of course, but any cert with the right hostname can be used to authenticate to puppet, because they share the same trust hierarchy (a “third way” option would be to have FreeIPA sign a puppet-specific CA, which is then used to sign the client certs, restricting the domain to just the puppet CA and the certs it has signed).
Lets assume that you’re OK with handling the issue of a single hardcoded authorization domain shared by multiple services (in this case, anything using an SSL cert from FreeIPA that has the DNS hostname in it), if for no other reason than this blog post would be a complete waste of time without that assumption. I’m also going to assume that you already have a working FreeIPA setup. Roughly, the steps we’re going to look into implementing are:
- Create service certificate for puppetmaster
- Setup the puppetmaster to use Apache with Passenger and mod_nss
- Create the service certificates for the puppet clients
- Setup the puppet clients
- Verify everything is working
The first step is to get get your puppet server into FreeIPA and create the service certificate, a fairly trivial process. On the FreeIPA server (or at least something with the admin tools installed), replace the IP address and hostname as appropriate, and season to taste:
ipa host-add puppet.example.com --ip-address=10.10.10.10 --password=secret ipa service-add puppetmaster/puppet.example.com ipa service-add puppet/puppet.example.com
On your puppetmaster:
yum --nogpgcheck --localinstall
http://passenger.stealthymonkeys.com/fedora/16/passenger-release.noarch.rpm
yum install mod_nss mod_passenger
ipa-client-install --password=secret
systemctl stop puppetmaster.service
ipa-getcert -K puppetmaster/puppet.example.com
-d /etc/httpd/alias
-n puppetmaster/puppet.example.com
ipa-getcert -K puppet/puppet.example.com
-D puppet.example.com
-k /etc/puppet/ssl/private_keys/puppet.example.com.pem
-f /etc/puppet/ssl/public_keys/puppet.example.com.pem
mkdir -p /var/www/puppet/public
cp /usr/share/puppet/ext/rack/files/config.ru /var/www/puppet
So now you’ve joined your puppetmaster to IPA, installed the relevant software, and gotten the SSL keys you need to identify the server to clients installed. Next step is to configure puppet.conf—it should look like this:
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
[master]
ca = false
certificate_revocation = false
So what all does this do? Well, firstly we’re removing the ssldir option, which will make puppet use /etc/puppet/ssl by default (i.e. where we had IPA install the SSL certs). The rest of it is pretty standard, save for the master config, which has two options—the first one disables the built-in PKI, and the second which disables certificate revocation (which we may be able to turn back on later).
That’s it for puppet’s configuration, lets move on to the HTTPd configuration. Firstly, lets clean up mod_nss, which installs a whole bunch of weirdness in it’s initial configuration. Suffice to say, you only need this:
LoadModule nss_module modules/libmodnss.so AddType application/x-x509-ca-cert .crt AddType application/x-pkcs7-crl .crl NSSPassPhraseDialog builtin NSSPassPhraseHelper /usr/sbin/nss_pcache NSSSessionCacheSize 10000 NSSSessionCacheTimeout 100 NSSSession3CacheTimeout 86400 NSSRandomSeed startup builtin NSSRenegotiation off NSSRequireSafeNegotiation off
The rest of the stuff in that file (including the bizarre decision to listen on port 8443 by default) is unnecessary, so you can simply cut it out. With that out of the way, we can move on to the puppetmaster-via-apache configuration. One thing to note here is that Puppet is written in ruby, and it operates via HTTP, so there’s little reason it can’t run just like any other ruby website, even though it’s not strictly intended for user-facing content.
Anyways, on to the configuration:
Listen 8140
<VirtualHost _default_:8140>
ServerName puppet.example.com
ServerAdmin puppetmaster@example.com
NSSEngine on
NSSCertificateDatabase /etc/httpd/alias
NSSNickname "puppetmaster/puppet.example.com"
NSSOptions +StdEnvVars
NSSEnforceValidCerts on
NSSVerifyClient require
NSSProtocol SSLv3,TLSv1
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN "/CN=%{SSL_CLIENT_S_DN_CN}e"
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
PassengerHighPerformance on
PassengerStatThrottleRate 120
PassengerUseGlobalQueue on
RackAutoDetect off
RailsAutoDetect off
RackBaseURI /
DocumentRoot /var/www/puppet/public
<Directory /var/www/puppet>
Options None
AllowOverride None
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
So the first bits are obvious: setup Apache to listen on port 8140 (the puppetmaster port), and setup SSL as provided by mod_nss. Of particular note is the fact that we’re creating environment variables from the contents of the SSL certificate that we’re being presented. My understanding (or at least hope) is that mod_nss will verify the certificate has been signed, then generate some environment variables for the actual puppet application based on that. In particular, you need the +StdENvVars NSSOption to be set.
The next bit, setting the RequestHeaders, handles munging the SSL certificate names in such a way as to let puppet validate them. After that it’s setting some passenger configuration which I haven’t looked up, and pointing it at the proper directory. After that comes setting up the directory, which is fairly simple and already done—just create the directories and copy the config.ru file that ships with puppet to the application base.
And that’s it. Assuming you’ve setup everything properly, your clients (in this case, just the local puppet daemon running on your puppetmaster) should be able to connect and start pulling their catalogs (even if the catalog is empty you’ll still get an successful run notice for the client in /var/log/messages)
James Cape 
Thank you for getting me 90% of the way there on this. I did notice that you forgot to add ‘request’ on the lines that start ‘ipa-getcert’, but this article has still been very helpful.
However, I’m stuck in a few places and wondered if you had thoughts. Currently, when I add the service for puppetmaster and puppet, all appears to go to plan, except when I attempt to start httpd, I get ”Unable to verify certificate ‘puppetmaster/pdx-caf-puppet.int.codeaurora.org’. Add “NSSEnforceValidCerts off” to nss.conf so the server can start until the problem can be resolved.’
I have done this and httpd successfully starts. But it only results in further problems down the road. Puppet can then not see the proper certificate. I get ‘puppet-agent[12648]: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read finished A: sslv3 alert bad certificate’.
I’m a bit lost as to why this is happening and any thoughts toward what I might be missing would be immensely helpful.
Thanks,
Clint (irc:herlo)
@Clint,
For validating certs, the issue is that certmonger doesn’t (by default) put the CA cert into the nssdb. You’ll have to manually do it with certutil.
IIRC, the ‘alert bad certificate’ is actually a bug in the latest 2.7 series in Puppet—it rejects any cert or CA with a name that doesn’t look like a hostname. Unfortunately, the default IPA setup includes them, so puppet improperly rejects them. I fixed it by downgrading to 2.7.13 (don’t feel bad, Mozilla Foundation got bit as well :-)).
The cert verification issue does not impact 2.6, of course.
@James
Thank you for the reply. Unfortunately, that doesn’t appear to be the puppet bug I’m experiencing. I’m actually running puppet-server-2.6.17-2.el6. I’m currently testing your suggestion (but was already in the middle of unenrolling and reenrolling the server). I’ll let you know how it goes.
Thanks very much for the response,
Clint
@James,
Further followup with you, I sure appreciate any feedback you can give. I did finally get httpd to start properly by adding the IPA Certificate Authority Certificate into the nssdb at /etc/http/alias. However, I am still a bit stuck on the bad certificate issue.
I think I have a handle on things enough to describe the issues. I’m pretty sure I’m *very* close, but am just missing something specific.
I’ve been able to successfully grab new certificates from the puppet/puppet.example.com service and put them onto the puppet master. The certificate resides in /etc/puppet/ssl/public_keys as descirbed above.
The puppet cert print command cannot find them. I’m guessing this is because we’re not using puppet as a CA, so I have to validate through FreeIPA, correct? If so, I’m assuming the certs were signed by the FreeIPA CA and should be good. It appears this way when I use openssl s_client to verify, but I’m not 100% sure.
The other part of this is that I’m supposing I should be testing the setup with a command similar to ‘puppet agent –test’. I am doing this on the same box as the master (since I eventually want to manage the puppetmaster server with puppet).
I’m going to hit up the #puppet channel on irc.freenode.net and ask the folks in #freeipa as well, but I thought I’d see if you had any other suggestions / direction for me to go.
Thanks again,
Clint