Skip to main content

Setting up your own Root Certificate Authority - the right way!



Setting up your own Root Certificate Authority, aka Root CA, can be a difficult process. Web browsers and e-mail clients won't recognize your CA out-of-the-box, so most people opt to use public CA infrastructure. When security matters, using a public CA is the wrong solution. Privately owned and controlled CAs can be infinitely more secure than their public counterparts. However, most people who set up a private CA don't set up their CA infrastructure correctly. Here is what most private CAs look like:

Root CA cert -> Server cert

This is wrong because the server certificate has to be regenerated regularly (e.g. annually). If the root certificate is compromised, then it involves fairly significant effort to replace all of the certificates, including the root. What should be built is this:

Root CA cert -> Intermediate cert -> Server cert

In fact, this is the format that most public CAs use. The root CA cert is generated on a machine that isn't connected to any network. Then it is used to generate any necessary intermediate certs. Both the root and intermediates are signed for long periods of time - typically about 10-30 years for the root and 2-5 years for the intermediates. The root CA certificate private key is then physically secured - a physical vault of some sort helps here. The root CA is never, ever used on a network-connected machine. It is always used offline and it is only ever used to generate intermediate certificates. Only when specific conditions are met is the private key ever accessed. Usually those conditions entail a paper trail of accountability with multiple people who are physically present and are authorized to access the key for purposes declared in advance.

After the root CA generates the intermediate certificates and is secured, the intermediate certificates are then used to generate other certificates. The intermediates can possibly sit on a network connected machine, but that machine is behind a very well-guarded firewall.

So how does one create this setup, while implementing it easily and securely, AND relatively cheaply? First, you are going to need a few things:

Wall-powered USB hub with at least 4 open ports (you can't use built-in USB ports on a computer for this since they are usually underpowered - blame your motherboard manufacturer)
Six USB thumbdrives (ideally, brand new - they can have tiny storage and therefore be cheap - if you are doing this for personal use only, you can scale it back to three thumbdrives)
VirtualBox or VMware
KeePass
An ISO of a public Linux distro of some sort - smaller is better (e.g. Tiny Core Linux)
1 CD-R + CD burner
Labels and a pen or marker
Multiple, physically secure locations for storing the various thumbdrives

If you are familiar with virtual machines and constructing a root CA the right way, perhaps you can see where this is going. Hopefully, you will at least agree with me that where I'm going with this is a step above what most people do for their root CA and it is FAR easier and cheaper to isolate/secure a USB thumbdrive than it is a whole computer system.

Now let's get this started:

Make sure everyone is in the room who needs to sign off on the construction of the root CA and they are well-watered/coffee'd and have been to the restroom recently. They aren't going anywhere for a while. If someone leaves during this time, there's a pretty good chance the entire process will have to be restarted depending on how serious you are about the root CA. It will help speed things up considerably if the individual doing the technical side has gone through a dry run on their computer in advance to get familiar with the process. Also decide in advance which of the four people present will receive thumbdrives containing important data required to decrypt the virtual machine and the root CA private key.

Start an audit log file on the USB thumbdrive where the virtual machine will reside. Don't forget to label this thumbdrive with something like "Virtual machine, audit log". In the audit log, document the date, time, and notes for each operation - including each and every executed command against the virtual machine. The start of the audit log should declare who is present for the CA signing process as the first note. This file should be updated accordingly any time the thumbdrive is accessed in the future (e.g. to generate a new intermediate cert every couple of years).

The next step is to make the Linux distribution ISO read only to all users. Depending on the host OS, this step will vary. A simple solution to making the ISO file read only is to burn the ISO to a CD as an ordinary file (i.e. don't burn a bootable CD). If you plan to burn a CD, you might as well burn a copy of the verification information mentioned in the next step. This portion of the process can be done in advance but should be noted in the audit log that the ISO is confirmed to be read only.

Next, verify the ISO against public verification information such as a hash or PGP. Then compare the same public verification information from the same source at a secondary location in such a way that defends against a man-in-the-middle attack. A different computer or device on a completely different network satisfies this (e.g. a smartphone visiting the same URL over a cell network). If that isn't possible and the same machine as the one that retrieved the ISO is used, a secure session to a temporary VPS (e.g. a Digital Ocean droplet), 'wget' the URL containing the information, and then using 'cat' on the retrieved information may be sufficient for all present.

At this point, randomly generate the two main passwords for the virtual machine and CA root key using KeePass and store the database on the thumbdrive(s). If using a single thumbdrive for personal use, use the master password option. Otherwise, use the encryption key option and store the password database on one thumbdrive and the encryption key for the database on another. When operating in a group, clone each password database thumbdrive to another thumbdrive (up to four drives attached to the hub at this point - Hint: Plug in and remove drives one at a time to make labeling and cloning easier). Label each password database thumbdrive appropriately. They will be given to responsible individuals at the end of the process to keep secure along with instructions on use.

Now build the virtual machine with the ISO on the first thumbdrive, disabling unnecessary features such as audio. Enable encryption on the virtual machine disk image file and use the appropriate password from KeePass. Leave networking enabled for the moment. Install the OS, making sure that files are persistent across reboots. Install OpenSSL and verify that it runs. Then power off the virtual machine.

Disable networking. Physically disconnect the host's Ethernet cable/WiFi/etc. Disable/remove the optical drive controller in the virtual machine software. Connect the thumbdrive that will hold the certificates and keys to the host. Boot the virtual machine back up. Attach the newly connected thumbdrive to the guest using the virtual machine software (if it hasn't done so already) and then mount the thumbdrive inside the guest OS:

mkdir /mnt/thumbdrive
blkid
mount -t [TYPE] /dev/[something] /mnt/thumbdrive

Now you are ready to construct your root CA. From the terminal, run the following as root:

mkdir root_ca
chmod 710 root_ca
cd root_ca
openssl req -new -newkey rsa:4096 -x509 -days 3650 -keyout ca.key.pem -out ca.cert.pem

You will be asked a series of questions. Use the appropriate password from KeePass to protect the CA private key. Common Name should be something like "[Organization] Root Certificate Authority". Email Address should be left blank. 3650 is roughly 10 years. That's the length of time until you have to go through the whole process again. 7300 days is ~20 years, 10950 is ~30 years. Shorter times are, of course, more secure but create more hassle.

Then, run:

chmod 400 *.pem
openssl x509 -noout -text -in ca.cert.pem
cat ca.cert.pem
cp ca.cert.pem /mnt/thumbdrive/

The first command makes it so only the root user can read the files - why OpenSSL doesn't automatically chmod 400 everything that it generates is a security vulnerability that should be fixed. The second command dumps out the information about the certificate (be sure to verify that "CA:TRUE" appears). The 'cat' command dumps the raw PEM certificate data to the screen (a sanity check for PEM formatted data). The last command copies the signed certificate to the thumbdrive. If you ever accidentally dump or copy the 'ca.key.pem' file, then you will have to start over.

Now we are ready to generate an intermediate certificate which will be used to sign all other certificates. Run:

sed s/CA:FALSE/CA:TRUE/ < /etc/ssl/openssl.cnf > openssl.cnf
openssl req -config openssl.cnf -new -newkey rsa:4096 -days 1095 -keyout intermediate_01.enckey.pem -out intermediate_01.req.pem
chmod 400 *.pem

The first line alters the OpenSSL configuration so that you can generate an intermediate certificate that can be used to sign other certificates. Depending on the OS, the openssl.cnf file might not be at /etc/ssl/. The second command again asks a series of questions is asked. Similar sorts of responses should be used. Common Name should be something like "[Organization] Intermediate Certificate Authority". Leave the challenge password and company name empty. 1095 is about 3 years. That's the length of time until the virtual machine will have to be fired up again to generate a new intermediate certificate.

openssl x509 -req -days 1095 -in intermediate_01.req.pem -CA ca.cert.pem -CAkey ca.key.pem -extfile openssl.cnf -extensions v3_ca -set_serial 01 -out intermediate_01.cert.pem
chmod 400 *.pem
openssl x509 -noout -text -in intermediate_01.cert.pem
cat intermediate_01.cert.pem
cp intermediate_01.cert.pem /mnt/thumbdrive/
cp intermediate_01.enckey.pem /mnt/thumbdrive/

Finally, power down the virtual machine. Disconnect all thumbdrives. The thumbdrive labeled with the encrypted virtual machine and is moved into a physically secure location (e.g. a vault). The thumbdrive with the CA public key and intermediate CA files move over to other, properly firewalled, network-connected infrastructure. Once moved over, if automation is desired and authorized, the password can be removed from the intermediate CA private key:

openssl rsa -in intermediate_01.enckey.pem -out intermediate_01.key.pem

If you need more than one intermediate certificate or are renewing the certificate, adjust the above commands to increment all the 01's to the next available number (i.e. 02, 03, 04, 05, etc).

Four of the people present each receive one of the four thumbdrives containing the password database/decryption key after they sign for them. How this happens is up to the organization. Here is some sample verbiage: "I, ________, hereby accept one of the required components of the [Organization Name] root CA password data store. I will keep this device and the data on it in a secure location at all times. I will not copy, duplicate, or clone the data for any reason. I will not use the device or the data on the device on any machine connected to a network. If any of these terms are violated, disciplinary action may be taken against me up to and including termination. If I ever leave [Organization Name], I agree to transfer stewardship of this password data store to another [Organization Name] employee. Date/Signature"

This concludes how to correctly protect your nuclear arsenal and create a root CA on the cheap with OpenSSL.

Comments

  1. Hey Thomas, so I read your post. So after I do all this, how do I install it on my VPS Linux server?
    Thanks, Corey

    ReplyDelete
    Replies
    1. Okay. Let's assume that you make a root certificate and your first intermediate certificate as the start to the three-layer chain as per the article and video. From here you sign your server certificates with the intermediate certificate and private key. Since server software varies widely, there is no one-size-fits-all solution so I can't say "do this specific thing and it will work". However, the majority of software out there lets you configure it to use the server private key in one file and a "certificate chain" in another file containing: The root certificate, the intermediate certificate, and the server certificate. They could be in that order or in reverse order - it basically depends on the server software.

      Here's what I do with all of my certificates on Linux servers for server software: I create a directory in /var called 'certs', so /var/certs. It is chmod 700 and owned by the root user. Then I generate a private key and CSR as one does. Then I get the CSR signed and put the resulting certificate into /var/certs. I clearly name each file so its purpose is clear. I build the certificate chain by 'cat'ing the root + intermediate + certificate. I also chmod 400 all of the files in /var/certs to make sure they are read-only and only the root user can see them. Most server software starts life running under root, so it has access to the files.

      Of course, you still have to configure clients to accept your new root CA. You do that by installing the root certificate in the OS or browser trust store. Eventually we will use something like DNSSEC TSLA records, which would allow us to eliminate the root certificate stores in web browsers and OSes and finally secure our infrastructure.

      Hope this helps you in your root CA endeavors!

      Delete
  2. Hello. Not sure if you will get this as I have just came across your post (it is fantastic by the way) but I am really struggling to figure out what goes on what USB stick. I am doing this for personal/hobby use. Please could you give me a clear bullet point of what goes on each of the 3 USB drives, if you don't mind, of course. Thank you very much in advance. Regards, Steve

    ReplyDelete
    Replies
    1. Thumbdrive 1: Audit log, KeePass portable, KeePass encryption keys (if any), and the virtual machine. Physically secure this somewhere. (Note that the KeePass encryption keys file in an enterprise setting store the file on a separate, physically secured thumbdrive.)

      Thumbdrive 2: Encrypted KeePass database. Physically secure this somewhere in a different location.

      (Don't forget where the first two thumbdrives are stored or you will have to start over whenever the intermediate or root cert expires! You can't access the virtual machine without the passwords and you can't access the passwords without the password database. The password database is also useless without the decryption keys from the first thumbdrive if you use that option in KeePass.)

      Thumbdrive 3: Root CA (public key), intermediate CA.

      I prefer to use brand new thumbdrives from trusted manufacturers as USB thumbdrives can be compromised (e.g. malware installed). For personal use or for experimenting, you can get away with using some random thumbdrives lying around.

      Delete
  3. Hi Thomas,
    Excellent instructions. You have obviously spent some time in a highly classified working environment. The term for a lot of what you presented, which I'm positive you are familiar with, is: "Air Gap".
    I caught your youtube vid and followed it to your blogspot.

    I setup a two tiered CA, no intermediary CA, for our VMware Horizon environment. I did this since we offer VMware IT Academy courses globally. www.cccti.edu/vmware

    We use Horizon View as the portal into our virtual lab environment.
    Oh, btw: I noticed you had VMware in you document as: "VMWare". They are somewhat particular on the way to present their name: either all lower case, or just: VMware.

    After building the two tiered CA it became obvious that I need the intermediary CA.
    I wanted to shorten the time for participants gaining access into our lab environment.
    Not easy to do with with a two tiered CA.

    I'm working on another project that you may find of interest, but I don't want to detract.
    I have a tendency to chase rabbits!
    I needed to revisit my CA deployment and thanks for doing your post which makes complete sense. I will be introducing the intermediary CA in the new design.

    The one area of concern for me is the download of keepass from sourceforge since it does not have a checksum validation.
    Not that significant of an issue. I'll just create my own password.

    I'll update you on my progress, but it will be awhile, still have a few other fires burning.

    Pete

    ReplyDelete
    Replies
    1. Thanks for the tip about VMware. I've updated the post. I don't like it when people use CubicleSoft with a lowercase 's' but I understand why they do it.

      Whoops. I didn't even think about double-verifying the KeePass download as part of the process. I already have it ready to go and use it as part of my workflow. KeePass also downloads over HTTPS while the OS portion comes down over plain ol' HTTP. So I was more concerned with verifying the OS, which would be housing the most critical components of the CA but you are definitely right, I should have verified KeePass too. I was also concerned with making a video of the process.

      SourceForge does calculate a SHA-1 and MD5 hash if you alter the download URL by chopping off bits of the URL until it stops trying to download it. For example, the various versions of the latest 2.x release (at the time of this comment) can be seen here: https://sourceforge.net/projects/keepass/files/KeePass%202.x/2.38/

      There I see a screen with all downloads for that version. There is a little 'i' icon next to each download. Each icon shows the calculated hashes.

      The passwords for the virtual machine should never be known or shown. How secure you want your CA setup to be is up to you. Then again, I attempted to present the strictest environment and, even then, I missed the KeePass verification bit - actual security is hard!

      I see a lot of two-tier CA setups and I'm pretty sure that is partly due to laziness, partly due to how the technology is built (e.g. having to alter config files to get an intermediate CA), and partly due to misunderstanding of how root CAs work by the broader public.

      One technology to keep a close eye on is DNSSEC DANE TLSA. Under "Certificate usage 3", it is possible to run a private root CA and have it trusted by a remote client (e.g. a web browser would display the lock icon without having to install the root CA public key). All the more reason to figure out how to do a root CA the "right way" now and work out the bugs before a lot of people start building CAs for TLSA and set them up in a way that doesn't future-proof their environment (i.e. a two-tier vs. a three tier CA, not validating the chain of software used, etc).

      Delete
    2. Thomas,

      Thanks again for the rapid reply.
      I woke up this morning with an additional thought.
      btw: you are correct regarding laziness, but it can also be due to the lack of support/understanding from within the organization one is associated.

      My thought/question: can the process be repeated to create an additional CA link?

      Reason for this in my/our case is that for participants to gain access into our environment I need to have them install the CA cert.

      I would like to have the "long-term" intermediary CA, and an additional "short-term" (3-month) CA.

      Totally agree with your comment on securing the VM.
      Were you ever familiar with another super small footprint linux called: dsl (damn small linux)?
      I still have an iso, (for what it's worth?).

      Thanks for the heads up, I'll investigate DNSSEC. Already on it.
      fyi: I have developed a complete VMware Lab environment that can run using VMware Workstation. It does require (32G RAM) or (2x16G). It will fit on 256G SSD.

      Delete
    3. If you are referring to creating a 4-tier CA, the answer is yes. You can build chains however long you need them to be. The intermediate can be configured to sign another intermediate CA that can be used to generate server certificates.

      However, you should be aware that a lot of software won't work too much beyond a 4-tier setup. The maximum allowed "certificate depth" is frequently hardcoded into the software itself, so changing it is usually impossible without recompiling the executable.

      TinyCore was started by at least one of the core developers of DSL. That's part of why DSL kind of died off.

      Delete

Post a Comment