Using OpenVPN Access Server to Access AWS VPCs

If you use AWS for anything non-trivial, you are likely using VPCs and keeping some non-public subnets for resources you don't want to be accessible to the open Internet.

Usually when this is done, a bastion host is set up on a VPC public subnet and accessed via SSH to reach those internal instances or resources, through secondary SSH connections or SSH tunnels. That is a good and simple solution that will work well for a simple environment with few internal services and few users that need to access it.

However, I would like to explore an alternative: using OpenVPN Access Server to create a client-to-site VPN connection to the VPC. This approach is really easy to implement and has some pretty interesting benefits:

  • Easy set up of two-factor authentication with Google Authenticator or similar TOTP solutions. This is something you could also achieve with the bastion host as well using some PAM magic if you really want to, by the way.

  • Use of different IP address ranges to users of different groups, which allows different Security Group inbound rules on assets for internal resources based on who the authenticated VPN users are.

  • A web interface for your users that simplifies the task of downloading the VPN client with the proper connection profile, which is far more manageable for users that are not of a strong technical background.

  • Commercial support from the team at OpenVPN.

One of the downsides is that this solution will require you to buy licenses for the OpenVPN Access Server if you plan to have more than two client connections at a time. But I find them to be reasonably priced.

For the record, this was documented based on OpenVPN Access Server 2.1.4 and is meant as a rough but coherent guide to the external references I used to get this working. Hopefully it will save someone some time.

OpenVPN Access Server Instance

First, go to the AWS Marketplace and find the BYOL AMI for OpenVPN Access Server. Then, follow their Quick Start Guide to perform the initial configuration.

A few things to keep in mind:

  • Make sure you install this on a public subnet of your VPC;

  • Assign the OpenVPN server EC2 instance an elastic IP address so that if you stop and restart it the IP address won't change.

Updates, Updates and More Updates

The first thing you'll want to do after you have finished the SSH questions and are logged into the instance is to fully update it and then reboot it by running the following commands:

sudo su -
apt-get update
apt-get dist-upgrade
shutdown -r now

Please don't skip the reboot, so that all updates actually take effect, such as updates to the kernel and to libraries that are currently being used by running processes.

Web Server Certificates

Now let's set up the OpenVPN Access Server web server so use Let's Encrypt certificates with automated renewal.

First, as root, install certbot:

cd /root
wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

Installing the certificates will require issuing custom commands using an OpenVPN Access Server script, so we'll need to request the certificate manually.

Choose a hostname whithin a domain whose DNS records you control. Then, do the following as root:

./certbot-auto certonly --manual --preferred-challenges dns

Follow the on-screen instructions, and when requested for the domain of the certificate enter the hostname you'll use for the OpenVPN server. It will ask you to create a TXT record on the DNS configuration of the domain so that it validates that you do own the domain.

If you use Route53 then follow these instructions to create the TXT record and you'll end up with something like this:

TXT record

Take this time to also create an A record set that maps the hostname of your OpenVPN server to the Elastic IP address you assigned to it in the previous section.

Finally, we need to configure OpenVPN Access Server to use these certificates. In order to be ready for automated renewal later on, let's do this by creating two scripts that I adapted from elsewhere:

cat > /root/pre-hook.sh << EOF
#!/bin/sh
/usr/local/openvpn_as/scripts/sacli stop
EOF

chmod a+x /root/pre-hook.sh

cat > /root/post-hook.sh << EOF
#!/bin/sh
/usr/local/openvpn_as/scripts/confdba -mk cs.ca_bundle -v "`cat /etc/letsencrypt/live/openvpn.example.com/fullchain.pem`"
/usr/local/openvpn_as/scripts/confdba -mk cs.priv_key -v "`cat /etc/letsencrypt/live/openvpn.example.com/privkey.pem`" > /dev/null
/usr/local/openvpn_as/scripts/confdba -mk cs.cert -v "`cat /etc/letsencrypt/live/openvpn.example.com/cert.pem`"
/usr/local/openvpn_as/scripts/sacli start
EOF

chmod a+x /root/pre-hook.sh

Please replace openvpn.example.com in the code above with the actual hostname you chose for your OpenVPN server. Then, you immediately run the scripts to load the newly created certificate:

/root/pre-hook.sh
/root/post-hook.sh

If you use Route 53 we can setup the automated renewal of these certificates by using certbot-route53. First, download the script:

curl -sL https://git.io/vylLx -o /root/certbot-route53.sh
chmod a+x /root/certbot-route53.sh

Then, ensure the IAM role associated with the OpenVPN instance has the following policy attached to it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:ListHostedZonesByName"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:GetHostedZone",
                "route53:ListResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/<your hosted zone ID>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:GetChange"
            ],
            "Resource": [
                "arn:aws:route53:::change/*"
            ]
        }
    ]
}

Finally, create a regular task executed twice a day by running crontab -e as root and entering the following new entry:

0 1,13 * * * /root/certbot-auto renew --pre-hook "/root/pre-hook.sh" --post-hook "/root/post-hook.sh" --manual-auth-hook "/root/certbot-route53.sh" --manual-cleanup-hook "/root/certbot-route53.sh" --quiet --manual-public-ip-logging-ok

Please choose an arbitrary number between 1 and 60 and replace the first 0 in the above line. Also choose different hours (instead of 1,13) for executing this. If lots of people configure their cron jobs to hit Let's Encrypt at exactly the same times we might end up with an accidental DDoS.

At this point, you should be able to access the admin interface (by default in our example it would be https://openvpn.example.com:943/admin), and not get a warning about an invalid certificate. Just remove the /admin from the URL and you'll land on the user-focused page instead, where they can log in to download the client software binaries and profile configuration, and also scan Google Authenticator QR codes.

Go into the admin interface and perform the configuration that best suits your needs. We'll discuss two particularly interesting aspects of the configuration next.

Routing

You may choose to have OpenVPN use routing instead of NAT when it forwards traffic from the connected clients into the VPC environment. If you do that, you can use its ability to assign different users or groups of users IP addresses in different ranges. This, in turn, allows you to configure Security Group rules inside the VPC that will only allow certain users or groups of users to communicate with specific services.

If you do that, however, keep in mind that you'll need to take some additional steps:

  • Ensure that the IP address ranges you assign VPN clients are outside the CIDR range for the entire VPC. So if your VPC has addresses on 172.16.0.0/16, you can't assign any part of that address space to VPN clients even if there are no VPC subnets currently using them.

  • You'll need to update your VPC routing configuration. For each CIDR of addresses that can be assigned to VPN clients, add a new route with that CIDR as the Destination and the OpenVPN server instance as the Target.

  • Disable the source/destination check on the OpenVPN Access Server instance so that it can "answer for" other ranges of IP address in the AWS network stack.

Google Authenticator

If everything else is working at this point, requiring users to have two-factor authentication using Google Authenticator should be pretty easy by now. First, on the admin interface go to Configuration  →  Client Settings and check the box that says Require that users provide a Google Authenticator one-time password for every VPN login.

What it took me a while to figure out was how to enroll users. Basically what you do is create the user normally in the admin interface in User Management  →  User Permissions, and assign the user a password.

Then, the end user must log in to the non-admin web interface (https://openvpn.example.com:943 in our example), and choose Login instead of Connect:

user login

In this page, they'll the given the choice to download a connection client or a profile configuration they can import into an existing client. But they will also be displayed the QR code they'll scan using Google Authenticator. Once they do that, everything should be good to go.

CloudWatch Logs Agent

Next let's configure the CloudWatch Logs Agent so that the OpenVPN Server logs are centralized and easy to query if necessary, and also protected from any attackers that might gain access to the server.

Firstly, a minor adjustment is needed to allow the agent installation to go properly. The provided AWS installation script checks /etc/issue to determine which Linux distribution the machine uses. And OpenVPN Access Server has a custom string in that file. So we'll need to edit it to make sure the first line starts with Ubuntu or else the script will refuse to run.

I also recommend assigning the machine its proper hostname by issuing the following command as root:

hostname openvpn.example.com

Then, make sure you edit /etc/hostname and /etc/hosts to make sure they also match the the full hostname.

Finally, you can then execute the official AWS installation instructions with the following three recommendations:

  • Make sure you include /var/log/openvpnas.log, /var/log/syslog and /var/log/auth.log to the log collection.

  • When asked what the log stream name should be, I recommend you use the newly-configured machine name for clarity.

  • When asked about the timestamp format, choose to use a custom format and enter %Y-%m-%dT%H:%M:%S%z.

Then, update /etc/rsyslog.conf as per these instructions and restart the rsyslog service:

service rsyslog restart

Finally, you'll need to enable the service to run via systemd (as root):

systemctl enable awslogs

You should now see the new log groups on the AWS Console.

I would also like to suggest that you create a new file /etc/logrotate.d/openvpnas with the following content:

/var/log/apt/openvpnas.log {
  rotate 90
  daily
  compress
  missingok
  notifempty
}

Additional Recommendations

A few security-minded tips that I would recommend you implement:

  • Ensure you close off SSH access to the OpenVPN box after you're done configuring it, by removing the applicable Security Group inbound rule. If you need in you can always do that through the VPN access itself later on. And if you lock yourself out, you can always allow it temporarily again on the Security Group if and when you need it.

  • On the web admin interface, go to Configuration  →  SSL Settings and select TLS 1.1 as the minimum for both the web server and OpenVPN server, as per industry best practice.

  • On the web admin interface, go to Configuration  →  Advanced VPN Settings. Under Additional OpenVPN Config Directives (Advanced), replace AES-128-CBC with AES-256-CBC in both the client and server configuration directives for increased encryption security.

  • Harden the operating system and make sure to keep install security updates as they become available.

Go Top