Archive for December, 2010

Lights-out Football in New York

  • Date: 13 December 2010
  • Author: Trevor
  • Category: Smart Grid
  • Comments: 0

On Sunday, November 14, 2010 the Dallas Cowboys defeated the New York Giants 33-20. Fans of the NFL had a number of reasons to watch this game; a historically bitter NFC East rivalry, the Cowboys’ new coach Jason Garrett testing his coaching prowess against a strong opponent, or simply wanting to catch a glimpse of the Giants’ new $1.6 billion New Meadowlands stadium. I personally was watching the game, or at least as much that was covered by the NFL’s RedZone channel.

You might be asking yourself why I am talking about a sports event on an Information Security blog. Well for those of you who don’t know, the New Meadowlands stadium, filled with 80,851 fans, completely lost power during the 3rd quarter of the game. The complete outage lasted only 5-6 seconds, but for someone with a Smart Grid security background, the first thing I thought was terrorism. Thankfully this was not the case, and there are no indications of such; rather a New Jersey Sports and Exposition Authority transformer failure caused the outage.

Power Outage at the New Meadowlands

Power Outage at the New Meadowlands

What concerns me is that while the stadium was able to restore power from the total blackout in a timely manner, the outage itself may have provided terrorists with a previously unidentified target: large stadiums filled with Americans. I have read about sporting events being the target of terrorist attacks before, but not when the power (or lack their of) was the target. Imagine the chaos that would have ensued if the stadium was without power for several minutes or hours. Several of the NFL players that night voiced similar concerns.

  • The Giant’s star Justin Tuck commented that, “You start worrying about is your family all right.”

As I was watching the situation unfold on the RedZone channel, Fox’s Joe Buck, Troy Aikman, and Pam Oliver (who were covering the game) made the following observations:

  • Troy Aikman – “They didn’t know what was going on nor did anyone else. But they hit the ground. A number of those guys did.”
  • Pam Oliver – Described the scene as “organized chaos” and “extremely scary” when the lights went out.

Interestingly, when Buck and Aikman were describing the scene at the New Meadowlands, they failed to mention that during the blackout, fire alarms also sounded. They did mention that an announcement came over the PA system telling fans to remain in their seats and to stay tuned for evacuation instructions, if necessary.

Fortunately, no major incidents occurred as a result of the power outage. However, I hope that those responsible for securing the New Meadowlands, and all public venues, use this as an opportunity to better understand their own weaknesses and how they may now have an additional threat from terrorists; power outages.

Securing The Smart Grid

Attacks like this are covered in my book, Securing the Smart Grid. Syngress has made part of the chapter covering threats like these, entitled “Threats and Impacts: Utility Companies and Beyond,” available here. It is unfortunate that situations like these are sometimes necessary to identify “unknown, unknowns” to our national security.

Shearing FireSheep with the Cloud

If your laptop ever connects to a network behind enemy lines (e.g. hhonors, attwifi, panera), this post is for you. The step-by-step directions below allow you to stand up a portable, cloud-based private VPN that you can use from anywhere – for around $0.50 a month. Once you get everything setup, you can feel good connecting to a hotspot and laugh at the guy running FireSheep.


Speaking of Firesheep, I’ve actually had some people close to me (including my wife) ask how they can prevent these types of attacks from happening. There are some nice “off-the-shelf” solutions like HTTPS Everywhere and BlackSheep but as a security professional I wanted to give a recommendation that would provide broader coverage than these solutions.


Enter Amazon’s recently introduced Free Tier for EC2. I’ll save my thoughts and comments on “The Cloud” and security for a later date (and after a couple of beers), but for the purposes of this solution, it works great to help you increase your security while using open wireless networks. Quite simply, the solution I came up with was to create an EC2 instance with Ubuntu 10.04 LTS server and setup OpenVPN and SideStep. This allows me to route all of my traffic over an SSL or SSH VPN to my EC2 instance and then out to the Internet.


To graphically represent what this solution offers, below is a picture of your laptop while surfing on an Open Wi-Fi network such as those at Starbucks.

Your Laptop @ Starbucks

The second image is the guy running Firesheep at Starbucks.

The Guy @ Starbucks Running FireSheep

The last image depicts your laptop running OpenVPN or SideStep at Starbucks.

Your Laptop Armed with OpenVPN or SideStep @ Starbucks

Enough with the ‘Behind Enemy Lines’ comparisons…I swear. I installed other services on my EC2 instance, like Privoxy and iodine (see my post on tunneling traffic via iodine), but for the purpose of this post, I will limit the scope to creating an EC2 instance, installing and configuring OpenVPN, and installing and configuring SideStep.


A couple of notes before we get started. While the instructions that follow utilize Amazon’s Free Tier, this setup will cost you roughly $.50 per month. There are ways to shrink your EC2 ami to fit within the Free Tier’s EBS limit of 10GB, but I will pay around $.50 a month to have this service available to me (the Ubuntu AMI we will use utilizes 15GB of EBS). Thanks to Martin’s post in the comments below, I have updated this post to utilize an 8GB ami, which is less than the 10GB allotted in the free tier for EBS storage.



So let’s get started…


1. If you haven’t already, head over to Amazon EC2 and create an Amazon EC2 account.


2. Once you have created an account, visit the AWS Management Console and click on the ‘Key Pairs’ link on the left side of the screen. Here you will create a Key Pair that will allow you to login to your EC2 instances. Click on the ‘Create Key Pair’ button and name the Key Pair something unique. I chose ‘JustinsAllEC2Key’. Save the file in your ~/Download folders and move it to your ~/.ssh/ folder by issuing the following commands:


Your Mac
jmorehouse@Old-Trafford:~$ cd Downloads
jmorehouse@Old-Trafford:Downloads$ mv JustinsAllEC2Key.pem ~/.ssh/
jmorehouse@Old-Trafford:Downloads$ chmod 400 ~/.ssh/JustinsAllEC2Key.pem


3. Now that you have a key pair, it is time to create and launch an instance. Click on the ‘AMIs’ link on the left side. Then select All Images from the ‘Viewing’ drop-down (it takes a minute to load all of the available instances), and search for ami-4a0df923 ‘ami-3e02f257’. This is an EBS instance of Ubuntu 10.04 LTS Server 64-bit 32-bit from Alestic. EBS allows for persistent storage, so that your setting will remain even when you power-cycle your instance.


4. Select the AMI and then click the ‘Launch’ button at the top. You will be prompted with a number of options, and I recommend using the following:
  • Number of Instances: 1
  • Availability Zone: No Preference
  • Instance Type: Micro
  • Launch Instances
  • Click ‘Continue’


  • Kernel ID: Default
  • RAM Disk ID: Default
  • No Monitoring
  • No User Data
  • Click ‘Continue’


  • Key = ‘Name’
  • Value = ‘Free Tier EC2 Ubuntu 10.04 Instance’
  • Click ‘Continue’


  • Choose from your existing Key Pairs – ‘JustinsAllEC2Key’ -> This is the key you previously created in Step 2 and moved to your ~/.ssh/ folder.
  • Create a new Security Group – ‘InternetAccessible’ -> This akin to a firewall ruleset group. I created a new once called ‘InternetAccessible’, but you can just as simply use and edit the ‘Default’ group.
  • Describe your security group – ‘Services allowed from the Internet’
  • Select ‘SSH’ from the drop-down ‘Applications’ menu -> I left ‘All Internet’ as we want to access this instance from wherever we are on the Internet.
  • Click ‘Add Rule’
  • Select ‘HTTPS’ from the drop-down ‘Applications’ menu -> This will give us access to our OpenVPN server. I also left this open to ‘All Internet’ for the same reason we configured SSH this way.
  • Click ‘Add Rule’
  • Click ‘Continue’


5. You are then be presented with a confirmation page where you should confirm your setting and make any necessary changes. If everything looks good, go ahead and launch your instance.


6. Your instance is now launching. Click on the ‘View your instances on the Instances page’ link to access information about your instance.


7. Now we will assign a static IP address to your instance as Amazon makes this feature available for free (what IPv4 shortage?). Click on the ‘Elastic IPs’ link on the left side. Then click on the ‘Allocate New Address’ button in the center of the page. Click the ‘Yes, Allocate’ button, and then click the checkbox infront of the newly added IP address. We want to associate this IP with your newly created instance. You can do this by now clicking on the ‘Associate’ button at the top. Select the ‘Instance ID’ for the instance you just created (there should be only one Instance ID in the drop-down) and click ‘Associate’. Copy the IP address somewhere handy as we will need it in a couple of minutes.


8. Once you have done this, it’s time to login to your EC2 instance! You can perform this from Terminal using the following:


Your Mac
jmorehouse@Old-Trafford:Downloads$ cd ~
jmorehouse@Old-Trafford:~$ ssh -i ~/.ssh/<filename>.pem ubuntu@IPAddress


9. Type ‘yes’ to accept the RSA key fingerprint and you should see something akin to the following:

Linux ec2 2.6.32-309-ec2 #18-Ubuntu SMP Mon Oct 18 21:00:50 UTC 2010 x86_64 GNU/Linux
Ubuntu 10.04.1 LTS

Welcome to Ubuntu!
* Documentation:

System information as of Fri Dec 3 00:40:20 UTC 2010

System load: 0.0 Processes: 60
Usage of /: 6.2% of 14.76GB Users logged in: 1
Memory usage: 6% IP address for eth0: 10.XX.XX.XX
Swap usage: 0% IP address for tun0: 10.X.XX.X

Graph this data and manage this system at
At the moment, only the core of the system is installed. To tune the
system to your needs, you can choose to install one or more
predefined collections of software by running the following

sudo tasksel –section server

14 packages can be updated.
4 updates are security updates.

Last login: Thu Dec 2 23:22:38 2010 from

10. At this point you want to perform some hardening and maintenance on the box.


Update passwords
EC2 Instance
ubuntu@ec2:~$ sudo su -
ubuntu@ec2:~$ passwd ubuntu

(Enter in a new password for the ‘ubuntu’ account. This is the default account on your EC2 instance. I recommend storing these passwords in KeePassX)

ubuntu@ec2:~$ passwd

(Enter in a new password for the ‘root’ account. This account should be need no explanation.)


Update packages
EC2 Instance
ubuntu@ec2:~$ exit
ubuntu@ec2:~$ sudo apt-get update

(This updates the list of known packages.)

ubuntu@ec2:~$ sudo apt-get upgrade -y

(This upgrades the installed packages to their latest version.)


If you are prompted for grub-pc config update, just hit enter. Also select ‘Yes’ at the next Grub message window.


Time Zone
EC2 Instance
ubuntu@ec2:~$ sudo dpkg-reconfigure tzdata


Follow the instructions to setup the proper timezone information for your EC2 instance.


ubuntu@ec2:~$ sudo reboot now

(This will reboot the sytem. Wait about 2 minutes before you try and reconnect to the EC2 instance via Terminal using the above ssh command.)


11. At this point I setup a host record for my EC2 instance so that I could use DNS to access it. I also configured the hostname on the system to match the DNS record. This is an optional step, and if you aren’t sure what I am talking about or aren’t sure how to do it, don’t worry about it.


12. Now that we have our EC2 instance configured and ready to go, it is time to install and configure OpenVPN. To install OpenVPN on your EC2 instance, simply type the following from within your SSH session:


EC2 Instance
ubuntu@ec2:~$ sudo apt-get -y install openvpn libssl-dev openssl


13. Now we need to create the certificates to use with OpenVPN. First let’s copy the easy-rsa tool to the OpenVPN folder.


EC2 Instance
ubuntu@ec2:~$ cd /etc/openvpn/
ubuntu@ec2:/etc/openvpn$ sudo mkdir easy-rsa
ubuntu@ec2:/etc/openvpn$ sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/
ubuntu@ec2:/etc/openvpn$ sudo chown -R $USER /etc/openvpn/easy-rsa/
ubuntu@ec2:/etc/openvpn$ cd /etc/openvpn/easy-rsa/


14. We now need to edit the ‘vars’ file to provide some information for our SSL certificates. You will need to know how to use the ‘vi’ text editor. If you don’t know how to use it, I recommend this tutorial.


EC2 Instance
ubuntu@ec2:/etc/openvpn/easy-rsa$ sudo vi vars


Change export ‘KEY_SIZE=1024’ to ‘export KEY_SIZE=2048’
Change export KEY_COUNTRY=”US” to your country.
Change export KEY_PROVINCE=”CA” to your state. I.e. ‘KEY_PROVINCE=”FL”‘
Change export KEY_CITY=”SanFrancisco” to your city. I.e. ‘KEY_CITY=”Tampa”‘
Change export KEY_ORG=”Fort-Funston” to your organization or something else. I did my family (‘KEY_ORG:”Morehouse-Family”‘)
Change export KEY_EMAIL=”me@myhost.mydomain” to your email address.


Save the file by hitting the ‘ESC’ key and then typing ‘:wq’ and press enter.


ubuntu@ec2:/etc/openvpn/easy-rsa$ source vars
ubuntu@ec2:/etc/openvpn/easy-rsa$ ./clean-all
ubuntu@ec2:/etc/openvpn/easy-rsa$ source vars
ubuntu@ec2:/etc/openvpn/easy-rsa$ ./build-ca


You should be prompted for the following. You can hit ‘enter’ to keep the default value you already setup by editing the ‘vars’ file.


Country Name (2 letter code) [US]:
State or Province Name (full name) [FL]:
Locality Name (eg, city) [Tampa]:
Organization Name (eg, company) [Morehouse-Family]:
Organizational Unit Name (eg, section) []:Personal
Common Name (eg, your name or your server’s hostname) []: -> Enter your hostname here if you created a DNS record. Otherwise enter your EC2’s Elastic IP address from Step 7.
Name []:Justin Morehouse
Email Address []:


Now execute the following commands:


ubuntu@ec2:/etc/openvpn/easy-rsa$ ./build-dh

(This takes some time. Like 2 minutes.)

ubuntu@ec2:/etc/openvpn/easy-rsa$ source vars
ubuntu@ec2:/etc/openvpn/easy-rsa$ ./pkitool --server server
ubuntu@ec2:/etc/openvpn/easy-rsa$ cd keys
ubuntu@ec2:/etc/openvpn/easy-rsa/keys$ openvpn --genkey --secret ta.key
ubuntu@ec2:/etc/openvpn/easy-rsa/keys$ sudo cp server.crt server.key ca.crt dh2048.pem ta.key /etc/openvpn/


15. Now we have created the CA and Server certificates. Now we need to create keys for our users. For the purpose of this blog, we will create one key for one user. You can repeat this step for each additional user you wish to allow to access your OpenVPN server.


EC2 Instance
ubuntu@ec2:/etc/openvpn/easy-rsa/keys$ cd..
ubuntu@ec2:/etc/openvpn/easy-rsa$ source vars
ubuntu@ec2:/etc/openvpn/easy-rsa$ ./pkitool <yourname>

(I typed ‘./pkitool justin’)

ubuntu@ec2:/etc/openvpn/easy-rsa$ cd ..


16. Now we need to create an archive to download all of the necessary files from the server to the system you want to configure to use OpenVPN (Your laptop). I recommend using Cyberduck to access the .tar file we create. Remember to use your EC2 key to login with Cyberduck. It is the key we created in Step 2 and stored in your ~/.ssh/ folder (JustinsAllEC2Key.pem). Remember, the keys.tar file will be located in the /etc/openvpn/ directory. Download the keys.tar file to your Downloads directory.


EC2 Instance
ubuntu@ec2:/etc/openvpn$ sudo tar czf keys.tgz ca.crt ta.key easy-rsa/keys/<em>yourname.crt</em> easy-rsa/keys/<em>yourname.key</em>


17. Now it’s time to configure your OpenVPN server. You can most likely use the pre-configured template I posted online. It uses the IP address scheme of for VPN clients, so unless you are using that network somewhere else, you don’t need to change a thing in the configuration. If you do need to edit the network, you can download the server.conf file here or issue the commands below and use vi to edit it as you would like. Use the commands below to download the server.conf file to the /etc/openvpn folder on your EC2 instance.


EC2 Instance
ubuntu@ec2:/etc/openvpn$ sudo wget


18. Now we have to setup ip forwarding on your EC2 instance. We’ll use sudo to perform these commands.


EC2 Instance
ubuntu@ec2:~$ sudo su -
root@ec2:~$ modprobe iptable_nat
root@ec2:~$ echo 1 &gt; /proc/sys/net/ipv4/ip_forward
root@ec2:~$ iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE
root@ec2:~$ iptables-save &gt; /etc/iptables.conf
root@ec2:~$ echo '#!/bin/sh' &gt; /etc/network/if-up.d/iptables
root@ec2:~$ echo "iptables-restore &lt; /etc/iptables.conf" &gt;&gt; /etc/network/if-up.d/iptables
root@ec2:~$ chmod +x /etc/network/if-up.d/iptables
root@ec2:~$ echo "net.ipv4.ip_forward=1" &gt;&gt; /etc/sysctl.conf
root@ec2:~$ reboot now


19. Back on your Mac, download and install Tunnelblick. It is is a free, open source Graphic User Interface (GUI) for OpenVPN on Mac OS X. You can download the latest stable version from here.


20. Once you have installed Tunnel blick, go do your ‘Downloads’ folder and extract your keys.tar files. Copy the ca.crt, ta.key, <yourname>.crt, and <yourname.key> files from the extracted .tar file to the Tunnelblick directory located at ‘~/Library/Application\ Support/Tunnelblick/Configurations/‘. (<yourname>.crt and <yourname.key> will be in the ‘easy-rsa/keys’ folder. Make sure all of the extracted files are in the ‘~/Library/Application\ Support/Tunnelblick/Configurations/‘ folder!)


21. You will now need to edit the client template that I have posted here. Download the file to ‘~/Library/Application\ Support/Tunnelblick/Configurations/‘ and edit the following three items:
  • Line 42: Change ‘<IP or hostname>’ to your EC2 instance’s IP address, from Step 7, or the DNS name you gave it.
  • Lines 89 & 90: Change cert <yourname>.crt & key <yourname>.key to the names of the .crt and .key files you extracted from the keys.tar file. This the client certificate you created for yourself in Step 15.
22. Once this is done, open up a web browser and go to IP Chicken. Obesrve your current source IP address. Then open Tunnelblick and from the menu bar at the top, select Connect ‘ec2’. Reload your browser and notice that you now have a source IP address of your EC2 instance! Congratulations on getting OpenVPN on an EC2 instance setup. Now let’s setup SideStep.


23. While Tunnelblick allows you to create an on-demand SSL tunnel to proxy all of your network traffic through your EC2 instance (for both wired and wireless) networks, SideStep takes the guess work out of when to use a proxy to secure your network when you are on an open wireless network (it currently only works on wireless networks, but Chetan is going add the capability to use it on an wired network as well). First download and install SideStep.


24. SideStep uses passwords or keys to create an on-demand SSH tunnel that proxies your traffic. As our EC2 instance doesn’t allow for password logins via SSH, we need to create a new keypair to use with SideStep. Using Terminal on your Mac, issue the following commands:


Your Mac
jmorehouse@Old-Trafford:~$ cd ~
jmorehouse@Old-Trafford:~$ ssh-keygen -t rsa -f ~/.ssh/id_ec2


Enter in a passphrase twice, and store it some place safe (KeePassX) because you will need it later.


jmorehouse@Old-Trafford:~$ scp -i .ssh/JustinsAllEC2Key.pem .ssh/ ubuntu@IP:~/.ssh/

(Key created in Step 2 and IP address from Step 7.)


25. Still within Terminal, log back into your EC2 instance and append the public key to your authorized_keys file.


Your Mac
jmorehouse@Old-Trafford:~$ cd ~
jmorehouse@Old-Trafford:~$ ssh -i ~/.ssh/&lt;filename&gt;.pem ubuntu@IPAddress

(Key created in Step 2 and IP address from Step 7.)


EC2 Instance
ubuntu@ec2:~$ cd .ssh/
ubuntu@ec2:~/.ssh/$ cat &gt;&gt; authorized_keys
ubuntu@ec2:~/.ssh/$ chmod 640 authorized_keys
ubuntu@ec2:~/.ssh/$ exit


26. Now we need OSX to prompt us for the passphrase for the id_ec2 key, so from Terminal, enter the following:


Your Mac
jmorehouse@Old-Trafford:~$ cd ~
jmorehouse@Old-Trafford:~$ ssh -i .ssh/id_ec2 ubuntu@IP


You should be prompted for a password. Check the save the password to your Key Chain and hit ok. You should now have an SSH session to your EC2 box using your new key. You can go ahead and exit from your SSH session and close out all of your Terminal sessions and quit the Terminal application.


27. Now fire up SideStep and click the ‘Next’ button. Under ‘I already have one’ enter ‘ubuntu’ as the Username, your IP address from Step 7 as the hostname, and press ‘Test Connection to Server.’ You should receive a ‘Connection to server succeeded!’ message. Now click the ‘Next’ button. Read the notes and check the box that reads ‘Run SideStep on login.’ Click ‘Finish.’


28. SideStep is now on the menu bar next to Tunnelblick. I added Tunnelblick to my login items so that it is launched when I boot. Understand the differences between these two tools (Tunnelblick and SideStep) and when to use each.


Congratulations! If you made it this far, pat yourself on the back. This was a long tutorial, but it should work if you followed each step. If you have any problems, hit me up on Twitter (@Mascasa).


Enjoy surfing open wireless networks or hostile wired network securely!