If this sounds like a glowing review of Packet.net – it is. I found myself re-reading this post over and over, trying to make it sound less shrilly – but I can’t. It’s just a ridiculously good product and value – EC2 containers just don’t make sense anymore.

A friend once told me, “Rishi – sometimes if you don’t advocate a product aggressively – you can be doing society a disservice in your attempt to be neutral. If the value is so good, you must tell everybody about it.”

This is one of those times.

EDIT: Feeling really grateful the HackerNews community decided to link to Tiingo a second time. In the first HackerNews posting many of you asked for an API, which is what led to me finding the AWS bottleneck. The API launched [quietly] this week at: https://api.tiingo.com where Tiingo is now the first company to bring IEX (anti-HFT exchange/darkpool) data to mainstream Financial Tech firms. Kind of went full-circle as this post wouldn’t have existed without the original HN coverage.

If you’ve noticed Tiingo being particularly snappy these days, it’s because I couldn’t stand it anymore. I had tried everything – buying more expensive instances on AWS, allocating more space, scaling horizontally, but it wasn’t matching up to my local dev machines. And so I started searching for the bottleneck – only to realize it was AWS.

I started researching AWS, I found I wasn’t alone. Many people experienced what I had but I tried prolonging the switch. Trying to change cloud service providers is  frustrating: scripts break, performance temporarily suffers, you experience downtime, and you know there will be unknown-unknowns.

Recently we just got real-time market data and this exacerbated the issues. Our websockets were being overwhelmed in queues and throwing more nodes at the problem was becoming expensive. We were trying to put a bandaid over a burst pipe. I finally decided on Packet.net and I want to share the reasons why. I’ve included benchmarking results to help emphasize the point.

Our search was motivated by two major reasons:

1. The costs were getting out-of-hand
2. After reading the below Reddit post on AWS’s [lack of] network stability, we started asking around and realized the experts were right… AWS’s network is slow. If we are going to give our users real-time data directly from the exchanges, that’s a heck-of-a-lot of data and we need it to be as fast as possible.

The Reddit/Blog Post was from an engineer at Stack Overflow.

http://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/.

More specifically, this Reddit comment on AWS’s network stability that seemed echo’d by many:

We explored options like DigitalOcean, but Tiingo, like all financial data/analytics companies, is very data heavy and their plans didn’t allow for flexible data storage (EBS on Amazon for example). We looked into Rackspace and Azure, but the cost differentials didn’t make it seem worth the transition. Admittedly, having used Rackspace in the past – I’ve always loved their customer support and was personally disappointed I couldn’t justify the cost.

Eventually I came across Packet and spoke to their engineers since I hadn’t heard of them before.

I took a chance. It paid off.

I told them my concerns and what I was trying to solve (market data connectivity and high data transfer rates). One of the co-founders, who was a networking engineer, personally oversaw my connectivity project to the data exchanges. I’m pretty sure this was Paul Graham 101 on start-ups and customer service.

Ultimately though – I’m a data nut and so I decided to benchmark AWS vs Packet and was really curious about the Reddit comments on AWS’s network stability. The benchmarks closed the deal for us. It was a no-brainer. Part of the major reason being that Packet.net is bare metal (dedicated physical machines) whereas AWS tends to be focused on virtual machines. The hardware/pricepoint is actually even cheaper on Packet. We are paying 1/3rd of what it would cost to get a similar, less performant, system on AWS.

SO here you have it!

The tests below compare AWS vs Packet for disk, network, and CPU benchmarking – and also cost.

I’ve outlined and commented the results below so you can reproduce the tests.

Hardware

Since we are testing Packet vs AWS, we started off with the Packet hardware and found the AWS price equivalent. We started with the Type 1 and worked backwards to find the equivalent in performance/price on AWS.

Note: For the network test, we also test a smaller machine. The reason for the lighter hardware is for load balancing (HAProxy in this sense). If all of the back-end servers can have high network throughput, but we need to send it to the end-user, the load-balancer’s networking performance will be the determining factor. This is especially important in cases like real-time data.

Packet:

Instance Name	CPU	Memory/RAM	Price	Price/Month	Notes
Type 1 (Server)	(4 core, 8-threaded)  3.4ghz Intel Xeon E3-1240 v3	32gb	$0.40/hr $0.37/hr if reserved for 1 month	$292.80/month
Type 0 (Load Balancer)	(4 core) 2.4ghz Intel Atom C2550	8gb	$0.05/hr $0.0459/hr if reserved for 1 month	$36.60/month	What somebody may choose as their load balancer

*Note:We assume 732 hours in a month; but if you reserve a Packet instance for a month, they will only charge you 672 hours per month. However, to make apples-to-apples comparisons, all calcs in Price/Month assume you choose hourly pricing (732 hours for 1 month) to keep things normalized.

AWS:

Instance Name	CPU	Memory/RAM	Price	Notes
m4.2xlarge (Server)	8 VCPU  (2.4ghz Intel Xeon E5-2676)	32gb	$0.479/hr	$350.63/month	xlarge was chosen for it’s optimized network performance
t2.medium (Load Balancer)	2 VCPU  (Xeon processors burstable to 3.3ghz)	4gb	$0.052/hr	$38.07/month	What somebody may choose as their load balancer

OS:

Ubuntu 14.04 server

The Benchmarks

Network:

For this test, we used iperf3 as per the AWS documentation

(https://aws.amazon.com/premiumsupport/knowledge-center/network-throughput-benchmark-linux-ec2/)

We wanted to simulate a very real-world network configuration for ourselves – basically what our site looks behind a load balancer. Load balancers tend to require very low processing power, and serve as a network bottleneck to the user.

We are testing:

• Internet -> Load-balancer (Haproxy)
• Load-balancer (HAProxy) -> Server
• Server -> Server

The “Internet” machine used was an Azure machine. Not perfect, but we figured it was a good 3rd party control.

You can view the detailed methodology in the Appendix below.

Results:

Performance:

AWS came out incredibly inconsistent – with a high std. deviation and low mean transfer rates. What AWS considered a “High” performance network tier, was the least expensive tier on Packet. Why didn’t we use AWS Elastic-Load-Balancer (ELB)? For our use case with websockets, – we found ELB to be lacking what we needed. This will be a blog post for a later day.

What was particularly interesting was the inconsistency of the lower tier machines. We ran our benchmarks over an hour, and here is what the rates looked like when making requests to-and-from the lower tier (t2.medium) EC2 Instance. This seems consistent with their “burstable” instance – which is great and all…except Packet’s lowest tier outperforms it:

Pricing:

The above AWS configuration is $.081/hour more expensive than Packet and also less performant.

Another consideration is bandwidth costs. AWS charges $0.09/GB (for the first 10TB) out to the internet. Packet.net charges $0.05/GB out to the internet. Within the same data centers (availability zones in AWS), both Packet and AWS are free. However, when transferring to a different availability zone, AWS charges $0.02/GB and Packet.net charges $0.05/GB.

Conclusion:

Packet is the clear winner in this. In both absolute speed and stability. In terms of price, Packet is cheaper by $.081/hour in the above configuration, or 15% cheaper – and for the majority of our bandwidth we go external to the internet. In outbound internet traffic, Packet is 44% cheaper.

Disk:

Packet offers two storage types: Basic (500 IOPS) and Performance (15,000 IOPS).

We created a EBS volume on both Packet & AWS with provisioned IOPS of 500 and then 15,000. Then we used sysbench to run an I/O test (see Appendix below for methodology).

Results:

Performance:

When getting to the 15k IOPS, we saw a more significant performance differential favoring Packet. At Tiingo we used the performance tier given the amount of data we store and calculate.

Price:

Provisioning 15,000 IOPS on AWS @ $0.065/IOPS  = $975. But wait, that’s not all! They also charge $0.125/hour per GB.  So a 15k IOPS 500GB HDD on AWS would be $1037.50

On Packet it would be 500GB * $0.15 = $75.

Doing  a bit of algebra, the cost for 15k IOPS on AWS would be cost effective if you have >39TB of storage. That’s right – Packet is cheaper until you hit 39TB of storage….

Conclusion:

Packet is literally 92.3% cheaper than AWS for 15k IOPS performance, and Packet is even more performant. It’s the victor in disk performance as well.

CPU:

CPUs cannot be benchmarked purely on the speed of the processor [clock] alone.  For these reasons, we ran a sysbench test as well on different threads.

Results:

Performance:

The results are damning for AWS. On an 8 processor machine, the benchmark ran slower on 8 cores than 4. I ran this multiple times, double checked to make sure this was an m4.2xlarge. Then I spun up another m4.2xlarge and the results were more in line with what I expected (still slower than Packet).
However, I am going to keep the original instance’s benchmark below to highlight the point of noisy neighbors. With AWS, you can get a shared machine with other neighbors who are processor intensive and reduce your performance. This is what virtualization is. With Packet, you get a dedicated system. What most likely happened was that our original machine had a noisy neighbor.

Here are the results – you can see at 8 threads Packet performs 4x faster than AWS.

OK OK – I will show the second instance’s performance – even when there are no noisy neighbors.

Even with a non-noisy neighbor machine, Packet is 30-40% faster in processor benchmarks.

EDIT: A user asked me to run the benchmark using a compute-optimized EC2 instance. I decided on c4.2xlarge which has 8 threads, but half as much memory (16gb). It cost $0.419/hour ($0.019/hr more expensive than a Type1 Packet server). Here are the results (Packet wins again but less drastic of a margin)

Price:

On the above setup, Packet is $0.079/hour cheaper.

Conclusion:

There really is no way around it – the above benchmarks show the issues with virtualization. Even with those issues aside, AWS is slower and more expensive. Packet wins this one again.

Conclusion

Even giving AWS the benefit of the doubt, there is no way around it – Packet is faster and SIGNIFICANTLY cheaper.

Let’s take a very real-world example of our server set-up:

Packet:

Instance Name	CPU	Memory/RAM	Price	Price/Month
Type 1 (Server)	(4 core, 8-threaded)  3.4ghz Intel Xeon E3-1240 v3	32gb	$0.40/hr $0.37/hr if reserved for 1 month*	$292.80/month
Type 0 (Load Balancer)	(4 core) 2.4ghz Intel Atom C2550	8gb	$0.05/hr $0.0459/hr if reserved for 1 month*	$36.60/month
15k IOPS 1TB HDD			$0.15/GB	$150/month
2TB Outbound Bandwidth			$.05/GB	$100/month
Total				$579.40/month

*Note:We assume 732 hours in a month; but if you reserve a Packet instance for a month, they will only charge you 672 hours per month. However, to make apples-to-apples comparisons, all calcs in Price/Month assume you choose hourly pricing (732 hours for 1 month) to keep things normalized.

AWS:

Instance Name	CPU	Memory/RAM	Price	Price/Month
m4.2xlarge (Server)	8 VCPU  (2.4ghz Intel Xeon E5-2676)	32gb	$0.479/hr	$350.63/month
t2.medium (Load Balancer)	2 VCPU  (Xeon processors burstable to 3.3ghz)	4gb	$0.052/hr	$38.07/month
15k IOPS 1TB HDD			$0.125/GB + $0.065/provisioned IO	$1,100/month
2TB Outbound Bandwidth			$.09/GB	$184.23/month
Total				$1,838.84/month

Packet is literally less than 1/3rd the price and is more performant than AWS.

It’s allowed us to deploy resources we didn’t think would be affordable before.

Thank you to everyone @ Packet for making this product possible.

Further Steps:

If anybody wants to continue this study, I would love to hear your results. AWS does allow you dedicated machines for extra $, but we didn’t bother testing them since Packet is already cheaper than their virtual machines.

Appendix:

Methodology:

Networking:

Setting up AWS:

We want to make sure we give AWS the best chance. First, we have to make sure enhanced networking is enabled. Running the command:

modinfo ixgbevf

Will give us the output, and look for “version”. In our instance we have version 2.11.3-k. Amazon recommends we upgrade. For the ubuntu users out there, follow this gist and run the commands:

ixgbevf 2.16.1 upgrade for AWS EC2 SR-IOV “Enhanced Networking” on Ubuntu 14.04 (Trusty) LTS

After rebooting run:

modinfo ixgbevf

Again to make sure the version now reads: 2.16.1

Let’s also check via command line to make sure enhanced networking is supported (Ubuntu 14.04):

sudo apt-get install python-pip
sudo python pip --upgrade pip
sudo pip install awscli
#Note: Create an IAM user and attach the policy: AmazonEC2ReadOnlyAccess
#Use the security credentials in the configure policy
aws configure
#after configuring run (replacing instance_id with your instance_id):
aws ec2 describe-instance-attribute --instance-id instance_id --attribute sriovNetSupport

If you get the output, you’re good:

"SriovNetSupport": 
{ 
"Value": "simple" 
},

Next, we used iperf3 to run the diagnostic scripts and scrapy bench. iperf3 is a common network benchmarking tool and scrapy is the framework that powers Tiingo’s scraper farm. We figured Scrapy would be another real-time test to see how things flow.

the iperf3 command was:

iperf3 -B internal_ip_of_current_machine -c internal_ip_of_iperf_server -i 1 -t 3600 -V -p 80 -P 10 --logfile test.txt

Meaning we ran the tests for one hour (3600 seconds), and with 10 processors in parallel. Also note to set the -B option on Packet machines as it takes advantage of the full bonding algo and increases thoroughput.

Note: make sure to use the internal IP addresses to give the best benefit of doubt 🙂

Disk:

First install/update sysbench on your Ubuntu machine using the code:

echo "deb http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
echo "deb-src http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
apt-get update
apt-get install sysbench

Then we used the command:

sysbench --test=fileio --file-total-size=150G --file-test-mode=rndwr --max-time=720 --max-requests=0 --num-threads=8 --file-num=64 --file-io-mode=async --file-extra-flags=direct --file-fsync-freq=0 run

The file size must be greater than the RAM size for this test to properly work.

CPU:

See the above “Disk” section to set up sysbench.

We then ran the command below, replacing “num-threads” with 1, 4, and 8 respectively

sysbench --test=cpu --cpu-max-prime=20000 run --num-threads=1