Browsed by
Category: Algorithms

AWS vs Packet.net
Why we left AWS

Benchmarking AWS’s Network, Disk, and CPU Performance

AWS vs Packet.net
Why we left AWS

Benchmarking AWS’s Network, Disk, and CPU Performance

If this sounds like a glowing review of Packet.net – it is. I found myself re-reading this post over and over, trying to make it sound less shrilly – but I can’t. It’s just a ridiculously good product and value – EC2 containers just don’t make sense anymore.

A friend once told me, “Rishi – sometimes if you don’t advocate a product aggressively – you can be doing society a disservice in your attempt to be neutral. If the value is so good, you must tell everybody about it.”

This is one of those times.

EDIT: Feeling really grateful the HackerNews community decided to link to Tiingo a second time. In the first HackerNews posting many of you asked for an API, which is what led to me finding the AWS bottleneck. The API launched [quietly] this week at: https://api.tiingo.com where Tiingo is now the first company to bring IEX (anti-HFT exchange/darkpool) data to mainstream Financial Tech firms. Kind of went full-circle as this post wouldn’t have existed without the original HN coverage.

TL;DR:

The performance of AWS on network speed, disk speed, and CPU performance are quantitatively just “not good,” for what we needed. When we introduced real-time market data, we were in search of our bottleneck and realized it was AWS. We made the decision to switch to Packet.net and the below reflects on our decision and explains why. The benchmarks continue to reaffirm our decision. Having said all of this, certain features of AWS remain incredibly convenient like S3, Cloudfront, and Route53 – but we can’t justify using EC2.

In Networking: Packet is significantly faster, more stable, and 15%-44% cheaper

In Disk Usage: Packet is more performant and 92% cheaper

In CPU: Packet is 30-40% more performant and 15% cheaper

In machines: Packet’s systems are all bare-metal/dedicated, whereas AWS charges extra for dedicated machines

If you’ve noticed Tiingo being particularly snappy these days, it’s because I couldn’t stand it anymore. I had tried everything – buying more expensive instances on AWS, allocating more space, scaling horizontally, but it wasn’t matching up to my local dev machines. And so I started searching for the bottleneck – only to realize it was AWS.

I started researching AWS, I found I wasn’t alone. Many people experienced what I had but I tried prolonging the switch. Trying to change cloud service providers is  frustrating: scripts break, performance temporarily suffers, you experience downtime, and you know there will be unknown-unknowns.

Recently we just got real-time market data and this exacerbated the issues. Our websockets were being overwhelmed in queues and throwing more nodes at the problem was becoming expensive. We were trying to put a bandaid over a burst pipe. I finally decided on Packet.net and I want to share the reasons why. I’ve included benchmarking results to help emphasize the point.

Our search was motivated by two major reasons:

  1. The costs were getting out-of-hand
  2. After reading the below Reddit post on AWS’s [lack of] network stability, we started asking around and realized the experts were right… AWS’s network is slow. If we are going to give our users real-time data directly from the exchanges, that’s a heck-of-a-lot of data and we need it to be as fast as possible.

The Reddit/Blog Post was from an engineer at Stack Overflow.

http://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/.

More specifically, this Reddit comment on AWS’s network stability that seemed echo’d by many:

Reddit Comment Inspiring Inquisition
https://www.reddit.com/comments/468p2m

We explored options like DigitalOcean, but Tiingo, like all financial data/analytics companies, is very data heavy and their plans didn’t allow for flexible data storage (EBS on Amazon for example). We looked into Rackspace and Azure, but the cost differentials didn’t make it seem worth the transition. Admittedly, having used Rackspace in the past – I’ve always loved their customer support and was personally disappointed I couldn’t justify the cost.

Eventually I came across Packet and spoke to their engineers since I hadn’t heard of them before.

I took a chance. It paid off.

I told them my concerns and what I was trying to solve (market data connectivity and high data transfer rates). One of the co-founders, who was a networking engineer, personally oversaw my connectivity project to the data exchanges. I’m pretty sure this was Paul Graham 101 on start-ups and customer service.

Ultimately though – I’m a data nut and so I decided to benchmark AWS vs Packet and was really curious about the Reddit comments on AWS’s network stability. The benchmarks closed the deal for us. It was a no-brainer. Part of the major reason being that Packet.net is bare metal (dedicated physical machines) whereas AWS tends to be focused on virtual machines. The hardware/pricepoint is actually even cheaper on Packet. We are paying 1/3rd of what it would cost to get a similar, less performant, system on AWS.

SO here you have it!

The tests below compare AWS vs Packet for disk, network, and CPU benchmarking – and also cost.

I’ve outlined and commented the results below so you can reproduce the tests.

Hardware

Since we are testing Packet vs AWS, we started off with the Packet hardware and found the AWS price equivalent. We started with the Type 1 and worked backwards to find the equivalent in performance/price on AWS.

Note: For the network test, we also test a smaller machine. The reason for the lighter hardware is for load balancing (HAProxy in this sense). If all of the back-end servers can have high network throughput, but we need to send it to the end-user, the load-balancer’s networking performance will be the determining factor. This is especially important in cases like real-time data.

Packet:

Instance Name CPU Memory/RAM Price Price/Month Notes
Type 1 (Server) (4 core, 8-threaded)  3.4ghz Intel Xeon E3-1240 v3 32gb $0.40/hr

$0.37/hr if reserved for 1 month

$292.80/month
Type 0 (Load Balancer) (4 core) 2.4ghz Intel Atom C2550 8gb $0.05/hr

$0.0459/hr if reserved for 1 month

$36.60/month What somebody may choose as their load balancer

*Note:We assume 732 hours in a month; but if you reserve a Packet instance for a month, they will only charge you 672 hours per month. However, to make apples-to-apples comparisons, all calcs in Price/Month assume you choose hourly pricing (732 hours for 1 month) to keep things normalized.

AWS:

Instance Name CPU Memory/RAM Price Notes
m4.2xlarge (Server) 8 VCPU  (2.4ghz Intel Xeon E5-2676) 32gb $0.479/hr $350.63/month xlarge was chosen for it’s optimized network performance
t2.medium (Load Balancer) 2 VCPU  (Xeon processors burstable to 3.3ghz) 4gb $0.052/hr $38.07/month What somebody may choose as their load balancer

OS:

Ubuntu 14.04 server

The Benchmarks

Network:

For this test, we used iperf3 as per the AWS documentation

(https://aws.amazon.com/premiumsupport/knowledge-center/network-throughput-benchmark-linux-ec2/)

We wanted to simulate a very real-world network configuration for ourselves – basically what our site looks behind a load balancer. Load balancers tend to require very low processing power, and serve as a network bottleneck to the user.

We are testing:

  • Internet -> Load-balancer (Haproxy)
  • Load-balancer (HAProxy) -> Server
  • Server -> Server

The “Internet” machine used was an Azure machine. Not perfect, but we figured it was a good 3rd party control.

You can view the detailed methodology in the Appendix below.

Results:

Performance:

AWS came out incredibly inconsistent – with a high std. deviation and low mean transfer rates. What AWS considered a “High” performance network tier, was the least expensive tier on Packet. Why didn’t we use AWS Elastic-Load-Balancer (ELB)? For our use case with websockets, – we found ELB to be lacking what we needed. This will be a blog post for a later day.

Comparing transfer speeds across machines on Packet vs AWS

What was particularly interesting was the inconsistency of the lower tier machines. We ran our benchmarks over an hour, and here is what the rates looked like when making requests to-and-from the lower tier (t2.medium) EC2 Instance. This seems consistent with their “burstable” instance – which is great and all…except Packet’s lowest tier outperforms it:

AWS's speeds decay significantly over time for the T2.medium instance- making it a poor choice for a load balancer
AWS’s speeds decay significantly over time for the T2.medium instance- making it a poor choice for a load balancer

Pricing:

The above AWS configuration is $.081/hour more expensive than Packet and also less performant.

Another consideration is bandwidth costs. AWS charges $0.09/GB (for the first 10TB) out to the internet. Packet.net charges $0.05/GB out to the internet. Within the same data centers (availability zones in AWS), both Packet and AWS are free. However, when transferring to a different availability zone, AWS charges $0.02/GB and Packet.net charges $0.05/GB.

Conclusion:

Packet is the clear winner in this. In both absolute speed and stability. In terms of price, Packet is cheaper by $.081/hour in the above configuration, or 15% cheaper – and for the majority of our bandwidth we go external to the internet. In outbound internet traffic, Packet is 44% cheaper.

Disk:

Packet offers two storage types: Basic (500 IOPS) and Performance (15,000 IOPS).

We created a EBS volume on both Packet & AWS with provisioned IOPS of 500 and then 15,000. Then we used sysbench to run an I/O test (see Appendix below for methodology).

Results:

Performance:

When getting to the 15k IOPS, we saw a more significant performance differential favoring Packet. At Tiingo we used the performance tier given the amount of data we store and calculate.

Disk Trasnfer Rates

Price:

Provisioning 15,000 IOPS on AWS @ $0.065/IOPS  = $975. But wait, that’s not all! They also charge $0.125/hour per GB.  So a 15k IOPS 500GB HDD on AWS would be $1037.50

On Packet it would be 500GB * $0.15 = $75.

Doing  a bit of algebra, the cost for 15k IOPS on AWS would be cost effective if you have >39TB of storage. That’s right – Packet is cheaper until you hit 39TB of storage….

Conclusion:

Packet is literally 92.3% cheaper than AWS for 15k IOPS performance, and Packet is even more performant. It’s the victor in disk performance as well.

CPU:

CPUs cannot be benchmarked purely on the speed of the processor [clock] alone.  For these reasons, we ran a sysbench test as well on different threads.

Results:

Performance:

The results are damning for AWS. On an 8 processor machine, the benchmark ran slower on 8 cores than 4. I ran this multiple times, double checked to make sure this was an m4.2xlarge. Then I spun up another m4.2xlarge and the results were more in line with what I expected (still slower than Packet).
However, I am going to keep the original instance’s benchmark below to highlight the point of noisy neighbors. With AWS, you can get a shared machine with other neighbors who are processor intensive and reduce your performance. This is what virtualization is. With Packet, you get a dedicated system. What most likely happened was that our original machine had a noisy neighbor.

Here are the results – you can see at 8 threads Packet performs 4x faster than AWS.

Packet is 4x Faster on This Noisy Neighbor AWS Machine

OK OK – I will show the second instance’s performance – even when there are no noisy neighbors.

Packet is 30-40% faster even with a better AWS instance
Packet is 30-40% faster even with a better AWS instance

Even with a non-noisy neighbor machine, Packet is 30-40% faster in processor benchmarks.

EDIT: A user asked me to run the benchmark using a compute-optimized EC2 instance. I decided on c4.2xlarge which has 8 threads, but half as much memory (16gb). It cost $0.419/hour ($0.019/hr more expensive than a Type1 Packet server). Here are the results (Packet wins again but less drastic of a margin)

Even using AWS Compute-optimized, Packet Type 1 outperforms it

Price:

On the above setup, Packet is $0.079/hour cheaper.

Conclusion:

There really is no way around it – the above benchmarks show the issues with virtualization. Even with those issues aside, AWS is slower and more expensive. Packet wins this one again.

Conclusion

Even giving AWS the benefit of the doubt, there is no way around it – Packet is faster and SIGNIFICANTLY cheaper.

Let’s take a very real-world example of our server set-up:

Packet:

Instance Name CPU Memory/RAM Price Price/Month
Type 1 (Server) (4 core, 8-threaded)  3.4ghz Intel Xeon E3-1240 v3 32gb $0.40/hr

$0.37/hr if reserved for 1 month*

$292.80/month
Type 0 (Load Balancer) (4 core) 2.4ghz Intel Atom C2550 8gb $0.05/hr

$0.0459/hr if reserved for 1 month*

$36.60/month
15k IOPS 1TB HDD $0.15/GB $150/month
2TB Outbound Bandwidth $.05/GB $100/month
Total $579.40/month

*Note:We assume 732 hours in a month; but if you reserve a Packet instance for a month, they will only charge you 672 hours per month. However, to make apples-to-apples comparisons, all calcs in Price/Month assume you choose hourly pricing (732 hours for 1 month) to keep things normalized.

AWS:

Instance Name CPU Memory/RAM Price Price/Month
m4.2xlarge (Server) 8 VCPU  (2.4ghz Intel Xeon E5-2676) 32gb $0.479/hr $350.63/month
t2.medium (Load Balancer) 2 VCPU  (Xeon processors burstable to 3.3ghz) 4gb $0.052/hr $38.07/month
15k IOPS 1TB HDD $0.125/GB + $0.065/provisioned IO $1,100/month
2TB Outbound Bandwidth $.09/GB $184.23/month
Total $1,838.84/month

Packet is literally less than 1/3rd the price and is more performant than AWS.

It’s allowed us to deploy resources we didn’t think would be affordable before.

Thank you to everyone @ Packet for making this product possible.

Further Steps:

If anybody wants to continue this study, I would love to hear your results. AWS does allow you dedicated machines for extra $, but we didn’t bother testing them since Packet is already cheaper than their virtual machines.

Appendix:

Methodology:

Networking:

Setting up AWS:

We want to make sure we give AWS the best chance. First, we have to make sure enhanced networking is enabled. Running the command:

modinfo ixgbevf

Will give us the output, and look for “version”. In our instance we have version 2.11.3-k. Amazon recommends we upgrade. For the ubuntu users out there, follow this gist and run the commands:

ixgbevf 2.16.1 upgrade for AWS EC2 SR-IOV “Enhanced Networking” on Ubuntu 14.04 (Trusty) LTS

After rebooting run:

modinfo ixgbevf

Again to make sure the version now reads: 2.16.1

Let’s also check via command line to make sure enhanced networking is supported (Ubuntu 14.04):

sudo apt-get install python-pip
sudo python pip --upgrade pip
sudo pip install awscli
#Note: Create an IAM user and attach the policy: AmazonEC2ReadOnlyAccess
#Use the security credentials in the configure policy
aws configure
#after configuring run (replacing instance_id with your instance_id):
aws ec2 describe-instance-attribute --instance-id instance_id --attribute sriovNetSupport

If you get the output, you’re good:

"SriovNetSupport": 
{ 
"Value": "simple" 
},

Next, we used iperf3 to run the diagnostic scripts and scrapy bench. iperf3 is a common network benchmarking tool and scrapy is the framework that powers Tiingo’s scraper farm. We figured Scrapy would be another real-time test to see how things flow.

the iperf3 command was:

iperf3 -B internal_ip_of_current_machine -c internal_ip_of_iperf_server -i 1 -t 3600 -V -p 80 -P 10 --logfile test.txt

Meaning we ran the tests for one hour (3600 seconds), and with 10 processors in parallel. Also note to set the -B option on Packet machines as it takes advantage of the full bonding algo and increases thoroughput.

Note: make sure to use the internal IP addresses to give the best benefit of doubt 🙂

Disk:

First install/update sysbench on your Ubuntu machine using the code:

echo "deb http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
echo "deb-src http://repo.percona.com/apt trusty main" >> /etc/apt/sources.list.d/percona.list
apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A
apt-get update
apt-get install sysbench

Then we used the command:

sysbench --test=fileio --file-total-size=150G --file-test-mode=rndwr --max-time=720 --max-requests=0 --num-threads=8 --file-num=64 --file-io-mode=async --file-extra-flags=direct --file-fsync-freq=0 run

The file size must be greater than the RAM size for this test to properly work.

CPU:

See the above “Disk” section to set up sysbench.

We then ran the command below, replacing “num-threads” with 1, 4, and 8 respectively

sysbench --test=cpu --cpu-max-prime=20000 run --num-threads=1
Facebooktwitterredditpinterestlinkedinmail
Launching the Tiingo Open Data Initiative

Launching the Tiingo Open Data Initiative

Since day one, Tiingo has been committed to providing you with top quality data that is more accurate than companies charging $30k+ a year.

How in the world is a company with a “set your own price” model going to pull this off?

Because we’re going to do something unheard of.

Presenting the Tiingo Open Data Initiative

Sound sexy?

No? Good, because clean data should be boring. Except we’re still going to try and make it sexy (we like doing the impossible).

aapl div

 

We’ve all been there, looking at a number on a financial tool and wondering, “is that number actually right?” Then we might go to another website or source and double check. Even if we see two equal numbers, we think, “hmm… ok.”

And our skepticism is well-warranted. Often there will be 1 or 2 main vendors for the same source of data. If one vendor is wrong, then many financial sites are wrong.

So what if a company could show you where, when, and how they got their numbers? This is what our Open Data Initiative is about: transparency.

Now within less than a second, you can verify Tiingo’s numbers straight from the official source: press releases. Either hover your mouse over the orange binoculars or click the “Source” link directly to see where, when, and how we got our data.

Try it here:
AAPL Dividend History

Cool huh?

Dividends are just the start.

BUT WAIT THERE’S MORE!
Since the ethos of Tiingo is to “Actively Do Good,” when we are ready we will open all of this data to the world via an API. Right now when we catch mistakes, we are notifying our data vendor so they can fix the data for all their users. We don’t believe in holding good data hostage.

A quick aside: back-populating dividend data and sourcing it is a data intensive process but we are working around the clock to load in this data historically. However, future dividends are being monitored and added in real-time.

From Tiingo with Love

-Rishi

 

Facebooktwitterredditpinterestlinkedinmail
That’s Enough Machine Learning – thanks!

That’s Enough Machine Learning – thanks!

Alright – so I’m going to hammer on one specific topic that’s been bothering me in the tech scene and that’s just machine learning being thrown everywhere. “Need a t-shirt? Let’s use machine learning to find our different habits and predict our tastes.” Or, you know, you could go to a store and see what appeals to you. OK that’s an exaggeration and going to stores and checking merchandise doesn’t scale across variety the web offers you. But I like this analogy so I’m going to keep it.

The problem I see with machine learning, and why I think it’s overused in markets inappropriately, is that it cannot explain in the same way human consciousness can. What I mean by that is that traditional science tells us to form a hypothesis before conducting an experiment. The idea being that by forming an explanation before seeing the data, we are forced to take current observations and make a rational expectation. This of course leads to biases which is shown quantitatively by the inability to replicate research as well as the number of papers that seem to support their hypothesis. What “big data” (I throw up a little in my mouth when I use that phrase) presents us though is the ability to get instant iterative feedback and A/B testing lets us test our samples in the real-world and see if our models hold up.

This is how it “should” be done. What happens though is that machine learning instead of being used as an optimization method becomes used as a method of find explanations. Many of us are using it to find relationships and then we are are backfilling a hypothesis and shows to be the case. While the current method of science is far from perfect, this approach seems far far worse. I have seen some who can master this, but they often have very strict processes in place to ensure the models hold up. Some enforce it via risk management while others run statistical tests – usually a combination of the two.

But do we really need to use advanced machine learning to create explanatory relationships instead of being an optimization method? After speaking with many people using it this way and reading papers on it, it seems like many doing it drastically overfits and their live results/trading do not match their out-of-sample. A common response to this idea is that, “machine learning should work if we properly out-of-sample tests.” Well, something taught to me by Josh + Steve @ AlphaParity (on this list), was that many people inappropriately run out-of-sample tests. What people often do is they initially have an in-sample and out-of-sample but when out-of-sample doesn’t match the in-sample performance, they parameterize the in-sample until the out-of-sample matches what they want. This creates just one in-sample and no out-of-sample.

Using machine learning as an explanatory relationship finder often leads to complexity of models, which just further adds the probability of overfitting. A secondary problem with markets is that regime shifts can happen rapidly, making machine learning less effective on larger time periods where there become new macro drivers. While it absolutely can be done, I know only one who has pulled it off and I have no idea how they do it. The question is, that all of this complexity worth it? The largest hedge funds out there like AQR do not use it to find explanatory relationships but use it for what it was meant to be: an optimization algorithm that slightly boosts performance. The simplicity of models like this reduce the chances of overfitting and also allow us to know when a model will break – when there will a regime shift. This knowing-when-it-fails allow us to assign higher odds as to when to size down risk (or weighting in non-market cases), or use portfolio construction to provide correlation/diversification benefit.

So before we go crazy with machine learning trying to be predictive from the start, I think it’s worthwhile to test the relationships and run studies and then consider ML at a “tweaking” stage. When used properly, it can be an effective tool, I just don’t think as effective as the mass-adoption of this phrase implies for the vast majority of cases. I think a good example of those who properly used it were the winners behind the Netflix Prize, where their solution is public. Their initial papers explored biases and preferences people had when initially ranking movies. Their final solution contained different ML and statistical methods to push results over the edge. Reading Team BellKor’s Pragmatic Chaos’s papers in sequential order is good fun: Direct link to final paper. Ignoring the math, their logic and explanations are fantastic displays of the scientific method + optimization methods.

Facebooktwitterredditpinterestlinkedinmail