All Clouds Are Not Created Equal – A Unix Benchmark of Amazon EC2 vs AgileCLOUD

While many things have changed in the cloud computing arena over the last three years, at the end of the day what you’re buying hasn’t: you’re getting a server with CPU, RAM, and some kind of disk.  Nowhere is this truer than over at Amazon, where you buy CPU power that’s measured in EC2 Compute Units. Amazon defines these as “… equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor“.

On the surface, it seems quite helpful that they’re abstracting something as complicated as a CPU and offering you a simple to understand unit.  Who wants to look at front side bus speeds?  Who cares where the memory controller lives?  Storage is storage and you buy it in Gigabytes. RAM is RAM, and you buy it in Gigabytes. So why not sell CPUs the same way?  Well, as Intel and AMD will attest, not all 2.4GHz CPUs are created equal.

Suppose you want to buy an EC2 instance with four compute units. Since we’re basing this on 2007 technology, this means that the power might be equivalent to a 2.4GHz AMD Opteron 250 (2×1.2GHz), or it might be equivalent to a 2.0GHz Xeon (2×1GHz). Both of these CPUs offer you two compute units.  In the real world though, the AMD Opteron 250 is more than twice as powerful as the Xeon 2.0 GHz. All of a sudden our yardstick can stretch from two feet to six feet.

The problem with trying to base a unit of CPU on clockspeed is that it’s no longer (and hasn’t been for some time) the sole determining factor in CPU performance. It’s like buying a car based only on horsepower.  On paper, the 403hp Cadillac Escalade is more powerful than a 375hp Ferrari F355 – which do you suppose is faster accelerating to 100mph?

With AgileCLOUD, you’re buying the Ferrari, and it’s made by Intel.  All AgileCLOUD instances are powered by the latest Intel Nehalem CPUs.

Let’s compare an Amazon instance with an AgileCLOUD instance and see what this actually means.  We’ll take a quick look and compare an entry-level AgileCLOUD instance with an Amazon Large instance.

• The Amazon Large Instance is $0.34/hr, and it offers 7.5GB RAM and 4 EC2 Compute units (2 cores, 2 compute units each, meaning 4-4.8GHz of 2007 processing power).
• The AgileCLOUD instance is $0.10/hr, starting with 2GB RAM and a single Nehalem core.

With twice the number of logical cores and twice the clock cycles, you’d probably expect Amazon to come out on top of a multi-threaded Unixbench shootout… you’d be wrong though:

Unixbench Raw Scores (higher is better)

In 10 runs spanning two days, the lower specced AgileCLOUD instance outperformed the EC2 instance in 2 key metrics:

• Raw Performance – On average, the AgileCLOUD instance outperformed EC2 by 7%.
• Consistency – AgileCLOUD’s performance varied by less than 1% through all 12 runs, while EC2 varied by more than 5%. During these random EC2 performance drops, AgileCLOUD pulls ahead by more than 12%.

The most important number of all isn’t found on this graph though, and that’s the cost.  AgileCLOUD outperformed EC2 by an average of 7%, while at the same time coming in at less than one third the cost of the EC2 instance.

Stay tuned, as next week we’ll have a more comprehensive set of benchmarks that take a look at more than just the CPU.