With the recent media attention Volkswagen has gotten, a lot of folks are wondering, how could that happen? Aren’t there checks & balances?
Join 32,000 others and follow Sean Hull on twitter @hullsean.
Then I ran across this observation on Todd Hoff’s brilliant blog High Scalability
Is what Volkswagen did really any different that what happens on benchmarks all the time? Cheating and benchmarks go together like a clear conscience and rationalization. Clever subterfuge is part of the software ethos. There are many many (search google) examples. Cars are now software is a slick meme, but that transformation has deep implications. The software culture and the manufacturing culture are radically different.
What exactly does all of this mean?
1. MySQL & Aurora
I was recently chatting with a colleague of mine Bret Miller who runs DeepSQL an adaptive database platform compatible with MySQL. He said:
“We’re actually doing testing against Aurora, but we recently had a couple customers do it independently with more challenging loads. Didn’t see the performance stated in the marketing stuff. ”
My response was… “Yeah. Aurora looks to be a win on the HIGH AVAILABILITY front.
On the scalability front, MySQL has certain limitations in it’s core. So i’m not surprised that the marketing material was grandiose in it’s promises.
The best way to improve mysql performance is to tune queries. As you’re writing your application, and when you want to boost performance. ”
And so it goes.
2. Redis & Memcached
Then I stumbled upon Salvatore Sanfilippo. He is the author of the brilliant & phenomenally successful NoSQL database called Redis. Turns out that another famous blogger was making some sweeping statements about Memcached & Redis and Salvatore ended up defending Redis in a blog post titled Clarifications about Redis.
The topic turned to benchmarks. Which lead me to another post titled
Why we don’t have benchmarks.
Heard this before?
Related: Did Airbnb have to fail?
3. Is Mongo webscale?
When Mongo was first releasing it’s benchmarks, the media went wild. And DBAs were scratching their heads. This fabulous video captures the sentiment of the time. 🙂
4. Oracle meets David DeWitt
In the 80’s Oracle began to forbid publishing benchmarks. After seeing a research paper by David DeWitt, Larry Ellison amended the End-user-license-agreement to include the DeWitt Clause. Later other database vendors followed.
It’s easy to see why. Benchmarks by their very nature depend on so many factors. It’s inevitable that those factors will be carefully picked by each platform to highlight it’s strengths.
Also: Are SQL databases dead?
5. Product versus disks
It is inevitable that all of this continues. When we reside at the level of the business, we perceive the product & its performance through that lens.
When we dive down to the level of disks, buses, cpus, network latency, multi-tenant clouds and a myriad of other factors, the waters are never so clear.
So remember your mileage may vary and buyer beware are as true today as they ever were.