Some recent NPD data showing the Android platform overtaking the iPhone in the US has set the web on fire. Apple apologists like John Gruber are working overtime to try to spin it.

Remember back when Daring Fireball was actually an interesting site? Now it’s like I imagine Pravda was during the Cold War.

In any case, I was a little shocked to see the gains come this quickly. Android was obviously making a dent, but I expected another generation of products to flow through the market – like the Incredible and the EVO 4G, and eventually the long-anticipated onslaught of Dell vapourware – before Android really took hold with mainstream consumers.

And of course iPhones are still selling. They’re selling like hotcakes, for that matter: Apple reported a more than doubling of sales during the period in question (kind of undermining the notion that it’s some sort of calm before the 4G storm).

At this point smartphones have a limited market saturation, which is why it’s a critical juncture and why there’s a bit of a race to perform a land grab. It remains anyone’s game.

In a year we might be looking at the emergent dominance of Windows Mobile 7 phones. Or maybe the next iPhone model will storm the market. Or WebOS will be reborn into a real contender. Or maybe Blackberry – the strangely unmentioned leader of the pack – will really wow the market with their next cycle.

It is critically important that there is competition in the space. Competition is great for everyone, including for iPhone users (see the upcoming multitasking that is a virtual clone of the implementation in Android). The market can’t be dominated by one vendor, especially one with walled gardens.

Speaking of walled gardens, as much as HTML5 got talked up during the Great Jobs-Adobe War of 2010, in reality HTML5 performance on the iPhone and iPad is abysmal and is close to unusable for rich content.

The real alternative to Flash on the iDevices has been native, single-platform, completely-proprietary apps. Which is arguably a perfectly fine approach (take full advantage of the platform and all), but it was incredibly frustrating seeing HTML5 used like such a cheap sacrificial prop during the debacle when it really had perilously little to do with the debate. It was so grossly dishonest.

Of course I’m speaking outside of videos. For videos Flash is almost always a completely unnecessary wrapper around an h.264 stream (which is what many Flash videos come encoded in these days). Separating the video from the container is a no-brainer.

Obviously HTML5 needs to keep improving. With Android 2.2 around the corner there are wide expectations that they’ll introduce a JIT compiler for the Dalvik engine (“native” apps largely run in a VM runtime and are currently seriously hobbled by the lack of optimizations in the same), and it would be fantastic to see something like V8 for the browser as well. While the platform has far-and-away leading canvas and dynamic graphics support, it falls behind in scripting and that would be a nice gap to eliminate. It would be nice to see a decent dynamic HTML 5 capability built into iPhone OS 4, as currently it is seriously deficient.

Why the web should be destroyed, by Steve Jobs:

“We’ve been there before, and intermediate layers between the platform and the developer ultimately produces sub-standard apps and hinders the progress of the platform.”

I tried to ignore the whole section 3.3.1 debacle, as I’ve already commented on Apple’s moves and motives too many times, and really this change is just a new wrinkle of existing restrictions: developers for the iPhone are already subject to the capriciousness and fickle whim of Apple, with no recourse.

If Apple said that you have to wear a purple bow-tie while developing for the iPhone it would arguably be a change for the better. At least then you wouldn’t have uncertainty on the whole bow-tie color issue. From that perspective, Section 3.3.1 should be a welcomed clarifier.

While claims that it’s for the greater good of quality are discountable as a ludicrous smoke-screen, if you were gullible enough to believe that, and you accept the asinine notion that development technology dictates app quality, is Apple promising to filter app submissions by quality?

Given what is already in their app store they have a lot of pruning to do.

They could carpet-bomb out the crapulence, with acceptable collateral damage, by banning any apps made by small development shops. This would be great for everyone, right?

Kudos to them for removing consumer choice: If someone liked an app created via one of the targeted tools, clearly it’s because they don’t know what’s good for them. Personally I choose based upon reviews and user ratings, but it’s a win if these sorts of personal decisions are made for me.

Like I said, I’m not going to get drawn into this whole section 3.3.1 debate.

So let’s get back to Steve Jobs’ statement above.

“We’ve been there before, and intermediate layers between the platform and the developer ultimately produces sub-standard apps and hinders the progress of the platform.”

Where do web standards fit into this equation, given that web standards are almost perfectly defined as intermediate layers between the platform and the developer.

Stop. Pause. Seriously think about this. Imagine that Steve Ballmer was saying this about SVG (just use GDI and DirectX or at worst VML) and JavaScript and the canvas element, while discussing banning non-ActiveX controls from the Windows ecosystem.

Where does Steve Jobs think the web fits? Is it just a convenient stop gap?

We know that Apple’s existence through the lean years, and then its resurgence, was made possible because of the cross-platform web, and paradoxically because of the cross-platform Flash technology. iMacs sold to the general population only because they knew they could still use the open web, consuming and interacting with apps that could certainly have been built richer and better if they targeted just the Windows platform.

They could use the popular sites, read the news, do their banking, pay bills, and send eCards, despite being on a platform used by only the very few.

Yet now Jobs has made it plenty clear that the web is for trivial, simple stuff. For richer apps you need to target the iPhone alone, using a process that in no way can allow you to target other platforms as well.

Flash is a no go because it enriches the web – there’s a lot of hate out there by people who know Flash only as a simple video player, where it punted Apple's QuickTime it’s worth noting, and as a platform for irritating ads, however as a parent of young kids I see Flash as the enabler of an incredible array of rich and entertaining educational tools for kids (PBS Kids, TVO Kids, CBC Kids, Disney Kids, among countless others) – and Apple has done nothing in the mobile web space to make up for the gap…because at that point you’re supposed to bridge over to the iPhone market and embrace it with fervor and loyalty. Sorry, but no thanks.

The iPad represents a great opportunity for HTML5, with many large web properties already shifting gears to ensure that they take advantage of the platform. The iPad as a web consumer is all about modern, open standards that empower and enrich the user experience. It has one of the best mobile web experiences going.

This is expected, really, as Apple's very existence hinged upon an open standards web. There was a dangerous period in the late 90s when the web almost got Windowsified — look to South Korea as an example of this happening — and if that came to fruition Apple would have been dead in the water. Thankfully a few who saw past short term interests rallied around keeping the ecosystem open for innovative companies like Apple to thrive.

yafla is finally going to be born into a remarkable web application that exploits the rich functionality of the iPad and iPhone's web platform, along with Android, the Blackberry webkit browser, and virtually any other modern HTML 5 consumer, from big to small. I'm putting my actions where my mouth has been (this includes architecturally on the back-end, with decisions that I will document and explain along the way).

The arguably proprietary Flash platform has rough times ahead. While the actual numbers of iPhone and iPad users combined represent a small percentage of the web consumer ecosystem, it occupies a disproportionately large area of the mindspace.

Boo for the iPad! It bring us back to the obsolete era of walled gardens

While the iPad supports HTML5, that isn't the primary focus of content providers: The vast majority of them are rolling out solutions that target the walled garden of the iPad. Video content, books, magazines, or newspapers, you're entering the land of made-for-the-iPlatform solutions that nothing to do with the modern web.

And of course it isn't just big media. Many web sites — like Engadget, Digg, and so on — are rolling out apps for the platform as quickly as they can. In most cases those apps do nothing that can't be done as well or better as simple web apps, but such is the return-to-mistakes-of-the-past era that we're in. In the cases mentioned they also built Android apps (others, including Big Banks, are far more myopic about this), but there's still the question of why they built anything platform specific at all, beyond the obvious explanation that they're hoping on the bandwagon.

Everything old is new again.

John Gruber argues that the iPad represents a more, open and innovative ecosystem than with the Atari 2600 circa 1978. Hard to argue with that. But what about the 32 years in between, John?

It's a clear sign that there's something seriously wrong when you have to base your comparison on an early home game machine from three decades ago.

For the past 20 years we've had a computing market where anyone and everyone could build applications for the vast majority of devices. Since the incarnation of the web, those creators have had the ability to have just as much presence as makes like EA. There is nothing new there. The only "advantage" that the iPhone cum iPad offers the little guy is that the market was so nascent and novel that a million made-on-a-weekend apps could sell thousands. That early ease is quickly disappearing, and the natural size advantage of shops like EA is coming to fruition. Small-shop, single-trick apps are going to very quickly get crowded into an unlit corner.

The Apple app model is horribly, horribly broken, though they have enough goodwill, and still get by with many deluded into thinking that they're the underdog little company, that they'll be able to float with it for a while longer while apologists continue to present their questionable defense. The iPhone and now the iPad are not simple game machines (whether from 2010 or 1978) — comparisons with the Atari 2600 or even the Xbox 360 are highly deluded — but represent a serious movement into the domain of general computing, and against that they should be compared. Pretty remarkable how Apple has managed to retroactively turn Microsoft into the good guys.

The Apple web model is brilliant, with it representing a fantastic web appliance of the best kind.

Let's just hope the web survives through this, and there isn't a rush from open standards to the opposite-of-open-standards walled garden.

Just a couple of minor notes:

• Tonight I switched data centers. There may be a small outage for some because I forgot to lower the TTL on the old DNS entries before the move: I guess being the, uh, "world's most pre-eminent domainologist" doesn't mean I'm always on top of such things, at least not for what now is just a wrapping around a blog.
• I made the move because I'm finally making use of the domain, and the reason for needing a more powerful platform will become clear.
• I often do minor edits of published pieces after the fact, without noting it as an edit. Often I'll read back through the archives and see typos, wording that I'd like to correct, etc, so I fix it for future readers. The most common edits see me removing parenthetical asides that add nothing. The software I wrote to run this does store every revision, so technically I could provide a diff'able history and have long considered doing that.
• Speaking of blog software, one solved problems that strangely causes havoc again and again are dynamic sites that fall over whenever they get any attention. Over the past while I've had days with torrents of incoming visitors, and the CPU needle barely ever spikes to the point of being measureable. It is just inconceivable that a simple blog dies when it gets on Reddit or the like. Seriously - caching, figure out how to use it. If you can't get it into your software itself, stick your site behind an nginx instance and use its wonderful functionality.
• Every entry that I post is automatically run through tidy. I really consider it important that mark-up claiming to subscribe to a given standard actually honors that standard.
• I've withdrawn from the whole NoSQL debate because it got incredibly boring. My closing note is simply to say that the way that many are using NoSQL is like discovering the buggy whip at the beginning of the automotive era.

Incredible times ahead.

Joe Stump – the former Digg lead architect with the coolest name in tech – posted a peripheral response to my recent entry about SSDs and NoSQL.

Rebuttal in tl;dr; Form

The original post was motivated by claims found on Digg’s technology blog.

• They say that the RDBMS “mindset” favours writes over reads: BLATANTLY WRONG CLAIM.
• They show poor index and schema use: WRONG DATABASE USAGE.
• They show that their database product can’t join: BAD DATABASE SERVER. RED FLAG.
• They report very poor performance without adequate detail: MEANINGLESS PROPAGANDA.
• They use this to show that the RDBMS can’t cope: SEE ABOVE.
• They say that if you don’t use all of an RDBMS’ feature set, you’re essentially using NoSQL: ABSURD.
• They describe scaling out issues with databases: TRUE FOR MYSQL.
• They described their move to NoSQL: GREAT FOR THEM. THOUGH REALLY THEIR SOLUTION WAS EXTREME DENORMALIZATION.

And on Joe’s post.

• You need an expensive DBA with the RDBMS, not with NoSQL: SPECIOUS, FLAWED REASONING.
• Capital expenses suck. Services are better: BUSINESSES GENERALLY LEASE THESE DAYS.
• $7,500 “just for disks”: FOR A SaaS BUSINESS THIS IS CHEAP.
• 50 node cluster: 50 NODES IS A COMPENSATION FOR ABHORRENT I/O RATES.
• SSD drives are expensive: NO THEY AREN’T. YOUR ARGUMENT IS OBSOLETE.
• Commercial database products are pricey: VIGOROUS AGREEMENT.
• NoSQL $/read and $/write win: MAYBE, MAYBE NOT. DIGG COULD LIKELY DO MORE WITH A COUPLE OF SSDs THAN THEY CAN WITH THEIR MASSIVE DENORMALIZATION

The Non-ADD Version

Joe has been in the Web 2.01 trenches. He built a solution that powers one of the top sites on the net.

Remember when getting "Slashdotted" was a big deal? Getting on the front-page of Digg makes a Slashdotting-at-its-peak look like a little traffic bump. There are probably a hundred PR reps busy trying to botnet their clients onto the front-page of Digg for every one punished into spamming Slashdot these days.

Far more people know Joe’s all-out-of-bubblegum name than will ever know mine, and rightly so.

A Strawman Built on Cliches and Appeals To...

Joe comes out of the gate resorting to the venerable old-versus-new tactic: "It's just those old-school DBAs upset that us kids are rewriting the rules," he says in not so many words, while nailing himself and his peers onto a cross, seeking pity for the flames they doth receive for their unconventional, rebellious ways.

This is a bit strange, really. Barely a day goes by lately without Hacker's News or Reddit’s /r/programming featuring another front-pager about how the Incredible NoSQL is rewriting the rules of, well, everything. The general demeanour is one that, I think, is far more sympathetic to completely unsupported and undemonstrated pro-NoSQL claims than it is to anything that questions the hype.

Countless NoSQL blogs have appeared (though if you browse them looking for actual content you’ll instead find that most feature few facts but lots of zealous punditry. Advocacy seems to be the primary focus right now). Anyone involved with any sort of NoSQL initiative is spinning off their own start-up to capitalize on this sure-win formula, acting like it’s some sort of magic ingredient that will assure them of success.

It is very reminiscent of the XML heyday – I’m a very big fan of XML in its place, as an aside – when countless start-ups appeared with business models that could be boiled down to “something to do with XML”.

The big database vendors have remained quiet, largely because the miniscule-budget operations all clamouring for their piece of the NoSQL pie aren’t worth bothering with.

“But what about Google, Amazon, and Twitter!” you say. Joe resorted to that same appeal to authority by incanting the same magical trio (say it three times quickly and your TPS rate will quadruple!). Not really much to bother with there, beyond pointing out what a cargo cult is. Your bamboo headset won't make you successful like Google. It really won’t.

Unless you are targeting the same problem space as those companies – say like providing very low performance but highly “scalable” database solutions for countless low-value start-ups – their solution choices are utterly irrelevant.

I'm not a DBA (though knowing how indexes work now strangely qualifies one for such a title). I'm just a technically curious solutions guy that has an innate need to keep asking questions and probing deeper until the Want-To-Believe fog that often hides hype dissipates.

On Rinky-Dink Operations

In Joe’s entry he focuses a lot of attention on the costs of RDBMS solutions.

One such argument is that it’s better to use computing hardware as a service than to buy, seemingly implying that while you can buy good hardware to run a RDBMS, it is better to rent less-good virtual hardware to run your NoSQL instances.

Yet leasing is what all the cool kids are doing these days, largely for the same financial reason. Writing it all off beats dealing with depreciation BS, and it makes financial planning a lot easier.

On the leasing front, $600 a month gets you an insanely powerful, makes-an-Extra-Memory-Quadruple-Extra-Large-EC2-Instance-Look-Like-A-Pile-Of-Puke server.

You’ll probably be paying 20x that for every developer you have working on your solutions. Is this really so astronomically high?

That less-than-the-cost-of-the-office-cleaners price tag gets you a server that with a bank of striped SSDs that will almost certainly demolish your impressive-in-count-but-not-in-throughput big scale out cluster, at least with a non-broken RDBMS system.

No really, it will. Of course for any sort of reliable system you’d have to pay for some DB licenses (presuming you aren’t going with PostgreSQL), and then you’ll want to double everything up into mirrors or some other reliable setup, so triple the price.

And really, is the $7,500 spent by 37signals on a disk array really even worth mentioning? I suspect that sort of number ends up almost as a rounding error on their expense sheets, and given that it's pivotal to their operation – it sits under the very foundations of their business – I doubt they spent many sleepless nights over it.

What sort of rinky-dink operations are we talking about here? Does Digg still qualify as a start-up? Don't they have a payroll and all of that, yet they're clamouring to wire up a collection of discount bin servers?

I posted the SSD entry because SSDs really do fascinate me, and I do think they change a lot of the rules of the game. It just happened to dovetail nicely with my investigation of the Digg scenario, where Digg solved their very real I/O issue by essentially pre-caching every possible query result for a targeted need.

Through extreme denormalization they traded storage to reduce I/O needs.

This is a very important point, because it’s far more pivotal to Digg’s solution that the NoSQL versus RDBMS debate.

Call up your old Digg coworkers, Joe, and have them setup a real database server with a couple of SSD drives and see how it compares with their impressive cluster. I’ll bet Dell would happily lend them a real server.

All of this is a bit humorous, really: The whole point of my original entry on this NoSQL topic was simply to say "what is good for Digg isn't necessarily appropriate for all database needs”, so it’s a bit unfortunate that it has come to this, with Digg’s former architect justifying their decision when they were held as a scenario where it is likely the perfect solution.

Then, after seeing the Digg case-study, I felt obliged to respond to their RDBMS claims because I saw them as flawed, indicative that the movement should really be called NoMySQL instead of NoSQL. It still doesn’t diminish the correctness of their choice.

But really, while I originally entered into this debate believing simply that NoSQL is being oversold (it is grossly inappropriate for the vast majority of non web 2.0 projects), the more I investigate the more I’m coming to think that it is a solution for the rapidly disappearing problem of pathetic I/O rates, at least assuming that you aren’t running on several of the cloud solutions where that is your only choice.

There are many other differences that come with NoSQL (many strongly questionable, like the oft lauded “no schema” claim for some of the solutions), but the I/O restriction is by far what sold it on the high end, and the high end is what convinced the little guy that it’s the way to go.

Oracle, DB2, SQL Server, Teradata, Vertica, Greenplum, Sybase and Friends All Cost Way Too Much

I very strongly agree with Joe about one thing: the licensing costs of the big RDBMS products are way too high.

They know that 2% of their potential customer base have giant budgets, and that they can squeeze more from that 2% than they could ever get from the other 98% who then get relegated to fighting over scraps like MySQL.

Not really sure how to solve that problem, but I concede that it is a non-trivial issue. PostgreSQL is probably the best low-to-no-cost database server, but even then quite a few performance features are missing (like real-time materialized views or SQL Server style clustered indexes).

Seven years ago I had an article published in MSDN Magazine demonstrating how to target SVG from ASP.NET. It was a speculative submission that I sent in because I really believed in the technology and its importance to the web.

In my various drafts I had included details about Microsoft's participation in the various SVG groups, and how that bodes well to its eventual adoption by Microsoft. Those got edited out before publication.

Alas, while it took far longer than it should have, it is a wonderful development to see that Internet Explorer 9 finally adds SVG to the browser.

Fighting The NoSQL Mindset, Though This Isn't an anti-NoSQL Piece

Shocked by the incredibly poor database performance described on the Digg technology blog, baffled that they cast it as demonstrative of performance issues with RDBMS’ in general, I was motivated to create a simile of their database problem.

While they posted that entry six months ago, they recently followed up with more statements on the NoSQL / RDBMS divide, and are now being heavily used as a citation of sorts.

For instance Dare Obasanjo held Digg's moves as a rebuttal of my prior entry on SQL scaling (though my entry actually explicitly excluded incredibly rare edge cases like Digg's, and my core point was that the majority of database uses don't have the needs of a site like Digg, I'm always one to take on a challenge), which then got picked up in other blogs.

Digg's case is an example of an entry-level RDBMS product used arguably suboptimally on under-powered hardware, and it seems questionable whether it proves anything of substance about either database technology. Yet it's held as demonstrative of something — in particular the failing of the RDBMS — which is why I focus on it. They are different tools in the toolbox, arguably for different purposes, and that isn't the focus of this entry.

So let's take a look at Digg's scenario.

I do this to evaluate their performance claims, to confirm my previous statements about indexing improvements, and to determine the impact that SSDs have on the problem space, because I strongly believe that SSDs (and cheap memory) completely change the equation.

The focus on this entry is not to question or answer whether NoSQL is the right choice for Digg — though there are some ramifications as SSDs take over, which is, I think, an interesting aside — or whether Google or Amazon or anyone else should use it, etc.

SQL Server 2008 Developer Edition, itself viewed as almost a training-wheels RDBMS by many, on Windows 7 was the most convenient platform for me when I ran this test, so I created a quick script to create what I think is a pretty accurate reproduction of the database described in their blog entry—

• 500,000 users, having…
• 10,000,000 friend relationships (using a power law distribution)
• ..and 500,000,000 “Diggs”, randomly distributed among 500,000 virtual “items” (which might be comments, submissions, etc) with a date range covering four years.

The database weighed in at a svelte 30GB.

I ran this on a two-year-old desktop machine with a Q6600 processor and 6GB of RAM, on a standard 7200 RPM consumer drive. You can easily find laptops with more processing and I/O power.

I opted against running it on a real server (you know, like a 24-core, 128GB, banks-of-SSDs monster than many real databases run on) simply because I knew it wasn’t necessary, and went contrary to the demonstration that even a mediocre machine can beat their results.

DISCLAIMER: This is not a high-fidelity reproduction of Digg's situation, as is pointed out many times in many ways in this post. However Digg took the time to post metrics to support their claims that they are some sort of extreme case, at the edge of database limits, and I simply don't believe that is true. Digg's data quantities are relatively small and lend themselves to sharding. The second point, which again is hammered home many times, is that SSDs present a solution that changes the equation, and, I think, provides some interesting inputs to the situation.

The First Clue That Something Isn’t Right: You Can’t Do a Simple Join

The Digg blog entry detailed how they had to manually build an IN clause given their selected database product’s inability to adequately run a trivial join, with the resulting query taking 14 seconds to find the Diggs for a given user’s friends against a single selected item.

This yielded a results return rate of 0.07 per second.

If you can add an IN clause that solves a database join issue that functionally should achieve the same, there is a much larger underlying issue that needs to be dealt with. I'm not a MySQL user, but apparently it offers minimal plan investigation tools, so there aren't the options to fully flesch-out what the query engine is doing. Nonetheless, it is a warning sign of a foundational product issue.

I ran a similar query in SQL Server, albeit without the hand-coded SQL builder, looking up friend Diggs for randomly selected combinations of users and items. It returned so quickly relative to Digg’s experience, even from a cold cache, that I had to up the iteration count to 1000 to get good test durations.

SQL Server was returning a fairly constant 36 result sets per second, probing the friend table’s ten million relationships to find the selected user’s friends, and then probing the five hundred million Diggs for the pertinent records, sorting it in the manner that Digg sorted their results. The query needed to draw data from all over each of the respective table populations, ensuring that it wouldn’t benefit from localized hot-spot caching. To prove this, limiting SQL Server to only have access to 1GB of memory had a negligible impact on the performance.

CPU usage was marginal, with the limiting factor being the slovenly I/O of the lowly magnetic disk. The iterations were run sequentially, as parallel runs yielded no net benefit, the magnetic disk moving as quickly as it possibly could already.

Already we’re running at close to 500x the rate reported by Digg, without doing anything beyond using an arguably better database product, at least in that it can join properly. MySQL's many weaknesses are well, well known, so the core point from that is not to question Digg (though their indexes were suboptimal), but to put their database product under a cloud, as they themselves often do when posting about their move (usually openly declaring their restricted option set given that they limited themselves to open-source products, obviously eliminating from consideration many of the clusterable, very high performance RDBMS options, even if that were a better choice which is completely uncertain).

Their dataset distribution may be entirely different, however even if I doubled or quadrupled or octupled the count in every table it would only marginally impact performance.

At this point I implemented the indexing changes I described in my prior entry – removing the surrogate keys and cluster-indexing on the unique columns – and the lookup rate jumped to 71 result sets per second, or around 1000x the speed reported by Digg. If I massively increased the data quantity and return counts, the difference between their poor indexing and proper indexing would dramatically widen, with the proper indexed solution showing little difference with significantly increased data counts.

If the database was cached in memory those index changes would have had a much more profound impact.

What If Localized Data Isn’t Your Primary Optimization Strategy?

I had been meaning to get an SSD for Eclipse Android development, so when my new 100GB SSD arrived (it’s an MLC unit that did well on an Anandtech review, though I won’t mention specifics as it isn’t pertinent – any decent SSD will perform at a similar level. Of course for real-world production use you would want an SLC drive) I detached and moved the database files over.

A quick reattach later and the 30GB was very amply hosted in the 100GB MLC SSD.

I fired up the benchmark to be pleasantly surprised to find it returning results at a rate of 4100 result sets per second. The write performance, while not a focus of this test, also hit extraordinarily high levels (which would conveniently lubricate the use of copious indexes).

Correcting the indexes and moving the database to a single inexpensive consumer-grade SSD, running on a dated desktop, had results coming back at a rate 60,000x what Digg reported.

None of this is intended to be a serious benchmark of SQL Server (I don’t wish to fall on the wrong side of a DeWitt clause), or even Digg's use of MySQL: This is not a disciplined benchmark, and during parts of it I hopped into some windowed online matches of Battlefield:Bad Company 2 while tests ran, after seeing that it had a limited impact on the results. I knew that the primary weakness was simply the movement of the hard drive head, and different technology choices (NoSQL versus RDBMS, normalized versus denormalized, clustered versus heap, etc) primarily impact how often and how far that head has to move.

And of course I don’t have Digg’s data, so it is completely speculative on my part based upon some rough descriptions given in the Digg posting. Maybe he hugely underestimated their data counts, or their data entropy is vastly different.

This is a macro-benchmark: Digg’s claimed results were so poor that I went in knowing that the difference would be very large.

Their described data quantities are small in the world of large databases. Most decent relational database products don’t even start to sweat with tens or hundreds of millions, or billions, of rows.

The key, of course, is proper indexing, trading write performance for read performance targeting your specific needs.

Indexes could be viewed as ways of creating “virtual tables” that are maintained in lock-step with your base table. Decent database products like SQL Server even allow you to include unordered but included columns in your index to ensure that you have a covering index (the best kind) for all scenarios. And that’s before you even get to the magical world of materialized views.

So either MySQL is an atrociously bad product at the larger limits, which ample evidence seems to point as a truism, or the Digg staffers simply weren’t getting the most out of their systems, but it’s hard to take their statements about the RDBMS field with seriousness, and their arguments more correctly invalidates MySQL more than it invalidates the RDBMS.

The fact that Digg is a large site says nothing to their technical leadership or mastery. Their site has sped up by leaps and bounds over the past year, so I suspect that they know what they are doing, but I'm wary of any cargo-cult like "they did {x} and they're a big site therefore it must be a good option" appeals to example.

On The Role of the DBA

What is most disturbing about this whole database debate are the number of commentators who excuse horrific database usage (not in relation to Digg's issue, but as a general conversation point whenever people make comments about proper database use in virtually any online discussion), ascribing rudimentary database performance design and knowledge as something that is limited to the elusive “DBA”.

This is ignorant and frightful.

You don’t know what a b-tree is? Don’t know how indexes work? Don’t know what a red-black tree is? Please get away from the compiler and save the world from your monstrosities until you have some knowledge of these basic concepts.

This is not esoteric knowledge, and instead is rudimentary comp. sci. knowledge.

DBAs are the guys who setup user accounts and monitor security, schedule backups and determine macro-optimizations like how to allocate file groups on the SAN arrays. They might probe lowest-hanging fruit performance issues and flag offenders or offer up suggestions.

Rudimentary database design and proper usage is the basic responsibility of developers, and if you don’t know it then it is your responsibility to learn it. Alternately you can just clutch onto NoSQL and bleat about how it changes all of the rules anyways, which is the route quite a few have decided to pursue (I fully expect to get the standard angry responses from those who take this like a religion).

Is NoSQL a Solution for Yesterday’s Problems?

Database servers really like having a lot of RAM. Ideally you should have more RAM than you have data, allowing it to cache the entirety of your DB (or at least the working-set quantity of DB on that partition) making incredible read performance achievable.

Joining rows is not a hard activity for database servers. It can do it at unfathomable rates if the data can be fed to it at the appropriate pace and in the right form. Even heavily normalized databases can be high performance.

What normally makes joins a performance issue is data locality: if you have to load two rows from different places on the disk, that’s two seeks instead of just one (or three, four, five or more instead of one). When seeks are as costly as they are on a magnetic disk, you avoid it (either by striving for a database that fits in memory, which paradoxical often calls for heavy normalization, or by de-normalizing).

Writes are obviously important too, yet on a site like Digg I would guess that reads outweighs writes — from a user interaction perspective — by a factor of 10000:1 or more outside of logging (which usually goes to a log-specific technology anyways).

In contrast to all of the “everyone is a publisher and the internet changes everything” bluster that is used to herald the wave of change that NoSQL brings, the reality is that it’s a very small percentage of users that post submissions and add comments, or even that do the simplest possible action of clicking an arrow.

Users overwhelmingly simply consume data, whether it’s the latest tech news, Asthon Kutcher’s tweets, or just browsing through the comments on a Slashdot article to see if they add any additional insight.

Despite Digg’s recent claim that they are “write intensive” (maybe because they’ve decided to dramatically explode the number of writes a simple action causes?), at its root their platform is primarily read focused, which is why they pursued Cassandra in the first place. Take note that their NoSQL solution for friend Digg lookups is to take every Digg and massively explode the number of writes it causes to happen (in the case of a Digg by Kevin Rose, a single write becomes 40,000+ large writes).

Hardware Is Cheap. Manpower is Expensive

If I had 48GB of RAM in the test machine (which is fairly pedestrian outside of gerbil-sized cloud instances. Note that you can now add 128GB of RAM to servers for around $4000 in some cases), outside of the initial caching period the select rates would be stratospheric regardless of storage medium, though SSDs would still come in a very, very strong lead when it came to write performance.

For the same $4000 you could chain five Intel X25-E drives for 320GB of intensely high performance – and persistent – storage. Just keep going up until you have more throughput, I/O and storage than you could dream of.

Some high-end enterprise solutions now tier storage and automatically place data as appropriate, choosing between magnetic, SSD, and memory caching systems. The pages of the table that are never touched end up on the magnetic storage while the hot area – say Diggs within the last 6 months – are moved to SSDs or to huge banks of memory caching.

There are bountiful options to achieve incredible performance, even on a budget, balancing memory and high performance storage systems.

Throwing Storage at the Problem

I didn’t waste the disk space, but as mentioned before I could do a simple join between the tables, materialize the view, and the performance would be very high even on magnetic disk, although it would add a serious cost to writes: When a user with a large number of people who befriended them dug something, their record creation would branch out into the write of potentially thousands of records.

That was the route that Digg took: They are pre-computing the sets of data that a user might possibly want, even apparently for reams and reams of long abandoned accounts.

They do this because looking up data that can’t be cached in memory is an expensive operation. Yet as has been shown, SSDs, which are getting faster and cheaper regularly, completely flip the I/O equation.

SSDs change everything.

Turning a small amount of data into a massive amount of data to improve performance paradoxically makes SSDs much less attainable (because the cost per GB is so much higher), and humorously may thwart the end goal. It also reduces the ability to memory-cache the relevant data.

By pursuing this solution, Digg has limited their ability to choose other solutions that are clearly hitting the mainstream.

Coming Next – PostgreSQL versus Cassandra

There is a complete absence of objective measures of the performance of Cassandra. In place of real performance comparisons and load metrics are a lot of hand-waving and comparisons against completely broken database products (never, ever hold MySQL as the vanguard of the RDBMS world. It is comical to do that) running horrendously malignant queries.

Not anymore. I’m on the case.

My goal is not to belittle the product (which I think is elegant, beautiful and concise, and serves a very important role), but simply to bring some rationality to the argument, as it is currently missing.

[EDIT: The following statement has been proven to be a wrong interpretation, but I leave it here out of humbled shame]Digg claims that Cassandra brings them “linear scalability”, yet every one of their Cassandra nodes is 100% replica of the other, meaning that a write (or 40,000 writes) on one is communicated and then replicated on every single other instance.

Response to Criticisms - 2010-03-25

This entry got picked up on a couple of excellent tech-oriented sites: Hacker's News and Reddit r/programming. Included in the comments of a lot of very smart people are a couple of common criticisms that I thought worthy of specific response.

"Your benchmark stinks. How about you..."

My benchmark, if you can even call it that, was focused on O(n) complexity and the difficulty of joins among very large tables with a half-decent database product, with the core take-away being "it's a solved problem. With proper indexing and a decent database system most datasets are `small'."

On the topic of concurrency, I mentioned that in the entry, noting that executing many parallel runs of the test yielded the same net output on the magnetic disk, while it actually significantly improved performance on the SSD and then leveled off. Database servers are fairly smart about concurrency and task queing.

The top result of 4100 resultsets per second, which was achieved using many simultaneous runs, still wasn't fully exploiting the I/O capabilities of the SSD, owing to the tuned-for-magnetic-disks nature of the database server that I didn't bother resolving.

However the Digg case study lacked significant details beyond a couple of spurious size details there to indicate, I believe, that "we think our data is large and the RDBMS can't service our needs". What I based my run on was quantity-of-data (which is not large in the land of databases) and key phrases like "from a cold-cache" (which can be reasonably interpreted as "on a test instance"). There is a lack of details in the Digg benchmark, given that I don't think they were intending it to be a industry standard metric, so it isn't reasonable to expect so much more regimented discipline from mine. However let me say that I did take the meager stats that were given and, where possible, erred to the high end — where "hundreds of million" appeared, I went with 500 million (if I went with a billion it would have barely impacted the results, but I was impatient and didn't want to wait for the data setup script to run that long). Where they said "millions" I went with 10 million. Some of the responses are demonstrative of how fact-free the debate has become, so it's not particularly surprising that NoSQL blogs group-hug around it.

This is not a replication of Digg's runtime environment, and any illusions that it pretended to be intentionally misinterprets. Though if it were a serious apples-to-apples comparison I would have run it on a serious server with serious load simulations, where the only orders-of-magnitude would be the difference between the results and what I achieved on a dated desktop.

In fact, 36 or even 71 results per second is still far too slow for Digg's use (especially given that they are stuck with a web technology that forces synchronous database calls), and I'm not even purporting that to be a viable option for them as they add out a lot of data-intensive personalization options. It's simply to contrast against their abhorrent performance number which I think are grossly misleading.

"But Google and Facebook and..."

Sure. That has nothing to do with this.

"So you're an RDBMS guy who hopes SSD prevents change..."

I'm not an "RDBMS guy". In a prior outing I was declared a DBA because I didn't just roll over for the NoSQL propaganda, and now I'm cast as a guy who holds himself as a database expert. Actually neither are my primary competency, and I think that's the point: I don't purport to be Joe Celko, or even a remote approximation, yet even I can see some massive issues in Digg's case study.

I'm just a solutions guy that looks at technologies and tries to digg (har har) through the cruft and get to the truth, which can be tough amidst tech religions: Warp back to 2001 and try to have a rational discussion about XML. In the case of Cassandra (and many NoSQL solutions) there is stunning ease with which many make absolute statements about RDBMS, such as the many "relational databases can't handle large amounts of data, just look at Digg" claims that litter the web, while cheering on vaguery and unsubstantiated hype about NoSQL solutions.

Show me real performance numbers for NoSQL solutions: They are disturbingly rare. Instead the argument is dominated by noise comments and hand-waving about how grand NoSQL is because it just simply solves everything and makes everything great.

Digg's NoSQL performance advantage is achieved by localizing all of the data necessary for a given request — in this case "tell me all of my friends who Dugg this item/parent item" which they had precomputed and cast out — ignoring the problem of MySQL not competently doing joins (apparently it has troubles sorting as well).

That is overwhelmingly a storage seek issue, and Digg's solution was to turn many seek actions into one or two by massively exploding their core dataset so the data for every need is repeated and persisted for every possible use. I can say right now that there is no question that if I performed the same benchmark on Cassandra, drawing randomly distributed user-item buckets from the same magnetic disk, my performance would max out at the number of seeks per second of the disk, which in the case of a normal desktop drive is somewhere in the range of 100-200 seeks per second.

Of course NoSQL yields the same massive seek gain of SSDs, but that's where you encounter the competing optimizations: By massively exploding data to optimize seek patterns, SSD solutions become that much more expensive. Digg mentioned that they turned their friend data, which I would estimate to be about 30GB of data (or a single X25-E 64GB with room to spare per "shard") with the denormalizing they did, into 1.5TB, which in the same case blows up to 24 X25-Es per shard.

This is interesting, is it not? Maybe it rains a little on the NoSQL parade, but to me it's a pretty fascinating development.

"Why do you have to insult the Digg crew?"

I don't intend to insult them, but at the same time I don't fall in the camp that gives them credibility simply because they're behind a large site. Many of the largest sites on the internet made technology mistake after mistake, yet succeeded regardless because they have a good product: These are some serious examples where ideas beat out execution. PHP somehow formed the base of a good number of the internet's largest sites, yet are there many that will seriously argue for its technical superiority?

"These are two different tools. I'm sick of this argument! Let's get back to the NoSQL Propaganda Parade."

In many cases they are used to solve the same problem. In the specific entry I refer to for this post the whole point was "we were using an RDBMS, and now we're using a NoSQL and it's so much better", so is it really rational to claim that they're two completely different worlds?

"NoSQL solves different problems like scaling out, data centers, etc."

Orthogonal. Cassandra solves problems that you can't as easily do with MySQL. MySQL != the RDBMS industry.