I was anti-Flash before Steve Jobs made it cool. I have as much anti-Flash credibility as anyone.

For over a decade I've steered organization after organization away from building solutions in Flash, often against great resistance. I was evangelizing SVG — what I saw as the biggest opportunity for a more illuminated, open solution than Flash — back when it's strongest corporate sponsor was none other than Adobe.

Aside: Humorously Adobe was leveraging SVG in their battle against Macromedia's Flash/Shockwave empire, before finally giving in and buying 'em out.

I have railed against Flash many times on many occasions.

The Great Flash of Propriety

Up until around 2005 and the birth of YouTube, Flash had very little presence in the video space: That realm was dominated by Real Player, Windows Media Player, Quicktime, among others. Flash fit in the RIA niche where it displaced the short-lived empire of ActiveX and Java Applets. It was the tool for great web entertainment like You Don't Know Jack, the Net Show.

Then YouTube rose with Flash at its core, and the rest is history. Soon Flash was the oddball foundation for video.

Mention Flash in the common era and the topic will veer to the great Flash/Apple battle of 2010, where Apple has violently and loudly rejected Flash (where it represents a proxy for "rich applications they don't control"), and Google, perhaps being a bit antagonistic, has embraced it with their Android and Chrome solutions.

Ok, embraced is probably a bit of an overstatement. In reality Google's initiatives have top notch HTML5 support, the best JavaScript performance, and are probably the best mobile/thin platforms for rich HTML5 applications, but they just happen to optionally support Flash as well, perhaps providing a bridge from the past.

A Very Personal Flash Experience

I've been running Flash on my Nexus One for several months now. I have it configured to On Demand so the scourge of animated, CPU-clogging Flash adverts don't destroy my web browsing experience. When there is Flash content that I want to view, I click the little arrow and voila, Flash is running on my mobile phone.

Overall it has been a very welcome addition to the device. From restaurant sites, to small videos like Zero Punctuation reviews, to games for my children, to product information pages, I like having the option of engaging Flash when the need arises.

So when I saw entry titled "Just How Bad Is Flash on Android" on Daring Fireball, of course I was drawn in. There Gruber indirectly linked to some demonstrations of Flash failing miserably on several Flash video sites.

To which, I say, no frigging kidding.

Anyone under any illusion that having Flash on their mobile device opened up any and all content is willfully or technically incompetent. That or it's link bait trying to herd in the people who desperately need their biases confirmed, and a lot of people desperately need to believe that Flash isn't necessary on mobiles. I'd probably go with that second option.

Of course not all content will play. This is, after all, a mobile device with a little power-sipping mobile processor. While 1Ghz sounds pretty pimp, the Snapdragon in the Nexus One is in many ways a weakling, particularly in video decoding and 3D graphics tasks where it fell behind even the iPhone 3GS.

On this 1Ghz processor I've had trouble playing 720x480 h264 videos encoded at 1Mbps, encountering frequent stuttering and dropped frames. Compare that to the iPhone 4 which allows for up to 7Mbps+ videos at 1280x720, which it plays flawlessly, owing to the excellent hardware decoding.

When it comes to video the iPhone 4 simply kicks the Nexus One (and virtually every other HTC Android device, as HTC is addicted to Snapdragons) to the curb. Then again, the Samsung Galaxy S kicks the iPhone 4 around and calls it Sally, while the OMAP in the Droid 2/X yields a good competitor, but the point remains: Snapdragon processor devices are not the top of the pile by a long measure.

So the Nexus One really isn't a great platform to highlight video prowess anyways.

And when it comes to Flash videos, Flash is almost always its own worst enemy: When you enable the plug-in, your enable is browser wide for that page, meaning it simultaneously enables the twelve punch-the-monkey animated ads surrounding it, often destroying the experience.

So no, trying to run a high bitrate 1000x500 video on ABC that clogs a high performance 2.4Ghz core on a modern x86 PC (which would be equal to about a 7Ghz snapdragon), probably with it being bundled up in DRM as well, is not reasonable.

"So Steve Jobs Was Right! Flash won't work on mobile!"

That's not what I'm saying. In fact, given that I've enjoyed Flash on many occasions, I find such claims ludicrous. When I see people make statements (direct from the talking points) like "Flash on mobile doesn't exist", I just marvel at the fervent, desperate doublespeak.

Mobile Flash exists. It's far from perfect. The available content usually isn't even aware it exists. Yet still, it is.

When sites provide HTML5 video, they do so often specifically for the iPhone/iPad. Most sites don't even feature sniff for the functionality — they don't care if you are running Chrome or Safari or IE 9 and can run HTML Video — but will instead refuse to serve up HTML 5 video for anything but those Apple devices. Given that, they encode specifically and only for that platform, with appropriate bitrates and complexity profiles ideal for those devices.

There's no big surprise that it often runs well.

That isn't a very encompassing, scalable solution though. One device family isn't meant to rule, and at some future point HTML5 will start to leave the iOS devices behind. The iPhone lived through a special moment where it was handled as a blessed child, getting its own special treatment, however that moment is passing.

Of course the Flash people have a solution for disparate devices, using dynamic streaming that is based upon the consuming device (resolution and bitrate scaling based upon the capabilities of the device). Not surprisingly there is little use of it yet given that mobile Flash devices just started appearing very recently, and still comprise a tiny market.

"So...Steve Jobs Was Right?"

No.

When I originally thought about how to respond to this, my first idea was to make a video demonstrating a smartphone failing miserably at rendering and interacting with various HTML5 sites. But then I thought better because someone might confuse that as a serious indictment of HTML5, when I love the technology stack and pragmatically realize that it can't (and shouldn't) always be universally accessible.

Yet still, contemplate the failings of HTML5 on mobile devices:

• Layout can be ill suited for the screen dimensions and resolution of a smartphone.
• Functionality can be overwrought and too heavy for a smartphone.
• Input techniques might not work on smartphone.
• etc.

Which is of course why many sites, like the atrocious script pig TechCrunch, created special mobile pages that they force you to, against your will, when you try to visit on a mobile device.

The next time you see a TOP 10 CANVAS DEMOS on your social news site of choice, pull them up on your tablet or smartphone and check out how well they work. Chances are overwhelmingly that they will fail miserably. Almost every HTML5 game demands that you use a keyboard to control it, if they aren't simply too resource demanding for a little mobile device.

So do we write off web browsers on mobile devices? Of course not: That would be asinine.

Punching Monkeys Coming in HTML5 Soon To You

One trend that is accelerating is the movement of many ads to HTML5 and away from Flash. That short period where iOS devices came with an almost intrinsic AdBlock+ in the form of the lack of Flash capabilities is rapidly disappearing. Soon every page you visit is going to be a mishmash of computationally demanding canvas renders and DOM manipulations, and it's going to become much more difficult to avoid their cost.

Closing Notes

For those who are on the fence about the whole Flash thing — whether they care about it on their mobile or tablet — take a look a the quick video I put up above. The videos did not run perfectly, the auto configurator didn't provide a seamless, perfect experience...but if the option is that or nothing, I think most people would agree that it's better than nothing.

What a thoroughly boring discussion that has distracted the entire tech industry. So many incredible innovations happening, yet everyone's talking this minor bit of errata. Move on!

It is further evidence that smartphones dominate the technology discussion now, with desktop technology fading into the noise. Intel is quickly preparing their very competent assault on the field with the Moorestown processor. It promises some exciting times ahead.

Backtracking on 802.11n with the N1

On the topics of product issues, in a prior entry I noted that I had been enjoying the fruits of 802.11n with the Froyo update on the Nexus One. Shortly after that entry, another update was pushed, FRF91, and my connectivity has become almost unusable unless I disable 802.11n on my wireless router (which was actually the case when I first got the phone, but FRF85 had provided salvation). So now I run two WAPs, with one to serve the 802.11n devices so I needn't cripple it for the N1, and a 802.11g one just to service the smartphone.

On High Density Smartphone Screens

With all of the talk about very high pixel density phones, and the downside of larger-screen phones, the focus of most discussions seems to be purely on the visual clarity of the screen. The actual usability of the on-screen content seems to be ignored.

On the Nexus One I'm dealing with a 252dpi (there is some fakery with the pentile pixel arrangement, but ignoring that), 3.7" screen. A site like the New York Times looks fantastic in landscape mode, with a fully readable presentation of the entire real-web contents. Yet the downside of that high-density, small screen becomes evident when I actually want to interact with the content on the screen. Something as simple as clicking on the category links down the left hand side is an exercise of extreme precision: Each link is approximately the size of a single fingerprint ridge, and they're so densely packed that a mm this way or that way yields the annoyance of a wrongly followed link. Zooming and unzooming just to interact with the screen isn't very enjoyable, and it erases many of the advantages of the pixel density.

Screensize is important, and smaller isn't better when you're talking about a device that doubles as a mini-web appliance. There is a balance to be achieved or the experience is compromised.

Sure, there are methods of rooting and rolling back and then forward to get it working, but I wasn't interested in them simply because this phone is too useful for me to bother: It is already a very decent phone, and I was willing to bide my time until Google refined the final bits and pushed it officially, at least when the alternative started with rooting.

Alas, a version has finally been unviled for my version of Android. I downloaded it, did the manual update, and initial impressions are incredibly positive.

I've complained numerous times before about the stuttering, seemingly overwhelmed feel of the platform, at least relative to the buttery smoothness of an iPhone, and I can happily say that most of those complaints are no longer true.

One possible facet of the Froyo update that has been debated back and forth as myth has been support for 802.11n. I can gladly say that FRF85b does indeed enable 802.11n on my handset.

Taken after switching the WAP to 802.11n only, though it establishes the same connection in mixed g/n. Yes, I named my WAP "dlink", even though it isn't a dlink.

While much has been made about 802.11n's higher theoretical speed, how often is greater than 54Mbps really needed on a smartphone? Is the max 54Mbps of 802.11g just too constrained?

Hardly. The increased max theoretical throughput is a non-issue for this usage.

Instead the real benefit of 802.11n is that it maintains a decent throughput at the edges of the connection, where 802.11g would stutter and disconnect. I can already say that testing this throughout the house and the yard has demonstrated a more consistent, more usable experience. With the WAP in the basement, connectivity in the bedroom was often more hit and miss, whereas now it appears to be rock solid and ever so speedy.

Viewing a static web page, one reviewer found the device used a miniscule 6W of power.

The PC that I’m typing this entry on – a very typical example of ordinary PCs – is wasting some 40W of power in power supply inefficiency alone at idle. Yet the Mac Mini is managing to provide a good computing experience with 1/6^th that, under a load using 1/6^th the power a typical PC consumes under a load.

Amazing.

The Mac Mini isn’t the first to move us in the right environmental direction, after years of escalating power demands (just look at the power consumption of current generation consoles to see how gluttonous devices have become. Watching a movie on an Xbox 360 could literally cost you $0.20 in electricity alone, not to mention the environmental impact).

The more powerful my smartphones have become, for instance, the more often my PCs are frozen in S3 Sleep. The effect is amplified as people move to very low power devices like the iPad, doing many of the tasks that they would have powered up the home PC for (or worse, kept it running around the clock to have it available) instead on that ultra-low power device.

The impact is measurable. The financial impact is significant (leaving a PC on around the clock costs some $20 a month). The environmental impact – or reduced impact – is invaluable.

The Android Developer’s Blog ignited a firestorm recently by publishing an entry on the purported fragmentation of the Android ecosystem.

It’s worth a read. Too bad the same can’t be said to most of the responses to it.

When the topic concerns either Android or the iPhone, and the arguable failings of each platform, it’s seldom debated by people genuinely interested in seeing the target of their criticisms improve in any meaningful way. Instead it’s pundits (less generously called fanboys) hoisting their flag, pushing in for their chance to line drive some talking points.

Rabble rabble rabble, Steve Jobs will eat your firstborn

Rabble rabble rabble, Android is fragmented and anarchist.

It is noise that seldom inspires or enlightens.

On Android Fragmentation

We have a problem, Mountain View.

Some handset vendors are lagging on upgrading their devices, if they do at all. There is a wide disparity in performance and functionality among devices, leading to unpredictable user experiences.

It parallels the dark days of Windows Mobile, when unloved devices were shat onto the market and promptly forgotten, the vendors more interested in prepping you for the next iteration that, they assure you, will fix all of the problems of the last.

For those writing application targeting Android, such as myself, there is a need to carefully consider the current version breakdown, choosing the target that offers the minimum functionality you need without restricting your market too much. You need to perform the same analysis on the target hardware capabilities, as the simple reality is that a rich graphical 3D application won’t run acceptably on a wide range of deployed Android devices. A high-end game might run well on a Samsung Galaxy S, for instance, while being unusable on every other Android device.

Do you target 2.1 and take advantage of Quick Contacts, for instance, and limit yourself to 45% of the potential customer base, maybe preparing yourself for the future? Or do you target 1.6 and simply go without those refinements? Do you build a 3D rendering interface that will choke on all but the latest handset, or do you cater to the lower common denominator?

It’s a serious concern. There are brand new Android devices that are hitting shelves today with Android 1.5 on them. There are new devices with mediocre processors, limited memory, and barely functional GPUs.

Isn’t that weird? It isn’t optimal.

Add that there are all of the various skins that the vendors put on their devices to differentiate: HTC’s SenseUI, Motorola’s MotoBlur, Sony’s Rachael, among others.

Android devices come in all sizes and capabilities: There are big screens and little screens; devices with keyboards and those without; cutting-edge super phones and feature phones on steroids; tablets and netbooks; processors with NEON instruction sets and without; with SSE or without.

An Android device is not a homogenous experience for either users or developers.

Crazy isn’t it? Anarchy reigns! Contrast this to the land of iPhone where there is a very strong uniformity and development consistency among experiences and development and devices.

Shouldn’t Android be more like the iPhone? Shouldn’t Google lay the hammer down and demand that vendors refrain from customizing, they upgrade in lockstep, they build a consistent, uniform platform, and nothing deviates from the reference implementation? Everyone building a Nexus One with a vendor label on it?

No way.

Fragmentation is the Reason the Android Platform Has Succeeded

The progress of Android over the past year, or even just the past 6 months, has been nothing short of extraordinary.

From being a marginal wannabe, largely adopted by those ideological few willing to turn a blind eye to its many, many faults, Android is now a very serious competitor among mainstream consumers.

It’s an accomplishment that Google could never have pulled off alone. They needed a lot of help from a lot of very capable partners, and their strategy with the Android platform and the Open Handset Alliance is entirely the reason we got to where we are today.

HTC, Motorola, LG, Samsung, Sony, nVidia, Intel, ARM, among a slew of others…a lot of very smart, very capable companies working to make their own success story, pulling up the Android platform at the same time.

Motorola wants to sell you a Droid, yet by investing so much into making their devices a success they’ve build an ecosystem where you benefit from their success even if you choose another vendor’s phone.

Fragmentation is progress.

Instead of releasing a single iteration yearly, a single lockstep uniform device and experience, a Darwinist battle is taking place, the best choices and experiences triumphing. The Evo 4G is hitting shelves today, yet already we’re hearing about the next superstar, a purported HTC Scorpion device with a 1.5Ghz processor coming soon.

When people argue that Android should be more like the iPhone, they lose sight of this very beneficial side effect of the so-called fragmentation.

If Google ran the Android platform like Apple, it would have been a Buzzesque flop. It would have failed miserably.

But they didn’t, and it wasn’t. We are where we are.

So where are we?

Well firstly, most of what you’ve read about Android fragmentation is bunk. Horror stories about incompatible apps and terrifying development processes are largely exaggerated.

But there are some issues.

Google Earth, Google Goggles and the official Android Twitter application (an excellent application that was built with Google’s help) all require Android 2.1 or above. It’s fairly obvious that Google intentionally limits the best to a recent version in efforts to motivate vendors to hasten their upgrades, and to showcase the most recent platform additions, such as the animated backgrounds in the Twitter applications.

Those are the reason that customers demand that vendors update their phone, and it helps keep them motivated to push out upgrades. Motorola just released the Backflip on 1.5, and Sony released the X10 on 1.6. Both of them have gotten a lot of criticisms in reviews for this choice, and they both will pay for lagging behind. Their motivation to get upgrades to the market quickly is very strong.

As a developer, when you develop an application you need to target a given version knowing that you’re cutting out anyone running something older (thus cutting out some prospective customers). In a subset of cases you can target an old version and optionally target newer features using runtime reflection and feature sniffing.

And there are compatibility quirks. In developing an app that integrates the camera I’ve encountered a number of issues, including devices that invert the preview output, some that only return greatly scaled down pictures when a picture is taken, among a variety of other little variations and surprises, and that’s just among the devices I’ve tested with.

There are known issues when dealing with OpenGL where some devices demand a certain magic quadrant of environment conditions to operate at their peak.

All in all, though, it is an absolute walk in the park compared to most other development efforts I have pursued. The Android SDK abstracts most everything to a shockingly effective degree, and the very clever, HTML-like (and thus XAML-like) layout manager allows you to very effectively run your application on a huge variety of displays.

You can build high-performance code in the NDK, including hand-crafted ARM assembly, but you can do it in a way that will only optionally be used if the right conditions are available (e.g. an ARM processor with NEON instructions), otherwise falling backed to a managed version.

In many ways writing to Android is a bit like writing an HTML5 application. There are quirks that you have to deal with on some clients, but they’re mostly known and documented and fade away as the clients move closer to the specification, without restricting the innovation and uniqueness of the individual clients.

Everything about Android is built to abstract from the device, from the virtual machine runtime to the density-independent pixels to the layout manager’s multi-dot-pitch resource utilization (I would have preferred that they used vector graphics, such as SVG, but it’s better than nothing).

If Android is HTML5, then the iPhone is ActiveX

When you develop for the iPhone, you develop for the iPhone. With the iPad there’s one additional form factor, but the universe of clients is extremely limited and of little variance. With the new iPhone there are rumors that it will see a simple doubling of each axis’ pixel count, allowing the system to simply pixel double previous applications.

It’s a bit like developing for Internet Explorer back in the early 90s: Everything was grand and entirely consistent and uniform, and in many cases you could just target an ActiveX control because you could make so many assumptions about your target.

There is limited variability build into the iPhone SDK or development process, and those hurdles are crossed when they are encountered. When the iPad came out, they updated the SDK to deal with the second screen size. The new iPhone SDK added the ability to deal with the new pixel density.

It makes a developer’s life simple. But it also seriously hobbles the rate of innovation and the ability to adapt to change.

It’s a Long Term Strategy

In the short run the single-target approach – the ActiveX approach – wins. In the longer run those early gains disappear, especially when you’re moving uniformly (“non-fragmented”) against many speedy opponents.

Just take a gander at the comments on any Android market share story. Just a few months ago it was gloating damnation as the iPhone ecosystem dominated the Android platform. Now it has subtly shifted, and the argument has moved from "iPhone dominates!" to “You can’t compare the many Android makers against just Apple! Apple still beats HTC!” It’s quite a shocking adaptation, and of dubious merit: As a developer I completely care about the platform saturation, not whether a phone is a Motorola or an LG.

And of course Apple does as well. There are millions of iTouch and iPad devices that don’t get counted in smartphone surveys, and they give iPhone targeting developers a staggering lead still, but we’ll see for how long.

Similar videos made the rounds for kids and the iPhone, kids and the iPod, all the ways back to kids and the original Mac.

If you want to show that something is easy to use, the thinking goes, record a child exploring it and post the results on YouTube.

It’s entirely backwards.

Children Are Uninhibited Explorers

Kids engage in active, uninhibited exploration to discover the functionality of a system, often through trial and error. They do this without self-doubt, prejudice or assumption.

Deviations from behaviours and interfaces that they already know don’t make them doubt their intelligence, or make them feel angry that their existing knowledge is obsolete.

While parents love to show off their children doing clever things, bragging about the superior essence of their DNA or parenting, the reality is that most children are shockingly clever.

I have three young children. They’ve had computers since each was a toddler. They have often seen me using computers. They watch each other using computers.

My youngest has, since well before his second birthday, gone to the office, turned on his computer, and logged into his account.

He launches Firefox and decides which site amongst his bookmarks that he wants to visit. There he explores the site-specific UI, picks the activity he wants to play, and masters its nuances.

There is little consistency among the sites or individual games, and they frequently change it up and add and remove content, yet that never discourages him or throws him off, but instead offers more opportunity for exploration. It plays a part in the enjoyment.

When he gets bored he goes to the next game or site.

If I’m at the office and I see him appear online through Skype, I can make a call to him and he’ll answer. He often exclaims “DADDY!”, then abruptly (how rude!) hanging up to get back to his game.

He figured out most of this on his own. There was no computer training course or bootcamp learning series.

The same technology aptitude certainly holds true for my other two. My daughter was beta testing a page when she was two. She had no problem quickly discovering how the elements interacted.

My children play videos from the network-attached storage. They take pictures of themselves with superimposed hats using the webcam. They scan their drawings for digital enhancements.

A year back I discovered my middle son, then a preschooler, on YouTube doing searches for Star Wars videos. I was sitting behind him on another PC and got curious when he asked how to spell Darth Vader.

Now that I’m onto my third Android phone – having gone through the HTC Dream, to the HTC Magic+, now onto the Nexus One, all over the course of less than six months – the Dream has been relegated as a portable media and internet access device for them, and again they figured out how to do almost everything it is capable of doing on their own, even on stodgy old Android 1.5.

Taking pictures, playing videos, navigating the market and installing and running applications (no pay account is associated, so thankfully they aren’t running up charges).

And it isn’t even an Apple product.

Children are brilliant. Of course I think my kids are especially awesome (as every parent does about their own sprogs), but most kids are capable of these feats if given the freedom to explore with the confidence that there are few negative ramifications of their actions.

Adults Are Dumb In Comparison

Adults, in contrast, usually approach anything new with self-doubt, biases, and prejudices. They have preconceived notions of how something should behave, and deviations from that leads them to grow angry and irritated that the subject is, essentially, making them feel stupid, while making their previously acquired knowledge obsolete.

If Facebook moves a button, there is mass outrage.

Linux desktops are generally considered deficient unless they closely mirror the behaviour of the Windows desktop that people are accustomed to. Individualizing, even where it is demonstrably superior, is almost always a negative return activity.

Just copy what people are accustomed to. Most adults actively avoid learning anything new, so the less “new” the better, generally.

Humanity suffered a decade of jokes about programming the VCR.

We’re seeing the same behaviour now in the smartphone ecosystem where Apple is held as the benchmark, and the way things should be done. While many bemoan everyone “copying” Apple, they need to keep in mind this people-are-dumb effect.

Of course this leads to a chicken-egg debate: how does Apple get away with doing things “Different”? How do they manage to get away with, and be heralded for, creating entirely new and different experiences? How do they manage to “make” markets?

Apple Treats Users like Children

Many users approach Apple devices as if they are children, just as the products are delivered as if to Children.

Watch Steve Jobs unveiling the iPad and seriously question whether that sort of presentation would have been tolerable from, say, Steve Ballmer or Eric Schmidt.

Listen as he soothingly delivers the goods like he’s a children’s author giving a reading from “Why Goombas Go Bananas”.

While that sounds like a negative statement about Apple, it is actually incredibly positive.

Given that there is a widespread expectation that there is a reward that will come from acclimation – they have been told that Steve Jobs has the Midas touch and Apple designers are the world’s best – people often approach Apple products with a wonderful child-like open-mindedness, willing to accept differences. There is knowledge that it will be a rewarding experience.

The same thing happened with the Nintendo Wii: Given positive press and a friendly impression of the company, many had an unexpectedly child-like response to it, and were willing to give it a chance.

This is seldom the case. Whether it’s a simple change of a web page, the use of the Ribbon bar in Office, a switch to Firefox – people have a strong gravity to things that they already know, and a powerful aversion to change.

It takes a lot of hype to overcome that friction.

Whether Apple products are really as intuitive and brilliant as claimed is debateable, but the belief that they are is enough to get them over that hump, yielding the same benefits.

On Acting like a Child and Letting a Child (or Adult) Act like a Child

One important note about children (or adults acting like a child) and exploration: To work there has to be an environment where there are few negatives to simple actions.

In the case above my kids use PCs running Windows 7 and limited user accounts, coupled with reasonable (but not overbearing) supervision, so there is little to no damage they could cause with a mouse and a keyboard.

Worst case I’d just wipe and reinstall.

They can’t accidentally order things on a credit card. They can’t install root kits. They can’t delete system files.

Having that freedom to try things risk free is critical to exploration.

It’s just important for adults who are learning a system, and perhaps a part of how we got to where we are was an ecosystem where simple exploration could be a destructive, damaging experience. Where an accidental keystroke launches a complete changed interface, for instance, with no obvious way to get back.

It destroys the user’s sense of adventure, probably forever.

Android 2.2 was announced at Google I/O just over a week ago. It is the highly anticipated update to the Android platform that brings a slew of very important improvements and refinements, including a reduced memory footprint, the addition of a JIT compiler to both the Dalvik Java-esque virtual machine and the V8 Javascript engine in the browser, push-messaging for any 2.2 or above targeting apps (with the reliable and powerful Google ecosystem to back it), streaming media, comprehensive ActiveSync support, among many other new features and functions.

Lots of refinements have taken place. It levels the ground with the iPhone, and provides some ammo against the soon-to-be-announced iPhone 4.

The Android platform is a very exciting place to be. Competition is excellent. Hopefully the iPhone 4 is a stunner as well.

A Terrible Roll Out

The (non-)roll-out of 2.2 could have been done better.

A pre-release of 2.2 got pushed to a subset of devices — those given to press or as handouts at Google I/O — shortly after I/O concluded, with no one seemingly knowing that it wasn’t the real deal, or that it was only going to a subset of devices.

The tech sites proclaimed that the roll out was underway. Purportedly there was a confirmation by a Google rep that the new version was being pushed.

Google had indicated, at the conclusion of I/O, that they planned on getting 2.2 out for the Nexus One in a “couple of weeks”, so this seemed like a case of setting expectations low and then wowing them. Everyone was eagerly checking their phones, suddenly convinced that it wasn’t worth using until Froyo was installed. Soon they could impress their friends with Linpack results.

But it wasn’t a real roll out. Nor was it the actual release version. Google didn’t seem to correct anyone until days later, after lots of people had downloaded and installed a covertly discovered pre-release version of it.

Android already suffers enough criticism for the purported need for “task management” (you don’t need to manage tasks! It is often simply a placebo), it doesn’t need people thinking that you need to root your phone, rollback to a prior version, and then roll forward, all to avoid being picked last for the team.

The pre-release clearly has some issues. There are observations about apps crashing, with commentators opining that those app devs need to “update their app for 2.2”, providing more misinformation to add to the talking points of Android detractors

Communicate better, Google. And stop treating the press (or even people sent to I/O by their employers) as your closest allies. They aren’t.

Android has succeeded despite sites like Gizmodo and Engadget and their brethren, a strong community and grassroots effort keeping it aloft in the face of a lot of effluence and misinformation.

This tarnished what should have been a stunning release of a stellar upgrade.

The Android Platform Today

It still is a bit rough for mainstream consumers. I own an HTC Dream, an HTC Magic+, and most recently added a Google (née HTC) Nexus One, and I love all three of them (and it is far and away the best platform for me and my needs), but there are still warts and issues.

Scrolling on the Nexus One, for instance, is often rough, like the device is over-extended. This includes scrolling down pages, between home screens, and so on. It seems like a trivial thing, and it doesn’t impede actual functionality at all, but it is one of those minor things that really favor the iPhone, where scrolling and transitions are brilliantly smooth.

The Sense UI added by HTC offers a much more iPhone-like smoothness, but the base OS shouldn’t falter on such powerful hardware (and Sense UI is why HTC lags behind the Google release cycle at such a distance).

If you add animations to interactions, they must be smooth or they become a detriment. I’d rather pages just instantly flipped rather than being poorly animated. Android does it well compared to devices that came before, but it’s the unparalleled smooth elegance of the iPhone that makes it pale in comparison.

The Android Market

The Android market is a mess. There are 50,000 apps, around 49,000 of them being horrendous make-money-fast junk apps (thousands of compasses, levels, and after a “success story” led to a gold rush, parked-car-finder apps).

The “Just In” section is simply depressing.

This is very similar to the Apple market, for those who might be gloating, where there are companies spamming the directory full of thousands of repackaged web pages and regurgitated filler. Apple might have strong opinions about what Adobe’s Flash-to-native compiler would do to quality, but they certainly have no qualms letting low quality junk saturate the app market.

Of the remaining 1,000 apps, many still have a pretty big “need work” sign hanging on them. As a contemporary example, Epicurious just pushed out an Android app to join their existing iPhone app, and while the iPhone app is lauded, the Android version is seriously deficient, almost to the point of being unusable.

Make it easier for us to separate the wheat from the chaff. The current rating system is broken, and seems to be primarily used by spammers trying to earn gems in garbage games.

Allow me to pick my own trusted rating authority to browse and sort the app market. This should be similar to looking at a movie listing and choosing whether you want to rate based upon Metacritic, IMDB, Rotten Tomatoes, Flixster, the community at large or just personally trusted reviewers and their network of trusts.

I don’t care what a million spammers have rated an app at, nor do I care to waste my time with apps that people whose opinion I value have given the thumbs-down.

The market is improving (the new system that will be coming after 2.2 is apparently going to be very similar to Xbox Live, where I can pick movies and games on their web interface from a desktop and they’ll instantly start pushing down to my powered on Xbox 360), but people need appropriate expectations going in.

It doesn’t matter how many apps there are in the Android market. Or the iPhone market, for that matter. It only matters how many satisfy your needs.

Taking on the Snaptic Challenge and Then Making it Personal

Snaptic launched an interesting challenge recently – the Move Your App! Developer Challenge – and it inspired me to get serious about Android development. More so than the cursory surface skimming I did previously.

It was mostly a nice external motivation and a bit of a deadline to get me taking the initial steps. Yet in the end I decided against submitting for the contest.

I didn’t submit for two reasons.

Firstly because I started two weeks before the contest started, and the odd evening or weekend just isn’t enough time to polish something in that amount of time. I don’t want to contribute to the pollution of the market.

Secondly because I couldn’t reasonably fulfill a basic requirement of the contest, which is that you have to integrate with their note/photo cloud service (the contest isn’t entirely selfless on their part). It didn’t lend itself to my app without seeming like a completely superficial tacked on requirement-filler.

I hugely applaud them for creating the contest, encouraging Android development and getting people to get on the move. Their prizes are incredibly generous and unique as well.

Alas, I’m going to keep on plugging away and will have a version 1.0 in the market in the coming weeks, and I will openly say that it was inspired by the Snaptic challenge.

Google and the Nexus One Channel “Failure”

A lot has been written about Google purportedly abandoning the Nexus One. In reality the market simply isn’t ready for unsubsidized phones, people much more willing to pay $199 for an iPhone, and then thousands in service fees, than to pay $500+ for an unsubsidized device.

It’s similar to bandwidth where people are more accepting of “unlimited” data plans, with onerous throttling and usage restrictions, vendors trying to game it to ensure that you use as little as possible, than to pay for actual consumption. It’s destructive for everyone, but it’s the Big Lie that we all live with.

It’s also true that Google is a terrible retailer.

When my Nexus One was ordered I quickly realized I should have joined two orders (the phone and the car dock) to pay a single shipping charge, but more importantly a single customs brokerage charge. Despite contacting them almost immediately I instantly got a stock reply saying that it couldn’t be done due to their ultra-expedient fulfillment process. The fulfillment wasn’t ultra-quick, and strangely I can make such changes to trivial, low-margin computer orders with no issues at all, but that’s the operation of Google where the earnings per employee is maximized, and saying no is easier (and cheaper) than being accommodating.

I also ordered an engraving (see below) on my phone and the phone came sans engraving. I filled out their support form to say “What happened with the engraving I asked for?” to instantly get an obviously mechanical Bayesian response that I needed to contact HTC for any order problems.

Google isn’t great at retailing directly to the public. I love the phone, but the experience was far from pleasant.

On Engraving One’s Phone

Several times prior I had almost completed an order but backed out once presented with the engraving options.

It seemed like something I should take advantage of.

But what to engrave? My name? My email address?

What if I transferred the phone to someone else once the next killer Android device came along? Surely they wouldn’t like having my name permanently engraved on a non-removable part of the phone.

What if I dropped the phone on a covert mission and wanted deniability?

At the same time I’d like for someone to be able to contact me if they find the phone, and maybe edge someone who is on the fence about whether to keep the found goods to doing the right thing.

So I went and registered a domain – mylost.mobi – and set my engraving to be yyz@mylost.mobi.

I “think bigger” in almost all that I do, and the idea was that such a site would offer a control panel and the ability to transfer the forwarding address, so if I transferred the phone I would simply transfer the email address.

Alas, my phone didn’t come engraved so I didn’t bother building it out. Neat idea, I think.

The scenario was that they needed to look up batches of geo-locations by postal code, passing in sets of 100 postal codes and retrieving the corresponding set(s) of latitudes and longitudes.

It is a real-world scenario, whether for mail processing system, census analysis, sales tracking, or many other common data processing needs.

You could simulate such a scenario with data like this. I did just that with the Canada Postal Code data.

In the reader's situation they were finding that batches of 100 postal code lookups took over a second.

That’s suboptimal, and not ideal for any system that needs to perform a large number of rapid lookups.

It is not the sort of task that I would normally do in a database. Hard to believe, perhaps, after the prior entries, but this is a process that I consider highly specialized – a unique snowflake, if you will.

The data is extremely static, and the usage is very specialized. Performance, rather than generalization, is paramount.

To validate the performance assumption I built a simple .NET test app that populated a Dictionary<string, List<Location>> with all 765,344 Canadian postal codes (there are, in some cases, multiple entries for a single postal code, so each dictionary element contains a list that contains 1..n results), and then looked up random sets of 4000 postal codes (hint: Create randomly sorted recordsets in SQL Server by ordering the results by NewId()).

It could lookup results at a rate of some 3,000,000+ lookups per second, with no parallelization running on a single mid-range core. Adding parallelization (extremely easy in .NET 4.0 using Parallel.ForEach) was of limited benefit as the reduce stage and thread safety efforts ate up any savings for all but the most unrealistically large test set.

That was an ultra simple solution with very few lines of code, specialized for the purpose. It did consume some 140MB of memory, but memory is bountiful and cheap.

Doing the same lookups in SQL Server, with optimized fill factors and a perfectly covering index — even after priming the cache (by doing a full select of the covering index) — yielding a return rate of approximately 5000 lookups per second on the same hardware, per core. The generalized execution engine simply isn't optimized for such a trivial lookup usage, and imposes significant overhead that isn't beneficial in this case.

3,000,000 versus 5,000 (per core) = an incredible reason to use a specialized solution, especially given that it’s the long solved problem of KV pairs.

The reader, after some correspondence, mentioned that they had settled on Redis, which is a solution that is midway between the custom in-application hashing solution and a generalized SQL solution (leaning much further to the former than the latter). The performance with such a solution will almost certainly be incredibly high, albeit bound by the overhead of IPC. Redis is a highly optimized solution for that task, and is quickly proving itself to be a viable part of most solutions.

It is the right solution for that problem. In no way is it a “new world order of social media and intraconnected graphs realigning the stars to herald the new way of using data”, but instead is a very appropriate use of the right tool. Redis, like Memcache, has a lot of metrics on its side, much unlike many of the other NoSQL solutions.

Using the right tool is what we should all strive to do.

Remember back when Daring Fireball was actually an interesting site? Now it’s like I imagine Pravda was during the Cold War.

In any case, I was a little shocked to see the gains come this quickly. Android was obviously making a dent, but I expected another generation of products to flow through the market – like the Incredible and the EVO 4G, and eventually the long-anticipated onslaught of Dell vapourware – before Android really took hold with mainstream consumers.

And of course iPhones are still selling. They’re selling like hotcakes, for that matter: Apple reported a more than doubling of sales during the period in question (kind of undermining the notion that it’s some sort of calm before the 4G storm).

At this point smartphones have a limited market saturation, which is why it’s a critical juncture and why there’s a bit of a race to perform a land grab. It remains anyone’s game.

In a year we might be looking at the emergent dominance of Windows Mobile 7 phones. Or maybe the next iPhone model will storm the market. Or WebOS will be reborn into a real contender. Or maybe Blackberry – the strangely unmentioned leader of the pack – will really wow the market with their next cycle.

It is critically important that there is competition in the space. Competition is great for everyone, including for iPhone users (see the upcoming multitasking that is a virtual clone of the implementation in Android). The market can’t be dominated by one vendor, especially one with walled gardens.

Speaking of walled gardens, as much as HTML5 got talked up during the Great Jobs-Adobe War of 2010, in reality HTML5 performance on the iPhone and iPad is abysmal and is close to unusable for rich content.

The real alternative to Flash on the iDevices has been native, single-platform, completely-proprietary apps. Which is arguably a perfectly fine approach (take full advantage of the platform and all), but it was incredibly frustrating seeing HTML5 used like such a cheap sacrificial prop during the debacle when it really had perilously little to do with the debate. It was so grossly dishonest.

Of course I’m speaking outside of videos. For videos Flash is almost always a completely unnecessary wrapper around an h.264 stream (which is what many Flash videos come encoded in these days). Separating the video from the container is a no-brainer.

Obviously HTML5 needs to keep improving. With Android 2.2 around the corner there are wide expectations that they’ll introduce a JIT compiler for the Dalvik engine (“native” apps largely run in a VM runtime and are currently seriously hobbled by the lack of optimizations in the same), and it would be fantastic to see something like V8 for the browser as well. While the platform has far-and-away leading canvas and dynamic graphics support, it falls behind in scripting and that would be a nice gap to eliminate. It would be nice to see a decent dynamic HTML 5 capability built into iPhone OS 4, as currently it is seriously deficient.

“We’ve been there before, and intermediate layers between the platform and the developer ultimately produces sub-standard apps and hinders the progress of the platform.”

I tried to ignore the whole section 3.3.1 debacle, as I’ve already commented on Apple’s moves and motives too many times, and really this change is just a new wrinkle of existing restrictions: developers for the iPhone are already subject to the capriciousness and fickle whim of Apple, with no recourse.

If Apple said that you have to wear a purple bow-tie while developing for the iPhone it would arguably be a change for the better. At least then you wouldn’t have uncertainty on the whole bow-tie color issue. From that perspective, Section 3.3.1 should be a welcomed clarifier.

While claims that it’s for the greater good of quality are discountable as a ludicrous smoke-screen, if you were gullible enough to believe that, and you accept the asinine notion that development technology dictates app quality, is Apple promising to filter app submissions by quality?

Given what is already in their app store they have a lot of pruning to do.

They could carpet-bomb out the crapulence, with acceptable collateral damage, by banning any apps made by small development shops. This would be great for everyone, right?

Kudos to them for removing consumer choice: If someone liked an app created via one of the targeted tools, clearly it’s because they don’t know what’s good for them. Personally I choose based upon reviews and user ratings, but it’s a win if these sorts of personal decisions are made for me.

Like I said, I’m not going to get drawn into this whole section 3.3.1 debate.

So let’s get back to Steve Jobs’ statement above.

“We’ve been there before, and intermediate layers between the platform and the developer ultimately produces sub-standard apps and hinders the progress of the platform.”

Where do web standards fit into this equation, given that web standards are almost perfectly defined as intermediate layers between the platform and the developer.

Stop. Pause. Seriously think about this. Imagine that Steve Ballmer was saying this about SVG (just use GDI and DirectX or at worst VML) and JavaScript and the canvas element, while discussing banning non-ActiveX controls from the Windows ecosystem.

Where does Steve Jobs think the web fits? Is it just a convenient stop gap?

We know that Apple’s existence through the lean years, and then its resurgence, was made possible because of the cross-platform web, and paradoxically because of the cross-platform Flash technology. iMacs sold to the general population only because they knew they could still use the open web, consuming and interacting with apps that could certainly have been built richer and better if they targeted just the Windows platform.

They could use the popular sites, read the news, do their banking, pay bills, and send eCards, despite being on a platform used by only the very few.

Yet now Jobs has made it plenty clear that the web is for trivial, simple stuff. For richer apps you need to target the iPhone alone, using a process that in no way can allow you to target other platforms as well.

Flash is a no go because it enriches the web – there’s a lot of hate out there by people who know Flash only as a simple video player, where it punted Apple's QuickTime it’s worth noting, and as a platform for irritating ads, however as a parent of young kids I see Flash as the enabler of an incredible array of rich and entertaining educational tools for kids (PBS Kids, TVO Kids, CBC Kids, Disney Kids, among countless others) – and Apple has done nothing in the mobile web space to make up for the gap…because at that point you’re supposed to bridge over to the iPhone market and embrace it with fervor and loyalty. Sorry, but no thanks.

This is expected, really, as Apple's very existence hinged upon an open standards web. There was a dangerous period in the late 90s when the web almost got Windowsified — look to South Korea as an example of this happening — and if that came to fruition Apple would have been dead in the water. Thankfully a few who saw past short term interests rallied around keeping the ecosystem open for innovative companies like Apple to thrive.

yafla is finally going to be born into a remarkable web application that exploits the rich functionality of the iPad and iPhone's web platform, along with Android, the Blackberry webkit browser, and virtually any other modern HTML 5 consumer, from big to small. I'm putting my actions where my mouth has been (this includes architecturally on the back-end, with decisions that I will document and explain along the way).

The arguably proprietary Flash platform has rough times ahead. While the actual numbers of iPhone and iPad users combined represent a small percentage of the web consumer ecosystem, it occupies a disproportionately large area of the mindspace.

Boo for the iPad! It bring us back to the obsolete era of walled gardens

While the iPad supports HTML5, that isn't the primary focus of content providers: The vast majority of them are rolling out solutions that target the walled garden of the iPad. Video content, books, magazines, or newspapers, you're entering the land of made-for-the-iPlatform solutions that nothing to do with the modern web.

And of course it isn't just big media. Many web sites — like Engadget, Digg, and so on — are rolling out apps for the platform as quickly as they can. In most cases those apps do nothing that can't be done as well or better as simple web apps, but such is the return-to-mistakes-of-the-past era that we're in. In the cases mentioned they also built Android apps (others, including Big Banks, are far more myopic about this), but there's still the question of why they built anything platform specific at all, beyond the obvious explanation that they're hoping on the bandwagon.

Everything old is new again.

John Gruber argues that the iPad represents a more, open and innovative ecosystem than with the Atari 2600 circa 1978. Hard to argue with that. But what about the 32 years in between, John?

It's a clear sign that there's something seriously wrong when you have to base your comparison on an early home game machine from three decades ago.

For the past 20 years we've had a computing market where anyone and everyone could build applications for the vast majority of devices. Since the incarnation of the web, those creators have had the ability to have just as much presence as makes like EA. There is nothing new there. The only "advantage" that the iPhone cum iPad offers the little guy is that the market was so nascent and novel that a million made-on-a-weekend apps could sell thousands. That early ease is quickly disappearing, and the natural size advantage of shops like EA is coming to fruition. Small-shop, single-trick apps are going to very quickly get crowded into an unlit corner.

The Apple app model is horribly, horribly broken, though they have enough goodwill, and still get by with many deluded into thinking that they're the underdog little company, that they'll be able to float with it for a while longer while apologists continue to present their questionable defense. The iPhone and now the iPad are not simple game machines (whether from 2010 or 1978) — comparisons with the Atari 2600 or even the Xbox 360 are highly deluded — but represent a serious movement into the domain of general computing, and against that they should be compared. Pretty remarkable how Apple has managed to retroactively turn Microsoft into the good guys.

The Apple web model is brilliant, with it representing a fantastic web appliance of the best kind.

Let's just hope the web survives through this, and there isn't a rush from open standards to the opposite-of-open-standards walled garden.

• Tonight I switched data centers. There may be a small outage for some because I forgot to lower the TTL on the old DNS entries before the move: I guess being the, uh, "world's most pre-eminent domainologist" doesn't mean I'm always on top of such things, at least not for what now is just a wrapping around a blog.
• I made the move because I'm finally making use of the domain, and the reason for needing a more powerful platform will become clear.
• I often do minor edits of published pieces after the fact, without noting it as an edit. Often I'll read back through the archives and see typos, wording that I'd like to correct, etc, so I fix it for future readers. The most common edits see me removing parenthetical asides that add nothing. The software I wrote to run this does store every revision, so technically I could provide a diff'able history and have long considered doing that.
• Speaking of blog software, one solved problems that strangely causes havoc again and again are dynamic sites that fall over whenever they get any attention. Over the past while I've had days with torrents of incoming visitors, and the CPU needle barely ever spikes to the point of being measureable. It is just inconceivable that a simple blog dies when it gets on Reddit or the like. Seriously - caching, figure out how to use it. If you can't get it into your software itself, stick your site behind an nginx instance and use its wonderful functionality.
• Every entry that I post is automatically run through tidy. I really consider it important that mark-up claiming to subscribe to a given standard actually honors that standard.
• I've withdrawn from the whole NoSQL debate because it got incredibly boring. My closing note is simply to say that the way that many are using NoSQL is like discovering the buggy whip at the beginning of the automotive era.

Incredible times ahead.

Rebuttal in tl;dr; Form

The original post was motivated by claims found on Digg’s technology blog.

• They say that the RDBMS “mindset” favours writes over reads: BLATANTLY WRONG CLAIM.
• They show poor index and schema use: WRONG DATABASE USAGE.
• They show that their database product can’t join: BAD DATABASE SERVER. RED FLAG.
• They report very poor performance without adequate detail: MEANINGLESS PROPAGANDA.
• They use this to show that the RDBMS can’t cope: SEE ABOVE.
• They say that if you don’t use all of an RDBMS’ feature set, you’re essentially using NoSQL: ABSURD.
• They describe scaling out issues with databases: TRUE FOR MYSQL.
• They described their move to NoSQL: GREAT FOR THEM. THOUGH REALLY THEIR SOLUTION WAS EXTREME DENORMALIZATION.

And on Joe’s post.

• You need an expensive DBA with the RDBMS, not with NoSQL: SPECIOUS, FLAWED REASONING.
• Capital expenses suck. Services are better: BUSINESSES GENERALLY LEASE THESE DAYS.
• $7,500 “just for disks”: FOR A SaaS BUSINESS THIS IS CHEAP.
• 50 node cluster: 50 NODES IS A COMPENSATION FOR ABHORRENT I/O RATES.
• SSD drives are expensive: NO THEY AREN’T. YOUR ARGUMENT IS OBSOLETE.
• Commercial database products are pricey: VIGOROUS AGREEMENT.
• NoSQL $/read and $/write win: MAYBE, MAYBE NOT. DIGG COULD LIKELY DO MORE WITH A COUPLE OF SSDs THAN THEY CAN WITH THEIR MASSIVE DENORMALIZATION

The Non-ADD Version

Joe has been in the Web 2.01 trenches. He built a solution that powers one of the top sites on the net.

Remember when getting "Slashdotted" was a big deal? Getting on the front-page of Digg makes a Slashdotting-at-its-peak look like a little traffic bump. There are probably a hundred PR reps busy trying to botnet their clients onto the front-page of Digg for every one punished into spamming Slashdot these days.

Far more people know Joe’s all-out-of-bubblegum name than will ever know mine, and rightly so.

A Strawman Built on Cliches and Appeals To...

Joe comes out of the gate resorting to the venerable old-versus-new tactic: "It's just those old-school DBAs upset that us kids are rewriting the rules," he says in not so many words, while nailing himself and his peers onto a cross, seeking pity for the flames they doth receive for their unconventional, rebellious ways.

This is a bit strange, really. Barely a day goes by lately without Hacker's News or Reddit’s /r/programming featuring another front-pager about how the Incredible NoSQL is rewriting the rules of, well, everything. The general demeanour is one that, I think, is far more sympathetic to completely unsupported and undemonstrated pro-NoSQL claims than it is to anything that questions the hype.

Countless NoSQL blogs have appeared (though if you browse them looking for actual content you’ll instead find that most feature few facts but lots of zealous punditry. Advocacy seems to be the primary focus right now). Anyone involved with any sort of NoSQL initiative is spinning off their own start-up to capitalize on this sure-win formula, acting like it’s some sort of magic ingredient that will assure them of success.

It is very reminiscent of the XML heyday – I’m a very big fan of XML in its place, as an aside – when countless start-ups appeared with business models that could be boiled down to “something to do with XML”.

The big database vendors have remained quiet, largely because the miniscule-budget operations all clamouring for their piece of the NoSQL pie aren’t worth bothering with.

“But what about Google, Amazon, and Twitter!” you say. Joe resorted to that same appeal to authority by incanting the same magical trio (say it three times quickly and your TPS rate will quadruple!). Not really much to bother with there, beyond pointing out what a cargo cult is. Your bamboo headset won't make you successful like Google. It really won’t.

Unless you are targeting the same problem space as those companies – say like providing very low performance but highly “scalable” database solutions for countless low-value start-ups – their solution choices are utterly irrelevant.

I'm not a DBA (though knowing how indexes work now strangely qualifies one for such a title). I'm just a technically curious solutions guy that has an innate need to keep asking questions and probing deeper until the Want-To-Believe fog that often hides hype dissipates.

On Rinky-Dink Operations

In Joe’s entry he focuses a lot of attention on the costs of RDBMS solutions.

One such argument is that it’s better to use computing hardware as a service than to buy, seemingly implying that while you can buy good hardware to run a RDBMS, it is better to rent less-good virtual hardware to run your NoSQL instances.

Yet leasing is what all the cool kids are doing these days, largely for the same financial reason. Writing it all off beats dealing with depreciation BS, and it makes financial planning a lot easier.

On the leasing front, $600 a month gets you an insanely powerful, makes-an-Extra-Memory-Quadruple-Extra-Large-EC2-Instance-Look-Like-A-Pile-Of-Puke server.

You’ll probably be paying 20x that for every developer you have working on your solutions. Is this really so astronomically high?

That less-than-the-cost-of-the-office-cleaners price tag gets you a server that with a bank of striped SSDs that will almost certainly demolish your impressive-in-count-but-not-in-throughput big scale out cluster, at least with a non-broken RDBMS system.

No really, it will. Of course for any sort of reliable system you’d have to pay for some DB licenses (presuming you aren’t going with PostgreSQL), and then you’ll want to double everything up into mirrors or some other reliable setup, so triple the price.

And really, is the $7,500 spent by 37signals on a disk array really even worth mentioning? I suspect that sort of number ends up almost as a rounding error on their expense sheets, and given that it's pivotal to their operation – it sits under the very foundations of their business – I doubt they spent many sleepless nights over it.

What sort of rinky-dink operations are we talking about here? Does Digg still qualify as a start-up? Don't they have a payroll and all of that, yet they're clamouring to wire up a collection of discount bin servers?

I posted the SSD entry because SSDs really do fascinate me, and I do think they change a lot of the rules of the game. It just happened to dovetail nicely with my investigation of the Digg scenario, where Digg solved their very real I/O issue by essentially pre-caching every possible query result for a targeted need.

Through extreme denormalization they traded storage to reduce I/O needs.

This is a very important point, because it’s far more pivotal to Digg’s solution that the NoSQL versus RDBMS debate.

Call up your old Digg coworkers, Joe, and have them setup a real database server with a couple of SSD drives and see how it compares with their impressive cluster. I’ll bet Dell would happily lend them a real server.

All of this is a bit humorous, really: The whole point of my original entry on this NoSQL topic was simply to say "what is good for Digg isn't necessarily appropriate for all database needs”, so it’s a bit unfortunate that it has come to this, with Digg’s former architect justifying their decision when they were held as a scenario where it is likely the perfect solution.

Then, after seeing the Digg case-study, I felt obliged to respond to their RDBMS claims because I saw them as flawed, indicative that the movement should really be called NoMySQL instead of NoSQL. It still doesn’t diminish the correctness of their choice.

But really, while I originally entered into this debate believing simply that NoSQL is being oversold (it is grossly inappropriate for the vast majority of non web 2.0 projects), the more I investigate the more I’m coming to think that it is a solution for the rapidly disappearing problem of pathetic I/O rates, at least assuming that you aren’t running on several of the cloud solutions where that is your only choice.

There are many other differences that come with NoSQL (many strongly questionable, like the oft lauded “no schema” claim for some of the solutions), but the I/O restriction is by far what sold it on the high end, and the high end is what convinced the little guy that it’s the way to go.

Oracle, DB2, SQL Server, Teradata, Vertica, Greenplum, Sybase and Friends All Cost Way Too Much

I very strongly agree with Joe about one thing: the licensing costs of the big RDBMS products are way too high.

They know that 2% of their potential customer base have giant budgets, and that they can squeeze more from that 2% than they could ever get from the other 98% who then get relegated to fighting over scraps like MySQL.

Not really sure how to solve that problem, but I concede that it is a non-trivial issue. PostgreSQL is probably the best low-to-no-cost database server, but even then quite a few performance features are missing (like real-time materialized views or SQL Server style clustered indexes).

In my various drafts I had included details about Microsoft's participation in the various SVG groups, and how that bodes well to its eventual adoption by Microsoft. Those got edited out before publication.

Alas, while it took far longer than it should have, it is a wonderful development to see that Internet Explorer 9 finally adds SVG to the browser.

Shocked by the incredibly poor database performance described on the Digg technology blog, baffled that they cast it as demonstrative of performance issues with RDBMS’ in general, I was motivated to create a simile of their database problem.

While they posted that entry six months ago, they recently followed up with more statements on the NoSQL / RDBMS divide, and are now being heavily used as a citation of sorts.

For instance Dare Obasanjo held Digg's moves as a rebuttal of my prior entry on SQL scaling (though my entry actually explicitly excluded incredibly rare edge cases like Digg's, and my core point was that the majority of database uses don't have the needs of a site like Digg, I'm always one to take on a challenge), which then got picked up in other blogs.

Digg's case is an example of an entry-level RDBMS product used arguably suboptimally on under-powered hardware, and it seems questionable whether it proves anything of substance about either database technology. Yet it's held as demonstrative of something — in particular the failing of the RDBMS — which is why I focus on it. They are different tools in the toolbox, arguably for different purposes, and that isn't the focus of this entry.

So let's take a look at Digg's scenario.

I do this to evaluate their performance claims, to confirm my previous statements about indexing improvements, and to determine the impact that SSDs have on the problem space, because I strongly believe that SSDs (and cheap memory) completely change the equation.

The focus on this entry is not to question or answer whether NoSQL is the right choice for Digg — though there are some ramifications as SSDs take over, which is, I think, an interesting aside — or whether Google or Amazon or anyone else should use it, etc.

SQL Server 2008 Developer Edition, itself viewed as almost a training-wheels RDBMS by many, on Windows 7 was the most convenient platform for me when I ran this test, so I created a quick script to create what I think is a pretty accurate reproduction of the database described in their blog entry—

• 500,000 users, having…
• 10,000,000 friend relationships (using a power law distribution)
• ..and 500,000,000 “Diggs”, randomly distributed among 500,000 virtual “items” (which might be comments, submissions, etc) with a date range covering four years.

The database weighed in at a svelte 30GB.

I ran this on a two-year-old desktop machine with a Q6600 processor and 6GB of RAM, on a standard 7200 RPM consumer drive. You can easily find laptops with more processing and I/O power.

I opted against running it on a real server (you know, like a 24-core, 128GB, banks-of-SSDs monster than many real databases run on) simply because I knew it wasn’t necessary, and went contrary to the demonstration that even a mediocre machine can beat their results.

DISCLAIMER: This is not a high-fidelity reproduction of Digg's situation, as is pointed out many times in many ways in this post. However Digg took the time to post metrics to support their claims that they are some sort of extreme case, at the edge of database limits, and I simply don't believe that is true. Digg's data quantities are relatively small and lend themselves to sharding. The second point, which again is hammered home many times, is that SSDs present a solution that changes the equation, and, I think, provides some interesting inputs to the situation.

The First Clue That Something Isn’t Right: You Can’t Do a Simple Join

The Digg blog entry detailed how they had to manually build an IN clause given their selected database product’s inability to adequately run a trivial join, with the resulting query taking 14 seconds to find the Diggs for a given user’s friends against a single selected item.

This yielded a results return rate of 0.07 per second.

If you can add an IN clause that solves a database join issue that functionally should achieve the same, there is a much larger underlying issue that needs to be dealt with. I'm not a MySQL user, but apparently it offers minimal plan investigation tools, so there aren't the options to fully flesch-out what the query engine is doing. Nonetheless, it is a warning sign of a foundational product issue.

I ran a similar query in SQL Server, albeit without the hand-coded SQL builder, looking up friend Diggs for randomly selected combinations of users and items. It returned so quickly relative to Digg’s experience, even from a cold cache, that I had to up the iteration count to 1000 to get good test durations.

SQL Server was returning a fairly constant 36 result sets per second, probing the friend table’s ten million relationships to find the selected user’s friends, and then probing the five hundred million Diggs for the pertinent records, sorting it in the manner that Digg sorted their results. The query needed to draw data from all over each of the respective table populations, ensuring that it wouldn’t benefit from localized hot-spot caching. To prove this, limiting SQL Server to only have access to 1GB of memory had a negligible impact on the performance.

CPU usage was marginal, with the limiting factor being the slovenly I/O of the lowly magnetic disk. The iterations were run sequentially, as parallel runs yielded no net benefit, the magnetic disk moving as quickly as it possibly could already.

Already we’re running at close to 500x the rate reported by Digg, without doing anything beyond using an arguably better database product, at least in that it can join properly. MySQL's many weaknesses are well, well known, so the core point from that is not to question Digg (though their indexes were suboptimal), but to put their database product under a cloud, as they themselves often do when posting about their move (usually openly declaring their restricted option set given that they limited themselves to open-source products, obviously eliminating from consideration many of the clusterable, very high performance RDBMS options, even if that were a better choice which is completely uncertain).

Their dataset distribution may be entirely different, however even if I doubled or quadrupled or octupled the count in every table it would only marginally impact performance.

At this point I implemented the indexing changes I described in my prior entry – removing the surrogate keys and cluster-indexing on the unique columns – and the lookup rate jumped to 71 result sets per second, or around 1000x the speed reported by Digg. If I massively increased the data quantity and return counts, the difference between their poor indexing and proper indexing would dramatically widen, with the proper indexed solution showing little difference with significantly increased data counts.

If the database was cached in memory those index changes would have had a much more profound impact.

What If Localized Data Isn’t Your Primary Optimization Strategy?

I had been meaning to get an SSD for Eclipse Android development, so when my new 100GB SSD arrived (it’s an MLC unit that did well on an Anandtech review, though I won’t mention specifics as it isn’t pertinent – any decent SSD will perform at a similar level. Of course for real-world production use you would want an SLC drive) I detached and moved the database files over.

A quick reattach later and the 30GB was very amply hosted in the 100GB MLC SSD.

I fired up the benchmark to be pleasantly surprised to find it returning results at a rate of 4100 result sets per second. The write performance, while not a focus of this test, also hit extraordinarily high levels (which would conveniently lubricate the use of copious indexes).

Correcting the indexes and moving the database to a single inexpensive consumer-grade SSD, running on a dated desktop, had results coming back at a rate 60,000x what Digg reported.

None of this is intended to be a serious benchmark of SQL Server (I don’t wish to fall on the wrong side of a DeWitt clause), or even Digg's use of MySQL: This is not a disciplined benchmark, and during parts of it I hopped into some windowed online matches of Battlefield:Bad Company 2 while tests ran, after seeing that it had a limited impact on the results. I knew that the primary weakness was simply the movement of the hard drive head, and different technology choices (NoSQL versus RDBMS, normalized versus denormalized, clustered versus heap, etc) primarily impact how often and how far that head has to move.

And of course I don’t have Digg’s data, so it is completely speculative on my part based upon some rough descriptions given in the Digg posting. Maybe he hugely underestimated their data counts, or their data entropy is vastly different.

This is a macro-benchmark: Digg’s claimed results were so poor that I went in knowing that the difference would be very large.

Their described data quantities are small in the world of large databases. Most decent relational database products don’t even start to sweat with tens or hundreds of millions, or billions, of rows.

The key, of course, is proper indexing, trading write performance for read performance targeting your specific needs.

Indexes could be viewed as ways of creating “virtual tables” that are maintained in lock-step with your base table. Decent database products like SQL Server even allow you to include unordered but included columns in your index to ensure that you have a covering index (the best kind) for all scenarios. And that’s before you even get to the magical world of materialized views.

So either MySQL is an atrociously bad product at the larger limits, which ample evidence seems to point as a truism, or the Digg staffers simply weren’t getting the most out of their systems, but it’s hard to take their statements about the RDBMS field with seriousness, and their arguments more correctly invalidates MySQL more than it invalidates the RDBMS.

The fact that Digg is a large site says nothing to their technical leadership or mastery. Their site has sped up by leaps and bounds over the past year, so I suspect that they know what they are doing, but I'm wary of any cargo-cult like "they did {x} and they're a big site therefore it must be a good option" appeals to example.

On The Role of the DBA

What is most disturbing about this whole database debate are the number of commentators who excuse horrific database usage (not in relation to Digg's issue, but as a general conversation point whenever people make comments about proper database use in virtually any online discussion), ascribing rudimentary database performance design and knowledge as something that is limited to the elusive “DBA”.

This is ignorant and frightful.

You don’t know what a b-tree is? Don’t know how indexes work? Don’t know what a red-black tree is? Please get away from the compiler and save the world from your monstrosities until you have some knowledge of these basic concepts.

This is not esoteric knowledge, and instead is rudimentary comp. sci. knowledge.

DBAs are the guys who setup user accounts and monitor security, schedule backups and determine macro-optimizations like how to allocate file groups on the SAN arrays. They might probe lowest-hanging fruit performance issues and flag offenders or offer up suggestions.

Rudimentary database design and proper usage is the basic responsibility of developers, and if you don’t know it then it is your responsibility to learn it. Alternately you can just clutch onto NoSQL and bleat about how it changes all of the rules anyways, which is the route quite a few have decided to pursue (I fully expect to get the standard angry responses from those who take this like a religion).

Is NoSQL a Solution for Yesterday’s Problems?

Database servers really like having a lot of RAM. Ideally you should have more RAM than you have data, allowing it to cache the entirety of your DB (or at least the working-set quantity of DB on that partition) making incredible read performance achievable.

Joining rows is not a hard activity for database servers. It can do it at unfathomable rates if the data can be fed to it at the appropriate pace and in the right form. Even heavily normalized databases can be high performance.

What normally makes joins a performance issue is data locality: if you have to load two rows from different places on the disk, that’s two seeks instead of just one (or three, four, five or more instead of one). When seeks are as costly as they are on a magnetic disk, you avoid it (either by striving for a database that fits in memory, which paradoxical often calls for heavy normalization, or by de-normalizing).

Writes are obviously important too, yet on a site like Digg I would guess that reads outweighs writes — from a user interaction perspective — by a factor of 10000:1 or more outside of logging (which usually goes to a log-specific technology anyways).

In contrast to all of the “everyone is a publisher and the internet changes everything” bluster that is used to herald the wave of change that NoSQL brings, the reality is that it’s a very small percentage of users that post submissions and add comments, or even that do the simplest possible action of clicking an arrow.

Users overwhelmingly simply consume data, whether it’s the latest tech news, Asthon Kutcher’s tweets, or just browsing through the comments on a Slashdot article to see if they add any additional insight.

Despite Digg’s recent claim that they are “write intensive” (maybe because they’ve decided to dramatically explode the number of writes a simple action causes?), at its root their platform is primarily read focused, which is why they pursued Cassandra in the first place. Take note that their NoSQL solution for friend Digg lookups is to take every Digg and massively explode the number of writes it causes to happen (in the case of a Digg by Kevin Rose, a single write becomes 40,000+ large writes).

Hardware Is Cheap. Manpower is Expensive

If I had 48GB of RAM in the test machine (which is fairly pedestrian outside of gerbil-sized cloud instances. Note that you can now add 128GB of RAM to servers for around $4000 in some cases), outside of the initial caching period the select rates would be stratospheric regardless of storage medium, though SSDs would still come in a very, very strong lead when it came to write performance.

For the same $4000 you could chain five Intel X25-E drives for 320GB of intensely high performance – and persistent – storage. Just keep going up until you have more throughput, I/O and storage than you could dream of.

Some high-end enterprise solutions now tier storage and automatically place data as appropriate, choosing between magnetic, SSD, and memory caching systems. The pages of the table that are never touched end up on the magnetic storage while the hot area – say Diggs within the last 6 months – are moved to SSDs or to huge banks of memory caching.

There are bountiful options to achieve incredible performance, even on a budget, balancing memory and high performance storage systems.

Throwing Storage at the Problem

I didn’t waste the disk space, but as mentioned before I could do a simple join between the tables, materialize the view, and the performance would be very high even on magnetic disk, although it would add a serious cost to writes: When a user with a large number of people who befriended them dug something, their record creation would branch out into the write of potentially thousands of records.

That was the route that Digg took: They are pre-computing the sets of data that a user might possibly want, even apparently for reams and reams of long abandoned accounts.

They do this because looking up data that can’t be cached in memory is an expensive operation. Yet as has been shown, SSDs, which are getting faster and cheaper regularly, completely flip the I/O equation.

SSDs change everything.

Turning a small amount of data into a massive amount of data to improve performance paradoxically makes SSDs much less attainable (because the cost per GB is so much higher), and humorously may thwart the end goal. It also reduces the ability to memory-cache the relevant data.

By pursuing this solution, Digg has limited their ability to choose other solutions that are clearly hitting the mainstream.

Coming Next – PostgreSQL versus Cassandra

There is a complete absence of objective measures of the performance of Cassandra. In place of real performance comparisons and load metrics are a lot of hand-waving and comparisons against completely broken database products (never, ever hold MySQL as the vanguard of the RDBMS world. It is comical to do that) running horrendously malignant queries.

Not anymore. I’m on the case.

My goal is not to belittle the product (which I think is elegant, beautiful and concise, and serves a very important role), but simply to bring some rationality to the argument, as it is currently missing.

[EDIT: The following statement has been proven to be a wrong interpretation, but I leave it here out of humbled shame]Digg claims that Cassandra brings them “linear scalability”, yet every one of their Cassandra nodes is 100% replica of the other, meaning that a write (or 40,000 writes) on one is communicated and then replicated on every single other instance.

Response to Criticisms - 2010-03-25

This entry got picked up on a couple of excellent tech-oriented sites: Hacker's News and Reddit r/programming. Included in the comments of a lot of very smart people are a couple of common criticisms that I thought worthy of specific response.

"Your benchmark stinks. How about you..."

My benchmark, if you can even call it that, was focused on O(n) complexity and the difficulty of joins among very large tables with a half-decent database product, with the core take-away being "it's a solved problem. With proper indexing and a decent database system most datasets are `small'."

On the topic of concurrency, I mentioned that in the entry, noting that executing many parallel runs of the test yielded the same net output on the magnetic disk, while it actually significantly improved performance on the SSD and then leveled off. Database servers are fairly smart about concurrency and task queing.

The top result of 4100 resultsets per second, which was achieved using many simultaneous runs, still wasn't fully exploiting the I/O capabilities of the SSD, owing to the tuned-for-magnetic-disks nature of the database server that I didn't bother resolving.

However the Digg case study lacked significant details beyond a couple of spurious size details there to indicate, I believe, that "we think our data is large and the RDBMS can't service our needs". What I based my run on was quantity-of-data (which is not large in the land of databases) and key phrases like "from a cold-cache" (which can be reasonably interpreted as "on a test instance"). There is a lack of details in the Digg benchmark, given that I don't think they were intending it to be a industry standard metric, so it isn't reasonable to expect so much more regimented discipline from mine. However let me say that I did take the meager stats that were given and, where possible, erred to the high end — where "hundreds of million" appeared, I went with 500 million (if I went with a billion it would have barely impacted the results, but I was impatient and didn't want to wait for the data setup script to run that long). Where they said "millions" I went with 10 million. Some of the responses are demonstrative of how fact-free the debate has become, so it's not particularly surprising that NoSQL blogs group-hug around it.

This is not a replication of Digg's runtime environment, and any illusions that it pretended to be intentionally misinterprets. Though if it were a serious apples-to-apples comparison I would have run it on a serious server with serious load simulations, where the only orders-of-magnitude would be the difference between the results and what I achieved on a dated desktop.

In fact, 36 or even 71 results per second is still far too slow for Digg's use (especially given that they are stuck with a web technology that forces synchronous database calls), and I'm not even purporting that to be a viable option for them as they add out a lot of data-intensive personalization options. It's simply to contrast against their abhorrent performance number which I think are grossly misleading.

"But Google and Facebook and..."

Sure. That has nothing to do with this.

"So you're an RDBMS guy who hopes SSD prevents change..."

I'm not an "RDBMS guy". In a prior outing I was declared a DBA because I didn't just roll over for the NoSQL propaganda, and now I'm cast as a guy who holds himself as a database expert. Actually neither are my primary competency, and I think that's the point: I don't purport to be Joe Celko, or even a remote approximation, yet even I can see some massive issues in Digg's case study.

I'm just a solutions guy that looks at technologies and tries to digg (har har) through the cruft and get to the truth, which can be tough amidst tech religions: Warp back to 2001 and try to have a rational discussion about XML. In the case of Cassandra (and many NoSQL solutions) there is stunning ease with which many make absolute statements about RDBMS, such as the many "relational databases can't handle large amounts of data, just look at Digg" claims that litter the web, while cheering on vaguery and unsubstantiated hype about NoSQL solutions.

Show me real performance numbers for NoSQL solutions: They are disturbingly rare. Instead the argument is dominated by noise comments and hand-waving about how grand NoSQL is because it just simply solves everything and makes everything great.

Digg's NoSQL performance advantage is achieved by localizing all of the data necessary for a given request — in this case "tell me all of my friends who Dugg this item/parent item" which they had precomputed and cast out — ignoring the problem of MySQL not competently doing joins (apparently it has troubles sorting as well).

That is overwhelmingly a storage seek issue, and Digg's solution was to turn many seek actions into one or two by massively exploding their core dataset so the data for every need is repeated and persisted for every possible use. I can say right now that there is no question that if I performed the same benchmark on Cassandra, drawing randomly distributed user-item buckets from the same magnetic disk, my performance would max out at the number of seeks per second of the disk, which in the case of a normal desktop drive is somewhere in the range of 100-200 seeks per second.

Of course NoSQL yields the same massive seek gain of SSDs, but that's where you encounter the competing optimizations: By massively exploding data to optimize seek patterns, SSD solutions become that much more expensive. Digg mentioned that they turned their friend data, which I would estimate to be about 30GB of data (or a single X25-E 64GB with room to spare per "shard") with the denormalizing they did, into 1.5TB, which in the same case blows up to 24 X25-Es per shard.

This is interesting, is it not? Maybe it rains a little on the NoSQL parade, but to me it's a pretty fascinating development.

"Why do you have to insult the Digg crew?"

I don't intend to insult them, but at the same time I don't fall in the camp that gives them credibility simply because they're behind a large site. Many of the largest sites on the internet made technology mistake after mistake, yet succeeded regardless because they have a good product: These are some serious examples where ideas beat out execution. PHP somehow formed the base of a good number of the internet's largest sites, yet are there many that will seriously argue for its technical superiority?

"These are two different tools. I'm sick of this argument! Let's get back to the NoSQL Propaganda Parade."

In many cases they are used to solve the same problem. In the specific entry I refer to for this post the whole point was "we were using an RDBMS, and now we're using a NoSQL and it's so much better", so is it really rational to claim that they're two completely different worlds?

"NoSQL solves different problems like scaling out, data centers, etc."

Orthogonal. Cassandra solves problems that you can't as easily do with MySQL. MySQL != the RDBMS industry.