I passively monitored Apple’s much anticipated announcement yesterday via a TechCrunch live feed. Apple makes a lot of brilliant products, and their announcements have a big impact, so it's beneficial for anyone in this industry to keep interested.

The TechCrunch show consisted of a couple of people monitoring the twitter feed of someone actually invited to the event while incompetently dealing with technical challenges like “show a graphic” or “don’t abruptly inject a floor audio feed without warning”.

One of the hosts demonstrated why so many of us have an automatic skepticism about the critical reception of new Apple products: As the picture of the product came onto their feed – carried down from the mountain by Jobs – her reaction was “Uhhhhh....it’s gorrrrrrrrgeous!”

Her observation is only shared by the truly faithful, though surely the rest of the Apple herd will inevitably come around. You can be sure that going forward this nondescript rectangle will become the new benchmark of product beauty.

Everything that follows will either be ugly in comparison, or declared a rip off. I just discovered a digital photo frame beside me which I sadly must report is a rip-off of the pure, blessed genius of Apple.

Early Prototype

A Big iPhone

We now know that the iPad is essentially an iPhone with a larger (low resolution, 4:3 ratio) screen, minus voice. Clearly it runs an ARM-derived processor, with performance likely very similar to a Snapdragon 1Ghz. Apple is talking a big game about the A4 system on a chip (saying things like “Intel is looking to do this with their Atom”, ignoring all that came before to pretend that they lead the pack. It's like coming in last in the marathon yet talking about how you finished before next year’s winner), so it would be interesting to see it put to the test against, for instance, the Tegra 2.

One other feature of the iPad is that you can change the background. Apparently that’s a pretty big deal.

The iPad seems to be the continuation of Apple’s platform royalty play, and may be subsidized in the same way that Microsoft or Sony sell their consoles. With this device Apple is going upscale, moving beyond the repackaged web pages and novelty water cooler apps that overwhelmingly dominate the app store. Getting a cut of magazines and books and even more media will surely pad their pockets.

To repeat what I said before, Apple and Sony would be a perfect union. Their modus operandi is virtually identical. Aside from the common quest to act as the troll under the bridge collecting a toll, they share a profound propensity for endlessly reinventing things that came before, cluttering their devices with proprietary plugs and connectors and cards and slots.

The iPad puts into focus why Apple has been so vigilant about maintaining their strict ecosystem command and control of the iPlatform. While some points were debatable with the iPhone (and were cause for much stupidity when otherwise intelligent technical commentators made ridiculous excuses for the restrictions and limitations of the platform, trying to sell some piss water as lemonade), with the iPad it’s clear that it’s for the same reason that the console makers lock down their platform, though the lame excuses are already being doled out.

It certainly isn’t to benefit the consumer. We had shades of this years back as Microsoft built out the trusted-computing platform, and one feared possibility at the time was that we'd end up with a dominant platform where software had to pay a fee and pass a gatekeep ("DENIED! Competes with Excel!"). Thankfully the massive chill was unfounded, or the objection was so loud that it discouraged that initiative.

Alas, the iPad is real. The faithful are pouring forth to tell us that it’s the end of netbooks. It’s the end of eReaders. It’s the future of computing! While usually it’s the Mac faithful that preach the message, in this case it’s the tech media that is pouring on the unabashed praise with no critical perspective. They’re all afraid of posting something negative, only to be mocked when Apple inevitably succeeds. They point nervously at the Slashdot summary of lore.

As Jobs creepily says during his demonstration, “It’s that easy.” Then again Jobs also told us that it will be the “best browsing experience you’ve ever had”, while showing us the device rendering websites like the NY Times, sans Flash or other accoutrements, much slower and less usably than it takes for virtually any PC, including higher resolution, vastly more capable $400 netbooks, to do the same.

“Flash is so yesterday! HTML 5 is the future!” you say. I agree with you, at least if you’re talking through a wormhole from about two or three years in the future, and with a vastly more powerful device. JavaScript and the canvas element can almost yield usable Flash similes on a PC many magnitudes more powerful than this device. Even just for video it’s grossly premature, though Apple will be overjoyed if you’re restricted to their little ghetto, paying your toll while thanking them for it.

Alas, such is the pure innovation of the sort that only Apple can bring us.

A Blog Exclusive!

As a reader-of-my-blog exclusive, I want to let you all into some secret iPad specifications I stumbled across.

http://www.tabletpc2.com/Review-HPTC1100.htm

I knew I couldn’t fool you. That’s actually a tablet PC from 6 years ago. It’s a follow-up of tablet and hybrid PCs that existed since the turn of the century (and of course supermarkets and science centers have had touch screens, including the revered multi-touch, for much longer. Am I the only one who finds that people endless pinching and unpinching on the screen look positively ridiculous?)

Of course it was far more expensive than the coming iPad. It weighed more too, and had a much shorter battery life.

Then again, it was probably faster than the iPad. It was completely open and could run hundreds of thousands of very rich applications (applications not gimped to a smartphone). It also had lots of standard expansion ports and capabilities.

The market generally didn’t care for it or its ilk because the only people who really wanted a screen like that are inventory takers at Home Depot. Most of the demonstrations of it were laughable.

Despite all of Bill Gates’ prayers before he went to bed, the format floundered. They're trying once more to make it stick.

Of course that device used a screen technology that required a stylus. Apple is into the capacitive touch screen technology, so maybe that is super new and innovative for a device like this?

No, it isn’t unique. That touchscreen device came out before Apple’s very first iPod (you know the one. It was the “me too” music player that saved Apple from dying at the hands of a failed computer business — though some gimmickry with the iMac kept it on life-support for a while longer — which they’ve since rebirthed by rebadging PC components, amazingly fooling the faithful into believing that these somehow came from the premium bin).

Where is the Innovation?

The iPad isn't innovative. Everything it does has been done many times before. Claiming that its restrictions are a benefit are like saying North Korea has a more refined sense of freedom.

Executing well is not innovation. Apple executes very well indeed, and they put incredible care and attention into their products. That is hugely laudable and worthwhile, but it isn’t innovative.

As to predictions that the iPad will take over the eReader market, while it may come to pass it ignores precedent.

People don’t read books on LCD screens for the simple reason that people couldn’t accept that as a substitute for print when they wanted print. That led to the creation and adoption of e-Ink, mirroring how actual reflective print works. I have no doubt that a lot of teary-eyed iPad adopters will tell us that it’s the cat’s meow, but we’ve been down this path many times before. Yes, even with IPS screens.

That’s Apple innovation for you. If standards change for your product, how can you fail?

Of course, all of this is for naught. Apple has a precedent of going into markets with products that cost more while doing less, and achieving remarkable success. So this is my final cry before I smile and nod politely as told about how Apple invented IPS display technology, the ARM reference processor, flash memory, and so on. The leader is truly wise and great.

In March of last year I wrote about replacing the home NAS with a custom-built Linux box.  

Almost a year in and the device has served the purpose well, providing a solid foundation for a connected home. I’ve been very satisfied with the change.

The only downsides of the unit are the higher power consumption (averaging around 38W), and the groan of the two fans inside: the power supply and chipset fans. The audible part isn’t really an issue given that it’s stashed away, but considering that a probable failure point on most new electronics is the fans, it becomes a reliability concern.

I junked a laptop because of an impossible to repair broken fan. I’ve lost several video cards for the same reason.

I can even hear the irritating whirring of my blu-ray player’s fan (do not buy the Samsung BDP1600. The thing is complete junk even without factoring in the noisy fan trying to upstage the even noisier optical unit. Speaking of junk, the Sony alpha-200 is another garbage product that made me regret ever turning my back on Canon).

As promised in the original entry, I got around to replacing the power supply with a PicoPSU 90W unit, which was basically a plug and play swap.

In my original entry I estimated a 4-8W power reduction, which turned out to be an underestimation. With this PSU the power consumption dropped a whole 10W, going down to a constant 28W (only slightly spiking under load), making me feel a little less enviro-guilt. There’s still the noisy chipset fan, but that’ll be another project.

The case was built around the expectation of a power supply fan exhausting heat, so some extra natural ventilation was required. With that the sensor readings now hover at low operating levels.

Economically this is a change that will not pay off. From NCIX the new PSU cost me $73.49 all in. Given a savings of 0.01kWh per hour, and a fully loaded electric cost around $0.16/kWh, it would take 5 years for the 10W to pay for the change.

It would be nice if all power supplies were mandated to be efficient (they aren’t for most devices because they know it plays zero part in your purchasing criteria. It’s unfortunately one of those areas where legislation is really the only effective solution), because right now inefficiency is the standard. Of course environmental choices don't always yield the expected results.

The Dream is Over...Wake Up With New Phone

In July of last year I wrote about choosing a new smartphone to replace the MotoQ that I had been using. While the MotoQ served a good tour of duty, it was seriously showing its age and was falling behind in the empowering mobile revolution.

While I’d been using variants of Windows CE since before the turn of the century, Windows Mobile was obviously lost in the wilderness. Not only was each equipped device essentially abandoned right after being released, the clearest sign that Microsoft lost the plot could be seen in PocketIE, where the preloaded bookmarks to various Microsoft Mobile pages led to 404 errors.

The team moved onto something new and shiny and had no concern at all for the existing base. Microsoft has a very short attention span to products that don't earn them Windows Office type revenue numbers, so it wasn't a surprise.

For various reasons I did not want an iPhone (we don’t need another restrictive and innovation crushing Microsoft scenario playing out, and I want to develop for the device without embracing the whole cult), despite it being the easy choice. I opined in the first entry that Android seemed to have a very bright future ahead, which is a prediction that seems quite obvious now given that it is the platform of so many incredible devices recently released or on the horizon.

The future is so bright for Android that the robots have to wear shades.

The options in Canada were (and remain) limited, so I went with an HTC Dream (G1) given that it had a keyboard and otherwise had largely the same specs as the newer HTC Magic, aside from what seemed like a minor difference in memory capacity.

 I have to confess to being disappointed with the device.

Functionally it is amazing, and even with Android 1.5 the platform is simply brilliant. When everything operates correctly I am over the moon with the device.

The problem is that everything didn’t operate correctly. For whatever reason the device seems to be horrendously overloaded, so even with virtually no apps installed and nothing beyond the base system running, most actions are plagued by obnoxious pauses, even on a fresh start-up.

I hate pauses.

I stopped using brilliant apps like Weatherbug because they seemed to make the situation worse.

Alas, my long term plan was always that I would buy one of the newer, faster phones when they came to market, while using the starter device for development purposes until that time. If an unlocked Nexus One or Droid/Milestone worked on Rogers’ wireless band, I’d grab one of those when it was a possibility.

Nonetheless, I was pleasantly surprised recently to find that Rogers was offering all HTC Dream owners a free HTC Magic for $0, with the caveat that your term length pushes out. Given that Dream owners can only possibly be 6 or 7 months into their term, that isn’t that tough of a demand. I am on a very reasonable family plan that allows me 5GB / month (which I seldom use more than 1% of), so I feel fairly future-proofed with that foundation and for me it was all win.

So the next day a Magic arrived in the mail and moments later I was up and running with it. With the SIM card removed my existing Dream still works on wifi, where it can browse the web and play media and respond to emails and take pictures, and I can of course put another card in it and continue using it online. I’ll likely install Cyanogen on it now.

Quite pleased about that.

The most shocking thing, though, is that this Magic is much more responsive. It has the same processor as the Dream, so that doesn’t explain the difference. If I had to guess, I’d point to RAM, which on this device comes in at 288MB, compared to the 192MB in the Dream. For comparison both the Droid and the iPhone 3GS feature 256MB of RAM.

The extra headroom over the base OS seems to make all the difference in the world. On the Magic I can see that the free memory is usually less than 90MB, even on a fresh start-up, which notably would put it over the limits of the Dream.

HTC and Rogers claim that they’ll release Android 2.1 for this device in the near future, which makes me especially pleased.

Great move, Rogers. The new HTC Sense update and free month of data is icing on the cupcake.

Firefox 3.6 Released – Web Worker Performance Remains the Same

Back in June I wrote about Web Workers, a fantastic new method to move processing out of the UI thread. To support the entry I posted a variation of the SunSpider benchmark I named Moonbat.

Safari kicked Firefox around in this benchmark. I just tried it with the just released 3.6, and it doesn’t look like much has changed: FF 3.6 does 10 iterations with 4 threads in ~11 seconds, Chrome does it in 2.6 seconds, while Safari leads the pack at 2.3 seconds.

Alas, web worker performance isn’t a critical factor in choosing a browser (my favourite browser remains Firefox), but it would be nice to see it moving in the right direction.

Celebrating My First Home High Speed Overage

Got the cable bill — a bill that pushes into the $250 range per month these days — to find a surprising $11.25 "internet overage fee". Apparently I used 67.5GB last month, while my limit is 60GB. The Steam sales, several purchased HD movies and a couple of on-demand games for the kids on the 360, added to the normal internet usage apparently really added up to a very atypically throughput-intensive month.

I'm not going to cry many tears about it, even though I do think $1.50 a GB is a bit absurd (in an average month I doubt I use 10GB, so now I almost feel obligated to max it out), given that I think by usage pricing would lead to a far better, more open, more honest system for everyone.

On Slashdot, news of this was submitted and accepted as “London Stock Exchange Rejects .NET for Open Source”, with the statement that “The switch is a pretty savage indictment of the costs of a complex .NET system.” The Digg submitter went with the title “London Stock Exchange dumps Windows for Linux” — which they took directly from the linked article — with the description “Fed up with Windows' failures, one of world's major stock exchanges is joining many others in making the switch to reliable Linux“.

The heavily-linked columnist in both cases is a guy who has been riding this "LSE dumps Windows!" horse for a while now. It has certainly provided him with lots of quality incoming visitors, drawing in those looking for validation, and angry hordes baited by his trolls. Encouraged on he seems to be accelerating the unsubstantiated hyperbole.

Let’s take a moment to go back in history for a bit.

Microsoft made a Really Big Deal about the LSE originally switching to this custom, Accenture-built, SQL Server 2000/.NET-based solution. This was sort of Microsoft’s coming out party, in a way saying “look, we’re big boys too! No more pull-ups for us”.

When the LSE had a very public failure on one of the biggest trading days in history, the detractors were screaming “I told you this would happen!” until their throats were sore, despite the cause of the failure never having been publicly detailed.

Failures have happened on every platform, most commonly as a result of application failure. To automatically assume the worst of a novel solution simply because it is atypical is the thought process of annoying simpletons, anxiously and eagerly hoping to try to pin any fault on anything that doesn’t fit their vanilla perspective of how things are supposed to work.

“The software must have failed because Bob went with HP instead of IBM!”

“The payroll system miscalculated. It must be because they moved it to Linux from Solaris!”

So what really happened with the LSE?

Accenture built a very expensive, custom solution for the LSE, purportedly costing somewhere in the neighborhood of $65 million dollars. To operate this custom in-house (albeit designed by Accenture out-of-house) system the LSE built up a considerable technology workforce.

The worldwide recession hits and the LSE takes some financial hits. A swarming mass of competitors in Europe, many running off-the-shelf, superior systems that they’re paying less for, go live.

A new CEO takes over and immediately starts to swing the axe. He makes specific comments in the press about the high IT costs of the organization; both of the large number of technology workers in London, and the continuing significant payments to Accenture to finesse the TradElect platform.

He undoubtedly observed that all of this custom work hasn’t gained them any unique advantage in the relatively commodity task that they performed. In some ways it’s like writing and maintaining your own in-house operating system – if it doesn’t give you some advantage, and actually puts you behind as everyone else pools resources on a solution, then why would you do that?

So they go on the market for a replacement, eventually deciding to go with the product of a Sri Lankan company. The price is right, and the lure of low-priced Sri Lanken talent is enticing enough that they buy the whole company.

In the end they have switched from an extravagant, custom-developed solution built by a notoriously expensive consulting company, and a workforce of expensive talent in the West, to a basically off-the-shelf solution that has been subsidized to its current state by other organizations, in the process getting some low-paid talent in South-East Asia.

The new product isn’t open-source, and it runs on a range of non-open-source UNIX platforms. The Oracle database system it uses is the antithesis of open-source.

What about this story has anything to do with open source?

The LSE doesn’t think it has anything to do with open source, or even necessarily Linux.

Where this story gets legs among the zealots is that the LSE plans to deploy the new product on Linux, given that the underlying operating system in many cases has been commoditized. Who wouldn’t?

Zealots cheering on trolling columnists like Steven J. Vaughan-Nichols do the profession harm. Now this nonsense is going to be parroted by people who don’t know better, making them look worse for it, for years to come.

I love Linux. I love open source. And you know I also am even quite fond of .NET and SQL Server. I detest fanatics, fanboys, and hysterical columnist that distort or invent reality to get themselves hits.

The game is a stunning technical achievement, and it is amazing what they squeeze out of the almost half-decade-old era hardware of the device.

What makes the game spectacular isn't specific to some mystical art of console gaming, however, but is simply great software design and execution. While many in "mainstream" development (business processes, websites, etc) consider game development foreign to what they do, it's all just algorithms and code: One person does financial projections and another does particle effects, differing less than many imagine

The game was so excellent that I decided that I'd try to find who the talent behind it was, my quest thwarted because this game, like many recent releases by large game studios, has an apparently anonymous development team. My search for credits has yielded only a listing of artists responsible for the songs in the game.

It would be great if there was an industry credits site similar to imdb, where you could find out the people responsible for games and applications: I can easily discover who did the foley mixer work on Joe Dirt, but can't discover the team behind Dirt 2 after a lot of digging. Maybe I'll make one.

I did find a "studio tour" video, in which the only person deemed worthy of naming was the "Senior Executive Producer". Maybe if I finish the game I'll discover who did the magic to make this game happen. I'd like to read how these guys operate and do what they do, because they are clearly successful at their craft, and I imagine they'd have interesting things to say.

Are they just cogs in the gears of CodeMasters? Crank it and a great game pops out, quality determined only by your Senior Executive Director in charge of North American Marketing?

Are we past the era of superstars like John Carmack? Are we into an era where everyone is nameless "team players"...unless of course they're in senior management/marketing, in which case their contributions and name will be heralded everywhere.

As a mostly unrelated aside, the "all contributors are equal, but some are more equal than others" policy reminds me of a conversation I once had with a peer, during which they bragged about how their workplace followed a policy that strongly discouraged fancy-pants work titles (e.g. no lead architect, senior developer, etc). My appreciation for that egalitarian workplace dissolved, however, when I learned that the speaker had granted themselves a lofty, important sounding title, as did the other senior members, and they failed to see the hypocrisy in it.

Sidenote: The website for the game is mildly offensive to Canadians. They decided that the landing page would require you to first select a country, with the options being the Netherlands, Belgium, Germany, Spain, the UK, Italy, France, and the USA.

As a Canadian I'm left not knowing which I'm supposed to pick. Maybe I'm supposed to pick the UK to get words with superfluous 'u' still intact. Maybe I'm supposed to pick the US just because of proximity? Two of those countries (the Netherlands and Belgium) are significantly smaller than Canada, so I have to guess it's a hybrid language/proximity thing.

Lots of websites pull this cheap navigation technique and it's lame. Often a US flag really means "English", other times a Union Jack means English. Nationality and language aren't the same thing, so it's a lazy tactic, made especially confusing when both appear together.

Then again, if I recall correctly the old Codemasters site worked by having you select on a world map, where all of North America was labeled "United States of America". Us Canadians get accustomed to it.

Embarrassingly it got picked up by Slashdot and was put up as the awkward counterpoint to an earlier article by Clay Shirkey, in which the author competently, with research and everything, argued against the concept.

Arguments were had and forgotten, and we all moved on. Lots of great content disappeared from the web, advertisers got more and more desperate, often malicious, and a long and horrible content drought ensued.

Eventually Google came onto the scene, bringing advertising to the little guy, and the content market was reborn on the back of Adsense.

Clay was held up as the victor, or more correctly was considered the only contender, and has been used for citations countless times since, unquestionably proving the non-viability of any small-transaction system. I came across just such an article moments ago, as I do several times a month.

Was Clay right? Are people really psychologically unable to handle small payments? Is the idea of small-cost subscription packages for websites an unworkable model, or do people just say that because they like imagining that they're having a free lunch?

Lots of people seem to think so.

Then again, lots of people thought the Earth was flat, terrible things would happen when we passed the speed of sound, it would be impossible to full-text index the internet or to search it economically, and so on.

Take a moment to consider that there have been 2 billion iPhone apps sold, with a current average price hovering around a dollar. There are predictions that the average price will rise in the coming years, to a magnificent $2.37 by 2013.

This is for generally small, disposable apps that often do little, but because the cost is small and the transaction surprisingly well lubricated by the iTunes process, a lot of people just click "buy" and enjoy the experience. Look at Atwood himself — using him as an example given that he comes into play in the article I referenced earlier — who said:

My total bill for 3 screens worth of great iPhone software applications? About fifty bucks. I've paid more than that for Xbox 360 games I ended up playing for a total of maybe three hours! About half of the apps were free, and the rest were a few bucks. I think the most I paid was $9.99, and that was for an entire library. What's revolutionary here isn't just the development ecosystem, but the economics that support it, too. At these crazy low prices, why not fill your phone with cool and useful apps? You might wonder if developers can really make a living selling apps that only cost 99 cents. Sure you can, if you sell hundreds of thousands of copies:

That's impossible! Clay Shirkey says so! Or at least that's what people often interpret him as saying.

iTunes doesn't just service the iPhone app market. Aside from its start selling music (usually sold by the track), countless people are avoiding advertisers and buying network television content via the services. Marginally small amounts, but low transaction costs, technology, and the ease of purchase makes it a viable market.

iTunes doesn't own this nickle-and-dime market, though.

I enjoy the occasional bout of gaming on the xbox 360, and it takes every opportunity it can to try to get me to partake of tiny little purchases, some as low as $0.40. Want a game-specific theme? Want some bling for your avatar? Want that car before you've "earned" it? Want this amateur community game? Come on, it's just a couple of points from some nebulous points pool that you can spend simply by pressing A a couple of times, and when it empties you just add a bunch more.

Of course, all of this naturally leads to the semantics of what a "micropayment" is, and inevitably people will argue that a micropayment must be paying sub-pennies by the page, or the KB, or the image, or whatever. That isn't the spirit of it at all, however, and instead the origins of micropayments were easy to accommodate payments of amounts that were traditionally uneconomical to gather. In my mind iTunes absolutely supports micropayments the theory, because prior to that service it simply wasn't possible to sell applications for $1 or less. There was a fairly lofty minimum threshold below which it wasn't worth your time.

That is no longer the case within certain spaces.

Which again brings us to the possibility of micropayments for websites that hold actual value: People need to quit pretending that micropayments are some disproven, unworkable theory. There are a lot of us who simply abhor advertisement or economic coersion in all its forms (as you saw in the prior entry, the moment someone adds commission links to what is purportedly a subjective review, my opinion of their credibility drops precipitously, and I'm suddenly wondering if they actually liked the product, or if they just fumbled around for something and pushed the first thing they found, using it to effectively tax the readership indirectly and terribly inefficiently), and who aren't cheap bastards. If you have a site of value, and if there were a trustworthy, credible and lubricated system like iTunes for Websites, the idea could have legs.

In an earlier, more naïve era of my career, I had three software development “heroes”: Jamie Zawinski , John Carmack and Joel Spolsky.

Jamie Zawinski grabbed my attention because he was at the center of the internet revolution, right in the bowels of Netscape from the outset, and had established a pattern of posting surprisingly pragmatic comments that defied convention.

It was extraordinary to read someone openly critical of their own organization, especially without it being retracted or redacted the sobered-up or calmed-down next day, and where the author didn’t hide behind anonymity.

Jamie let us commoners see the sometimes ugly mechanics behind the curtain. He also revealed a very interesting workplace that was foreign to the gray-walled cube world that most of us lived in.

This was at a time when Microsoft really had almost unthinkable dominance over the industry, so to hear Jamie discuss the travails of cross-platform development was like going out of bounds at a tourist resort. Seeing what the brochure didn’t show you.

An SGI box? IRIX? How exotic!

Another of the kings, John Carmack, was blessed with “F-you money” from the incredible success of some of his earlier projects, along with a proven abundance of intelligence and skills, so he too had the luxury of entertaining a surprisingly realistic and pragmatic perspective. He was a principal driver of the evolution of GPUs and gaming hardware, and you can owe thanks for some of the capabilities of your console or dual-GPU rig to his desire to make shooting things in first-person shooters hyper realistic.

Carmack was also one of the original “bloggers”, regularly posting lengthy “blog entries” by repurposing the UNIX finger facilities.

Joel Spolsky is a bit of the odd-man out in this trio. While he did have the requisite first-initial, he wasn’t known for extraordinary technical acumen (beyond having worked on Excel in some earlier life), but hear me out, please.

Joel ascended to Kingship – at least in my personal hierarchy of industry royalty – just after the dot com crash, when CMM factory-line initiatives started to become the mythical silver-bullet: This was an era awash with articles gushing about the amazing adoption of CMM5 among offshoring firms.

Many organizations were striving to reduce software development to an assembly line of easily interchangeable cogs, both of code and people, achieving a utopia where the process would become perfectly predictable and repeatable if only you filed enough forms.

Joel spoke up for developers when most were absurdly blaming the .COM collapse on dual-monitors, Aeron chairs, and inflated developer egos, as if taking developers down a notch and having them sit on a cold rock would have made selling kitty litter online a good idea.

He was essentially an enlightened pointy-hair blogger, and while I wouldn’t look to his blog for technical advice (Wasabi!), he really understood developers and the process of getting software built. And he was willing to risk his own nest egg and put his money where his mouth is, having since built a reasonably successful company in Manhattan that most of us should be envious of.

Unbound by Convention

What made these three really stand apart in a sea of cheap advice-givers and pundits was that none of them were writing to get a job or even necessarily to keep one. Joel made his own bank while the other two were of such technical esteem that they had little to worry about professionally regardless of what they might say.

They weren’t coerced into railing off the latest buzzwords and best practices, deferring to the latest silver-bullet best practice pattern-based UML diagramming system 3NF data warehousing factory built on a n-tier service-oriented, aspect-oriented, polymorphic framework so that they can get the approving nods from the nervous masses and clueless PHBs.

They didn’t worry about offending a boss who held some sacred cow that if only you did it the way she read on some best practices blog, everything would be fabulous, at least until that initiative fails and you move on to the next cure-all.

The three kings were just saying it like they saw it, which was and still is rare.

Eventually Joel ran out of things to talk about and switched his blog to mechanically regurgitated repeats; Carmack got lost endlessly perfecting the noble quest of simulating head shots when he wasn’t reaching for the stars; and Zawinski decided to engage in endless battles with the city of San Francisco over his money sink of a late-night dance club (if you read his blog about DNA Lounge early on, you could almost smell the contractors taking this dotCOMinaire for a ride).

Maligning Metaphors

I was delighted to see Joel return from effective blogging retirement, and my enthusiasm exploding when I saw that it was a post about Zawinski!

A royal duet!

Okay really Joel was selling a book – like his partner-in-crime Atwood, he seems to be motivated to post by Amazon Affiliate bucks these days, credibility undermined by that kickback – however he chose the Zawinski chapter as his pitch, talking admiringly about how practical and “get ‘er done” (paraphrased) jwz was about his craft, doing so with a present tense that betrays a certain blissful ignorance about Jamie’s career path since.

Joel labeled Jamie the “Duct Tape Programmer”, which was a description that Jamie took as “damning with faint praise”. Joel has long been against architectural astronauts, so he seemed to excitedly hold up Jamie as the successful counterpoint.

Perhaps “duct tape” is a bit of a metaphorical overreach, causing many to envision some Tim the Toolman ‘Ar ar ar’ hack.  Pragmatic or practical probably would have been more accurate, though it would have made for a less contentious entry.

Never mind that Jamie worked within extremely tight timelines, using technology far less advanced than what we have now.

Joel’s entry raged across the social news sites, with the regular suspects popping out of the woodwork to declare it a grievous offense to all that is all that is good in the world of software development. Lots of blog replies parroting the standard best practices appeared, their authors clearly hoping that their boss and any future employers will see how studious and diligent of worker bees they are.

Who Decides on Best Practices?

The people who are the most certain about software development patterns, practices, and technologies are usually the people who have the least reason to have such certainty.

I’m going to be a bit trollish while I go to the extreme and say that many of the oft-quoted leaders of the field, responsible for much of the unquestioned wisdom-bites, have little to demonstrate why they’re in a position to preach.

The revered Fred Brooks, author of the Mythical Man-Month, came into a position of considerable influence largely by leading a project that was by most accounts a massive failure. That would be fine if there was but one way to fail and he found it for the benefit of all, but there are an infinite number of ways that a project can fail.

Of course you must learn from failures, but my experience has been that the explanations for failure are often a worse-than-useless distractionary tactic: When a team technically fails to accomplish what they set out to do, expect the post-mortem to be full of nonsensical misdirection about how everyone and everything else is to blame.

How many post-mortems include the statement “I grossly overestimated my own capabilities”? I suspect few.

Steve McConnell is another well-known author in the field, revered for his software development books (though many strangely overlook After the Gold Rush, where McConnell knee-jerk responds to the dot COM collapse by advocating an ill-considered licensing system for software developers), but his professional experience seems to be limited to working on TrueType at Microsoft, and some nebulous software development at Boeing, after which he took on the role of telling the world how software should be developed. Now he consults with pointy-hair bosses to unknown outcomes.

Don’t get me wrong, I have both of them in the bookshelf behind me, and read and greatly enjoy their opinions (Brooks’ observation about second systems is more profound and important, I think, that the over-quoted man-month snippet), but really, let’s keep some perspective and stop using it like they’re the incontestable word of truth.

I read them critically and with an open mind, not taking it as the voice from an all knowing deity, but instead the perspectives of a couple of guys drawing from their experiences.

Of course, the esteemed Fred Brooks and Steve McConnell exist in a realm far above most silver-bullet cheerleaders in our industry. These successful authors actually dirtied their hands with actual software development, refactoring their opinions over the years into refined perspectives. I select them merely as “absolute best case” examples.

More commonly the people who most ardently advocate certain practices and approaches have achieved little, usually having nothing to back their conviction but self-interest and a desire to look like they know what they’re talking about, having associated their id with “correct” approaches.

They just clutch onto whatever they hear is proper and start repeating it like a novelty birthday card repeatedly opened. They’ll tell you that should develop like an ecommerce site, despite not being an ecommerce site; like you’re NASA, despite not being NASA; like you make the software for a pacemaker, despite actually making an ebay auction sniping tool.

Why do I hear the word “pattern” from mediocre or non-developers more than I hear about it from experienced developers, always stated as some sort of conclusive statement?

Why do we accept that a chimp-level of software development skill is acceptable for maintenance programmers, capable of understanding only the most infantile code that is carefully decorated with “Coding for Dummies” comments?

Why is “We should use UML” the desperate last-ditch fallback of failing teams everywhere?

Unit testing, or the more early-loaded TDD, can be great, but it isn’t a panacea and is an extremely poor substitute for actual craftsmanship.

Moving beyond the non-developers giving their unwanted opinion on how software should be built, the other class of destructive noise is the advocacy of silver-bullet methodologies during the honeymoon period.

Great, you built a sample app on RoR/Haskell/Scheme/python or whatever else is the cure-all platform that profoundly changed your world view.

Here’s a nickel. Go build a real app then tell us how revolutionary it is now. I don’t discount the advantages, but advocacy based upon toying around is of little use to real projects. Extrapolating it up is foolhardy.

Oh look, another guy telling us how switching to the Dvorak keyboard layout made him regular and makes his code smell like cinnamon. Here’s someone saying that they slept 4 hours a night by taking 20 minute catnaps, proven out over their two day sample period. This guy says that having a 400x200 single-app screen on a netbook made him a perfectly focused coder. Here’s a dieter who is certain that they’re onto an incredible, beats-the-laws-of-thermodynamics diet now that they’ve followed it for a whole six hours.

The Emperor Has No Clothes!

The fairy tale “The Emperor’s New Clothes” has significant relevance to the software development field. To quote the plot summary from Wikipedia

An emperor of a prosperous city who cares more about clothes than military pursuits or entertainment hires two swindlers who promise him the finest suit of clothes from the most beautiful cloth. This cloth, they tell him, is invisible to anyone who was either stupid or unfit for his position. The Emperor cannot see the (non-existent) cloth, but pretends that he can for fear of appearing stupid; his ministers do the same. When the swindlers report that the suit is finished, they dress him in mime. The Emperor then goes on a procession through the capital showing off his new "clothes". During the course of the procession, a small child cries out, "the emperor is naked!" The crowd realizes the child is telling the truth. The Emperor, however, holds his head high and continues the procession.

Too often the software development industry suffers for lack of someone crying out. We often just go along with it, listening to the declarations of non-developers and maintenance programmers as if they speak unquestionable truth, all while discarding any counterexamples as mere aberration (“Well not every team has superstars you know! We aren’t all John Carmack!”).

Everyone withholds contrary, pragmatic “Well it isn’t quite so cut and dry…” opinion lest they look like a “hack” to a present or future employer or nervous, cargo-cult embracing peers, smiling politely while the never-coded, overconfident guy acronym drops about things he doesn’t understand in the daily stand-up.

The more you know, the more you’ve experienced, the less obvious the world becomes, and the more hesitation before offering up opinions. The less ease there is to criticize the path of others when it has yielded obvious success.

Opinions come quickly to experts and morons. Few of us are experts.

Jamie Zawinski had unique conditions under which he unquestionably succeeded. Many, with the seeming clarity of hindsight and the ability to project whatever imaginary timeline one desires, will look back and comment on how the codebase got rewritten, purportedly twice, and how eventually the product was squashed, stomped out of relevance by Microsoft (before being reborn as the game-changing Firefox), using that to draw the absurd conclusion that if it were produced “properly” at the outset, today we’d all be using Netscape 9. Then again, maybe it would have followed the disastrous arc of Chandler.

The road that leads to most successful apps is often an ugly, brutish affair filled with compromise and folly, risk-taking, detours-followed, and shortcuts pursued. That isn't to justify them, or to diminish alternative approaches, but we should always keep our minds open, being less quick to defensively guard whatever we're selling as the cure-all this week.

A Call Out For Success Stories

What I’d like to read more about are the success stories, and less about the professional pundits telling everyone how it ought to be done.

Of course here’s where we get into a common paradox that exists in most industries: The successes are usually off enjoying their success and wealth, less inclined to toil away their days writing blog entries extolling their "dart toss" method of architecture. We’re left with the conversation being dominated by the people who don’t actually make software at all, telling us how grand they could make software, if they ever actually did, by following their sure-win magic formula. The conversation boards are overrun with the people who actually have so little to do that they spend their time describing the ideal way that everyone else should write software.

Parallels can be drawn with the financial world, where the snake-oil salesmen pitching the ways to make money are usually doing so because their only way of making money is pitching how to make money (Want to know the secret to making big money? Send me $5, and I’ll tell you that it’s to get people to send you $5 to learn the secret of making big money). The guys actually making the money are off making the money.

This brings us full circle back to Joel’s recommendation of the book. A book that serves as one of the few opportunities we have to really read how projects succeeded, straight from the source.

It’s good if only to let the successes have a voice in the conversation.

One improvement that has gone largely unnoticed is that 3.0 brings Web Workers to Chrome (of course it already had it in Google Gears fashion, however now it's in the standard, cross-browser form), so you can run the web worker benchmark I previously put up and see Chrome 3 blazing a path. I'll update the comparison charts to include this browser shortly.

This means that web apps using Web Workers for enhanced performance on multi-core machines, or more importantly for better GUI responsiveness, now function on Firefox, Safari, and Chrome, leaving just Internet Explorer and Opera as the hold outs.

Quiet Upgrades

I really appreciate how quietly Google updates their browser when you already have it installed. No user-disrupting fanfare or attempts to use the opportunity to push new toolbars or side products on you (which, I suspect, drives the frequent updates to many products. I uninstalled NoScript way back after having the update notice seemingly every other day for what seemed to be the most trivial extension possible, each update bringing you to the advertisement riddled website where it incidentally told you that they made some sort of irrelevant non-update. That was before the notorious battle between NoScript and Adblock, where the former again was motivated primarily by ad impressions). Of course this model gets much more debatable once changes include major functionality changes, or breaking changes, however as is it's a seamless way of delivering updates, where each version simply improves and adds to what is there.

However you feel about Stallman, he should be thanked for eliciting this position declaration and finally helping to legitimize Mono. .NET / C# developers should rejoice at this development.

The first Android-based phones available in Canada are the HTC Dream and Magic, both recently introduced by Rogers Canada. The Dream is better known in the US as the G1 under its T-Mobile guise, where it has been available since October of last year. The Magic is a newer phone, known in the US as the G2, offering a bit more internal memory and flash storage, and a sleeker, lighter design made possible because it doesn’t have the physical keyboard that the Dream sports.

The phones are a bit late to the moose party, but at least they finally made it. We tend to get these things a little slower here in the Great White North, as we’re a lucrative enough market that these companies want to pursue some sort of strategy to aggressively monetize us, yet we’re a small enough market that they’re in absolutely no rush to do so. We’re often stuck in a bit of limbo, embargoed out of play until some grand strategy is developed.

Alas, the Android finally got their landed immigrant papers and moved in.

Apple Goes and Spoils the Party

The Android launch has been rumored to be a big disappointment for Rogers, and they've already resorted to slashing prices. The phones aren’t nearly as sexy as the iPhone, and their release in Canada came right as anticipation for the 3GS started swelling, exploding into a tumultuous wave of gooey pro-Apple fanaticism.

The poor reception isn't all that surprising. For many end-users the iPhone really is a better product, an assessment made even truer with the release of the 3GS. It was a close race with the 3G, but there seems to be a clear winner now that the 3GS is available.

The Dream, its 528Mhz processor purportedly underclocked to 384Mhz* — presumably for thermal reasons — fell behind the iPhone 3G, and is easily kicked around by the 3GS. The HTC’s design is comparably stodgy, and the app support isn’t as robust or polished (especially here where inexplicably there is no access to the commercial Android market apps. Instead we’re restricted to the free software). The hardware saw no upgrades at all over its year-old G1, so there’s still a restrictively small amount of on-phone storage, and yet Android 1.5 — the “cupcake” version installed on the Rogers phones — has yet to support installing to the SD card while they iron out the inevitably futile DRM efforts, so that remains a very real concern.

* - Note that the statement about the underclocked processor is based on information I discovered online while researching the phone, repeated across many sites, not that repeated assertion gives it any further truth. However I can say that when playing around in the shell on the phone with the excellent and free connectbot app, the scaling_max_freq is set to 528000, implying that the Rogers build of the phone will clock all the way up to the limits of the processor.

Rogers made the situation for the Dream worse by crippling the one meager advantage it holds over the iPhone (in my opinion a huge advantage, but subjective opinons differ), which is that it has a real keyboard. For reasons that are hard to understand — I have a strong suspicion that it relates to bilingualism, which is the reason why we also don't have voice search, a feature really sorely missed given how valuable it is in the latest Google maps update — the Rogers build of the Dream variant of the phone has no onscreen keyboard, forcing you to pull a Transformers routine on the phone every single time you need to enter so much as a letter. This turns the physical keyboard into as much a liability as an asset, and if it ended at that I would recommend that people go for the Magic and steer clear of the Dream. Not only does it make it a pain to do quick interactions with the device, but I'm sure the endless opening and closing (often in situations where you shouldn't be trying to open and close a device) isn't optimal for the long term reliability of the unit.

Thankfully this terribly stupid decision can be easily overcome, and you can get the incredible utility of both the software keyboard and the physical keyboard, as each best fits. When you just want to punch in the start of a business name to look it up in Google Maps, there’s no need to pull out the keyboard.

The HTC Android phones came to Canada with excellent Exchange ActiveSync support, so for corporate users it supports their Exchange installations with no unnecessary middleware software or services.

Enough about the HTC phones, I recently had the desire to upgrade my smartphone, and as you can see above I firmly believe that the iPhone 3GS is the superior phone. It is impossible to argue otherwise.

The choice is clear.

So I went and bought an HTC Dream, signing up for the 6GB / month data plan. Given that I'm usually around a WiFi point, I doubt I'll ever use even a fraction of that, but I'd rather not even have to worry about it.

It's All About Potential

I simply believe that the Android platform has a very bright future ahead. That it is poised to be unstoppable. I am of the belief that gaining the developer knowledge and comfort with the product now beats using the faster, slicker (and definitely cooler) iPhone 3GS. And I like having a real keyboard.

I am convinced that in the next two years we'll see the release of a variety of compelling and powerful products based on the platform.

And while it isn’t an iPhone 3GS, and I wouldn’t recommend it to my sister in-law, it’s still an amazing little phone in its own right. It is far from perfect (there are Windows Mobile-like interface delays throughout, and several apps have outright crashed on me), but it’s very decent and technically amazing.

The Google integration of the phone is a nice benefit of the device. A few years back I predicted the emergence of Satvertising, in particular as satellite imagery became available on navigation devices, so it's interesting to see it become a real possibility with Google maps on the Android. It’s pretty bizarre to pull up the map while sitting on the deck out back to find that Google maps had me literally at the table that I was sitting at.

The deep Gmail integration, especially for Google Apps users (including "Standard Edition"), is excellent. The Exchange integration is close to perfect, and it certainly beats what the MotoQ, a Windows Mobile device, offers. Using both on the same device is painless, with excellent integration (e.g. calendar events drawing and saving to either) and no confusion.

Several videos I encoded with h.264 AVC (one of the best codecs around) played brilliantly, and I've read that it's comfortable with a number of other video formats. The included player is a little buggy, so occasionally things like update spinners would get orphaned on the screen, overlaying your video for the duration, but its easy to get around. The unit plays MP3s well enough, so not much notable there. On WiFi some better YouTube videos look amazing, though some arbitrary limits of the software has the quality declining to somewhere between atrocious and terrible when you fall back to 3G, not because 3G can't feed the data quickly enough, but instead because the YouTube app starts sending a "mobile" bit, telling the server to send the degraded version regardless of the capacity of your pipe.

The onboard camera stinks, but so does the onboard camera integrated into any cell phone (yeah, I'm talking to you iPhone users as well. The lowest-end P&S camera does a much better job than your phone does). It works in a pinch to document some random taserings, but I wouldn't convince myself that it replaces a real still or video camera. It does allow for some pretty cool uses like ShopSavvy. The GPS-like integration of Google Maps is very cool, and again works in a pinch (and offers some really dramatic functionality like overlaid satellite imagery, and is great when walking in a big city), but I wouldn't depend too much on it, or imagine that it replaces a dedicated unit. Given that it's downloading on an as-needed basis, a weekend trip to a conservation area outside of town had it drawing a blank, and despite having a GPS lock, it had no context to know where it was (having the constant need for a data connection). Maybe there are full-out GPS apps with real maps you can store on your SD card, making it usable for off-the-pipe usage — I've heard that BigPlanet is just such an app — however just a warning for those thinking that this phone alone replaces the need for a real GPS.

The short battery life of the device is well known, and is similar to most other high-power smartphones. I grabbed a 1400mAh battery to replace the included 1050mAh unit, and hopefully that makes it a bit less of an issue.

The wireless speeds seem decent enough, but despite signing up for a 6GB/month plan (which was a closely-held secret that you could get for $30 up to June 30^th), it goes out of its way to ensure that it pretend that I’m on dial-up. YouTube, for instance, has spectacular quality when connected to WiFi, but the moment you’re on the supposedly 7.2Mbps 3.5G network, it unnecessarily degrades to a garbage quality level (maybe there’s a “quit skimping on the bits” setting somewhere, but I’ve yet to find it, beyond ridiculous workarounds including downloading videos fully first and then viewing locally). This is one of those consequences of illusions about throughput.

Other potential services like VoIP are simply servico-non-grata (err…), with vendors like Skype providing terse statements that they’ll never support pure VoIP, careful not to open that can of worms. I’ve heard such a use of the service has even been made they-come-and-cuff-you illegal in some countries, which is extraordinary. You’ll pay for those minutes and you’ll like it, damnit!

Developers, Developers, Developers!

Alas, what really compelled me to choose the an Android-based smartphone is my software developer urges. What incredible hacker fun the platform provides!

Getting started developing for the platform is surprisingly easy (I think I’m going to post a little “first steps” post about what it’s like for a non-Java, non-Eclipse developer to get started developing for the Android), and the free development ecosystem is remarkably robust and feature rich. The normal complexities of software development present themselves as you try to build something “real” — no magic bullet there — but there isn’t the demotivating waste of time dealing with the typical nonsense when starting on a new platform. And you don't have to go and buy a Mac or agree to let Apple name your firstborn (iConsumer) just to develop for it.

And you don't have to stay in the stratosphere of the Davlik virtual machine, but instead you can create native code, or even alter that base platform itself. The whole thing is open source after all.

Excuse me, is that 600 million Dhrystones in your pocket?

It really is a powerful little computer in your pocket. In just a few more years I’m sure such a device will be central to a user’s computing reality, on a need basis wirelessly connecting to keyboards, displays, mice, and so on.

What an amazing time. To have such a remarkably powerful device with accelerometers, a compass, GPS, a very powerful little processor, 3D hardware assists, WiFi, WiFi-like cellular network speeds, all targetable by anyone using extremely rich, yet free, development environments (the Eclipse integration is spectacular)…the potential is truly limitless.

Amazing things lie ahead.

If you're running Firefox 3.5 or Safari 4 [EDIT: Or Chrome 3.0], take a look at the "benchmark"/technology demo I just put up. [Safari 4 compatibility added based upon the great comment submitted by Oliver]

It's a modified variant of the SunSpider benchmark that I've written about before (in less than flattering terms), which I heavily altered to utilize the remarkable new Web Worker functionality you can now explore in Firefox 3.5. If you're really analyzing performance, be sure to disable Firebug as it significantly impacts the results.

Web Workers, a standardization of a feature of Google Gears, are a remarkably simple method of multi-threading JavaScript, not just to get it out of the UI thread — where it can be very detrimental to the user experience as the interface freezes while a script runs — but also to scale across multiple CPUs and cores on modern PCs, which while seemingly a ridiculous notion ("but it's just JavaScript! Multithreading?") is becoming a real concern as the JavaScript engines continue to advance and the usage and scope of the language and related technologies continue to expand.

Through a simple, synchronized message passing system and a minimalist API, the Web Workers model lends itself to robust, elegant code that isn't as prone to classic multi-threading pitfalls. While not a part of the current instantiations, in a theoretical implementation there is no reason why web workers couldn't be located on entirely different machines, given that each worker is essentially an isolated runtime, sharing very little (the navigator properties and some basic security info for things like enforcing XmlHttp restrictions), communicating via serialized messages.

Understanding the Benchmark

The benchmark/technology demo is operational in Chrome, Opera, and Internet Explorer, but only if you change Web Workers to 0. In that case it is sequentially running the set of tests in the main thread, as JavaScript has traditionally been run. I didn't intend for this to be used for cross-browser comparisons, even if I resort to presenting just such a comparison at the end of this entry, and instead the focus is really on the technology, so the real "power" is seen once you start to turn up the web worker dial, all the way to 11.

Web worker multithreading isn't limited to Firefox 3.5. Oliver left a comment pointing to a Safari-ready variant he threw up, so I modified the test accordingly (the difference being that when Safari implemented it, it didn't intrinsically include JSON encoding, so your caller and receiver had to do that themselves). I didn't realize that Safari had covered this ground, though it isn't shocking given how rapidly that browser has been advancing.

With one web worker, the UI remains fully responsive to user interaction, which is an experience quite unlike what was seen at 0 (where the browser essentially locks up during the run), and the actual run itself suffers little for the isolation. On a quad-core CPU, the CPU usage during the duration of the test cycle fluctuates around approximately 25%.

At two web workers, the individual tests take slightly longer to run, however the actual completion and pace of the tests in the whole is greatly improved. Not quite a halving of the runtime, but not too far off. Two cores are saturated during the duration of the test.

At three web workers, three of the cores are filled with work, and the total elapsed time improves somewhat, albeit not by the ratio that correlates with the 50% increase in computation power.

At four web workers, we've tapped out the parallelism and despite all four cores being saturated for most of the duration, the total runtime actually suffers slightly. Going above four doesn't cost much, but it also brings no real gain (beyond possible algorithm gain isolated various parts of the application).

You can also run a mode where instead of running a modified js directly in the worker thread, the code is passed as a string parameter, eval'd into a function reference, and the function is run. There are some interesting observations to be observed by this test, such as the lack of tracemonkey loop optimizations on eval'd code (see bitwise-and in particular. It suffers dramatically when run as an eval'd function relative to running as literal JavaScript). This surprised me as the eval merely instantiates a function in the current context, but doesn't run it, yet the performance penalty remains because it was sourced from an eval.

Here are some results for 1-8 threads, running 10 cycles of each test, gathering the total elapsed time in Safari 4 and Firefox 3.5 RC2. This was run on a quad-core Q9400 machine, and of course your mileage will vary. While it is evident that Firefox 3.5 is using more of the available processing power as you move past 1 thread, with it increasing from 25%, 50%, 75%, to 100% at 1, 2, 3, and 4 threads respectively, it doesn't fully benefit from the additional resources, yielding a greatly diminished rate of return. Safari, on the other hand, already started with a considerable lead, and it pulled away with each thread up to the optimal 4, really hitting its stride.

I'll add some charts and the like to this entry later, but just thought I'd drop a line on that demo of a very promising technology that will soon see fairly robust deploymet (one huge benefit of Firefox -- shared by Chrome and Opera -- is that the uptake rate for new versions is extremely high).

Firefox 3.5 – You Should be Trying It

Have you tried out Firefox 3.5 yet? It’ll happily install side-by-side, so there’s very little risk giving it a spin.

While currently a Release Candidate, it is already supported by most popular add-ins, such as Firebug and Adblock Plus, and brings a bounty of new functionality, more comprehensive emerging standards support, all while improving performance.

Performance has never been Firefox’s strong suit. While some incorrectly believe that Firefox has just recently been overtaken by upstarts like Webkit and its bastard child Chrome – the lore being that Firefox’s creaky “old” legacy code is not as nimble as the newer projects – the truth is that Firefox has always been a performance laggard compared to all but Internet Explorer.

It has always used more memory, done things more slowly, with benchmarking and resource consumption metrics that seldom earned it praise. The history of Firefox (aka Mozilla Suite, Phoenix and Firebird) is a tortured legacy littered with complaints about slow start-up times, a slow page layout engine, massive memory consumption, and middling JavaScript performance.

These are the costs of building on the XUL abstraction, along with some expensive add-on scaffolding. The ability for an add-on to dramatically change the behaviour of the browser (all from within a world of JavaScript and chrome) isn’t just an accidental happenchance, but instead is the result of considered, pervasive design choices.

The EU Fines Opera for Their Monopolistic Browser Stranglehold

If a small footprint and snappy response were the keys to success, Opera would have taken the market by storm given that it’s been evangelizing and focusing on that pursuit, to a great degree of technical success and leadership, for well over a decade now. Yet most users care more about a slightly better bookmark manager than they do about even magnitude differences in JavaScript performance.

But performance does matter. As web applications grow richer and more elaborate, the weakest of the contenders will get pushed out of the race. Already web applications like Slashdot and Facebook are getting unpleasant with a few of the current contenders, yet some of the new features of standards like HTML 5 promise dramatic new functionality that will completely blow the top off of the domain restrictions of the browser world.

So with recent releases the Firefox crew has reduced the toll you have to pay for these architectural choices, and through a lot of hard work it’s getting to be a memory lightweight, as far as web browsers go, and performance has improved across the board, sometimes dramatically.

Your Browser Is a Remarkably Complex Software Product

And to sidetrack for a moment about a personal peeve, let me kick down the frequent whiners’ complaint that it’s “just a web browser”. Every time someone make such an ignorant, misinformed statement -- usually during some entitled complaint about resource consumption on their obsolete PC -- I think a universe blinks out of existence in some alternate reality.

There is very likely nothing that you do on your PC that is as rich and complex as the things that you’re doing right now just browsing the web. It’s isn’t “just a web browser”, it should be “I can’t believe how much is happening for me to loll around on the web for a bit!”.

Taking a couple of paragraphs to reminisce, I signed up for a Slashdot account not long after the site opened for business. Following my normal tradition at the time, I suffixed my newly created account with the two-digit year of creation, in that case 98. It was an easy way to choose account names that were less likely to be rejected as already in use.

I didn't imagine that I'd be using the same account over a decade later, so now I have a nick that makes me look like I’m pretending to be an eleven year old (if you ignore the 4-digit user ID, it is conceivable that an eleven year old might visit Slashdot).

I didn’t join the ranks of Slashdot to cheerlead Linux and open source, despite it being the overwhelming direction of the site at the time.

My intentions were much less idealistic.

I actually signed up to troll a co-worker: a Linux-loving long-haired hippy named Warren Postma. The guy was always going on about the greatness of open source software and the related initiatives, about the evils of Microsoft and closed-source, and so on, and of course he was an early fan of Slashdot.

[Over time I came to appreciate that much of what he had to say was right on the money, and in reality he was one of the best developers I’ve had the pleasure of working with.

And he wasn’t really long-haired or a hippy, though maybe my perception imagined him that way at the time, what with all the free-software rhetoric]

That was an era when being up on the latest meme was a lot more difficult, so a site like Slashdot was a welcome source, allowing you to knowingly declare that you'd already seen the dancing baby when a less-connected co-worker gets it emailed to them and comes over acting all connected-like. Though to be fair Slashdot came a little late to give a heads up of the dancing baby craze, but it was there for All Your Base Natalie Portman hot grits, though along the way to enlightment came some less appealing memes involving tubs and goats.

Slashdot no longer fills the industry-dominating position that it once did (where being "Slashdotted" was a term used in the mainstream press) — I suspect it has more to do with its competitors pandering to the common, all degrading eventually to funny pictures, "facepalms" and non-stop political posturing and claims of oppression, with sites like Digg and later Reddit escaping from the technology ghetto, while Slashdot remained chained to its original niche — but it often still has interesting and elucidating discussions throughout its comments, even if the UI of the site continues to grow worse with every passing month. Seriously, guys, the endless page thing was demonstrastably lame on DZone, so why did you think it was a good idea?

So during a moment of mental relaxation yesterday I thought I’d see what was up in the world of /., where the top story at the time detailed a congressman’s bill to “Ban ISP caps”. I found the submission hard to believe, so I RTFA to find that it had been very poorly interpreted.

What the congressman really proposed is more of a fee oversight body similar to those that many utilities have to work within.

Here in Canada the monopoly phone companies, for instance, have to apply to the CRTC every time they want to change fees (whether to increase or decrease) or service levels, justifying why they need to make those changes. Of course they’ve always been grotesquely profitable, and seldom get denied their desires (except, humorously, when it came to reducing fees to compete better against new upstarts), but just pretend that such bureaucracies make them more competitive or accountable or economical or something.

So I posted a comment saying that the summary was inaccurate. I also tossed in a little aside about utilities, drawing from the congressman and the submission’s analogy comparing ISPs with utilities, speculating that soon ISPs might just switch to a pure consumption model more analogous with the utilities that they are being compared to, saying "I don't get carte blanche from the electric company to use it all for free, complaining that "they provide 20A to the house so I should be able to use 20A around the clock for free!"."

I touched a sore spot, which I have to confess to doing intentionally. Sometimes I can’t help but troll a bit, and given that it’s a pretty common position on sites like Slashdot that it is a basic human right to torrent a pipe full around the clock, I thought I’d pick that scab a bit.

I got the typical “the bits are free and the man is just keeping us down.” variety of responses, including my favourite.

Anyone who is comparing ISPs and bandwidth to other utilities such as electricity, water, or anything tangible should simply stop posting… immediately. The Internet is not a series of tubes, your comparisons are invalid, and you are ignorant.

Please cease in the proliferation of these ludicrous analogies.

Ignoring that the utility analogy came directly from the congressman himself, this comment has made me personally invested in this spurious analogy, so now I must embrace it with vigour.

And, for that matter, the internet is effectively a series of tubes, as fun as it might have been to point and laugh at Mr. Stevens. Those tubes have a finite capacity, and if you want to push more through, you have to lay more or “wider” tubes.

Conceptually it isn’t all that different from a power or water network, where many of the same circuit principals apply.

And those “tubes” didn’t magically appear from nowhere. They aren’t public infrastructure. The internet isn’t free. Bandwidth isn’t free. Throughput isn’t free. All of it costs money to lay and service, and as demands grow, there is the capital required to increase capacity.

So let’s go back to the electricity analogy (even though I had no intention to draw it out so literally, I’m invested now).

Back to the responses again, a less stupid one said:

The total amount of power you use in a month directly affects the amount of fuel a power utility has to burn, or the amount of water you consumer affects how much water the utility has to treat. Bits on a connection aren't like that. If you don't use a bit on their fibre link to the backbone, that doesn't leave them with an extra bit, and if you use a bit, the next one is coming at the same time and same cost anyway.

Fair point. When you use electricity, the variable costs increase. When you use water they have to buy a little more chlorine and pay the electric company to run the pumps.

Yet here in Ontario most of my electricity comes from either hydro-electric (our power company was historically called “Ontario Hydro”, causing confusion for decades as children tried to understand why the power company was effectively named Ontario Water), or nuclear.

The variable costs with either are a small percentage of the total cost to operate those services, and instead the vast majority of the cost comes in the form of capital expenditure – what they had to build and maintain to handle the current load, and to handle tomorrow’s load.

Most of the “conservation” efforts – incremental and accelerated fees for heavy consumers, and rebates for power saving appliances and technologies — have been aimed at avoiding exceeding the gross capacity of the system right now, because the next step will cost billions, and it’s not to buy a few more fuel rods.

Nuclear power plants and giant dam projects don’t come cheap; nor does building a water purification system to service a city of 200,000. It’s one of those perverse situations where if demand dropped significantly, they’d actually have to increase costs.

To wring every last bit of ridiculousness out of this analogy, imagine that you live in a village of 100 people and @Tesla just twittered about a cool new tech called electricity.

Hyped about the latest fad, you get a bank loan and put up a wind generator, connecting it to anyone willing to pay an even split of the costs (a loan you amortized to get paid off just as the turbine needs replacement, plus basic maintenance and upkeep).

Your new turbine is spinning around the clock (it’s a very windy valley near the ocean), putting out a constant 100kW, and everyone is enjoying their 1kW space heater, all paying their 1/100^th cost.

Every now and then someone turns on a hairdryer, but it balances out because other people sometimes turn off their space heater, and in the end the load can be handled due to its natural distribution.

But then someone decides that they’d also like to run their 5kW back massager around the clock – you did say that electricity could be used around the clock, and the wind is free, right? — so they crank it up and your system starts to suffer brownouts and everyone suffers.

You face some tough choices: You can either put a 1kW limit on everyone (“dedicated” capacity), which would be unfortunate given that most of your customers benefit from sporadically spiking their load while they use their motorized hair curler, or you can start charging or limiting only the outliers, maybe targeting the growing trend of back massagers specifically, putting limiters on that specific use of your power.

Or you can add capacity.

So you decide to do the latter in the most cost-effective way, arranging a power sharing agreement with a neighbouring village: When you have left over capacity you send it to them, and they do the same in return.

It all balances out, and everything is fine in the world again, and that one customer gets to enjoy his back massager, essentially freeloading on the standard customer load.

But then another user decides that he, too, might as well get a back massager. And then another. And then another.

The neighbouring village starts rethinking the power sharing agreement because it is no longer serving both partners equally. They demand either payment for their inconvenience, or they’re going to sever the line.

So now, despite the fact that there is “no cost” for each incremental kW, the village is faced with a massive capital (or loan servicing) cost of putting up another wind turbine.

This is grossly simplified, as analogies often are (and there are countless similar scenarios, from toll roads to bridges to movie rentals…this is like beginner economics, but so many are so blissfully ignorant of it), but there are a lot of parallels with internet backbones, where end-users are getting 10Mbps connections to a grossly oversold network at a heavily discounted price. Any delusion that there’s some 10Mbps dedicated channel available for every subscriber is absurd. Go and buy a commercial dedicated T1 at 1/6^th the speed and see how the pricing stacks up.

Consider that my little suburban town has about 50,000 high speed customers, all with 10Mbps service. If we all used our throughput, this single town would be a 400Gbps load on the network. Try laying a 400Gbps city wide network, much less province wide, much less country wide, much less continent wide, much less…that is an enormously expensive proposition.

This is the reality of the internet today. Maybe the backbones should be a public bit of infrastructure, with massive, limitless pipes from coast to coast, but that isn’t currently the case. If people want to argue for that – I think a next generation, coast-to-coast Canada backbone would be a great project for some of those stimulus billions — then they should argue for that.

Things are getting better. There are billions of dollars constantly being spent improving the backbone of the net (YouTube and Hulu today alone today account for more traffic than the entire backbone could handle in 2000), and I’m sure in a few years we’ll all being using our 1Gbps connections to play a live-streamed 1080p never-obsolete video game system.

I have no desire to defend ISPs, or to support some of the various tactics they have pursued (traffic shaping, for instance, is not at all justifiable, nor should a carrier care or restrict which sites you visit, trying to monetize the fact that you visit Hulu, or trying to extort Google. Restrict or charge based on the throughput, not specific uses of it that should be none of your concern), but some of the rhetoric on this topic is just so absurdly ridiculous, and can only be drawn from the delusions of self-serving fantasy.

The denial of the reality of overselling and fair use is how we end up with traffic shaping, or stupid constraints like cellular carriers refusing to allow uses like tethering (which is because they rely upon most of their customers not using their "alloted" throughput, and tethering would completely throw off that assumption. I'd rather they base all of it around actual usage and cease trying to indirectly massage usage through unnecessary secondary restrictions and throttles.

The standard tripe about how one’s usage somehow only ever uses excess capacity (is there some “only if available” NoQuality-of-Service bit?), where people assume that their actions somehow exist in a vacuum of inconsequence, unbound by the limits or constraints of anyone else, offends common sense.

Eight year old Victoria Stafford went missing from the nearby town of Woodstock on April 8th. She had left her school at the end of the day, starting the short walk home alone, but purportedly never made it.

Video from a high school up the street (see the enhanced version as well), discovered the next day, shows Victoria walking with a thus far unidentified female (a sketch, purportedly based upon a witness account, has been released. Given the sudden appearance of this supposed witness account two weeks later, it seems to be an attempt by the police to put a little pressure on a suspect to see if they break or make a panicky misstep).

The town in question, Woodstock, Ontario, sits on a major highway that goes from Windsor to Toronto and Montreal and beyond, with a nearby branch going to the Buffalo area. It seems noteworthy that Victoria’s public school is a very short distance from an onramp.

I find this case distressing. These things generally don’t turn out well.

Whether the abduction is real or not (there’s a general cynicism about cases like this because of vile, murderous sociopaths creating a cry wolf situation, leading many to automatically disbelieve), it usually forebodes very bad things.

Of all of the video cameras that blanket our society, the best they’ve got — at least that they’ve publicly announced — is the single grainy video from the high school (see the graphic I made of approximately the zone covered by this video, as determined by correlating landmarks with satellite imagery.) There were lots of people around, but history has shown time and time again that people are really, really terrible witnesses of anything, and that seemed to have held true in this case.

Who is the woman in the video? Apparently no one knows.

This has me thinking about pervasive, distributed monitoring. Where in time of need – like an incident like this (yes, “think of the children”) – swarm media and electronic capture can be combined to zero in on the truth. Preferrably in a way that utilizes the enormous talent and load distribution of the public.

High definition video capture is becoming dirt cheap. Solid state storage is rapidly evolving.

I’m not talking about the Big Brother 1984-style of central government monitoring, with the endless pitfalls and abuses that entails, but rather a situation where almost incidentally most everything is recorded by the public using a distributed array of devices.

It seems inevitable, for instance, that in just a few years every car on the road will have forward and rear facing cameras. The former is already in place on most police vehicles (helping to keep police in line as much as to capture public malfeasance), and the latter is making inroads on large vehicles that are prone to backing over people and things.

Video retention will inevitably come next, under the auspices of road safety (similar to how your car is ready and willing to rat on you for speeding if you get in an accident). Soon it won’t be an option, but will be a legal requirement for using the roads.

“Sorry, bub, but your car’s video system clearly shows you blowing through that red light. The other guy’s video system shows that he was in the right.”

99.999% of the time the video loops over and is erased and inconsequential and irrelevant, but every now and then it serves an important purpose in getting to the truth.

Imagine, for instance, that they could put out a call for anyone who drove down that street or that neighbourhood in the time period in question (presuming they don't already know from telemetrics via systems like OnStar, again soon to be the legal norm as every road becomes a toll road with vehicle self-reporting), from which they got a number of different time and position videos.

Videos of vehicles parked in the vicinity. Of the woman in the white coat waiting.

Add the video capture on private buildings of all sorts (homes, businesses, parking lots, etc). Merge it all together into a exhaustively documented, fact-based accounting of what happened.

Cellphones of course play a part as well.

Someone premeditating a crime would likely leave their cell phone at home (though I would wager that the woman in the white coat probably had one on her), or disable it in some way, but it would nonetheless allow the net of discovery to close in on the truth. The lack of certain data is often indicative.

Say to contact a guy in the area (via text message of course) and find out that he was in the neighbourhood taking pictures of doors, and wouldn’t you know it but he happened to have one that has a suspect matching the description walking towards the crime scene 9 minutes before the crime.

That’s if you even need to contact him. Maybe he uploaded his GPS-and-time-tagged photos to Flickr, and a police investigation entails a photo search, again drawing from the enormous distributed capture that happens every day.

It is neither a utopia nor a dystopia, but I think pervasive, distributed, decentralized recording and archiving will be a good thing for society.

Should we bother paying out big dollars for arrays of magnetic drives, or should we push the envelope and go with an array of SSDs? Should we wait a while? Will our vendors and the storage systems support this technology? Will existing products make optimal use of it? Will the SSDs burn out under our usage models?

Dell, for instance, still has nary a mention of SSDs in their servers and storage products site. Their reps still telling you that SSDs are unsupported.

Yet the evidence is obvious that in the year ahead SSDs are going to absolutely annihilate the existing field of storage vendors. Suddenly "outsiders" like Intel (now really known for storage products) and Fusion-Io are the leaders, making the existing market look like a bunch of chumps. Paying big dollars for a large array of magnetic drives seems like a choice that will certainly yield some serious buyer's regret a few months down the line.

SSDs change everything.

In a similar way, the extraordinary advances in JavaScript over the past year have completely changed the scope of what a "web application" could entail, and we as developers still haven't fully come to realize what this means. Opera, Webkit, Tracemonkey, and now, jumping to a big lead, the supercharged V8 engine of Chrome 2 (I have some serious misgivings about Google's browser given that it's the product of an ad company, but it is uncontestably becoming a real contender. I will warn that on the Chrome download page linked before this parenthetical aside, they put the agreement to send...cough...anonymous statistics in exactly the position where people have habitually learned to click to agree to the ToS), the advances have been truly spectacular.

Exciting times ahead. These innovations aren't simply evolutionary, but change the scope and rules of the game.

I have my work machines. My wife has her laptop. My kids have their PCs.

Family and friends that comes to visit connect their iPhones and laptops to our 802.11g/n.

We have shared media that we play on devices like the Xbox 360 and PCs over protocols like uPnP, DAAP and SMB. We have shared files, such as financial info, that we need to access from any machine, all while confident that they remain as secure as we want them to be.

We do VPNing from remote locations like coffee shops to access files and media. We have a shared printer.

To optimize this situation without leaving my power guzzling pig of a PC on around the clock (I’ve setup and torn down a number of home servers over the years, all the ways back to a FreeBSD-based device in the mid-90s that also doubled as a firewall), a while back I bought a Synology DS-106j NAS unit and dropped it on the network, deciding on it after looking at some Windows Home Server products, finding them overpriced and uncompelling (not least because of concern about licensing and DRM BS.)

One of the selling points of this ~$200 unit (to which you add whatever 3.5” hard drive fits your fancy) was the gigabit network port. To me that implied that it must be capable of at least keeping a 100Mbps pipe saturated with goodness.

I also liked that it gently sipped power (13W or so with the hard drive), allaying any guilty about leaving it on around the clock.

The Synology unit is fantastic in many ways.

It has a fantastic interface. It is rock solid (zero downtime with it) and is very feature rich, with several free “firmware” updates since I purchased it adding a number of new features and interface improvements. It is easy to setup and use, and to a superficial degree secure. It comes with great desktop software for common tasks like backing up (and versioning) files on your Windows machines. You can offload torrent downloading to it if you’re into that sort of thing.

But the hardware seems to be underpowered. The 200Mhz MPC8241 CPU that powers it can’t deliver on the extensive feature list, in my opinion.

The 1Gbps network port turned out to be brave talk that it couldn’t back up with action. Normal SMB/CIFS (the protocol that Windows utilizes to access network file shares) averaged about 2.5MB/second throughput with the device, which factoring in overhead is around 22Mbps, leaving plenty of headroom before it hit the limits of even a lowly 100Mbps network.

I got it up to 8MB/second or so by turning off all of the features like media sharing, and then disabling all security on the share and enabling guest access – it seems that access control security significantly adds to the computational burden – but that isn’t a viable day-to-day option so I had to revert and it was back to 2.5MB/second. FTP was a little better, but was still way below expectations given that the same hard drive got a rough 40MB/second from a direct-attached USB 2 enclosure.

Given that I am now downloading from the web at large at 1.1MB/second, this seemed silly. As we’ve started to amass monstrous AVCHD video files of a GB or more from the home video camera (the latest GB or so features my youngest son herding some chickens that had escaped from their pen at a local farmhouse), it was getting unsustainable.

Synology is now selling a purportedly faster unit, but given that I saw much lower real-world speeds than what they graph for the old unit, and their new unit really doesn’t do that much better anyways, it didn’t compel me to upgrade. Many competitive low end home NAS devices have reportedly similar performance in the wild.

And it isn’t just waiting while moving large files or having SyncBack jobs run. Navigating the music directory from a uPnP client made it obvious that the slowness impacted all activities, with basic operations having intolerable delays. Attaching to its network shares from client PCs inexplicably had multi-second waits before responding.

Then again, I’m the guy who thinks that the biggest crisis in the electronics world is the excessive lag before DVD players will eject a disc, so maybe I’m an outlier, but it seemed like a pretty big weakness of what would otherwise be a great product.

It really bothers me when units doing tens or hundreds or thousands of millions of operations per second take perversely long times to do simple things.

On the bright side, the small dimensions of the enclosure made it easy to find it a home, but it also had the S.M.A.R.T. monitoring of the hard drive complaining about borderline heat incessantly. And the printer sharing never worked properly, but given the endless variety of printers I won’t blame that on the NAS unit.

So I decided that it’s time to replace the device, with the following criteria for its successor.

• Relatively cheap, because I’ll probably just be replacing it soon enough anyways
• CIFS and FTP at a minimum
  
• uPnP would be nice. DLNA wouldn’t be bad. DAAP would be gravy
  
• Some method to backup its own files to a remote location, or at worst to an attached USB drive, with security options like encryption
• Good performance. On a switched 1Gbps network I would hope to see read throughput times of 20MB/second or better, and writes not much worse, with a very low latency on activities: Network shares should enumerate close to instantly
• Small enough that it can be physically secured such that the alarm responders would have arrived before a thief could have gotten to and removed it
• Low enough power usage that I don’t have squirrels giving me a beatdown
• A bit of fun. I look for these things to be educational and challenging to a degree. The easiest solution doesn’t provide me the entertainment value, so turnkey is a negative. At the same time, I have a very finite amount of time to deal with things like this, so it can’t require too much coddling

Local RAID isn’t important to me, as availability isn’t a primary consideration (and multiple drives go contrary to power conservation / heat reduction goals, and generally increase maintenance.) Every file that is on the unit always exists on at least one other drive on a separate system, so if the drive died I could quickly rebuild and repopulate and life would be grand again. And it is vastly more likely that two drives in one unit would catastrophically fail than it is that two drives in physically separated devices would fail, especially when the separation between can be thousands of kilometres.

Nor do I want this to function as a media station, which I consider an entirely different function, with an entirely different requirements list. I don’t need or want HDMI output, blu-ray decoding computational power, 3D offloading, a MAME box, or anything of that sort. That’s a different project, with a different set of goals.

After looking at the available market options of targeted NAS devices, and hardware solutions like Shuttle computers, I decided to have a little fun and build my own (in the minimalist stick-some-lego-like pieces together way that is "building" a PC), so I grabbed a couple of components.

• $102.30 - An Intel D945GCLF2 mini-ITX motherboard, featuring an on-board passive cooled Atom 330 dual-core processor. The Atom is not a powerhouse processor by any measure of the imagination, but for this purpose it should be fine.
  
  My only real hesitation with this board is that it uses a relatively power hungry chipset, leading to the mixed-up situation where the chipset has an active fan, yet the CPU has just a heatsink. I’d prefer a lower power board with zero moving parts, but didn’t want to risk going too low on processing power  
• $24.75 - A 2GB DDR2-667 RAM module. Even 2GB is excessive for the targeted purpose
• $55.22 - An Apex MI-008 MINI-ITX Case. The case is small enough that it’s stashable, but big enough that it won’t be a hard drive cooker. Like the motherboard, the case isn't a paragon of efficiency, instead coming with a less efficient power supply that is capable of a grossly excessive 250W, so that isn’t optimal, but I’ll worry about that at a later point
• $126.90 – Western Digital WD10EADS Caviar Green 1TB drive. It is intentionally a bit slower of a drive, putting energy efficiency (and with that reduced heat) ahead of raw speed, but it’s still very fast. This would be a great place to use an SSD drive, so maybe I'll swap to one of those as they mature and become more cost effective for this amount of storage

<$200 for the “NAS enclosure” part, then $127 for a good hard drive with room to grow. Pretty inexpensive, and would probably even make a decent internet appliance, which presumably is the intended use of the motherboard/CPU combo.

Putting it together was a snap, with dutiful oversight from my children, followed by a quick flash to the latest BIOS release.

I installed Ubuntu 8.10 Server from a USB stick (I considered installing FreeNAS, but decided that I wanted something a bit more custom), after which I discovered that the hard drive wasn’t set to bootable and the system wouldn’t boot without the USB key, so I had to boot with the key and use parted to manually set the sda device to the boot device, after which the USB key was no longer necessary.

Installation complete, I stashed it away, sans keyboard or mouse or monitor connection (which it makes no complaints about, happily existing without head, keys or mouse), and access since has been via putty. A magical source and destination of media and files, out of sight and hopefully usually out of mind.

I installed Samba (during the OS install by choosing the file server option), created the users and groups and perms, added some shares, and started accessing it from my workstation. It was all very easy and straightforward, including even setting up AppArmor to enforce mode against the Samba services, adding a small additional guarantee of security.

I started the first file move, ready to witness some low-cost awesomeness.

11MB / second to write to it, averaged out over the copy of a GB sized video file. I got about the same speed reading back from it.

Pretty good compared to 2.5MB/second (and now I had much better security granularity than I had before, on a much more versatile platform), but not quite what I was expecting. Awesomeness not witnessed.

The hard drive isn’t the fastest of the bunch, but it’s still very decent.

I’m far from an expert when it comes to Linux (or more generally Unix) systems, seemingly achieving some small level of localized expertise at intervals in my career, but then the solutions simply work and I don’t have to touch them again, so the knowledge rusts and each time I return I feel like I'm a Linux virgin (who would have ever thought that Linux and virgin would used in the same sentence?) It is similar to my proficiencies with Perl or the equally obtuse Powershell, where I put together very decent solutions, but then several months later look in disbelief that I actually wrote the code that now looks foreign.

Nonetheless, I quickly determined that the network adapter was auto-negotiating itself incorrectly to 100Mbps, and 11MB/second is about the max you could achieve over such a link.

After a quick bit of searching, I added ethtool -s eth0 speed 1000 duplex full autoneg off to my interfaces file (after validating that it worked at the command line). It was now at full 1Gbps glory.

So I performed the file test again.

47MB / second to write to it. 55MB / second to read back from it.

Sweet! That isn’t that far off of local storage, and is entirely with the acceptable zone. It was achieved on a $200 host device. File sharing is hardly the most demanding application nowadays, but I was happy to get something much more usable.

Just to take a side trail here for a moment, I just wanted to mention that Linux, and the evolution and progress of it and related projects, really is amazing. It is astounding that such a solution evolved the way it did. Even people who don’t think they benefit from it regularly use devices run on the OS. The amount of functionality and technology available, including the source so you can roll your own, truly is remarkable. Linux didn’t start this movement, and projects like FreeBSD, sendmail and prior initiatives blazed a successful trail before Linux really took off, but it is the most evident element of that development model.

Back to setting up the homebrew NAS, virtually every step of the way I had to consult the wizard, Mr. Google, but at least the answer was usually only a page 1 or 2 hit away (though Google Groups has taken a perilous dive into close-to-uselessness, and proved of little use.)

To this point I still haven’t had luck finding a good uPnP server for the device. uShare simply didn’t work. Mediatomb inexplicably doesn’t work with the xbox360, and even if it did the fact that its anonymous web user interface has the default behaviour of exposing your entire file system makes me question the wisdom of the developers.

Various other projects didn’t fit the bill or didn’t work for me, and most seemed to have been abandoned.

I thought I’d try the MythTV backend, as it is purported to double as an uPnP AV mediaserver, to find that it bizarrely needs an X server running and an X client on the other end just to perform the setup. I decided to purge it from the system to find it wouldn’t uninstall due to /dev/.static/dev being mounted read only. I found a workaround involving stopping and starting udev, which remounted it as read/write, but at that point the damage was apparently done, and it refused to gracefully leave the system.

So I went MythTV huntin’, committing a moronic error in the process (it tried hard to warn me, but I persisted in my foolishness.) I’ll let the putty log speak to it.

Installing everything once was so much fun that I decided to do it again!

So I reinstalled again. The second time around it was a very quick process (even with a nuked /usr/bin I still had a chance to backup customized config files to a client box before reinstalling, so I really didn’t lose much.)

Power wise, I’ve gone from ~13W for the DS-106j, to 35W to 40W for the new device (measured with a Kill-A-Watt. Handy device.) This represents a pretty big move in the wrong direction, but it wasn’t unexpected. 4-8W or so could likely be saved going to a more efficient, fanless power supply. That’ll be a future experiment. The Caviar Green, at 3-4W, isn’t much more power hungry than an SSD, so that wouldn’t make a huge difference. The motherboard and its chipset is the real piggy of this farmyard.

Alas, in return for the extra power consumption I now have a vastly more interesting network device, serving files at a much more usable pace.

So my first adventure is beginning the work on a .NET-based uPnP server, ensuring that it works on Mono along the way. Thus far it has been nothing but remarkable success, and it still astounds me seeing fairly complex .NET applications running well on a Linux box. The thing really, really works!