In March of last year I wrote about replacing the home NAS with a custom-built Linux box.
Almost a year in and the device has served the purpose well, providing a solid foundation for a connected home. I’ve been very satisfied with the change.
The only downsides of the unit are the higher power consumption (averaging around 38W), and the groan of the two fans inside: the power supply and chipset fans. The audible part isn’t really an issue given that it’s stashed away, but considering that a probable failure point on most new electronics is the fans, it becomes a reliability concern.
I junked a laptop because of an impossible to repair broken fan. I’ve lost several video cards for the same reason.
I can even hear the irritating whirring of my blu-ray player’s fan (do not buy the Samsung BDP1600. The thing is complete junk even without factoring in the noisy fan trying to upstage the even noisier optical unit. Speaking of junk, the Sony alpha-200 is another garbage product that made me regret ever turning my back on Canon).
As promised in the original entry, I got around to replacing the power supply with a PicoPSU 90W unit, which was basically a plug and play swap.
In my original entry I estimated a 4-8W power reduction, which turned out to be an underestimation. With this PSU the power consumption dropped a whole 10W, going down to a constant 28W (only slightly spiking under load), making me feel a little less enviro-guilt. There’s still the noisy chipset fan, but that’ll be another project.
The case was built around the expectation of a power supply fan exhausting heat, so some extra natural ventilation was required. With that the sensor readings now hover at low operating levels.
Economically this is a change that will not pay off. From NCIX the new PSU cost me $73.49 all in. Given a savings of 0.01kWh per hour, and a fully loaded electric cost around $0.16/kWh, it would take 5 years for the 10W to pay for the change.
It would be nice if all power supplies were mandated to be efficient (they aren’t for most devices because they know it plays zero part in your purchasing criteria. It’s unfortunately one of those areas where legislation is really the only effective solution), because right now inefficiency is the standard. Of course environmental choices don't always yield the expected results.
In July of last year I wrote about choosing a new smartphone to replace the MotoQ that I had been using. While the MotoQ served a good tour of duty, it was seriously showing its age and was falling behind in the empowering mobile revolution.
While I’d been using variants of Windows CE since before the turn of the century, Windows Mobile was obviously lost in the wilderness. Not only was each equipped device essentially abandoned right after being released, the clearest sign that Microsoft lost the plot could be seen in PocketIE, where the preloaded bookmarks to various Microsoft Mobile pages led to 404 errors.
The team moved onto something new and shiny and had no concern at all for the existing base. Microsoft has a very short attention span to products that don't earn them Windows Office type revenue numbers, so it wasn't a surprise.
For various reasons I did not want an iPhone (we don’t need another restrictive and innovation crushing Microsoft scenario playing out, and I want to develop for the device without embracing the whole cult), despite it being the easy choice. I opined in the first entry that Android seemed to have a very bright future ahead, which is a prediction that seems quite obvious now given that it is the platform of so many incredible devices recently released or on the horizon.
The future is so bright for Android that the robots have to wear shades.
The options in Canada were (and remain) limited, so I went with an HTC Dream (G1) given that it had a keyboard and otherwise had largely the same specs as the newer HTC Magic, aside from what seemed like a minor difference in memory capacity.
I have to confess to being disappointed with the device.
Functionally it is amazing, and even with Android 1.5 the platform is simply brilliant. When everything operates correctly I am over the moon with the device.
The problem is that everything didn’t operate correctly. For whatever reason the device seems to be horrendously overloaded, so even with virtually no apps installed and nothing beyond the base system running, most actions are plagued by obnoxious pauses, even on a fresh start-up.
I stopped using brilliant apps like Weatherbug because they seemed to make the situation worse.
Alas, my long term plan was always that I would buy one of the newer, faster phones when they came to market, while using the starter device for development purposes until that time. If an unlocked Nexus One or Droid/Milestone worked on Rogers’ wireless band, I’d grab one of those when it was a possibility.
Nonetheless, I was pleasantly surprised recently to find that Rogers was offering all HTC Dream owners a free HTC Magic for $0, with the caveat that your term length pushes out. Given that Dream owners can only possibly be 6 or 7 months into their term, that isn’t that tough of a demand. I am on a very reasonable family plan that allows me 5GB / month (which I seldom use more than 1% of), so I feel fairly future-proofed with that foundation and for me it was all win.
So the next day a Magic arrived in the mail and moments later I was up and running with it. With the SIM card removed my existing Dream still works on wifi, where it can browse the web and play media and respond to emails and take pictures, and I can of course put another card in it and continue using it online. I’ll likely install Cyanogen on it now.
Quite pleased about that.
The most shocking thing, though, is that this Magic is much more responsive. It has the same processor as the Dream, so that doesn’t explain the difference. If I had to guess, I’d point to RAM, which on this device comes in at 288MB, compared to the 192MB in the Dream. For comparison both the Droid and the iPhone 3GS feature 256MB of RAM.
The extra headroom over the base OS seems to make all the difference in the world. On the Magic I can see that the free memory is usually less than 90MB, even on a fresh start-up, which notably would put it over the limits of the Dream.
HTC and Rogers claim that they’ll release Android 2.1 for this device in the near future, which makes me especially pleased.
Great move, Rogers. The new HTC Sense update and free month of data is icing on the cupcake.
Back in June I wrote about Web Workers, a fantastic new method to move processing out of the UI thread. To support the entry I posted a variation of the SunSpider benchmark I named Moonbat.
Safari kicked Firefox around in this benchmark. I just tried it with the just released 3.6, and it doesn’t look like much has changed: FF 3.6 does 10 iterations with 4 threads in ~11 seconds, Chrome does it in 2.6 seconds, while Safari leads the pack at 2.3 seconds.
Alas, web worker performance isn’t a critical factor in choosing a browser (my favourite browser remains Firefox), but it would be nice to see it moving in the right direction.
Got the cable bill — a bill that pushes into the $250 range per month these days — to find a surprising $11.25 "internet overage fee". Apparently I used 67.5GB last month, while my limit is 60GB. The Steam sales, several purchased HD movies and a couple of on-demand games for the kids on the 360, added to the normal internet usage apparently really added up to a very atypically throughput-intensive month.
I'm not going to cry many tears about it, even though I do think $1.50 a GB is a bit absurd (in an average month I doubt I use 10GB, so now I almost feel obligated to max it out), given that I think by usage pricing would lead to a far better, more open, more honest system for everyone.
.NET/Microsoft detractors got an early Christmas present recently when the London Stock Exchange, under a relatively new CEO, decided to dump their .NET/SQL Server –based trading platform, TradElect, to replace it with the product of a being-acquired company.
On
Slashdot, news of this was submitted and accepted as “London
Stock Exchange Rejects .NET for Open Source”, with the
statement that “The switch is a pretty savage indictment
of the costs of a complex .NET system.” The Digg submitter went
with the title “London
Stock Exchange dumps Windows for Linux” — which they took
directly from the linked article — with the description “Fed
up with Windows' failures, one of world's major stock exchanges is
joining many others in making the switch to reliable
Linux“.
The heavily-linked columnist in both cases is a guy who has been riding this "LSE dumps Windows!" horse for a while now. It has certainly provided him with lots of quality incoming visitors, drawing in those looking for validation, and angry hordes baited by his trolls. Encouraged on he seems to be accelerating the unsubstantiated hyperbole.
Let’s take a moment to go back in history for a bit.
Microsoft made a Really Big Deal about the LSE originally switching to this custom, Accenture-built, SQL Server 2000/.NET-based solution. This was sort of Microsoft’s coming out party, in a way saying “look, we’re big boys too! No more pull-ups for us”.
When the LSE had a very public failure on one of the biggest trading days in history, the detractors were screaming “I told you this would happen!” until their throats were sore, despite the cause of the failure never having been publicly detailed.
Failures have happened on every platform, most commonly as a result of application failure. To automatically assume the worst of a novel solution simply because it is atypical is the thought process of annoying simpletons, anxiously and eagerly hoping to try to pin any fault on anything that doesn’t fit their vanilla perspective of how things are supposed to work.
“The software must have failed because Bob went with HP instead of IBM!”
“The payroll system miscalculated. It must be because they moved it to Linux from Solaris!”
So what really happened with the LSE?
Accenture built a very expensive, custom solution for the LSE, purportedly costing somewhere in the neighborhood of $65 million dollars. To operate this custom in-house (albeit designed by Accenture out-of-house) system the LSE built up a considerable technology workforce.
The worldwide recession hits and the LSE takes some financial hits. A swarming mass of competitors in Europe, many running off-the-shelf, superior systems that they’re paying less for, go live.
A new CEO takes over and immediately starts to swing the axe. He makes specific comments in the press about the high IT costs of the organization; both of the large number of technology workers in London, and the continuing significant payments to Accenture to finesse the TradElect platform.
He undoubtedly observed that all of this custom work hasn’t gained them any unique advantage in the relatively commodity task that they performed. In some ways it’s like writing and maintaining your own in-house operating system – if it doesn’t give you some advantage, and actually puts you behind as everyone else pools resources on a solution, then why would you do that?
So they go on the market for a replacement, eventually deciding to go with the product of a Sri Lankan company. The price is right, and the lure of low-priced Sri Lanken talent is enticing enough that they buy the whole company.
In the end they have switched from an extravagant, custom-developed solution built by a notoriously expensive consulting company, and a workforce of expensive talent in the West, to a basically off-the-shelf solution that has been subsidized to its current state by other organizations, in the process getting some low-paid talent in South-East Asia.
The new product isn’t open-source, and it runs on a range of non-open-source UNIX platforms. The Oracle database system it uses is the antithesis of open-source.
What about this story has anything to do with open source?
The LSE doesn’t think it has anything to do with open source, or even necessarily Linux.
Where this story gets legs among the zealots is that the LSE plans to deploy the new product on Linux, given that the underlying operating system in many cases has been commoditized. Who wouldn’t?
Zealots cheering on trolling columnists like Steven J. Vaughan-Nichols do the profession harm. Now this nonsense is going to be parroted by people who don’t know better, making them look worse for it, for years to come.
I love Linux. I love open source. And you know I also am even quite fond of .NET and SQL Server. I detest fanatics, fanboys, and hysterical columnist that distort or invent reality to get themselves hits.
I grabbed "Dirt 2" for the xbox 360 recently, looking for an accessible late-night gaming distraction from coding.
The game is a stunning technical achievement, and it is amazing what they squeeze out of the almost half-decade-old era hardware of the device.
What makes the game spectacular isn't specific to some mystical art of console gaming, however, but is simply great software design and execution. While many in "mainstream" development (business processes, websites, etc) consider game development foreign to what they do, it's all just algorithms and code: One person does financial projections and another does particle effects, differing less than many imagine
The game was so excellent that
I decided that I'd try to find who the talent behind it was, my
quest thwarted because this game, like many recent releases by
large game studios, has an apparently anonymous development team.
My search for credits has yielded only a listing of artists
responsible for the songs in the game.
It would be great if there was an industry credits site similar to imdb, where you could find out the people responsible for games and applications: I can easily discover who did the foley mixer work on Joe Dirt, but can't discover the team behind Dirt 2 after a lot of digging. Maybe I'll make one.
I did find a "studio tour" video, in which the only person deemed worthy of naming was the "Senior Executive Producer". Maybe if I finish the game I'll discover who did the magic to make this game happen. I'd like to read how these guys operate and do what they do, because they are clearly successful at their craft, and I imagine they'd have interesting things to say.
Are they just cogs in the gears of CodeMasters? Crank it and a great game pops out, quality determined only by your Senior Executive Director in charge of North American Marketing?
Are we past the era of superstars like John Carmack? Are we into an era where everyone is nameless "team players"...unless of course they're in senior management/marketing, in which case their contributions and name will be heralded everywhere.
As a mostly unrelated aside, the "all contributors are equal, but some are more equal than others" policy reminds me of a conversation I once had with a peer, during which they bragged about how their workplace followed a policy that strongly discouraged fancy-pants work titles (e.g. no lead architect, senior developer, etc). My appreciation for that egalitarian workplace dissolved, however, when I learned that the speaker had granted themselves a lofty, important sounding title, as did the other senior members, and they failed to see the hypocrisy in it.
Sidenote: The website for the game is mildly offensive to Canadians. They decided that the landing page would require you to first select a country, with the options being the Netherlands, Belgium, Germany, Spain, the UK, Italy, France, and the USA.
As a Canadian I'm left not knowing which I'm supposed to pick. Maybe I'm supposed to pick the UK to get words with superfluous 'u' still intact. Maybe I'm supposed to pick the US just because of proximity? Two of those countries (the Netherlands and Belgium) are significantly smaller than Canada, so I have to guess it's a hybrid language/proximity thing.
Lots of websites pull this cheap navigation technique and it's lame. Often a US flag really means "English", other times a Union Jack means English. Nationality and language aren't the same thing, so it's a lazy tactic, made especially confusing when both appear together.
Then again, if I recall correctly the old Codemasters site worked by having you select on a world map, where all of North America was labeled "United States of America". Us Canadians get accustomed to it.
In an earlier, more naïve era of my career, I had three software development “heroes”: Jamie Zawinski , John Carmack and Joel Spolsky.
Jamie Zawinski grabbed my attention because he was at the center of the internet revolution, right in the bowels of Netscape from the outset, and had established a pattern of posting surprisingly pragmatic comments that defied convention.
It was extraordinary to read someone openly critical of their own organization, especially without it being retracted or redacted the sobered-up or calmed-down next day, and where the author didn’t hide behind anonymity.
Jamie let us commoners see the sometimes ugly mechanics behind the curtain. He also revealed a very interesting workplace that was foreign to the gray-walled cube world that most of us lived in.
This was at a time when Microsoft
really had almost unthinkable dominance over the industry, so to
hear Jamie discuss the travails of cross-platform development was
like going out of bounds at a tourist resort. Seeing what the
brochure didn’t show you.
An SGI box? IRIX? How exotic!
Another of the kings, John Carmack, was blessed with “F-you money” from the incredible success of some of his earlier projects, along with a proven abundance of intelligence and skills, so he too had the luxury of entertaining a surprisingly realistic and pragmatic perspective. He was a principal driver of the evolution of GPUs and gaming hardware, and you can owe thanks for some of the capabilities of your console or dual-GPU rig to his desire to make shooting things in first-person shooters hyper realistic.
Carmack was also one of the original “bloggers”, regularly posting lengthy “blog entries” by repurposing the UNIX finger facilities.
Joel Spolsky is a bit of the odd-man out in this trio. While he did have the requisite first-initial, he wasn’t known for extraordinary technical acumen (beyond having worked on Excel in some earlier life), but hear me out, please.
Joel ascended to Kingship – at least in my personal hierarchy of industry royalty – just after the dot com crash, when CMM factory-line initiatives started to become the mythical silver-bullet: This was an era awash with articles gushing about the amazing adoption of CMM5 among offshoring firms.
Many organizations were striving to reduce software development to an assembly line of easily interchangeable cogs, both of code and people, achieving a utopia where the process would become perfectly predictable and repeatable if only you filed enough forms.
Joel spoke up for developers when most were absurdly blaming the .COM collapse on dual-monitors, Aeron chairs, and inflated developer egos, as if taking developers down a notch and having them sit on a cold rock would have made selling kitty litter online a good idea.
He was essentially an enlightened pointy-hair blogger, and while I wouldn’t look to his blog for technical advice (Wasabi!), he really understood developers and the process of getting software built. And he was willing to risk his own nest egg and put his money where his mouth is, having since built a reasonably successful company in Manhattan that most of us should be envious of.
What made these three really stand apart in a sea of cheap advice-givers and pundits was that none of them were writing to get a job or even necessarily to keep one. Joel made his own bank while the other two were of such technical esteem that they had little to worry about professionally regardless of what they might say.
They weren’t coerced into railing off the latest buzzwords and best practices, deferring to the latest silver-bullet best practice pattern-based UML diagramming system 3NF data warehousing factory built on a n-tier service-oriented, aspect-oriented, polymorphic framework so that they can get the approving nods from the nervous masses and clueless PHBs.
They didn’t worry about offending a boss who held some sacred cow that if only you did it the way she read on some best practices blog, everything would be fabulous, at least until that initiative fails and you move on to the next cure-all.
The three kings were just saying it like they saw it, which was and still is rare.
Eventually Joel ran out of things to talk about and switched his blog to mechanically regurgitated repeats; Carmack got lost endlessly perfecting the noble quest of simulating head shots when he wasn’t reaching for the stars; and Zawinski decided to engage in endless battles with the city of San Francisco over his money sink of a late-night dance club (if you read his blog about DNA Lounge early on, you could almost smell the contractors taking this dotCOMinaire for a ride).
I was delighted to see Joel return from effective blogging retirement, and my enthusiasm exploding when I saw that it was a post about Zawinski!
A royal duet!
Okay really Joel was selling a book – like his partner-in-crime Atwood, he seems to be motivated to post by Amazon Affiliate bucks these days, credibility undermined by that kickback – however he chose the Zawinski chapter as his pitch, talking admiringly about how practical and “get ‘er done” (paraphrased) jwz was about his craft, doing so with a present tense that betrays a certain blissful ignorance about Jamie’s career path since.
Joel labeled Jamie the “Duct Tape Programmer”, which was a description that Jamie took as “damning with faint praise”. Joel has long been against architectural astronauts, so he seemed to excitedly hold up Jamie as the successful counterpoint.
Perhaps “duct tape” is a bit of a metaphorical overreach, causing many to envision some Tim the Toolman ‘Ar ar ar’ hack. Pragmatic or practical probably would have been more accurate, though it would have made for a less contentious entry.
Never mind that Jamie worked within extremely tight timelines, using technology far less advanced than what we have now.
Joel’s entry raged across the social news sites, with the regular suspects popping out of the woodwork to declare it a grievous offense to all that is all that is good in the world of software development. Lots of blog replies parroting the standard best practices appeared, their authors clearly hoping that their boss and any future employers will see how studious and diligent of worker bees they are.
The people who are the most certain about software development patterns, practices, and technologies are usually the people who have the least reason to have such certainty.
I’m going to be a bit trollish while I go to the extreme and say that many of the oft-quoted leaders of the field, responsible for much of the unquestioned wisdom-bites, have little to demonstrate why they’re in a position to preach.
The revered Fred Brooks, author of the Mythical Man-Month, came into a position of considerable influence largely by leading a project that was by most accounts a massive failure. That would be fine if there was but one way to fail and he found it for the benefit of all, but there are an infinite number of ways that a project can fail.
Of course you must learn from failures, but my experience has been that the explanations for failure are often a worse-than-useless distractionary tactic: When a team technically fails to accomplish what they set out to do, expect the post-mortem to be full of nonsensical misdirection about how everyone and everything else is to blame.
How many post-mortems include the statement “I grossly overestimated my own capabilities”? I suspect few.
Steve McConnell is another well-known author in the field, revered for his software development books (though many strangely overlook After the Gold Rush, where McConnell knee-jerk responds to the dot COM collapse by advocating an ill-considered licensing system for software developers), but his professional experience seems to be limited to working on TrueType at Microsoft, and some nebulous software development at Boeing, after which he took on the role of telling the world how software should be developed. Now he consults with pointy-hair bosses to unknown outcomes.
Don’t get me wrong, I have both of them in the bookshelf behind me, and read and greatly enjoy their opinions (Brooks’ observation about second systems is more profound and important, I think, that the over-quoted man-month snippet), but really, let’s keep some perspective and stop using it like they’re the incontestable word of truth.
I read them critically and with an open mind, not taking it as the voice from an all knowing deity, but instead the perspectives of a couple of guys drawing from their experiences.
Of course, the esteemed Fred Brooks
and Steve McConnell exist in a realm far above most silver-bullet
cheerleaders in our industry. These successful authors actually
dirtied their hands with actual software development, refactoring
their opinions over the years into refined perspectives. I select
them merely as “absolute best case” examples.
More commonly the people who most ardently advocate certain practices and approaches have achieved little, usually having nothing to back their conviction but self-interest and a desire to look like they know what they’re talking about, having associated their id with “correct” approaches.
They just clutch onto whatever they hear is proper and start repeating it like a novelty birthday card repeatedly opened. They’ll tell you that should develop like an ecommerce site, despite not being an ecommerce site; like you’re NASA, despite not being NASA; like you make the software for a pacemaker, despite actually making an ebay auction sniping tool.
Why do I hear the word “pattern” from mediocre or non-developers more than I hear about it from experienced developers, always stated as some sort of conclusive statement?
Why do we accept that a chimp-level of software development skill is acceptable for maintenance programmers, capable of understanding only the most infantile code that is carefully decorated with “Coding for Dummies” comments?
Why is “We should use UML” the desperate last-ditch fallback of failing teams everywhere?
Unit testing, or the more early-loaded TDD, can be great, but it isn’t a panacea and is an extremely poor substitute for actual craftsmanship.
Moving beyond the non-developers giving their unwanted opinion on how software should be built, the other class of destructive noise is the advocacy of silver-bullet methodologies during the honeymoon period.
Great, you built a sample app on RoR/Haskell/Scheme/python or whatever else is the cure-all platform that profoundly changed your world view.
Here’s a nickel. Go build a real app then tell us how revolutionary it is now. I don’t discount the advantages, but advocacy based upon toying around is of little use to real projects. Extrapolating it up is foolhardy.
Oh look, another guy telling us how switching to the Dvorak keyboard layout made him regular and makes his code smell like cinnamon. Here’s someone saying that they slept 4 hours a night by taking 20 minute catnaps, proven out over their two day sample period. This guy says that having a 400x200 single-app screen on a netbook made him a perfectly focused coder. Here’s a dieter who is certain that they’re onto an incredible, beats-the-laws-of-thermodynamics diet now that they’ve followed it for a whole six hours.
The fairy tale “The Emperor’s New Clothes” has significant relevance to the software development field. To quote the plot summary from Wikipedia
An emperor of a prosperous city who cares more about clothes than military pursuits or entertainment hires two swindlers who promise him the finest suit of clothes from the most beautiful cloth. This cloth, they tell him, is invisible to anyone who was either stupid or unfit for his position. The Emperor cannot see the (non-existent) cloth, but pretends that he can for fear of appearing stupid; his ministers do the same. When the swindlers report that the suit is finished, they dress him in mime. The Emperor then goes on a procession through the capital showing off his new "clothes". During the course of the procession, a small child cries out, "the emperor is naked!" The crowd realizes the child is telling the truth. The Emperor, however, holds his head high and continues the procession.
Too often the software development industry suffers for lack of someone crying out. We often just go along with it, listening to the declarations of non-developers and maintenance programmers as if they speak unquestionable truth, all while discarding any counterexamples as mere aberration (“Well not every team has superstars you know! We aren’t all John Carmack!”).
Everyone withholds contrary, pragmatic “Well it isn’t quite so cut and dry…” opinion lest they look like a “hack” to a present or future employer or nervous, cargo-cult embracing peers, smiling politely while the never-coded, overconfident guy acronym drops about things he doesn’t understand in the daily stand-up.
The more you know, the more you’ve experienced, the less obvious the world becomes, and the more hesitation before offering up opinions. The less ease there is to criticize the path of others when it has yielded obvious success.
Opinions come quickly to experts and morons. Few of us are experts.
Jamie Zawinski had unique conditions under which he unquestionably succeeded. Many, with the seeming clarity of hindsight and the ability to project whatever imaginary timeline one desires, will look back and comment on how the codebase got rewritten, purportedly twice, and how eventually the product was squashed, stomped out of relevance by Microsoft (before being reborn as the game-changing Firefox), using that to draw the absurd conclusion that if it were produced “properly” at the outset, today we’d all be using Netscape 9. Then again, maybe it would have followed the disastrous arc of Chandler.
The road that leads to most successful apps is often an ugly, brutish affair filled with compromise and folly, risk-taking, detours-followed, and shortcuts pursued. That isn't to justify them, or to diminish alternative approaches, but we should always keep our minds open, being less quick to defensively guard whatever we're selling as the cure-all this week.
What I’d like to read more about are the success stories, and less about the professional pundits telling everyone how it ought to be done.
Of course here’s where we get into a common paradox that exists in most industries: The successes are usually off enjoying their success and wealth, less inclined to toil away their days writing blog entries extolling their "dart toss" method of architecture. We’re left with the conversation being dominated by the people who don’t actually make software at all, telling us how grand they could make software, if they ever actually did, by following their sure-win magic formula. The conversation boards are overrun with the people who actually have so little to do that they spend their time describing the ideal way that everyone else should write software.
Parallels can be drawn with the financial world, where the snake-oil salesmen pitching the ways to make money are usually doing so because their only way of making money is pitching how to make money (Want to know the secret to making big money? Send me $5, and I’ll tell you that it’s to get people to send you $5 to learn the secret of making big money). The guys actually making the money are off making the money.
This brings us full circle back to Joel’s recommendation of the book. A book that serves as one of the few opportunities we have to really read how projects succeeded, straight from the source.
It’s good if only to let the successes have a voice in the conversation.
We have a number of network devices at play in my household.
I have my work machines. My wife has her laptop. My kids have their PCs.
Family and friends that comes to visit connect their iPhones and laptops to our 802.11g/n.
We have shared media that we play on devices like the Xbox 360 and PCs over protocols like uPnP, DAAP and SMB. We have shared files, such as financial info, that we need to access from any machine, all while confident that they remain as secure as we want them to be.
We do VPNing from remote locations like coffee shops to access files and media. We have a shared printer.
To optimize this situation without leaving my power guzzling pig of a PC on around the clock (I’ve setup and torn down a number of home servers over the years, all the ways back to a FreeBSD-based device in the mid-90s that also doubled as a firewall), a while back I bought a Synology DS-106j NAS unit and dropped it on the network, deciding on it after looking at some Windows Home Server products, finding them overpriced and uncompelling (not least because of concern about licensing and DRM BS.)
One of the selling points of this ~$200 unit (to which you add whatever 3.5” hard drive fits your fancy) was the gigabit network port. To me that implied that it must be capable of at least keeping a 100Mbps pipe saturated with goodness.
I also liked that it gently sipped power (13W or so with the hard drive), allaying any guilty about leaving it on around the clock.
The Synology unit is fantastic in many ways.
It has a fantastic interface. It is rock solid (zero downtime with it) and is very feature rich, with several free “firmware” updates since I purchased it adding a number of new features and interface improvements. It is easy to setup and use, and to a superficial degree secure. It comes with great desktop software for common tasks like backing up (and versioning) files on your Windows machines. You can offload torrent downloading to it if you’re into that sort of thing.
But the hardware seems to be underpowered. The 200Mhz MPC8241 CPU that powers it can’t deliver on the extensive feature list, in my opinion.
The 1Gbps network port turned out to be brave talk that it couldn’t back up with action. Normal SMB/CIFS (the protocol that Windows utilizes to access network file shares) averaged about 2.5MB/second throughput with the device, which factoring in overhead is around 22Mbps, leaving plenty of headroom before it hit the limits of even a lowly 100Mbps network.
I got it up to 8MB/second or so by turning off all of the features like media sharing, and then disabling all security on the share and enabling guest access – it seems that access control security significantly adds to the computational burden – but that isn’t a viable day-to-day option so I had to revert and it was back to 2.5MB/second. FTP was a little better, but was still way below expectations given that the same hard drive got a rough 40MB/second from a direct-attached USB 2 enclosure.
Given that I am now downloading from the web at large at 1.1MB/second, this seemed silly. As we’ve started to amass monstrous AVCHD video files of a GB or more from the home video camera (the latest GB or so features my youngest son herding some chickens that had escaped from their pen at a local farmhouse), it was getting unsustainable.
Synology is now selling a purportedly faster unit, but given that I saw much lower real-world speeds than what they graph for the old unit, and their new unit really doesn’t do that much better anyways, it didn’t compel me to upgrade. Many competitive low end home NAS devices have reportedly similar performance in the wild.
And it isn’t just waiting while moving large files or having SyncBack jobs run. Navigating the music directory from a uPnP client made it obvious that the slowness impacted all activities, with basic operations having intolerable delays. Attaching to its network shares from client PCs inexplicably had multi-second waits before responding.
Then again, I’m the guy who thinks that the biggest crisis in the electronics world is the excessive lag before DVD players will eject a disc, so maybe I’m an outlier, but it seemed like a pretty big weakness of what would otherwise be a great product.
It really bothers me when units doing tens or hundreds or thousands of millions of operations per second take perversely long times to do simple things.
On the bright side, the small dimensions of the enclosure made it easy to find it a home, but it also had the S.M.A.R.T. monitoring of the hard drive complaining about borderline heat incessantly. And the printer sharing never worked properly, but given the endless variety of printers I won’t blame that on the NAS unit.
So I decided that it’s time to replace the device, with the following criteria for its successor.
Local RAID isn’t important to me, as availability isn’t a primary consideration (and multiple drives go contrary to power conservation / heat reduction goals, and generally increase maintenance.) Every file that is on the unit always exists on at least one other drive on a separate system, so if the drive died I could quickly rebuild and repopulate and life would be grand again. And it is vastly more likely that two drives in one unit would catastrophically fail than it is that two drives in physically separated devices would fail, especially when the separation between can be thousands of kilometres.
Nor do I want this to function as a media station, which I consider an entirely different function, with an entirely different requirements list. I don’t need or want HDMI output, blu-ray decoding computational power, 3D offloading, a MAME box, or anything of that sort. That’s a different project, with a different set of goals.
After looking at the available market options of targeted NAS devices, and hardware solutions like Shuttle computers, I decided to have a little fun and build my own (in the minimalist stick-some-lego-like pieces together way that is "building" a PC), so I grabbed a couple of components.
<$200 for the “NAS enclosure” part, then $127 for a good hard drive with room to grow. Pretty inexpensive, and would probably even make a decent internet appliance, which presumably is the intended use of the motherboard/CPU combo.
Putting it together was a snap, with dutiful oversight from my children, followed by a quick flash to the latest BIOS release.
I installed Ubuntu 8.10 Server from a USB stick (I considered installing FreeNAS, but decided that I wanted something a bit more custom), after which I discovered that the hard drive wasn’t set to bootable and the system wouldn’t boot without the USB key, so I had to boot with the key and use parted to manually set the sda device to the boot device, after which the USB key was no longer necessary.
Installation complete, I stashed it away, sans keyboard or mouse or monitor connection (which it makes no complaints about, happily existing without head, keys or mouse), and access since has been via putty. A magical source and destination of media and files, out of sight and hopefully usually out of mind.
I installed Samba (during the OS install by choosing the file server option), created the users and groups and perms, added some shares, and started accessing it from my workstation. It was all very easy and straightforward, including even setting up AppArmor to enforce mode against the Samba services, adding a small additional guarantee of security.
I started the first file move, ready to witness some low-cost awesomeness.
11MB / second to write to it, averaged out over the copy of a GB sized video file. I got about the same speed reading back from it.
Pretty good compared to 2.5MB/second (and now I had much better security granularity than I had before, on a much more versatile platform), but not quite what I was expecting. Awesomeness not witnessed.
The hard drive isn’t the fastest of the bunch, but it’s still very decent.
I’m far from an expert when it comes to Linux (or more generally Unix) systems, seemingly achieving some small level of localized expertise at intervals in my career, but then the solutions simply work and I don’t have to touch them again, so the knowledge rusts and each time I return I feel like I'm a Linux virgin (who would have ever thought that Linux and virgin would used in the same sentence?) It is similar to my proficiencies with Perl or the equally obtuse Powershell, where I put together very decent solutions, but then several months later look in disbelief that I actually wrote the code that now looks foreign.
Nonetheless, I quickly determined that the network adapter was auto-negotiating itself incorrectly to 100Mbps, and 11MB/second is about the max you could achieve over such a link.
After a quick bit of searching, I added ethtool -s eth0
speed 1000 duplex full autoneg off to my interfaces file
(after validating that it worked at the command line). It was now
at full 1Gbps glory.
So I performed the file test again.
47MB / second to write to it. 55MB / second to read back from it.
Sweet! That isn’t that far off of local storage, and is entirely with the acceptable zone. It was achieved on a $200 host device. File sharing is hardly the most demanding application nowadays, but I was happy to get something much more usable.
Just to take a side trail here for a moment, I just wanted to mention that Linux, and the evolution and progress of it and related projects, really is amazing. It is astounding that such a solution evolved the way it did. Even people who don’t think they benefit from it regularly use devices run on the OS. The amount of functionality and technology available, including the source so you can roll your own, truly is remarkable. Linux didn’t start this movement, and projects like FreeBSD, sendmail and prior initiatives blazed a successful trail before Linux really took off, but it is the most evident element of that development model.
Back to setting up the homebrew NAS, virtually every step of the way I had to consult the wizard, Mr. Google, but at least the answer was usually only a page 1 or 2 hit away (though Google Groups has taken a perilous dive into close-to-uselessness, and proved of little use.)
To this point I still haven’t had luck finding a good uPnP server for the device. uShare simply didn’t work. Mediatomb inexplicably doesn’t work with the xbox360, and even if it did the fact that its anonymous web user interface has the default behaviour of exposing your entire file system makes me question the wisdom of the developers.
Various other projects didn’t fit the bill or didn’t work for me, and most seemed to have been abandoned.
I thought I’d try the MythTV backend, as it is purported to double as an uPnP AV mediaserver, to find that it bizarrely needs an X server running and an X client on the other end just to perform the setup. I decided to purge it from the system to find it wouldn’t uninstall due to /dev/.static/dev being mounted read only. I found a workaround involving stopping and starting udev, which remounted it as read/write, but at that point the damage was apparently done, and it refused to gracefully leave the system.
So I went MythTV huntin’, committing a moronic error in the process (it tried hard to warn me, but I persisted in my foolishness.) I’ll let the putty log speak to it.
dennis@mediaserver:/var/lib/dpkg$ sudo killall
mythbackend
dennis@mediaserver:/var/lib/dpkg$ whereis mythbackend
mythbackend: /usr/bin/mythbackend
dennis@mediaserver:/var/lib/dpkg$ cd /usr/bin
dennis@mediaserver:/usr/bin$ cd mythbackend
-bash: cd: mythbackend: Not a directory
dennis@mediaserver:/usr/bin$ rm -R *
rm: remove write-protected regular file `['? ^C
dennis@mediaserver:/usr/bin$ sudo rm -R *
dennis@mediaserver:/usr/bin$ ls
Installing everything once was so much fun that I decided to do it again!
So I reinstalled again. The second time around it was a very quick process (even with a nuked /usr/bin I still had a chance to backup customized config files to a client box before reinstalling, so I really didn’t lose much.)
Power wise, I’ve gone from ~13W for the DS-106j, to 35W to 40W for the new device (measured with a Kill-A-Watt. Handy device.) This represents a pretty big move in the wrong direction, but it wasn’t unexpected. 4-8W or so could likely be saved going to a more efficient, fanless power supply. That’ll be a future experiment. The Caviar Green, at 3-4W, isn’t much more power hungry than an SSD, so that wouldn’t make a huge difference. The motherboard and its chipset is the real piggy of this farmyard.
Alas, in return for the extra power consumption I now have a vastly more interesting network device, serving files at a much more usable pace.
So my first adventure is beginning the work on a .NET-based uPnP server, ensuring that it works on Mono along the way. Thus far it has been nothing but remarkable success, and it still astounds me seeing fairly complex .NET applications running well on a Linux box. The thing really, really works!
The story of Markus Frind is not a new one around development circles: Some guy creates a remarkably unpolished, seemingly unsophisticated dating website and in short order is bragging about the million dollar checks he's getting from Google Adsense payments.
Still, you owe it to yourself to read the article about his exploits in January's Inc. It is a fascinating story of internet success against the odds, and the site that is serving up 1.6 billion pages per month on an inexpensive modicum of hardware. That article references Markus' blog entry from 2006 where he explains how his extraordinary success story began.
[Imagine that I insert some drawn-out blowhard "lesson" to be learned from Mr. Frind's success here, allowing me to justify making an entry that is basically nothing more than a link, all while pretending that if you follow these simple steps you too can achieve the same results]
Coding Horror is an entertaining, sometimes even educational blog. Be careful diving in headfirst, though, as the technical depth is generally so shallow you'll be hitting the bottom before you've even broken through the surface tension.
It's always a danger — in the nerdly get-some-unkind-emails way — to question it. It has quite an army of loyal fans who, I presume, have had their ego carefully stroked over the years into loyal defensiveness ("Yes you are a top notch programmer! Yes you are!"): Any prior time I've disagreed with Jeff on here has resulted in a flurry of emails that are the text equivalent of the infamous Chris Crocker video.
Yet Jeff's latest entry has me unable to contain myself.
In that post, Jeff opines that Windows 7 might just be worth a look because, he says, in the latest outing Microsoft has taken to changing the visible parts of the OS, instead of, I guess, just improving the underlying awesomeness. The example Jeff draws from is that the calculator has visually changed, whereas before it was just the underlying mechanics of calculation that saw awe-inspiring improvements, all while gaggles of ungrateful goons continued to hurl insults at Microsoft, unaware of the great gift they had been handed. That Microsoft has decided to enlighten us to the great improvements they've made by visually making change apparent, instead of just doing their magic in secret.
As a newsgroup troll might say, errrrr...wut?!?!?!
What planet has Jeff been living on? What spaceship did he just hop off of, interstellar cruise of the Outer Gamagia quadrant completed, that leads him to be so completely out of touch with reality?
Here on planet Earth, Vista was seen as largely being about changing the UI — much like XP before it — and many of the complaints were that the actual utility of the OS suffered (even basic operations like moving files seemed to have missed being QA'd, slowing to a paralyzing crawl under completely ordinary uses). Functionality got lost under layers of paint, and interfaces seemed to be changed for the sake of change.
To many, Vista was 99% visual changes and 1% detrimental functional changes. But at least it brought the unwashed masses a calc.exe that had shaded buttons and a translucent title bar!
Conversely, a lot of the excitement about Windows 7, relative to Vista, is that it fixes stuff "under the hood" (better, strong, faster.)
But I'm no Vista basher, and actually believe that much of the anti-Vista vitriol is undeserved and unfounded. While I was on the record saying that it would be a product failure because it was wrongly focused and had little that compelled people to desire it, I'm actually somewhat of a fan of the OS, insofar as the comparison is with XP. Vista even has some very cool features under the hood, such as TxF, though that's the sort of structural change that isn't really useful until applications start using it, but they won't use it until it is available in a good percentage of deployed PCs.
Back to Jeff's entry, the ridiculous example of the accessory calculator being an example of...anything...really strikes me as absurd, and it seems to be the sort of "try to draw some big observation from some small example" space filler you end up resorting to when you're trying to hit a schedule.
To add to the march of absurdity, Jeff links to a ridiculous post by the occasionally interesting Raymond Chen.
In Raymond's post we hear about how tough it is for poor Microsoft (sidenote: what's with the bizarre victim complex that many Microsoft employees develop?) You see, prior to Windows 2000 someone at Microsoft made the choice that when you use a calculator in Windows, you really want to enjoy IEEE floating point rounding errors in all of your results, because the people that developed calc.exe — which would literally be a 8 hour project for an intern..you don't even need to make an installer — decided to take the laziest route possible, implementing it in the most naive way available. Raymond goes on the defensive, telling us that those critics just don't understanding floating-point. Not really, Raymond. They just don't understand how Microsoft could have ever thought that it was a reasonable decision for a calculator app to use and suffer from, versus the decimal math of virtually every other calculator app.
So Microsoft swapped out the embarrassing calculation "engine" of calc.exe (Jeff got the timing of the change seriously wrong. It wasn't between XP and Vista. It was before Windows 2000), put in the bignum-style implementation that should have been there from day one, and people were supposed to send them flowers or something? You ungrateful sons of...
Anyways, Windows 7 will invariably make a big impact, so I do plan on taking a look at it soon. But I'm certainly not motivated because calc.exe got some minor changes.