an overwhelming percentage of software developers imagine themselves to be far above the norm - they themselves conveniently fit that top 2% echelon, they believe

Several readers have emailed to ask how my "Optimal Software Development Processes and Practices" entry -- which was surprisingly well received -- compared to several better-known software development guideline papers. For instance Joel Spolsky's superlative "The Joel Test: 12 Steps To Better Code".

"Read them all and then adopt what sounds right to you," I replied.

To expand upon my answer a bit, my entry is a listing of high-level management practices that apply to virtually any team involved in virtually any type of software project. If you have accountable software developers; you're focused on solving the right problem, using the right platform, tools, and technologies (given that they've done the research, and they're not tied up having camelCase/PascalCase wars, developing a redundant GFA Basic solution for the Atari ST); you have a history of accurate, detailed time information to draw upon for estimates and to really know the capacity of your team -- which usually indicates a grizzled, experienced management that is more attuned with reality than is the norm; you have full transparency of how the project is progressing and what is getting done; then you have a much better probability of success.

Yet these basics -- these core fundamentals -- are sadly missing from many software development teams.

It is, however, fairly typical of papers of this genre to look for something seemingly unique -- some sort of hook -- that'll grab the masses as the silver bullet. One which they can quickly implement in their process (for instance a poorly used and abused bug tracking application, or a hashed out daily build script that has been failing for months but no one has noticed because it has no utility, or partnering up developers in pair programming, or implementing code reviews, or pursuing test-driven development) and claim success. "We score a 9! Awesome!" they gloat, high fiving and then returning to the continuing cycle of project failures.

One common silver-bullet hook these days is the classic "only hire the best!" mantra, and I thought it worthy of special mention.

"Only hire the 2% of software developers!" these advice papers crow. "We ensure that our software is top notch by rigorously putting our interviewees through 19 interviews, by asking clever brain-teasers from How Would You Move Mount Fuji, and then by having them design and implement an embedded operating system on a whiteboard!" This is usually accompanied by a "how to spot a great developer!" listing which could better be described as "How the author would find themselves in a pile of applicants".

This arrogance isn't limited to the authors of such silver-bullet recommendations, though. In conversations and forum threads discussing such a top X% recommendation, one quickly comes to realize that an overwhelming percentage of software developers imagine themselves to be far above the norm -- they themselves conveniently fit that top 2% echelon, they believe, so of course they agree: Let the banks and the IT shops take the other 98% (the duds), because the top 2% are too busy making webmail interfaces and bug tracking applications (as ridiculous as that sounds).

The problem with such a "only hire the best!" proclamation, however, is that it's meaningless from the start -- How do you select the top X%, given that the number of attributes of a software development team member is virtually infinite? How do you quantify prospective talent universally like this given that what really matters to you varies dramatically based upon what you're developing, what fit the developer will be in a team, and how you're going to manage them?

If you're a poor manager and you fail to utilize a talent properly, or if you choose the wrong talent for a particular position, it can be an absolute disaster, or at least a non-optimal situation.

The top 2% in one realm was the bottom 2% in the other.

To draw from personal observations, I've known some remarkably intelligent developers who would absolutely storm a path of success in one type of project, in one type of role, basically designing, implementing, and delivering a remarkable project almost single-handedly. Yet they would wallow fruitlessly for months when put on another project. Whether it was because they had no motivation or passion for the new project, or that they simply couldn't adapt to the new requirements -- which is remarkably common. Great developers, young and old, often have a niche where they are remarkably strong, outside of which they falter -- the results were the same: A great developer was being wasted on a project that didn't leverage their strengths, and a project had a deadweight superstar that couldn't catch on. The top 2% in one realm was the bottom 2% in the other.

There are many projects and situations in which John Carmack and Linus Torvalds would be complete disasters.

There are many who'll disagree with me. "That's hogwash!" they'll say. "I can do anything well, it's just that IIS is a piece of crap, and the MSDN documentation is all wrong, and Microsoft is evil, and there's this strange behaviour with the MSXML Library -- that wouldn't happen on Linux! -- and that really screwed up my timeline, and....". Sure.

Of course this could be countered by claiming that the "top 2%" refers to whatever pet dogma the speaker subscribes to. "The top 2% writes test cases before writing code, comments all code (in RoR of course), and does their timesheets by 4:30pm every day".

And thus it is explained how every developer can claim to be the top 2%, and how every elite-herder can claim to hire only the top 2%: They simply adapt the meaning to describe whatever practices they adhere to, whatever techniques they use, and their personal skillset, or by simply describing the candidates who made themselves available to them, and who accepted the position.

In that way we can all be #1.

I've described why I blog several times previously, including within my very first blog-style entry back on September 4th (this blog just passed its 4 month birthday). The motivations are the same motivations that have pushed me to post online "papers" for about a decade now: Reputation, a bit of an outlet for thoughts (it is therapeutic), and of course to maintain or gain some namespace in the internet world (which really means PageRank these days), or at least to avoid namespace loss.

The namespace competition is much more competitive these days than it was a few short years ago, with the technical ease of blogging encouraging a lot of very capable entrants. If you're stand still, then you're falling behind quickly.

It really has been an uphill battle trying to get my thoughts noticed, but I'm finally at a remarkable point where I post something -- something written late at night in a brief interlude between ending "real work" for the night and hitting the hibernate button -- and discover that many thousands of visitors have passed through the next day. Often something makes its way onto the meme sites courtesy of one of the readers who thinks it's worth sharing (thank you kindly to those good folks). That's pretty neat, and is very rewarding.

The Reddit Effect

It's especially satisfying given that I seldom spend more than 3 hours on an entry, even on the longer pieces, usually spreading it out over several days. On an average entry -- for instance the spelling entry (which also saw many thousands of visitors per day), or understanding daylight savings time -- I spend approximately 30 minutes, going back on re-reads to clean up the flow. There are also coffee-break length entries like this one.

I usually have the general concept fermenting in my mind for a while, and I type quite quickly so it's really just a process of transcribing it out and dealing with the technical errata (like Radio Userland's propensity for being overly helpful).

To make them a bit more visually appealing, I tend to take a few more oddball pictures during day to day activities than I normally would, but that's fun and is hardly a chore.

Entries are often much more readable after about a week, and if blogs had a wiki-style history you would see constant minor wording changes and paragraph reworks.

The few pieces that have earned considerable attention have been real eye-openers, and in some ways they encourage one to move to the dark side: It's very easy to see, for instance, exactly the type of content that earns a lot of attention, and it would be terribly easy to write pandering pieces, saying everything that the crowd wants to hear, and that which they want restated (which is what many use the popularity-measuring sites for -- to push their own agenda by promoting sites that state their opinion, and by silencing dissenting opinion by punishing those that don't. Look at Slashdot comment moderation as a great example of this).

It has been a great experience, and I greatly appreciate everyone who stops by. Thank you very much for a moment of your time.

There is a growing trend towards smaller and smaller fonts on blogs (e.g. 0.60em). While it's a simple matter of customizing your local stylesheets, or overriding the font size at the GUI level,it is a bit of a hassle.. Nonetheless, it seems to be a growing belief that using a small font someone implies professionalism or academic worth.

Yet when will the cycle end?

When Java first hit the development ecosystem, to many it wasn't just a method of doing efficient, high-level development, but rather it became a new religion: You couldn't only use Java as the glue between existing code, or even as the overwhelming bulk of your solution. A partial-Java solution simply wasn't good enough.

Instead your product had to be 100% Pure Java. The still sought-after eventual goal was a complete Java solution, from applications right down to the operating system, with only the smallest possible binary kernel, if even that. All of this would be running on a Java-aware processor, engineered specifically for Java.

Sun created a "100% Pure Java" campaign to push this philosophy, including banners and designations for appropriately certified software, and advocated it as a very desired moniker. Users were led to feel that mixed solutions were impure and somehow dirty: Are you some sort of nut running an impure solution, dirtied with some pointer munging, buffer overflow vulnerable C code? While there were (and remain) methods to call native code, they were discouraged.

Of course there is a lot of validity to this agenda. Primary being the fact that pure Java solutions are theoretically cross-platform, with no ties to external technologies. Compare this to a solution leveraging C libraries, which would require a rebuild or binary available for every distinct target platform. Additionally Java could only impose its sandbox and extensive security constraints if you stayed in the world of Java, and thus callouts to native code represented a risk.

In the real world, though, it often meant that developers were constantly solving long-conquered problems, redundantly reinventing solutions in Java that long existed elsewhere, or waiting until adequate libraries eventually appeared: Developers were pressured to use Java alone even when it was a hammer and the solution really needed a chisel.

Thankfully .NET hasn't been pushed in such a single-minded way (even if some of its champions have foolishly taken up such a misled cause, including some at Microsoft. Instead of a justified part of the solution, it becomes a religion. .NET! .NET! .NET! .NET!), and indeed Microsoft themselves has always facilitated, and even advocated, "impure" solutions. The majority of the .NET Framework, for example, is actually a very thin veneer over the existing Win32 facilities and libraries -- it was either that, or version 1.0 would have come with a much smaller, much less efficient library.

The "orchestration layer over native code" implementation is the reason .NET hasn't suffered the performance difficulties that Java has.

Microsoft chose to leverage what they'd already done, to maximize both performance, and to maximize the breadth of the library. 

This advantage isn't limited to Microsoft, though, and the developer can utilitize this functionality as well. .NET offers very simple COM and P/Invoke functionality to leverage "legacy" code (or even new code developed in a best-solution, non-.NET technology), allowing you to easily use your existing DLLs and/or COM libraries as first class partners in your .NET solutions. Even if they're created in "dirty" languages.

I take advantage of this functionality regularly, utilizing existing best-solution libraries and functions, regardless of whether they're pure .NET or not. For instance in creating the static version of the "best of" blog entries, I quickly -- maybe 2 hours -- wrote a quick transformation tool that basically imported the "best of" RSS feed (it isn't included in the normal category lists), then doing some XSL transformations (using extension objects in the XSL given that XSLT alone wasn't adequate for some special purposes -- for instance HTMLDecoding the description block of the RSS) to the resulting XHTML, as well as creating an index page.

One goal when creating this solution is that the resulting pages are all fully XHTML compliant, and they pass the W3C validity checks. While I could easily see how the pages rendered in Mozilla/Firefox/IE/Opera, and of course they all rendered fine, technically there were a couple of deviations from the spec. Some of these errors and warnings were caused by unavoidable transformation issues, while others were caused by minor mark-up errors in the original blog entries (both because of my own errors when doing it by hand, but also because of Radio Userland's "helpful" auto-"cleanup" of HTML. It is remarkable how often auto-formatting is detrimental).

HTML Tidy to the rescue.

I had several options with HTML Tidy, the easiest of which would be to ShellExecute out to the EXE, telling it to process an existing file. I could have taken more time and tried to make a managed C++ version of Tidy, but I really didn't want to spend that much time.

I decided to have a bit more fun, not to mention building a more integrated, higher performance solution, and use the Tidy dll from the micro-.NET utility. I grabbed the Tidy source code (Tortoise CVS is a great solution for this, in this case using :pserver:anonymous@cvs.sourceforge.net:/cvsroot/tidy), updated the included MSVC projects to Visual Studio 2005, and added them to the transformation utility solution. I set the Tidy dll project output to the build directory of my .NET utility (in this case $(SolutionDir)\blogStatic\bin\$(ConfigurationName)). The MSVC build worked perfectly right away, which is amazing given that Win32 isn't an officially supported build.

To reference the Tidy dll methods, of course I had to add the DLL import signatures, in this case adding only the ones I had a need for.

  [StructLayout(LayoutKind.Sequential)]
  struct TidyBuffer
  {
    public IntPtr bp;           /**< Pointer to bytes */
    public uint size;         /**< # bytes currently in use */
    public uint allocated;    /**< # bytes allocated */
    public uint next;         /**< Offset of current input position */
  };

  class FileClean
  {
    [DllImport("libtidy.dll")]
    public static extern IntPtr tidyCreate();

    [DllImport("libtidy.dll")]
    public static extern int tidyParseFile(IntPtr tidyPointer, [MarshalAs(UnmanagedType.LPStr)]string fileName);

    [DllImport("libtidy.dll")]
    public static extern int tidyParseBuffer(IntPtr tidyPointer, ref TidyBuffer tidyBuffer);

    [DllImport("libtidy.dll")]
    public static extern int tidyCleanAndRepair(IntPtr tidyPointer);

    [DllImport("libtidy.dll")]
    public static extern int tidySaveFile(IntPtr tidyPointer, [MarshalAs(UnmanagedType.LPStr)]string outFileName);

    [DllImport("libtidy.dll")]
    public static extern int tidyRelease(IntPtr tidyPointer);

    [DllImport("libtidy.dll")]
    public static extern int tidySetCharEncoding(IntPtr tidyPointer, [MarshalAs(UnmanagedType.LPStr)]string encoding);

    [DllImport("libtidy.dll")]
    public static extern int tidyOptSetBool(IntPtr tidyPointer, int value, int Bool);

    public static bool CleanFile(System.String outputfileName, System.IO.MemoryStream docDataStream)
    {

      int result = -1;

      IntPtr tidyPointer = tidyCreate();
      try
      {
        // We want the resulting file to be UTF8 encoded
        tidySetCharEncoding(tidyPointer, "utf8");

        byte[] docDataArray = docDataStream.ToArray();

        TidyBuffer tidyBuffer;
        tidyBuffer.size = (uint)docDataArray.Length;
        tidyBuffer.allocated = (uint)docDataArray.Length;
        tidyBuffer.next = 0;

        GCHandle pinHandle = GCHandle.Alloc(docDataArray, GCHandleType.Pinned);
        try
        {
          tidyBuffer.bp = Marshal.UnsafeAddrOfPinnedArrayElement(docDataArray, 0);

          if (tidyParseBuffer(tidyPointer, ref tidyBuffer) >= 0)
          {
            tidyOptSetBool(tidyPointer, 29, 1);
            tidyOptSetBool(tidyPointer, 23, 1);
            if (tidyCleanAndRepair(tidyPointer) >= 0)
            {
              result = tidySaveFile(tidyPointer, outputfileName);
            }
          }
        }
        finally
        {
          pinHandle.Free();
        }
      }
      finally
      {
        tidyRelease(tidyPointer);
      }

      return (result == 0);
    }
  }

Most of this should be self-evident, however the two tidyOptSetBool calls may be a little cryptic. For the sake of brevity I haven't used the constants, but 29 is the TidyMakeClean value of TidyOptionId enum (see tidyenum.h), and 23 is the TidyXhtmlOut value. Together these indicate that I want to clean the documenting, converting it to XHTML. Note that I've also set the encoding to UTF8.

Voila, after transforming the RSS to the memory stream as quasi-conformant HTML, I passed the stream to this function, along with the desired output filename, and out went a cleaned-up, valid XHTML document. Pedants everywhere were thwarted from pointing out minor deviances from the standard. I could have processed to another buffer, and then done follow-up processing in .NET as well, but this was sufficient.

This is a trivial example, but it really exemplifies the great value of the easy interoperation of .NET. With it I could instantly leverage existing code, without having to search out bastardized ported versions, and instead could go right to the source.

We hadn't gone to the Ontario Science Centre for quite a few years (maybe 7 or 8 years), but after a mini-trip to Ottawa, and then a trip to Niagara Falls, were called off due to inclement weather (the former because it was too warm, and the latter because it was too cold), we decided to just bring the kids to the Ontario Science Centre today.

The place has improved dramatically.

7 years ago the OSC was largely full of malfunctioning equipment - leftovers from the original installation - and a lot of rather terrible computer demonstrations powered by Apple Mac Classics (though of course they weren't called Classics when they first made them, but they still were eye-opening to a lot of visitors years back, when the idea of a mouse controlled GUI was pretty novel).

A few of the better old displays still exist, include some Mac Classics still going strong, but they are now supported by a lot of fabulous new displays that kept our 3 year old overloaded with excitement until closing time. She is on the young end for a visit to the OSC (although the KidSpark area was perfect for her), but even much older children were having a blast. Parents were learning something as well, and indeed parents were often the first to start pressing the buttons or pulling the levers on the interactive displays.

All in all it really has improved, and was a very enjoyable visit. It's great to see such an attraction getting a facelift (much of it is ongoing, with several huge new improvements being built right now).

Now if Toronto could get a real world-class Aquarium.

I don't image myself a great photographer by any means -- I've simply learned the rules of framing a shot, along with the technical details of capturing it well. Given that I take a lot of pictures, invariably some of them turn out quite nice (and a lot turn out terrible -- oh thank you digital camera technology. This was much more expensive with the 35mm).

Really, I consider myself a photography technician (in this case capturing a visit to the Science Centre. All shots in this entry were shot at ISO 1600 with no flash. Aperature was usually fairly fast - f/5.6 or so. Shutter speed varied between 1/20th to 1/60th of a second, so a firm, steady grasp of the camera was paramount), and my best shots aren't themselves a form of art, but are simply capturing the beauty of the world around us. Beautiful moments we see every day.

In any case, when asked to provide photography advice I have one peeve that I always mention: Flash photography.

Flash lit pictures are, as a general rule, terrible. They usually feature a starkly lit subject - one that's often both overexposed and underexposed at the same time -- overlaid atop a barely visible background (almost detached from it, as if it was photoshopped in). Most ambient lighting has disappeared, replaced by the cold, artificial light of the flash. Shadows are cast across the scene (the further the flash is from the lens, the more of a visible shadow there will be, yet the closer the flash is to the lens, the worse that direct reflection and red-eye becomes). 

My general disdain for flash photography was one of the reasons why I bought the Canon Digital Rebel XT -- It features up to a 1600 ISO (which refers to the sensitivity of the sensor - in this case equal to 1600 film, which is very "fast" film. Many sensors have amplifiers that you can engage to increase the ISO, but it also increases sensor noise), with a slightly grainy but otherwise excellent picture at that speed, coupled with "fast" (lots of light) lens options. I've captured many wonderful pictures simply by setting the ISO to 1600 and disabling the flash, holding the camera as steady as possible, and then taking multiple shots with the hope that one of them turns out.

These were all pictures that would have turned out terrible with a flash, but instead I've captured gorgeous ambient light streaming in, colour reflections off of the wall, and the detail of the environment around the subject. As sensor technology improves, and we get even higher ISOs with reduced noise (and larger sensors that make use of more of the light), a lot of great photographs will result.

Of course there is a place for flash photography: It's critical for capturing fast moving moment indoors, professionally staged scenes with lots of professional lights, and even a skillful use of fill flash on a harshly lit day outside; however flash photography in general, in particular indoors, is grossly overused, turning a lot of beautiful photographs into so-so snapshots. 

The sensitivity of a camera and lens combination is a selling point that gets far less attention than it should.

If you were expecting me to say something about Macromedia Flash...well I aim to please. Check out this link.

http://www.honda.ca/HondaCALive/noFlashEng.htm

That's what you get if you visit Honda Canada and you don't have Flash (or more correctly if their shoddy script can't detect Flash. I have Flash in Firefox, but alas I'm redirected to this "too bad" page). While I can appreciate elements of the site being implemented in Flash (e.g. the car configurator), forcing Flash to see anything at all is amazingly ignorant. The wording almost implies that without Flash you're not welcome to Honda cars, which is just astounding hubris.

22 years ago my home computing experience was defined by my eldest brother's Commodore 64, along with an old flat-keyboard Atari 400 (take a look at that keyboard. Imagine spending hours typing in entire applications, your fingers pulsing in pain. Now imagine your neighbour curiously pushing the button that opens the cartridge cover, an action which hilariously also turns off the power. That happens to be a very destructive act when you have no persistent storage). Primary school computing consisted of programming stick figures bouncing o's on Commodore PETs.

These were pretty exciting times in personal computing, with seemingly endless potential, and a wide range of publications had appeared to cater to the burgeoning market. One of the most successful of which was Compute!, a magazine that stood apart by being largely platform agnostic (versus the rags that catered to a specific zeal, advocating why, for instance, the Ti-99/4a was the greatest computer ever).

I headed to the library monthly to read each issue, eager to see what new innovations were happening in the industry. These scans are from the February 1984 issue (if you're price comparing with today, note that the CPI inflation since 1984 is 1.86x), one of many that I bought on eBay some time back to use as office wall art, as obviously this was a very shaping period of my life.

Each Compute! issue featured one or more type-in game or application (as did most other home computing magazines), often ported to several platform. Sometimes they were BASIC, while other times it was a BASIC loader followed by pages of machine language. Fun times, and I have a great memory of some of these games. I also have memories of spending hours typing in a game to have it lock up when we went to run it.

These type-in applications were actually my original motivation for learning to program. My dream was to eventually submit my own application, imagining the glory of having thousands of people across the land typing my work into their computers. I would be a hero to people everywhere!

Of course the whole type-in fad faded before I had a chance for such illustrious glory. For a short while there was a standard for printing programs as 2D barcode patterns, and with the appropriate scanner you could scan it right off the page. Neat idea, but some technical difficulties kept it from taking off.

Several trends seem obvious looking through this magazine. Educational software of all sorts was a huge market in those days (and it seemed to be the greatest intended use for home computers), with Spinnaker being one of the largest vendors.

And of course there were games. SSI was a significant publisher of games, featuring several ads in every issue. Though they originally started as a wargame company (hence the Strategic Simulations company moniker), they began generalizing into all sorts of (mostly terrible) games.

It's also evident that the still struggling "everyone will be a programmer" philosophy was strong. BASIC, for instance, was intended as a language that everyone would use to achieve their computing needs -- certainly not as the primary language of many corporate developers -- with the whole family writing programs to do what they needed to get done.

CompuServe was the dominant pre-Internet, allowing users in several countries to communicate, participate in discussions, and even play primitive multiplayer online games. Of course it was insanely expensive, so I stuck to local BBSs.

And of course one of the most interesting sections of these sorts of magazines were the mail-order shop ads (a trend that reached its pinnacle with Computer Shopper several years late -- 800 pages, 99% of which were full-page ads).

I hope you enjoyed this trip down memory lane, or in many cases pre-memory lane. I had to scan these images anyways, so I figured I might as well share. I've stuck to ads -- rather than actual magazine content -- to avoid copyright issues.