A couple of years back I wrote a short piece titled "Edit and Continue - Valuable Tool, or Sloppy Vice?"

I pondered whether some development tools and practices -- such as test-driven development (TDD) and the reduced cost of errors (in both time and personal reputation) -- were actually making us worse developers, paradoxically decreasing productivity and the suitability and correctness of solutions.

That entry was motivated by the outpouring of demands by my peers that a particular tool continue to feature edit-and-continue functionality: what I thought would be an infrequently used frill turned out to be something that many depended upon daily, correcting their flawed code at runtime as a regular part of their process.

Today I came across DevGrind's How not to solve a Sudoku entry -- itself linking to Ravi Mohan's "Leaning from Sodoku Solver" -- where he links to a gent who implemented a thoughtful, sober design carefully, and another who pursued a TDD-approach, building his test harness, and then, it appears, flailing about madly in the hopes that some random keypresses will generate a solution that passes the test.

To demonstrate the value of TDD.

...Today's blast from the past is To GUID or not to GUID in your Database, where I describe the benefits and pratfalls of GUIDs in the database.

Nearing the end of 2006 I put up a bit of a rant-- Two-Factor Authentication, Hashing, and Cell Phone Restrictions / J2ME -- concerning two-factor authentication, and the difficulties implementing a simple, no-cost solution on my handy new Motorola cell phone. I pretty much gave up in frustration, the many barriers and limitations just making it not worth the trouble.

I recently started using a Motorola Q, based upon Windows Mobile 5.0 and running on the Bell Canada network, and I have to say that the situation is night and day -- developing and deploying either native or .NET Compact Framework apps on it is ridiculously easy (and incredibly well supported in Visual Studio 2005 with the SDK add-ins), easily using the data network to communicate with sites over the net, and so on.

Absolutely wonderful device for developers and enlightened shops.

Microsoft recently released a new technology/product named Silverlight. Many in the Microsoft-enthusiast community, seemingly only seeing the world that Microsoft delivers to them, cheered in applause at this innovation, declaring that it completely changed the rules of the game: Soon we would see fat apps and web apps dancing in the streets together, no longer segregated.

The world was freed from the awful tyranny of HTML!

Microsoft, the story goes, innovated up some vector graphics and a way to interact with and transform them from script, changing the world from the boring, raster-graphics monotony of yesteryear.

Amazing!

Wait -- wasn't SVG already a pretty mature technology at the turn of the century? Why yes, now that I think of it, I recall deploying solutions that actually used glorious vector graphics, declaratively described and transformed with layers of vector goodness. Of course Microsoft, despite being on the SVG committee, didn't actually support it themselves (by then they realized that the whole web thing was probably unhealthy to their operating system stronghold, so such an incredibly rich addition to the browser was unlikely to occur), and instead one had to use third-party plug-ins like Adobe's SVG Viewer. Then of course the banner was passed to the Macromedia cum Adobe Flash product, where rich, cross-platform, vertically-scaled vector graphics rendering primarily occurs today (though SVG is revived through its inclusion in the standard release of Firefox, and is becoming a standard graphic format for a certain domain of images on Wikipedia).

I imagine the brainstorming session that yielded the name "Silverlight" consisted of some people thinking "We're innovating up a Flash clone, so think of anything that reminds you of Flash". Silver light, aka a light bouncing from a silver-color flash enclosure, is hardly a stretch.

The product itself might be entirely worthwhile, but thus far nothing I've seen makes it any more compelling than Flash. Indeed, the limited targeted platforms of Silverlight -- both from a breadth and depth perspective, the rank immaturity of the product, the grossly conflicted interests of its host, and the massive ubiquity of Flash makes Silverlight a contender that needs to really wow to sell itself.

Nothing I've seen wows me any more than the existing Flash demos wow me (and even those aren't very compelling, and many Flash implementations are abusive overkill). The asinine sales video certainly didn't convince me.

All of that is pretty much irrelevant, though: Microsoft wants to take a run at Adobe, hoping to get some strategic control to lever in the future. That's fine, and if the product really does provide some sort of advantage above and beyond Flash to excuse the massive disadvantages, then it's definitely a product worth considering.

What really gets me irked, though, is the perceptive of some of this industry to only see the merit of certain solutions when they come from their camp. I recall when .NET was in the initial betas and having a breathless, excited peer, fresh from their Visual Basic 6 nightmare, declaring the wonderful advantages that garbage collection, JIT compilation, reflection, and so on brought to development.

"If it's so great now, why haven't you been embracing Java for years?" I asked.

He had no answer. I truly think he was completely unaware that what he was describing was already available. This sort of virgin enthusiasm, with cheerleaders completely blind to other solutions available to them, is far too common.

WPF itself is a decent addition to the .NET Framework (although the name ".NET 3.0" is absurd), and it does add a lot of whiz-bang, but they need to do a better job of selling it than videos like this. For all the talk about improving user interaction during that video, everything they showed managed to slow user interaction, reduce data density in a detrimental way, and overall just add junk that detracted from the experience of the app, but remarkably it's what people always first try to use to sell things like this.

Hugh Macleod recently asked "How Well Does Open Source Currently Meet The Needs of Shareholders and CEO's?" It's an illogical, inconsistent post that should have scrolled out of the memory of mankind, but somehow it has been mentioned far and wide, including a mention on the popular Coding Horror, among other highly ranked blogs, many of which just muddied the waters of this misleading question even worse.

The original post by Hugh was baffling, first confusing the Microsoft Partner program with gross Microsoft sales, and then completely hodge-podging software as a commercial product with software as a consumable.

There aren't any open source billionaires selling software as a product, Hugh tells us, so therefore open source doesn't service shareholders and CEO's (of non software-as-a-product organizations) as a consumable.

Huh?

Hugh isn't even commenting on the financial viability of building a software as a product company around open source, where such a question merits sober consideration (though there are quite a few very successful open source businesses, whether a company can survive and thrive with their crown jewels open sourced needs to be considered on a case by case basis). Instead he's claiming that every organization needs to ensure that their suppliers are making boatloads of loot (preferrably suppliers with an ownership structure that centralizes it on one or two individuals), which is a rather odd consideration. Or worse, that their suppliers share some nonsensical correlation with someone else that is very successful.

Open source is a very contentious topic, and to many it is threatening to their world. It is often grossly oversold as a silver-bullet, when often it only plays a marginal part (both Firefox and MySQL, as examples, are overwhelmingly developed by a traditional team in a traditional way, and the source just happens to be available. It really isn't the driving force of their march forward).

Yet to correlate software as a consumable so arbitrarily holds no value whatsoever, and isn't a good foundation for any discussion.

And most remarkably it uses Bill Gates as the example of the merit of non-open source software as a consumable -- yet Bill Gates is perversely probably the single biggest reason the open source market is so vibrant and alive. The anti-Microsoft rallying cry was a critical early factor in the growth of the OSS community.

The success of Vista is of obvious importance to any developer targeting any Vista-only or Vista-enhanced technology, so the latest news that Vista sales are strong and higher than expected is of obvious interest.

It appears that Vista has been a stunning success for Microsoft, surprizing even the optimists inside the Redmond empire. This stands is a stark contrast with reports of general consumer apathy about this release, and many dire predictions about Vista's adoption (predictions that are dubious, given that Vista is pretty much assured a reasonable level of success given its automatic sale to virtually anyone buying a new PC. Corporations naturally delay adoption of new operating systems, as they have with every prior edition, so only a delusional expected it to storm across business desktops).

You can see their Q3 return on the SEC site. Under the Client division, you can clearly see that revenue has rocketed up, from $3.151 billion in Q3 2006, to $5.272 billion in Q3 2007.

There are a couple of massive, almost-Enronesque caveats to these numbers that deserve some serious scrutiny before you start your Vista-only product launch.

Firstly, they deferred $1.2 billion of XP sales from Q1 and Q2 2007 -- sales that qualified for the upgrade coupon -- rolling it into this result.

Client revenue increased for the three months ended March 31, 2007, primarily reflecting licensing of Windows Vista, including recognition of approximately $1.2 billion of revenue previously deferred in fiscal year 2007 pending the January 2007 release to consumers.

Counting out that rather dubious bit of accounting trickery drops the gross revenue to a growth of just $0.8 billion over the year earlier.

Secondly, some of the gain is attributed to the price premium applied to Vista Premium (effectively imposing a price increase over prior XP licenses, as most PC makers automatically deliver the Premium edition.)

During the quarter, the OEM premium mix increased 18 percentage points over the prior year to 71% driven by the demand for Windows Vista Home Premium.

Not to mention that the computer market in general has grown over a year earlier.

Based on our preliminary estimates, total worldwide PC shipments from all sources grew 10% to 12% from the third quarter of the previous year and approximately 8% to 10% from the first nine months of the previous year driven by strong consumer demand in both emerging and mature markets.

Not quite as successful as some reports are claiming: After a year of market growth, subtracting the hard-to-rationalize rolling-forward trickery, and considering that the price for the operating system was effectively raised via the Premium edition, and suddenly the situation doesn't look quite as rosey.

Rough back-of-a-napkin calculations have OS unit sales remaining relatively flat after incorporating in market growth. If this exceeded inside-of-Microsoft estimates, then clearly they're either lying, or they were expecting an implosion. Applauding about Vista's lofty percentage of OS sales should be quelled by the reality that virtually every new client PC OS sold now is Vista -- if a corporation does want XP, their recourse now is to buy a Vista license that grants them the right to install XP, though it'll carefully get added as another vote of confidence by the Microsoft beancounters.

There are several other surprizes in their Q3 report. For instance that the entertainment division (Xbox, Zune) saw revenue drop over a year earlier, and that the Online Services Business -- this is where Microsoft put a lot of attention recently, with the huge push of the Live platform -- saw a marginal revenue increase, with a significant loss increase.

The only bright spot of the whole quarterly report, from my perspective, is the business systems division: Office 2007 has seen good adoption and has very healthily contributed to earnings.

Reports that this quarterly report validates Vista's success are unfounded. Further, it puts a huge question mark over Microsoft's web and hardware initiatives (the complete failure of the diversification to actually add to the bottom line, instead of just drowning in losses -- excused early on as toothing pains, but there doesn't seem to be a point when they'll actually make money -- should raise serious concern. Microsoft is still held aloft by Windows and Office).

I mentioned Microsoft's new Silverlight platform a week ago, opining that it appeared to be nothing more than a Flash competitor...despite all the noise, enthusiasm and bluster that comes out of the gathering of the faithful (Mix '07).

If Flash wasn't already the answer for a particular problem space -- and it seldom is necessary or beneficial given the incredible power and versatility of the foundational web technologies -- why would Silverlight change the equation?

For those instances where Flash is an appropriate solution, the question then is what Silverlight brings to the table that Flash doesn't? What could possibly overcome the new entrant's already limited platform coverage and concerning vendor conflict of interest? Is Silverlight really just another attempt at pushing Windows Media, released as a last-ditch attempt to recapture some of the marketshare that WMV has lost, largely to Flash of all players.

The answer came as Mix progressed, when it was announced that the Silverlight platform will optionally come equipped with a .NET Common Language Runtime (the common language of course being Common Intermediate Language), equipped with a greatly pruned down Framework: the runtime and the framework occupy 2MB, so clearly it's only a tiny subset of the full .NET 2/3 framework.

Supposedly the CIL really is the same CIL used in our desktop and server .NET apps, so hypothetically a given assembly can be used on the desktop, in Silverlight on Windows, and even on Silverlight on the Mac (or any other platform that Microsoft decides to target), which is interesting.

In practice it's never that easy. The minimized framework included with Silverlight means that a lot of code will have dependencies that aren't satisfied in Silverlight, similar to the experiences many will have attempting to run .NET code on a Mono platform (despite Mono having a vastly larger Framework to draw from).

What it does mean, however, is that code (which before deployment will be "compiled" from whatever source language to the remarkably high-level Intermediate Language, much like you can use tools to "compile" Java to JavaScript, or Wasabi to PHP, or between any other Turing-complete language) run within Silverlight will likely demonstrate excellent performance: The .NET team has done a superb job of teasing great performance out of managed code, and I would expect that to carry over to Silverlight as well.

Silverlight was completely uninteresting before, but this new information makes me somewhat curious, and it does technologically differentiate the product.

This should light some fires under the team incorporating the JavaScript Tamarin engine (which humorously came from Flash) in Mozilla products. As it currently stands, the JavaScript interpreter in Mozilla is among the performance frontrunners, however that pack in general is proving to be unacceptably slow as the web becomes increasingly rich.

The Tamarin engine, and similar designs, hyper-accelerates things by doing JIT compilation of ECMAScript Edition 4 (JavaScript 2) to native code where possible, offering potentially enormous performance improvements.

Back to Silverlight, I laugh at those who are giddy with glee about the death of "HTML", or the coming of "Web 3.0" (which is an embarrassing comment being bandied about in regards to Silverlight).

Were Java Applets Web 3.0? Was ActiveX Web 3.0? Was VRML Web 3.0? Flash/Shockwave have been with us since Web 0.9.

Put a Remote Desktop ActiveX control on a page and people will declare it Web 3.0. Who knows, maybe Google will do just that.

Many garbage-collected languages feature "immutable" strings -- once a value is assigned, it can't be manipulated without creating a new instance (either explicitly or implicitly), with obvious performance implications. Consider the following operation, concatenating the string variable B and C into the resulting string A.

A = B + C;

It's for this reason that string-concatenation is often a ghastly slow affair. Many developers have faced this when building a string through concatenation in a loop, for instance dynamically generating a TABLE and its content, or building an XML document.

Consider the following:

myString = myString + "A";

As visualized above, generally that would cause the runtime to create a new memory allocation large enough to hold the concatenated length, then copy the first value and then the second value to the new buffer.

In a loop, where a string is continually adding new content to the end of itself -- a very common need -- the performance will generally take quadratic time (as it's repeatedly allocating and copying a larger and larger string).

I've encountered this frequently enough in .NET, deciding when the ugliness and heft of a StringBuilder is necessitated versus when to just suck up the terrible concatenation performance, however this came up recently in a web project requiring a large amount of display data to be dynamically constructed on the client end.

JavaScript doesn't have a native string builder (a big oversight), so I needed to evaluate how basic string concatenation compared with some of the hackish string builder similes for disparate cases. I was quickly reminded that while Firefox doesn't have great string concatenation performance, it absolutely annihilates Internet Explorer (incl. 7) doing the same.

Try this basic benchmark in both browsers to see for yourself: On the machine I'm typing this on, Firefox completes 131,072 iterations in 1984ms, while Internet Explorer had only completed 32768 iterations after 11078ms (not only did just 1/4 the iterations take 5 times as long, but the iterations it did complete were the smaller, much easier ones, making the disparity much greater than it appears).

I became curious how Firefox implemented string concatenation so performantly, and from that to extrapolate how the Internet Explorer team screwed it up so royally, and honestly to see if I could easily tease even more performance out of it. Given that we have the source (and the ability to easily change it and build it to test differences), I jumped into js/jsstr.c in the Firefox source to take a look (all code subject to the MPL and the Mozilla Foundation retains all rights).

JSString *
js_ConcatStrings(JSContext *cx, JSString *left, JSString *right)
{
    size_t rn, ln, lrdist, n;
    jschar *rs, *ls, *s;
    JSDependentString *ldep;    /* non-null if left should become dependent */
    JSString *str;
    if (JSSTRING_IS_DEPENDENT(right)) {
        rn = JSSTRDEP_LENGTH(right);
        rs = JSSTRDEP_CHARS(right);
    } else {
        rn = right->length;
        rs = right->chars;
    }
    if (rn == 0)
        return left;
    if (JSSTRING_IS_DEPENDENT(left) ||
        !(*js_GetGCThingFlags(left) & GCF_MUTABLE)) {
        /* We must copy if left does not own a buffer to realloc. */
        ln = JSSTRING_LENGTH(left);
        if (ln == 0)
            return right;
        ls = JSSTRING_CHARS(left);
        s = (jschar *) JS_malloc(cx, (ln + rn + 1) * sizeof(jschar));
        if (!s)
            return NULL;
        js_strncpy(s, ls, ln);
        ldep = NULL;
    } else {
        /* We can realloc left's space and make it depend on our result. */
        ln = left->length;
        if (ln == 0)
            return right;
        ls = left->chars;
        s = (jschar *) JS_realloc(cx, ls, (ln + rn + 1) * sizeof(jschar));
        if (!s)
            return NULL;
        /* Take care: right could depend on left! */
        lrdist = (size_t)(rs - ls);
        if (lrdist < ln)
            rs = s + lrdist;
        left->chars = ls = s;
        ldep = JSSTRDEP(left);
    }
    js_strncpy(s + ln, rs, rn);
    n = ln + rn;
    s[n] = 0;
    str = js_NewString(cx, s, n, GCF_MUTABLE);
    if (!str) {
        /* Out of memory: clean up any space we (re-)allocated. */
        if (!ldep) {
            JS_free(cx, s);
        } else {
            s = JS_realloc(cx, ls, (ln + 1) * sizeof(jschar));
            if (s)
                left->chars = s;
        }
    } else {
        /* Morph left into a dependent prefix if we realloc'd its buffer. */
        if (ldep) {
            JSPREFIX_SET_LENGTH(ldep, ln);
            JSPREFIX_SET_BASE(ldep, str);
#ifdef DEBUG
          {
            JSRuntime *rt = cx->runtime;
            JS_RUNTIME_METER(rt, liveDependentStrings);
            JS_RUNTIME_METER(rt, totalDependentStrings);
            JS_LOCK_RUNTIME_VOID(rt,
                (rt->strdepLengthSum += (double)ln,
                 rt->strdepLengthSquaredSum += (double)ln * (double)ln));
          }
#endif
        }
    }
    return str;
}

The key to Mozilla's performance is this line, and the surrounding concept of "dependant" strings:

s = (jschar *) JS_realloc(cx, ls, (ln + rn + 1) * sizeof(jschar));

In a nutshell, when you perform the operation A = B + C, where B and C are strings, it reallocates the memory assigned to B -- depending upon the memory manager and fragmentation, this can often be done in place with no memory copy -- setting B to indicate that it is now a dependant string (e.g. "don't free the associated memory when you garbage collect B"), then copying C to the tail.

A and B are now pointing to the same location in memory. In a loop where a string is adding to itself, it is just constantly realloc'ing -- copying to a new memory location when the memory manager can't expand the requested amount in place, otherwise just doing a land grab of memory trailing the string -- and copying the new data on the end.

Changing this bit of code to do a straight new malloc and copy on all concatenations yielded an executable with Internet Explorer-like glacially slow string concats, exactly as expected.

Which made me curious: What if we predictively accommodated the often growing nature of strings, allocating a bit of extra space at the outset, making it more likely that realloc's could be accommodated in place? In the grand scheme of things, the memory taken by JavaScript strings is marginally small, so a little extra padding on result-of-concatenation strings shouldn't hurt.

I implemented the following function in js/jsstr.c.

size_t js_GetNextSize(size_t desired_size)
{
  int max = 0, i;
  
  for (i = 1; i != JSSTRFLAG_DEPENDENT; i <<= 1)
  {
    if (desired_size & i)
    {
      max = i;
    }
  }
  if (max)
  {
    return max << 1;
  } else
  {
    return desired_size;
  }
}

That's nothing more than a trivial implementation of ceil(log2(n)), however I tried to implement in the most transparent, cross-platform way possible.

Now I changed the realloc line referenced above to read:

s = (jschar *) JS_realloc(cx, ls, js_GetNextSize((ln + rn + 1) * sizeof(jschar)));

All reallocs will be to the next power of 2 of the requested size -- if a realloc demands 37 bytes, it'll allocate 64 bytes, and if it demands 7 bytes then it'll allocate 8 -- trading a relatively small amount of memory usage for processing time and memory bandwidth.

Re-running the benchmark linked earlier now shows my custom build of Firefox completing 131,072 iterations in 60 to 90ms (versus 1984ms before, so it completed it in less than 1/20th the time of the already speedy Firefox. Internet Explorer would take minutes to perform that many iterations). Firefox memory usage has not changed to a measurable degree, even during heavy browsing.

This change -- allocating a little extra space to accommodate realloc calls -- is likely the technique used in Opera 9, as this turbo version of Firefox was neck and neck with the speedy Opera 9 on this bit of functionality.

Now to delve through other parts of the excellent JavaScript code...