A classic, forever repeating quandary when designing applications that store large numbers of data files (images, user files, supporting documents, etc) is whether to store the files in the database, or to store them in the filesystem with pointers in the database.

Consider a web application that tracks support tickets, where users can add supporting documents to their tickets. In many implementations the files are uploaded and stored in a filesystem location with a unique name or directory (often a GUID), and that unique filename is stored in the database correlated with the record. This means that record access necessitates both file system access along with the database access.

There are significant disadvantages to this, including the lack of transactional integrity of the filesystem objects, the difficulty of management (trying to coordinate file system and database backups to be able to restore to a consistent state), the security issues, the lack of relational integrity (files could be deleted, records could be deleted without cleaning up the related files, and so on), among others.

On the flip side, the advantage of this technique is reduced load on the database server (e.g. you could offload file storage to a very large scale NAS device), as well as immediate file system access where appropriate (e.g. an administrator needs no special tools to browse the files, although this could be considered a detriment as well). Many developers find it an easier model to implement using the file system for supporting files.

For those who prefer the file system in such scenarios, the transactional integrity deficiency of this technique will be fixed in Windows Vista (formerly Longhorn) and related technologies - It is introducing a transaction-capable variant of NTFS (TxF). NTFS is already a journaled and reliable filesystem, however TxF will add the ability of the filesystem to participate in distributed transactions - both intra-machine, and inter-machine - with standard two-phase commit functionality. This means, for instance, that when the user is adding the record that includes a supporting document, the file and record could be created under a shared transaction in the middle tier (or even in the database if you use it as a conduit, storing and retrieving filesystem objects in the database logic), and if either fail they both fail (avoiding dirty data). Add this with the easy ability to add complex database logic to probe and validate the correlating file system (now that SQL Server 2005 can host .NET functionality, meaning that your trigger can more robustly check for file existance when records are created, and delete them when they are removed), and it becomes a much more credible option.

(As an aside - Distributed transactions - transactions across heterogenous resource managers - have traditionally been very, very slow. This new file-system transaction functionality most certainly isn't free, but where the reliability is critical - which is almost always given the cost and uselessness of dirty data - it can represent a great improvement. Registry changes will also be boundable in distributed transactions)

Supporting Links
MSDN Page
Channel 9 Video on TxF (Given that the long form name is the correctly descriptive Transactional NTFS, shouldn't the abbreviation be TxNTFS? Longer to say, but it seems superior to me)

Tagged: [Software Development], [Programming], [Software-Development], [Vista]

One of the benefits of being in the industry for a few years (I've been professionally developing and providing system consulting services for 12 years now, and of course was in the amateur ranks for a decade before that) is that you get to see history revised firsthand. This is especially true in the web app world, where the history of the platform is being rewritten by people who want to change it for their own gain, or who simply weren't involved in the industry and thus have incomplete knowledge, rewriting it purely out of ignorance.

A frequent loser in this rewriting is Microsoft: Whether it's imagining Microsoft to be a web app laggard (I was developing for the Microsoft technology stack, making web apps that blow away what people are amazed by today...6 and 7 years ago. Microsoft was a web technology superstar, but because most shops remained committed to fat apps, or wanted cross platform capabilities, few embraced their innovations), or having no influence (a lot of the current platform was either invented or implemented first by Microsoft. From IFRAMEs to most of CSS to XMLHTTP. Others like behaviors and filters died an ignoble death). While Microsoft is far from a perfect netizen, a lot of what they did has significantly and positively affected the web that we use today.

Rumor has it, and I am prone to believing it, that the web app platform was getting so powerful that the Internet Explorer team was disbanded: It was becoming capable enough that many corporations were switching many of their in-house applications to web apps, and the worry was that even with IE-only web apps, tied to IE-specific functionality, it was just a short jump to making them cross-platform (or allowing for parallel, slightly less capable cross platform options), dramatically reducing the lock-in of the Windows platform.

In any case, one Microsoft technology that is being particularly maligned is the infamous ActiveX.

Of course the term itself is a bit of a mess, and offers a classic example of Microsoft marketing gone awry (just like the disaster of naming that was .NET. If people weren't fired over that debacle, then justice wasn't served) - According to some Microsoft sources, ActiveX was a set of interfaces that could be added to a COM (Component Object Model) object to allow it to interact with the interface of an application. Generally encapsulated in .OCX files (Ole Custom Controls), these provided a replacement to the venerable VBX controls of yesteryear, providing a binary, language-neutral visual control that could be used in any ActiveX environment: Whether a Visual Basic app, a Delphi app, a MS Access form, an Excel worksheet, or a Visual C++ app, you could make use of a single ActiveX control. At one gig we needed two synchronized animated graphs showing engine performance for a tradeshow presentation - one quick Delphi ActiveX control later, and it was in the presentation (integrated right in the PowerPoint) and working great. That was the power of ActiveX.

ActiveX was also the technology behind plug-ins in Internet Explorer - Instead of begging the Netscape cabal to let them into the inner circle of Netscape plug-ins, ActiveX controls could be created by anyone and used in web pages (presuming some security hurdles were jumped, such as getting the controls signed). It was a free and open world for web extensions, and of course they proliferated by the thousands, though only a few remained when the dust settled.

Another definition is that ActiveX refers simply to COM controls themselves - if it's a COM control, then it's an ActiveX control. Another variant is that ActiveX refers to COM controls marked "Safe For Scripting".

In any case, COM was a great advance for the platform. It provided high performance, binary, language neutral, object-oriented controls that could be used throughout the system in a truly modular fashion. They could even be proxied across systems, or hosted in service modules (MTS which became component services).

Seeing the value of this powerful, extensible, system-wide technology, the Internet Explorer team decided to implement a lot of its functionality via this mechanism - So long as you configured it with the proper registry entries, and optionally implemented an interface stating its safety level, these components were usable from scripting in Internet Explorer. An obvious, and incredibly powerful, example was the use of the XMLHTTP component (a part of the MSXML library, which itself is a variety of COM controls) from within Internet Explorer. Independently both sides could be upgraded and changed, automatically benefitting the other side where desired. If you implemented visual controls, you could implement specific functionality that couldn't be handled with traditional web technologies in something like Delphi or MFC/C++, and gain all of the advantages of the web model (such as the document flow layout) alongside extremely rich controls.

It helped a lot of shops start transitioning to web applications long before the web platform could do it on its own.

The problem with ActiveX, and the main reason why it's maligned (apart from the platform lock-in), is that several controls that were marked safe for scripting were not, in fact, safe for scripting: Either they were programmed sloppily, and opened holes for buffer overflow and other nefarious activities, or they had dangerous operations that should never have been allowed from within Internet Explorer. Whatever the case, they opened holes that shouldn't have been opened.

Specific implementations gave the whole technology - a modular, high-performance and highly extensible system - a bad name. It could be said that it deserved it, given that it didn't sandbox the operations of the scripted object, but that's an implementation detail: At the core it really is a fantastic foundation.

Tagged: [Software Development], [Programming], [Software-Development], [ActiveX]

Back on September 13th I declared that SVG was a dead technology. Since then, the release of Firefox 1.5, along with the free-as-in-beer state of Opera - both featuring native SVG rendering engines - has really spurred SVG activity. I've been getting dozens of SVG related search hits here a day, and that's for an old article that I wrote back in 2002. It could be that the community finally caught onto this fantastic technology.

SVG might not be dead afterall.

Tagged: [SVG], [Programming], [Software-Development]

Opera's Unwanted Functionality

I was just doing a bit of work in the Opera web browser, typing some information into a web app's text box, when I accidentally de-selected the input box in the process of jumping between applications. On my next keystroke the interface suddenly went to an archaic layout. It looked like something rendered in Netscape 3.

I had no idea how I did this, it was completely unwanted, and the impact was extremely disruptive. Closing and restarting the application didn't remove this sticky setting, and randomly (and systematically) selecting what I thought would be the accidental shortcut keys yielded no solution.

Now I had to waste time finding, and then turning off, a feature that I didn't want in the first place.

This brings to mind a couple of user interface issues:

• All applications should have an undo stack (a transaction log, of sorts) that clearly logs and lists every notable app change - minimizing, maximizing, resizing, or changing options like rendering mode. If there was such an undo stack, I could jump to it, see the "Switch to User Stylesheet Rendering" transaction just occurred moments ago - maybe even with a help link to see what that event type was and how I triggered it - and I could roll it back. This is something I've wanted for years (neigh...decades) but of course have never had the time or resources to implement such a non-standard technique in my own apps.
• Having options like this - switching rendering modes - configured for a standard key (in this case Shift-G) is questionable, and given the severity of the change it should really ask for a confirmation of use. "You have selected the shortcut to switch to User stylesheet mode (Shift-G). Are you sure?" (with a checkbox allowing brave users to avoid the confirmation in the future). It's a bit of education and avoidance wrapped up in one.
• Opera doesn't indicate the shortcuts on many of their menu items, including the style mode menu item (the culprits in this case), making it difficult to investigate possible causes.

While there is a minority of users who override site stylesheets with their own, justifying the feature in Opera (though I'm not convinced that it should be an everyday keystroke like Shift-G by default), this brings me to another user interface observation.

Highly Configurable Interfaces are Usually Detrimental

Drawing from personal experience, I worked on a project quite a few years back where one developer insisted upon absolute flexibility in the user interface - Every toolbar had to be movable and dockable anywhere, every sidebar item drag and droppable, every menu item configurable, every UI skinnable. It was a nice cop out for us because we didn't really have to put too much thought into the interface, and could always justify it with the stock "the user can configure it how they want". Stick some more toolbars, statusbars, and panels in there because the user can clean it up according to their own needs, the logic went.

In the field, about 99.9% (more likely 100%) of the time that people discovered this functionality it was to their detriment. Like the taskbar-stuck-perilously-on-the-side-of-the-screen on your Aunt's Windows 95 computer, it was just something that happened by accident, and they didn't know how to get it back the way it was: No one (or very few) did it on purpose, but there it was terrorizing every computer user.

The first step of any support call for our app was to determine in what innovative ways the user managed to mess up their user interface. After getting a visualization of the sidebar on the bottom, the icons all on the background, the toolbar on the right, some critical toolbars hidden, with the menus all jumbled and the icons all removed, the cleanup began.

On the next release a menu item to reset the interface to the initial defaults was added, and on further releases most interface flexibility was removed (or alternately made much more difficult to do - you had to be dedicated and informed if you really wanted to change things. Someone is much more likely to unintentionally hit Shift-G with no input box focused than they are to accidentally go into the advanced preferences and set an option).

The moral of the story is that customizable interfaces are seldom beneficial, and instead function as a lazy, non-committal cop-out by the developers and designers of the application.

Even the most fundamental element of our user experience - windowing - merits some analysis: Apart from Winamp and Media Player, how often are apps in any configuration other than maximized or minimized? I run with dual-monitors, and 99% of the time one or both of them has a full screen application on them. My "windowing" is alt-tabbing through full-screen windows, and I copy data between apps using copy/paste, or, where dragging is necessary, via the taskbar.

Tagged: [Software Development], [Programming], [Software-Development], [Usability]

I've pursued various Microsoft certifications over the years, starting with the MCP, and then acquiring an MCSE and MCDBA.

My motivation in pursuing these certifications was that they served as a destination of sorts, motivating me to learn products and technologies to a breadth and depth that I wouldn't have otherwise.

The knowledge has proven very handy: Even when I serve in a development/design role (especially when I serve a development/design role) the information gained is critical in making appropriate decisions. When I serve leadership and advisory roles, again I'm glad that I spent the time going through every esoteric option and alternative, because the knowledge does help to head off misdirected initiatives and wasted effort.

For those who think "Oh, but I know all of it anyways. I am a Linux super-guru and thus I can achieve anything on the Windows platform with ease". I've heard this sort of boast before, and the results weren't pretty. Go to the Microsoft certification site and take some assessment exams - you might be shocked. The platform is absolutely huge, and it is remarkable how much of it doesn't gain our attention or focus, yet it can help us make better apps, and deploy better solutions.

"If you're a software developer and development manager, why did you get administrative type certifications?" some might ask. Very good question, and the answer is found in the paragraphs above - I dealt with the coding side all day every day, so I didn't see as much of an advantage focusing on an area that I know so well (basically it would have been hundreds of dollars for Microsoft to anoint what I'd proven amply in the field), while I (like most development focused people) didn't really pay enough heed to the platform side of things. Now that I am often called upon for platform consulting as well, it was a nice foundation to build upon.

Nonetheless, now that Microsoft has revamped their certifications, I've decided to upgrade to the MCTS: SQL Server 2005 along with the MCPD. I had hoped to get the MCTS out of the way, but it looks like the exam isn't available yet (despite a November 2005 timeline). Alas. Already it has encouraged me to focus on esoteria of SQL Server 2005 that would have gone ignored.

And for those who protest "But I don't have time! I'm a very busy person!": You could very well be running to stand still. It is an epic problem in this industry that tremendous effort is expended because people don't spend enough time on the skills side of things, focusing all of their attention on the application side.

If you're thinking of providing a demo or limited-use version of your software, pay for the bandwidth and host it yourself. It is an enormous waste of time for potential customers - not to mention that it's incredibly insulting - when you host at one of the big "make you follow 7 links, then sit in a queue, and then download a potentially tampered executable at a reduced speed" 3rd party file hosts that seem to be all the rage these days.

Bandwidth is relatively cheap nowadays, coming in at less than 8 cents a GB at many providers.

What does this have to do with social proof? Well if you host your demo or lite version at one of the aforementioned file hosts, my immediate presumption is that a very tiny percentage of users actually pay for the software: What else could justify such an abuse of clients?

Given this obvious conclusion, the power of social proof pushes me to lean against purchasing it either.

A lot of my work - both system consulting and software architecture/development - relies upon Microsoft technologies: Whether it's re-engineering a legacy system to take advantage of new SQL Server features for performance or functionality, overhauling a network infrastructure to leverage ActiveDirectory and the extensive platform security functionality, or developing a performant and scalable time-tracking application for an enterprise client, Microsoft is often a very important part of the equation.

Partly due to specialization (it's the tools we target), coupled with simply being the best choice in a lot of scenarios in our target market, we heavily rely on the Microsoft platform for ourselves and our customers. As a professional I can say with confidence that the platform is a secure, high performance, extremely scalable, robust one that compares very favourably against all competitors.

That wasn't always an accurate statement, though. Indeed, it is remarkable looking at the history of Microsoft and learning from their success: On paper it really is hard to believe that Microsoft maintained the market dominance that they did, and it's amazing that competitors couldn't capitalize on Microsoft's late entrance into a lot of markets, and their missteps in others.

Was Microsoft a master of timing, holding off on technologies and advances until the perfect time, or were they simply the beneficiary of a captive audience that was willing to wait however long Microsoft took, blind to the available alternatives?

I'll provide a couple of examples that I recall marvelling at as they occurred- these are hardly exhaustive, however I think it's a nice sampling.

Microsoft Maladies

• Microsoft core offerings were crippled by real/virtual mode limits until long after the 386 and 486 were prevalent. In a nutshell, this made software development a lot less pleasant, and the resulting applications more limited and unstable - I remember being enormously unhappy learning real-mode assembly on the x86 after dealing with the elegant, 32-bit flat world of Motorola 68000 assembly. It seemed so primitive that it still existed, or that a software company continued to rely upon it long after it was obsolete and irrelevant in hardware.
• Microsoft's "operating system" for years was simply the DOS command line, and a set of utilities and software interrupt handlers. While Mac users were busy with a rich graphical user interface, we in the DOS world were anxiously awaiting fantastic new features like DELTREE, and maybe a new version of EMM386 to deal with real mode nonsense. It amazes me now to recall actually going to a store and paying real money for a stack of 3 1/2" DOS 5 upgrade disks...6 years after I was programming applications on a richer 4MB platform, here I was excited that himem.sys could free up some of the critical 640KB of low memory.
• Microsoft toyed with windowing systems, finally creating something credible and successful in 1990 (Windows 3.0). In contrast a variety of competitors had fully-integrated, rich, usable, robust Windowing systems many years before - The 1984 Apple Mac being an obvious example, along with the 1985 Atari ST and Amiga...even options on the Commodore 64. I was an Atari ST fanatic in those days, and I marvelled at how primitive the PC world remained even years later.
• It wasn't until Windows 95/NT that memory protection was utilized to avoid processes stomping on each other's memory. Again, many, many years after most competitors had implemented this basic functionality. Instead we dealt with the normal occurence of misbehaving apps taking down the entire system as a fact of life.
• It wasn't until Windows 95/NT that preemptive multitasking was available in Windows. Prior to this a single misbehaving application could capture the CPU's attention and never let it go (never yielding), which was a fairly typical event. The Amiga featured pre-emptive multitasking a decade earlier.
• Microsoft released Windows 95 without a web browser, remarkably enough, finally releasing a barely changed version of the NCSA's Mosaic in the Plus! pack.
• Microsoft 95 was pretty much a security nightmare. Not only was its software far-from-ready to be connected on the public internet - I remember being the unhappy victims of winnuke and friends when I made people unhappy on IRC (you can't please all of the people all of the time), it also had no real file/object security of consequence. While NT was built as a "multi-user" system from a security and kernel perspective, many of the shell and utilities were user unaware, undermining this capability.
• Microsoft's web technologies were far behind the times until Bill Gates' famous speech that changed their direction, reacting to Netscape's lead rather than charting the course. Internet Explorer quickly ramped up and became the dominant web platform - until it became so powerful that the team was disbanded.
• Alternative 3D rendering APIs (Glide and OpenGL) led the way in an area where eventually DirectX would emerge dominant.

I recall during my early courtship with the PC simply marvelling at how incredibly obsolete the platform seemed to be compared to competitors like the Amiga and the Mac introduced years earlier - from graphics capabilities to software to hardware: Everything about it seemed so backwards in comparison to the superior alternatives, yet customers stuck with it. This was the platform that Microsoft wed themselves to, so surely they would suffer as well, right?

Microsoft's insistence on legacy compatibility led to a platform that moved much slower than competitors - Competitors that had the liberty of just tossing it all out and starting from scratch with whatever whizz-bang feature the newest chips offered. Maybe they could run super-stable and super-fast, and offer the developers an elegant platform upon which to perform their magic...but could it run Commander Keen 1 through 3? Could it run that ancient text database app?

Not All Negatives

Of course it's easy to focus on the deficiencies and imagine that they wrote the whole story, but in reality the situation was much more complex. Windows, for instance, pioneered widely-used video card acceleration (I still remember that shiny new Diamond Speedstar 24x. 24-bit graphics, coupled with hardware acceleration of 3D primitives. It was good times running those benchmarks. Of course the Amiga fanatics will point out that it supported hardware acceleration, just as the STe featured a hardware blitter chip, but the interaction between acceleration and the GDI in Windows really set the bar), and Microsoft's push greatly accelerated the adoption of optical media. Windows For Workgroups brought inexpensive networking to a lot of shops (NetBEUI was imperfect, but it was an easy transition to TCP/IP), and Windows in general represented a "good enough" platform for a lot of users. Internet Explorer, for all of its ActiveX "holes" and CSS quirks introduced the rich web model that we rely upon today.

En Route to 64-bit x86

This all comes to mind as the x86-64 transition accelerates: More and more users are starting to switch to 64-bit capable systems, and the 2/4GB limits of our machines is actually becoming a rational limit among desktop users: Everyday users are shouldered against a limit that seemed almost theoretically large just a few short years ago.

Of course Microsoft has been releasing incomplete 64-bit options for years (for instance you could get a 64-bit version of SQL Server 2000 for the Itanium platform, barring a laundry list of exclusions and limitations, and way back with NT 3.1 Microsoft supported 64-bit processors, albeit in 32-bit mode). Now that 64-bit support is finally becoming a critical factor, Microsoft has a wide gauntlet of support ready, and is finally ready to deliver.

Once again when the market really cares, Microsoft is ready. For years some have been talking about the advantage of various operating systems, such as Linux, being availabile on cutting edge processors and 64-bit platforms. For years that has been paraded as an advantage to customers who continued to run their platform on a standard old x86-32 foundation. Yet now that those limits are being reached, and the platform needs to accommodate new levels of capability and performance, Microsoft is ready. Another deficiency overcome.

The Question

Looking at the platform now - the stability, security, and feature set of Windows 2003, a lot of it already existing in XP - it really does seem like a tremendous window of opportunity for the competition has passed: What used to be a crop full of delectable low hanging fruit is now a well protected enclave featuring armed guards.

If competitors couldn't make inroads before, how do they have a chance now? If Linux couldn't capture the desktop market against a monstrosity like Windows Me!, what chance does it really have against XP?

The most obvious answer is web applications - render the operating system irrelevant and you don't really have to compete.

Tagged: [Software Development], [Programming], [Software-Development]