[The static location of this piece can be found at this address]

Introduction

Work habits and conditions vary dramatically between software development groups.

At one extreme of the spectrum are productive, tuned development teams delivering solutions on schedule and on budget, staffed with passionate experts happily building their skills and careers while providing valuable cutting-edge solutions. At the other extreme lie dysfunctional, under-utilized teams. The latter group endlessly catapults magnitudes past estimates of time and money, with a revolving door roster of contemptuously-treated employees.

Most shops fall somewhere in-between -- imperfect, but continually working towards better, more efficient practices and process, all while trying to provide a rewarding, fulfilling, and career-building experience for team members.

This entry is targeted at that continually-optimizing audience, and is written largely for development group managers and team/group leaders, though it's still informative for the general development community (lateral and upward management is critically important, and is the force that drives most process change).

These observations are gleaned from my experience with teams of varying sizes and types, acting in a variety of roles including management, team lead, developer, process consultant, mentor, mentee, and technology consultant. I have directly participated in, or observed, the development process in small, tightly-knit engineering groups stocked with professional electrical engineers, to mega corporations with walking-dead teams awaiting the next executive shuffle to mete out some doom on their orphaned projects.

I've learned from the successes, but I learned even more from the failures.

I've focused on best-practices that are optimal in virtually any team, and which are frequently evident problems, while avoiding those that are more likely to vary significantly between teams and organizations. Work environment, for example, is very subjective and context specific: Some developers and teams work optimally in private, quiet offices with comfy chairs and scheduled communications, while others thrive in hectic, noisy open offices, interfacing directly with the "customer" throughout the day, eagerly setting up development kiosks on whatever can serve as a seat. Some organizations need armies of BAs and BSAs and QAs and UAs, with layers upon layers of heavy process and checks built atop a traditional waterfall development model, while other shops work best with a highly agile, frequent delivery model, completed by multi-role development resources. It isn't possible to declare universal best-practices within those domains.

I also appreciate that most real-world teams have budgets, and that feel-good lists of .COM-style excess aren't realistic or helpful in the current IT climate. Every developer doesn't need every tool available, or every electronic toy on the market, to do their job effectively. I haven't tried to pander to the development community, blowing smoke about how entitled and empowered they should be, beyond the level that is actually beneficial to their career and the role.

On to the list!

Optimizing Software Development - Executive Summary

• Customize Your Development Process - Choose standards and processes that fit your team
• Use Source Control, And Use It Properly- Source control is a requisite of good development
• Formulate Written Standards- Don't rely on by-example or word of mouth standards
• Mandate Regular Status Updates- Have daily status updates
• Organize Your Information- Standardize your information archive structure
• Allocate Appropriate Time For Research- Research appropriately, or enjoy wasted effort and redundancy
• Accurately Track Development Time- Build valid historical data, and time accountability
• Focus On Your Competencies- Now when to outsource tasks to stay focused on your competencies
• Focus On Results Instead of Effort and Sacrifice - Sacrifice and effort should only matter when they yield results

Customize Your Development Process

Select processes, practices, and standards based upon your particular type of organization, type of project, and workforce, and then customize them for your specific needs.

Of course this suggestion seems ridiculously obvious, yet this is an industry rife with groups foolishly mimicking the process of those who appear to be successful.

Whether it's the previously agile vertical-solution shop trying to act like they're a bank, overloaded with process and CMM levels far beyond necessity because that looks like what the "big boys" do, or the banking group trying to adopt XP practices and frequent deliveries because it's the hip new trend that all the blogs are talking about, when their product only requires bi-yearly deliveries under a heavy layer of checks and balances: What works for one type of organization doesn't necessarily work for another, and adopting those processes under the wrong environment will bring defeat rather than success.

Don't act like you're developing life-critical flight control software when you make a P2P app, just as you shouldn't do the opposite. Don't pretend that you're an ISV, or assume that much of what's right for a retail software vendor is transferrable, if you're actually an in-house IT development shop.

This sort of cargo-cult mentality exists at a lower level as well. During the .COM boom, success apparently came from stocking up on Aeron chairs -- Get the chairs, and you'll do well. After the collapse the idiocy continued, with the same chairs now being demonized: Now they could only bring doom, and had to be removed post haste. It's a chair people. In the grand scheme of things, it really isn't going to make that much of a difference, positive or negative.

The same could be said about foosball tables; catered lunches; big, corporate-style meetings; having big cozy offices for everyone, or tiny open plan offices for everyone. Someone seemingly did it to their success, so therefore it was perceived as an applicable practice for everyone.

Just because someone else benefited from it doesn't mean that it would benefit your organization. It could very well hurt your organization.

Use Source Control, And Use It Properly

Source control is the foundation of good coding practices.

Not only does source control keep your code centralized (facilitating organized access for developers, along with easy backups and automated integrity assurances), with the proper use of branching and labels it can greatly increase the agility of your team: Having the comfort of making major changes in a code branch, while retaining the ability to revert to a labeled release or trunk, is liberating, reducing developer paranoia and ameliorating the risk of change.

Whether you're using Microsoft Visual Sourcesafe, Perforce, CVS, or one of the countless other available source control products, it is a critical component in your toolset. It's critical even if you're a single developer working alone in your one-man software development shop.

Source control is also invaluable for change management and auditing, historically tracking how much has changed, and where, over a given period of time, which is a benefit that often pays off handsomely years later. For example auditing the code history to determine when an errant algorithm was introduced to gauge how long it's been screwing up the calculations.

Adequately trained quality assurance teams can even use source control to determine what has changed between releases to tightly focus their testing efforts without relying on the often incomplete word of the developers.

The technique to use source control properly depends upon the tool that you're using, and the standards and needs of the group: Some groups use source control only to check-in production releases, while other groups check code in frequently, with every developer checking in multiple times per day, merging branches at convergence checkpoints. Ensure that you know the feature-set of your source control package inside and out, and that you're following a rational, best-benefit standard of use for your scenario.

If your usage is non-optimal because of limitations of your SCM tool, consider a product change. It really is that important.

Tip: Frequently perform project builds on a clean machine with limited network access by following development environment configuration instructions, and then getting and building from source control. During this process many teams discover hard coded paths, missing dependencies, and solution defects. It's better to detect it early than for development to grind to a halt the day before a big release when the corporate IT department decommissions that old server that supposedly wasn't in use for the past year.

Formulate Written Standards

Standards can be extraordinarily detailed, or as simple as noting which industry standards a team subscribes to (e.g. "All .NET coding will be done in conformance with the Microsoft .NET Design Guidelines"). It could even be the formalization of the lack of specific guidelines in a certain realm (e.g. "Put the braces wherever you'd like, and either tabs or spaces are fine"). Documented standards should cover all aspects of development, including development environment, tools, accepted languages, best-practices, check-in behaviour, naming guidelines, and so on.

Of course standards should be living documents, with occasional reassessments leading to the removal and addition of new points.

If, on the other hand, your standards are word-of-mouth, or "by example" (usually with many conflicting examples throughout the organization), you don't really have standards -- more likely it's acting as a demotivating, inefficient confusion for newer developers to the team, and a constant political skirmish between the veterans of the group: While good developers will happily adapt and conform to the idiosyncrasies and preferences of a group or project, it is frustrating and unproductive when they have to constantly go back and rename tables and rework classes because of unwritten "standards" that exist only in the minds of a few. When developers lose confidence in their code because of uncertainty over trivia such as spacing standards, something is seriously wrong, and people are focusing their care and concern on something that should be thoughtless.

As an added benefit, the process of documenting standards is often a time for rational debate, when "that's the way we've always done it" standards have to be legitimately rationalized and justified, or rightfully discarded. Many teams have improved their process, finally jettisoning unjustified legacy standards, during just such an exercise.

Having concise, documented standards (which can be the documented absence of standards in a particular realm) saves everyone from having doubts about whether they're conforming, eases new developers into the fold, and increases the percentage of good standards while removing the excess baggage. It also facilitates external assistance more smoothly.

Mandate Regular Status Updates

Every project, and every team, should have a daily or bi-weekly status updates by every member to their team leader or manager, or better still to every other member of the team (including from the manager to the team). This is regardless of your development methodology.

This can be a scrum, or a simple daily email stating what work the person is planning for the day, and what they achieved during the past day/period. This practice not only keeps everyone focused on the task at hand, with better time accountability (just like everyone else, and possibly worse as a group, developers are procrastinators), but it also gives a transparency of development that let's the manager, leads, team members, and stakeholders see where impediments exist, providing them an opportunity to help, or to recognize and respond to schedule slip early on, before it becomes a crisis. This transparency is one of the fundamental goals of the CMM process, and the benefits can be largely gained just by frequent, scheduled, mandatory and accountable communications.

Optimally this information will also exist in an archived, centralized, widely accessible location, such as an audio recording of a scrum, or a centralized collection of update notes.

It is critically important not only that this information is conveyed, but that it's actually consumed and retained as well -- management and the rest of the team must pay it proper heed, and they shouldn't wallow over during the day to ask what was previously stated. That undermines the entire process when it turns into a wasted effort that no-one pays attention to.

Organize Your Information

Having lots of specifications, standards, code libraries, discussions, solutions, and examples strewn across the topology of the network is of little use to anyone, and can be detrimental. Outdated documents will continually reappear, and misinformation takes on an unstoppable life of its own.

Centralize and standardize, ensuring that everyone knows exactly where to find all pertinent information on a moment's notice. Everyone should know exactly where they can find, and thus where they can store, specific types of information.

As it is, many teams start off with the "dumping bin"-style repository when they begin a project, planning on sorting it out later, however soon enough there are GBs of files -- many of which have little or nothing to do with the product or project -- confusingly cluttering a massive mess of a directory structure. This puts a serious damper on productivity.

Documents can't be found. People have to constantly interrupt each other to ask where to find something or where to store something. Eventually an expensive initiative will have to be undertaken to separate the wheat from the chaff, at a time when most people have forgotten why all of the various unrelated information is there in the first place. Undoubtedly good data will be removed, and bad data will be retained.

Before you create the first project or product file, decide exactly how you're going to organize the information, planning for the long term. Who will have access to it, and what permissions will they have? How they will access it? What will be contained where?

Tip: Documents such as specifications yield the same benefits of source control.

Allocate Appropriate Time For Research

The first step of any project shouldn't be to hit the IDE, or even to start high level design work. The first step should be to research and understand all competing or similar products, and any and all libraries and tools that could be leveraged to empower your solution.

As I write this, countless teams are spinning their wheels reinventing the Microsoft Application Blocks, or they're developing a data transformation service that could be integrated in Biztalk or SQL Server Integration Services less expensively (and much more flexibly). Perhaps they're developing a web portal that's just an incomplete, second-rate clone of SharePoint, Community Server or Zope.

There are even teams duplicating functionality already existing in the .NET Framework or standard template library because they never spent the time to look.

For virtually any need there are libraries and existing solutions that could either replace the need for the project altogether, or more likely could serve as a foundation of a more robust, more capable solution. If you're in the solution business, the desire to propose and implement the best solution should trump any desire to code for the heck of it.

Developers can't complete that sort of competitive research in an afternoon, or as a footnote, nor should you rely upon off-the-cuff commentary provided in meetings by individuals vaguely aware of the options.

While it varies by project and industry, such research usually requires a significant upfront investment of time before coding begins, allowing the stakeholders and development group to all feel entirely confident that the project is justified and will yield the best possible results. Document every finding and decision to save endlessly having to explain why one decided to develop what seems to be a clone of Biztalk.

Accurately Track Development Time

One of the primary reasons why many shops continue to grossly under-estimate the time required for projects is that they have limited, or fictional, historical real-world time tracking to base future projects upon -- they might have half-a-dozen developers working full-time, but the best they can account for at the end of the week is that a couple of small app issues got worked on.

This is sadly very typical.

Where there is some form of time tracking system in place, often it is of little value: Developers, as a general observation, will try to find ways to under-report the amount of time they spent on a particular development problem or project, as it's a bit of a badge to claim to have "thrown together" complex solutions in a "couple of hours". Furthermore few shops properly partition times (e.g. by project stage and task), further blurring the value.

Implement a time tracking system, and coach your team to accurately and regularly post times (daily, if not more frequently. Any less frequent and accuracy suffers dramatically (as someone with the bad habit of filling out timesheets at the end of the month, anxiously looking at file timestamps to figure out what I worked on when, I say this from personal experience).

Ensure that there is enough granularity that the time reporting is usable. For instance that you can see that competitor research actually took two weeks, instead of the foolishly anticipated 2 hours.

Avoid tacitly encouraging false timesheet entries: Don't act concerned when real world values start appearing, and most tasks are much more time-intensive than originally thought. Adapt and accept that reality differs greatly from the mirage that most people imagine that they see.

Focus On Your Competencies

You have your team of six hyper-intelligent and eager software developers, overloaded creating the next great scrum-tracking application. Based upon some user feedback, you realize that a small percentage of your users may want to interact with the system in a minimal way using J2ME equipped cell-phones, but your team is entirely staffed with .NET experts. Do you a) add a J2ME developer to the team, and just try to expand the J2ME product line, b) let one of your existing developers try their hand at J2ME, c) outsource that non-core component. Most shops would pick b without hesitation, incorrectly justifying it under the guise that it's "free" because it's covered by the payroll.

If you decide to try your hand at it in-house -- despite the considerable domain difference -- the results will almost certainly be a second-rate solution, delivered more slowly and more expensively than just outsourcing the need. This is worsened in that a resource originally dedicated to the core product has been diverted to a non-core need, and will come back to the core product with a need to spend time reacclimating.

This isn't to say that this is always a bad solution: Sometimes it entirely makes sense to build such skills in the core team (if this is a component that will need regular modification, or if it's an expanding customer base), not to mention that the work diversion can prove entertaining and rewarding for a developer.

Undertake non-core competencies -- be it building a one-off data bridge to a partner, putting up a corporate website, designing logos and icons, or countless other non-core needs -- with eyes wide open, realistically assessing the cost to do it in-house. Estimation of the "knowns" is already notoriously bad in software development, but the situation is far worse for non-core competencies, where developers are prone to extraordinary underestimations.

Realistically consider leveraging external solution providers (or even other teams within the organization) where it can keep your group focused -- with appropriate communications, teams will understand that it enhances their value and effectiveness, rather than presenting a risk to their employment. [Note: My organization provides outsourced consulting and development, so I do have a conflict of interests. However we always go in with the intention of empowering and furthering the existing team, and never to supplant them]

Focus On Results Instead Of Effort and Sacrifice

Getting the team working on nights and weekends means little if it saps their passion to produce, expensively inflates the bug count, and increases turnover. Yet sadly a lot of managers work under the delusion that effort and sacrifice are worthy substitutes for results. Indeed, many management techniques are built entirely around deriving maximal effort and demanding complete sacrifice, inventing artificial deadlines and manufactured crises to prod the troops.

The results are invariable destructive and counter-productive.

This is rooted in basic human psychology. Just look at a famous cough syrup with a heavy advertising presence: Surely it must work better because of the sacrifice required to ingest it? Contrast that with the presumption of self-destruction that most attribute to activities that people find enjoyable.

If it's fun or requires little sacrifice, then it can't be beneficial. If it is unpleasant, then it must be doing good. These foolish and unfounded notions have sold a lot of snake oils over the years, and they've supported a lot of human folly.

Savvy Machiavellian developers know how to exploit this management myopia, under-delivering until crunch time, and then putting in some superficial extra effort to get the project done, yielding kudos all around for going "above and beyond". Just by finally delivering long overdue results that they could have achieved in the normal workday. This is especially the case in shops with limited time reporting, and no regular status updates: Show up early in the morning (sending emails as validation), grunt and groan and gripe, shuffle around regularly, and then leave late (again sending emails to validate how committed one is), maybe VPNing in during the middle of the night to again demonstrate great sacrifice.

In some shops that alone is a substitute for any results at all.

On the flip-side, the "slacker" that comes in at 10am, leaves at 4pm, and takes extended lunches somehow keeps generating the bulk of the product design and code, but his apparent lack of effort and sacrifice will be dealt with at the next performance review.

Many heavy and onerous processes are undertaken, and then maintained, not because they've empirically proven themselves to be useful, but rather because the effort involved gives a mentally cheap illusion of achievement. From volumes of documents that no one ever references or validates, to arduous signature gathering exercises. Most of the time they're the mindless completion of a task with no real benefit to the project or the process, but it goes unquestioned as a part of the flow.

Discard all processes that are mechanically completed with little actual benefit. Tweak those that require more effort than necessary, maximizing the results while minimizing the effort.

Effort alone doesn't make your quality better, software better, or team smarter. Focus on results, and welcome and encourage results that come with minimal apparent effort or sacrifice.

Conclusion

Building a top-notch, effective software development team is a lot of hard work, coupled with a bit of luck, but hopefully some of these points have given a bit of food for thought that might encourage some development in the right direction. This list most certainly isn't comprehensive, and I've stayed away from domain-specific practices, but it is a start.

Tagged: [Software Development], [Programming], [Software-Development]

an overwhelming percentage of software developers imagine themselves to be far above the norm - they themselves conveniently fit that top 2% echelon, they believe

Several readers have emailed to ask how my "Optimal Software Development Processes and Practices" entry -- which was surprisingly well received -- compared to several better-known software development guideline papers. For instance Joel Spolsky's superlative "The Joel Test: 12 Steps To Better Code".

"Read them all and then adopt what sounds right to you," I replied.

To expand upon my answer a bit, my entry is a listing of high-level management practices that apply to virtually any team involved in virtually any type of software project. If you have accountable software developers; you're focused on solving the right problem, using the right platform, tools, and technologies (given that they've done the research, and they're not tied up having camelCase/PascalCase wars, developing a redundant GFA Basic solution for the Atari ST); you have a history of accurate, detailed time information to draw upon for estimates and to really know the capacity of your team -- which usually indicates a grizzled, experienced management that is more attuned with reality than is the norm; you have full transparency of how the project is progressing and what is getting done; then you have a much better probability of success.

Yet these basics -- these core fundamentals -- are sadly missing from many software development teams.

It is, however, fairly typical of papers of this genre to look for something seemingly unique -- some sort of hook -- that'll grab the masses as the silver bullet. One which they can quickly implement in their process (for instance a poorly used and abused bug tracking application, or a hashed out daily build script that has been failing for months but no one has noticed because it has no utility, or partnering up developers in pair programming, or implementing code reviews, or pursuing test-driven development) and claim success. "We score a 9! Awesome!" they gloat, high fiving and then returning to the continuing cycle of project failures.

One common silver-bullet hook these days is the classic "only hire the best!" mantra, and I thought it worthy of special mention.

"Only hire the 2% of software developers!" these advice papers crow. "We ensure that our software is top notch by rigorously putting our interviewees through 19 interviews, by asking clever brain-teasers from How Would You Move Mount Fuji, and then by having them design and implement an embedded operating system on a whiteboard!" This is usually accompanied by a "how to spot a great developer!" listing which could better be described as "How the author would find themselves in a pile of applicants".

This arrogance isn't limited to the authors of such silver-bullet recommendations, though. In conversations and forum threads discussing such a top X% recommendation, one quickly comes to realize that an overwhelming percentage of software developers imagine themselves to be far above the norm -- they themselves conveniently fit that top 2% echelon, they believe, so of course they agree: Let the banks and the IT shops take the other 98% (the duds), because the top 2% are too busy making webmail interfaces and bug tracking applications (as ridiculous as that sounds).

The problem with such a "only hire the best!" proclamation, however, is that it's meaningless from the start -- How do you select the top X%, given that the number of attributes of a software development team member is virtually infinite? How do you quantify prospective talent universally like this given that what really matters to you varies dramatically based upon what you're developing, what fit the developer will be in a team, and how you're going to manage them?

If you're a poor manager and you fail to utilize a talent properly, or if you choose the wrong talent for a particular position, it can be an absolute disaster, or at least a non-optimal situation.

The top 2% in one realm was the bottom 2% in the other.

To draw from personal observations, I've known some remarkably intelligent developers who would absolutely storm a path of success in one type of project, in one type of role, basically designing, implementing, and delivering a remarkable project almost single-handedly. Yet they would wallow fruitlessly for months when put on another project. Whether it was because they had no motivation or passion for the new project, or that they simply couldn't adapt to the new requirements -- which is remarkably common. Great developers, young and old, often have a niche where they are remarkably strong, outside of which they falter -- the results were the same: A great developer was being wasted on a project that didn't leverage their strengths, and a project had a deadweight superstar that couldn't catch on. The top 2% in one realm was the bottom 2% in the other.

There are many projects and situations in which John Carmack and Linus Torvalds would be complete disasters.

There are many who'll disagree with me. "That's hogwash!" they'll say. "I can do anything well, it's just that IIS is a piece of crap, and the MSDN documentation is all wrong, and Microsoft is evil, and there's this strange behaviour with the MSXML Library -- that wouldn't happen on Linux! -- and that really screwed up my timeline, and....". Sure.

Of course this could be countered by claiming that the "top 2%" refers to whatever pet dogma the speaker subscribes to. "The top 2% writes test cases before writing code, comments all code (in RoR of course), and does their timesheets by 4:30pm every day".

And thus it is explained how every developer can claim to be the top 2%, and how every elite-herder can claim to hire only the top 2%: They simply adapt the meaning to describe whatever practices they adhere to, whatever techniques they use, and their personal skillset, or by simply describing the candidates who made themselves available to them, and who accepted the position.

In that way we can all be #1.

When Java first hit the development ecosystem, to many it wasn't just a method of doing efficient, high-level development, but rather it became a new religion: You couldn't only use Java as the glue between existing code, or even as the overwhelming bulk of your solution. A partial-Java solution simply wasn't good enough.

Instead your product had to be 100% Pure Java. The still sought-after eventual goal was a complete Java solution, from applications right down to the operating system, with only the smallest possible binary kernel, if even that. All of this would be running on a Java-aware processor, engineered specifically for Java.

Sun created a "100% Pure Java" campaign to push this philosophy, including banners and designations for appropriately certified software, and advocated it as a very desired moniker. Users were led to feel that mixed solutions were impure and somehow dirty: Are you some sort of nut running an impure solution, dirtied with some pointer munging, buffer overflow vulnerable C code? While there were (and remain) methods to call native code, they were discouraged.

Of course there is a lot of validity to this agenda. Primary being the fact that pure Java solutions are theoretically cross-platform, with no ties to external technologies. Compare this to a solution leveraging C libraries, which would require a rebuild or binary available for every distinct target platform. Additionally Java could only impose its sandbox and extensive security constraints if you stayed in the world of Java, and thus callouts to native code represented a risk.

In the real world, though, it often meant that developers were constantly solving long-conquered problems, redundantly reinventing solutions in Java that long existed elsewhere, or waiting until adequate libraries eventually appeared: Developers were pressured to use Java alone even when it was a hammer and the solution really needed a chisel.

Thankfully .NET hasn't been pushed in such a single-minded way (even if some of its champions have foolishly taken up such a misled cause, including some at Microsoft. Instead of a justified part of the solution, it becomes a religion. .NET! .NET! .NET! .NET!), and indeed Microsoft themselves has always facilitated, and even advocated, "impure" solutions. The majority of the .NET Framework, for example, is actually a very thin veneer over the existing Win32 facilities and libraries -- it was either that, or version 1.0 would have come with a much smaller, much less efficient library.

The "orchestration layer over native code" implementation is the reason .NET hasn't suffered the performance difficulties that Java has.

Microsoft chose to leverage what they'd already done, to maximize both performance, and to maximize the breadth of the library. 

This advantage isn't limited to Microsoft, though, and the developer can utilitize this functionality as well. .NET offers very simple COM and P/Invoke functionality to leverage "legacy" code (or even new code developed in a best-solution, non-.NET technology), allowing you to easily use your existing DLLs and/or COM libraries as first class partners in your .NET solutions. Even if they're created in "dirty" languages.

I take advantage of this functionality regularly, utilizing existing best-solution libraries and functions, regardless of whether they're pure .NET or not. For instance in creating the static version of the "best of" blog entries, I quickly -- maybe 2 hours -- wrote a quick transformation tool that basically imported the "best of" RSS feed (it isn't included in the normal category lists), then doing some XSL transformations (using extension objects in the XSL given that XSLT alone wasn't adequate for some special purposes -- for instance HTMLDecoding the description block of the RSS) to the resulting XHTML, as well as creating an index page.

One goal when creating this solution is that the resulting pages are all fully XHTML compliant, and they pass the W3C validity checks. While I could easily see how the pages rendered in Mozilla/Firefox/IE/Opera, and of course they all rendered fine, technically there were a couple of deviations from the spec. Some of these errors and warnings were caused by unavoidable transformation issues, while others were caused by minor mark-up errors in the original blog entries (both because of my own errors when doing it by hand, but also because of Radio Userland's "helpful" auto-"cleanup" of HTML. It is remarkable how often auto-formatting is detrimental).

HTML Tidy to the rescue.

I had several options with HTML Tidy, the easiest of which would be to ShellExecute out to the EXE, telling it to process an existing file. I could have taken more time and tried to make a managed C++ version of Tidy, but I really didn't want to spend that much time.

I decided to have a bit more fun, not to mention building a more integrated, higher performance solution, and use the Tidy dll from the micro-.NET utility. I grabbed the Tidy source code (Tortoise CVS is a great solution for this, in this case using :pserver:anonymous@cvs.sourceforge.net:/cvsroot/tidy), updated the included MSVC projects to Visual Studio 2005, and added them to the transformation utility solution. I set the Tidy dll project output to the build directory of my .NET utility (in this case $(SolutionDir)\blogStatic\bin\$(ConfigurationName)). The MSVC build worked perfectly right away, which is amazing given that Win32 isn't an officially supported build.

To reference the Tidy dll methods, of course I had to add the DLL import signatures, in this case adding only the ones I had a need for.

  [StructLayout(LayoutKind.Sequential)]
  struct TidyBuffer
  {
    public IntPtr bp;           /**< Pointer to bytes */
    public uint size;         /**< # bytes currently in use */
    public uint allocated;    /**< # bytes allocated */
    public uint next;         /**< Offset of current input position */
  };

  class FileClean
  {
    [DllImport("libtidy.dll")]
    public static extern IntPtr tidyCreate();

    [DllImport("libtidy.dll")]
    public static extern int tidyParseFile(IntPtr tidyPointer, [MarshalAs(UnmanagedType.LPStr)]string fileName);

    [DllImport("libtidy.dll")]
    public static extern int tidyParseBuffer(IntPtr tidyPointer, ref TidyBuffer tidyBuffer);

    [DllImport("libtidy.dll")]
    public static extern int tidyCleanAndRepair(IntPtr tidyPointer);

    [DllImport("libtidy.dll")]
    public static extern int tidySaveFile(IntPtr tidyPointer, [MarshalAs(UnmanagedType.LPStr)]string outFileName);

    [DllImport("libtidy.dll")]
    public static extern int tidyRelease(IntPtr tidyPointer);

    [DllImport("libtidy.dll")]
    public static extern int tidySetCharEncoding(IntPtr tidyPointer, [MarshalAs(UnmanagedType.LPStr)]string encoding);

    [DllImport("libtidy.dll")]
    public static extern int tidyOptSetBool(IntPtr tidyPointer, int value, int Bool);

    public static bool CleanFile(System.String outputfileName, System.IO.MemoryStream docDataStream)
    {

      int result = -1;

      IntPtr tidyPointer = tidyCreate();
      try
      {
        // We want the resulting file to be UTF8 encoded
        tidySetCharEncoding(tidyPointer, "utf8");

        byte[] docDataArray = docDataStream.ToArray();

        TidyBuffer tidyBuffer;
        tidyBuffer.size = (uint)docDataArray.Length;
        tidyBuffer.allocated = (uint)docDataArray.Length;
        tidyBuffer.next = 0;

        GCHandle pinHandle = GCHandle.Alloc(docDataArray, GCHandleType.Pinned);
        try
        {
          tidyBuffer.bp = Marshal.UnsafeAddrOfPinnedArrayElement(docDataArray, 0);

          if (tidyParseBuffer(tidyPointer, ref tidyBuffer) >= 0)
          {
            tidyOptSetBool(tidyPointer, 29, 1);
            tidyOptSetBool(tidyPointer, 23, 1);
            if (tidyCleanAndRepair(tidyPointer) >= 0)
            {
              result = tidySaveFile(tidyPointer, outputfileName);
            }
          }
        }
        finally
        {
          pinHandle.Free();
        }
      }
      finally
      {
        tidyRelease(tidyPointer);
      }

      return (result == 0);
    }
  }

Most of this should be self-evident, however the two tidyOptSetBool calls may be a little cryptic. For the sake of brevity I haven't used the constants, but 29 is the TidyMakeClean value of TidyOptionId enum (see tidyenum.h), and 23 is the TidyXhtmlOut value. Together these indicate that I want to clean the documenting, converting it to XHTML. Note that I've also set the encoding to UTF8.

Voila, after transforming the RSS to the memory stream as quasi-conformant HTML, I passed the stream to this function, along with the desired output filename, and out went a cleaned-up, valid XHTML document. Pedants everywhere were thwarted from pointing out minor deviances from the standard. I could have processed to another buffer, and then done follow-up processing in .NET as well, but this was sufficient.

This is a trivial example, but it really exemplifies the great value of the easy interoperation of .NET. With it I could instantly leverage existing code, without having to search out bastardized ported versions, and instead could go right to the source.

This is a preview of an article scheduled for completion before the week is out (and while it shares superficial similarities to a recent Paul Graham article, it was started before Paul "published", and has been a topic that I've long wanted to cover).

I've dedicated a little more time to writing now, and should finally complete another commissioned magazine article over the next month (this one on distributed/symmetrical computing with .NET), which will be satisfying to finally .

Introduction

Software development can be a tremendously rewarding and enjoyable career.

Few careers offer comparable opportunities to weave intricate, complex structures that, while virtual, have such a  positive impact on the world around them. Few offer the freedom and creativity that software development does, or the very real potential for entrepreneurial riches.

Whether it's building a new peer-to-peer application, control software for a massive power generator, or improving the workflow of the corporate scorecard system, done right this can be a very fulfilling, enjoyable, challenging pursuit.

This article describes the wonder and curiosity that many developers start out with, whether it's when they pick up their first JavaScript in 1 Hour book, when they start toying with the gcc compiler for the first time, or when they started their first Computer Science course in university. It describes how that natural enthusiasm can be crushed, and how it can hopefully be regained or maintained.

This is written for the developer, whether a new recruit or a veteran, motivated or unmotivated, spirited or crushed, yet it's also written for software development managers (who might identify how to make the workplace more enjoyable and more rewarding).

A Passion for Software Development

Does your mind race at all hours, abuzz with potential solutions for vexing software development challenges? Do you lie awake at night -- anxious like a preschooler on Christmas Eve -- eager for morning to arrive so you can implement the crafty coding structures you just thought up? Do you frequently find yourself powering up your system in the twilight hours to implement the fruits of an epiphany?

Or do you put in just enough face time and superficial effort that sacrifice makes up for undelivered results? Do you purge your mind of software development the moment the virtual end-of-day whistle goes off, sliding off your Aeron dinosaur satisfied that it's one day closer to the weekend? Do you dread Mondays, motivating yourself to keep going with the dream of a far off vacation?

Do you eagerly embrace new technologies, seeing it as a challenging opportunity to learn something new when a solution calls for a new skill? Would you voluntarily dive into the innards of the Firefox web browser if a solution demanded it and you'd never touched it before? Do you swim through documentation, thirstily absorbing new APIs, tools, and languages to expand your skill-set, eagerly embracing industry advances?

Or do you dread anything different, praying that you're tasked with challenges that require only the skills you've long held, allowing you to apply them in a mechanical, repetitious fashion? Do you hope every project is an echo of a prior project? Do you put off any task requiring research, and show disdain towards new languages, techniques and practices, hoping that they don't gain traction?

Are you really passionate about software development?

Be honest with yourself.

A desire to outshine a teammate isn't passion. Nor is a motivation to impress the boss. Neither is a combination of the two worn as a magic defensive cloak against downsizing spells. All of those are second-rate, artificial passion substitutes: Mixed into the recipe, they yield subpar results, often leaving a nasty aftertaste called burnout and dissatisfaction.

Instead I'm talking about a bona fide interest and enjoyment of the craft and challenge of software development, even outside of career or job security issues (though it benefits the same). This isn't a job ad demanding that you're "passionate about business reports!", but rather is just a moment for sober reflection on whether you're over-clocking life, or running idle instructions in a tight loop.

If you're like most software developers in the industry today, a feeling of enthusiasm and enjoyment for the pursuit is just a distant memory (often during the happy days of university and your first job). Instead it has become a career, and is just something you do from 9-5 (or more when passion is replaced by sacrifice). Skills have likely stagnated, moving just enough to compete with coworkers, or to avoid obsolescence.

Of course there are those who've never (and will never) enjoyed this career. The only advice I can offer to those people is a suggestion that life is too fleeting to spend so much time doing something you don't enjoy.

Many others, however, remember the passion, and sporatically get a taste of it again. For those people I propose some personal habits that coupled with workplace practices (for managers, as well as people who rightfully manage up), that will help recapture, and maintain, that passion.

Software developers who truly love what they are doing are the ones creating the most innovative code. They're the ones with productivity rates multiples of their peers in the industry. They're the ones getting paid to do what they love doing.

...

I have been considering the possibility of yafla providing training services, developing and delivering programs here in the Greater Toronto Area (and globally where the monetary return makes it worthwhile), adding this service to the existing software development, outsource management, and consulting options. Not only is it an additional revenue stream, much more importantly it's another potential avenue for making contacts and getting involved as a vendor with new clients, creating opportunities to more easily offer our other services.

I've done a lot of group training in the corporate space, have been involved in quite a few tutorials and online training guides, and find it to be a very rewarding pursuit. Several of my associates have been heavily involved in the training industry during periods of their career. The possibility of hosting an "Advanced SQL Server" seminar or workshop, for example, seems very exciting.

We're very equipped to perform this task, and we certainly can do a better job than all of the trainers I've been exposed to. Our approach would never be to dedicate anyone to training alone (or even as a substantial period of their time), as continuing and up-to-date real-world experience would be absolutely critical to the program.

As such, I'd greatly appreciate any input anyone might have regarding the technical training industry, and how external training programs work at their organization. I know many mid- to large-sized firms have an "authorized training vendor", basically ensuring a universally weak level of training throughout their organization, and that could present a significant barrier to entry.

Earlier this month -- January 4th to be precise -- I posted an entry regarding optimal software development practices (here in archived form), one of the most important points being that teams should "Focus On Results Instead of Effort and Sacrifice".

Focusing on results instead of effort and sacrifice can be realized in many ways; For instance by using the easiest possible tools and technologies that acceptably achieve your results. By cancelling long, drawn out meetings that everyone hates if the meetings don't achieve results. By ditching any process that is nothing more than cargo-cult remnants.

It's a simple perception change that forces one to evaluate the actual benefit yielded by extended efforts, rather than blindly applying brute force with hopes that it magically yields returns.

This rule isn't just for workplace practices, though, but applies to our day-to-day living as well.  For instance making a delicious cup of coffee.

I recently came across a widely referenced piece, "A Coder's Guide To Coffee", which details the various steps that one should take to achieve a drinkable cup of coffee. It's an interesting read, and serves as an entertaining bit of additional knowledge about the craft of brewing. Nonetheless, as I read it I imagined countless people creating subpar, or even just par, cups of coffee, confident that the additional care, concern, and manual effort they put into the effort guarantees them a better cup of coffee.

It doesn't.

In fact it could lead to much worse results, not to mention that it took a lot more effort to yield those worse results in the first place.

My Coffee Pedigree

Most of the time I drink packaged coffee, brewed in an often-dirty automatic drip coffee maker (it isn't the height of science getting water just below the boiling point, and in fact many automatic drips work by boiling water up from the reservoir, letting it cools the perfect amount while dispensing. In essence the water temperature is guaranteed perfect by the thermodynamics of the design). It requires close to no effort on my part, yet most of the time my coffee is (in my humble opinion) extremely good. For any normal coffee drinker it would be close to the "perfect cup", and unless you had dedicated your life to the classification of coffee, or you're on to drinking only coffee defecated from civets, you probably won't notice the difference from the most effort-encrusted specialty coffee. The only thing my coffee lacks is the placebo effect of imagined advantages.

Getting the "perfect cup" was incredibly easy: I found the perfect water/coffee ratio for my particular tastes, my cheap drip coffee maker does a very credible job (and provides water at the perfect temperature), and I brew small enough pots that coffee isn't sitting for very long before being consumed.

I tried a wide variety of grind brands and roasts, and eventually found a couple that are predictably good, so they're my staple. Occasionally I buy some of the "bulk" gourmet coffees (although the results there have been negative as often as they've been positive. The large coffee manufacturers seem to have the process down to much more of a science than the local coffee house).

Even the most common, most pedestrian, package ground coffee is made with 100% Arabica beans, so that isn't too much of a concern, and the whole Robusta red herring is a bit of cheap, disposable advice.

So without further ado. Here's the amazing magic of making the pragmatic perfect cup of coffee!

The Pragmatic Coder's Guide To The Perfect Cup of Coffee.

• Try various coffee brands until you find one that matches your taste. There is no "ultimate" coffee, and one person's winner is another person's dud. The major coffee brands use excellent beans, and have extremely tight quality control, so any illusion that it's a crapshot or that the quality is second rate is misguided. Of course you should care for the freshness of coffee as you would with any food product, ensuring it's in an airtight container in a non-spoiling environment.
• NEVER take anything from the pot/carafe while it's brewing, and apply corporal punishment to those who do. Most office coffee stinks because a jerk came and took a cup right after the brew started, taking with it most of the flavor. As grinds release flavor  second at a rate loosely like [Flavour Per Second]=1/([Seconds Into the Brew]^2), what's left is some discoloured water that smells like an ashtray.
• Try various ratios of coffee/water until you find what works for you. Many office coffee is supplied in little prepackaged packs that are too small for most tastes, so combine one and one quarter, or one and a third, or whatever quantity makes the perfect cup.
• Drink a batch quickly, and if it regularly sits then start brewing smaller batches. You can maintain the taste longer by brewing into, or transferring to, an insulated carafe.
• Never microwave coffee. Again there is no obvious reason for it - Microwaving is simply supposed to excite the water molecules, raising the temperature of the cup - but microwaved coffee always...ALWAYS...tastes terrible. If it cools too much, toss it.

That's it.

Focusing on hand roasting your beans, or manually brewing, is absurd when the overwhelming majority of the population can't even accomplish the basics consistently. Following those simple rules gets you to the point of extraordinarily diminished returns, and it is the Pragmatic Perfect Cup.

Of course you could hand craft your own gathering containers, walk 500 miles barefoot to hand pick only the cutest beans from the largest jar, brew with the most remarkably pure spring water after having it blessed by the saint of coffee in a pot made of the purest of silver, using beans ground with ancient Egyptian artifacts, but that doesn't mean that you'll yield a better cup of coffee, unless you're susceptible to the placebo false return effect.

Oh, and occasionally clean the pot. I think I'll go do that right now.

Out of Bounds : Avoiding Career Protection Faults

#define KNOW_IT_ALL_DEVELOPER 64 
#define TEAM_VIEW_OF_THE_NEW_GUY 8 
#define UNWANTED_INPUT 1

int team_integration[TEAM_VIEW_OF_THE_NEW_GUY]; 
team_integration[KNOW_IT_ALL_DEVELOPER] = UNWANTED_INPUT;

The following is an email I could have gotten recently.

Hi there. First time caller, long time listener. You're the greatest, and your words are like gold!

I recently took a new software development job, and I've been experiencing a lot of ill will from coworkers and peers. It's really been killing my job satisfaction.

To explain, I was recently brought in to help a software team get a product out the door, with a mandate of helping with some web app code. I've been trying my best to integrate with the team, trying to earn some credibility and respect by making myself useful.

I've been forwarding various Joel On Software essays to All, recommending that the office stock up on Code Complete, Peopleware, and the Mythical Man Month, and I make an effort to point out everything I believe could be done better. I regularly browse through the source repository to find ways that other members could be working better.

When other developers ask for my help, I try to maximize my input by broadening my assistance to cover the way they're developing, how they could improve their typing form, what naming standard they use, to advocate a better code editing tool, and to give my educated final word regarding the whole stored procedure/dynamic SQL debate.

Despite all of this, I keep facing resistance, and I don't think the team likes me very much. Many of my suggestions aren't adopted, and several people have replied with what I suspect is thinly veiled sarcasm.

What's going wrong?

Are these developers just afraid of change? Are they so stuck in their ways? Do they simply LIKE doing things the wrong way?

Yours truly,
The New Guy

Thanks for writing New Guy. It's a thrill when I can manufacture a cheap strawman to make a point, so I welcome the opportunity to address this scenario.

Let's get to why you're experiencing what you're experiencing.

The Politics of Software Development Teams

Many development shops operate as a zero sum game: there are a fixed number of promotions, raises, and apple-of-the-boss'-eye positions that everyone is competing for, and thus one developer's gain is at the cost of the rest of the group.

Given this sad reality, when a new recruit joins the team, there may be resistance, particularly among less confident groups and team members, and attempts could be made to limit the recruit's influence by deriding their ideas, limiting their input, and marginalizing their contributions.

The longer the existing team has been in place, and the less confident they are about their positions and their skill, the worse this problem usually is.

Of course, their fears may be well-founded. Many new hires are motivated to push silver-bullet methodologies, to lazily advocate quick-fix online essays, and to make superficial commentary about how to improve the code and the products, is the desire to earn themselves enhanced credibility with the overseers, and thus the opportunity to fast-track past the existing team members.

The Ugly History of Most Projects

About a decade back, I was tasked with building an aggregation system to pull together data from facilities across the continent, and to perform some rudimentary filtering upon it. The eventual result of this effort was a rather ugly creation that I was both proud of and ashamed of at the same time.

This system originally grew out of some Excel macros, growing into an Access solution with some simple automation, and then to a SQL Server database with on-demand manual data extractions, and finally to a DHTML web application reporting framework against a data warehouse. Through that evolution it incorporated a mish-mash of technology advances, methodology changes, and of course changes based upon my own knowledge increasing. 

Many of the project steps weren't actually authorized -- I built the original application because I was lazy, and wanted to automate some manual tasks I had been assigned -- and the project was always viewed by management as a short-term hack until another of the organization's products could render it irrelevant, so rewrites and major refactoring were out of the question. Requirements and user input was incredibly sparse, and most of the project was developer initiated, with the users choosing what color to paint the shed after the fact.

Remarkably that ugly hack job continues to process data today.

Yet having that product history under my belt made the position close to untenable: With each new developer I got to hear how the data should have been structured, whether more normalized or less normalized; how it should have incorporated as yet uninvented technologies like XML; how it doesn't conform with coding standards imposed years later; how it was theoretically rendered redundant by hypothetical solutions released far later; and how the hack-job code, one of my first projects created in Delphi and Object Pascal, was less than ideal; and so on. Many of these criticisms were made as widely and as loudly as possible, in standard To: All fashion.

Yet, despite all of its real-world warts and appendages, it was a critical part of the enterprise. Despite all of the bold talk about its shortcomings, no one could more than superficially improve it or deliver any real alternative.

And that's the reality that new recruits walk into -- products created in less than ideal conditions, with limited requirements and limited sponsor backing, under absurd time constraints, by developers not entirely adept at the technologies, and with limited or non-existing supporting technologies (ADO, for instance, didn't even exist yet). Developers who live through such a history are often resistant to brash or unrealistic commentary.

Of course, this doesn't mean that the project deserves no constructive criticisms. Indeed, I would be the first to say that lessons should have been learned from its implementation and failings, and a much better replacement built using all current generation technologies. I was always open to discussion regarding some of the choices made, and to explain its less than ideal architecture.

Yet I grew an instant dislike to anyone who haughtily pronounced its shortcomings.

This is the sort of reality that many new developers walk into, with ugly-offspring projects that are nonetheless loved. The legacy developers know its faults, and while they work to improve those that they can, they accept others as necessary compromises (often compromises of time -- it may be ugly, but it's worked for years so it's not the best place to spend critical time changing). They are naturally defensive of their legacy, while more pragmatic than a fresh "everything should leverage all of the advantages of .NET 2.0" recruit.

Real World Requirements

In the real world you probably don't need GUIDs for your employees table primary key. In the real world you can use basic authentication and extremely heavyweight reporting solutions (such as Excel automation) on a high value, single customer web server. In the real world client-server architectures can often suffice, and less-efficient code is often satisfactory. In the real world you can often get away with unilingual applications.

Of course in the artificial one-sized-fits-all ecommerce world, all of these are sins. In the artificial world every solution has to be database vendor netural, platform neutral, infinitely vertically and horizontally scalable, immediately multilingual, using n-tier connection pooling via a hierarchy of web services.

The conflict between the real world and the artificial one-sized-fits-all world is a frequent point of contention between new hires and existing teams. One organization's Enterprise solution is another organization's short sighted mistake.

Practical Knowledge

To draw from personal experience again, I farmed some code off to a new recruit once, asking them to build some extensions. Some time later they returned to tell me that the code I sent was clearly flawed, and that it couldn't possibly work due to some perceived flaws. I assured them that the code worked, and they were off again. Once again they returned to tell me that I was wrong. I imagined myself a bumblebee with a distraught aerodynamics scientist informing me that it was impossible that I could fly.

The code I sent them was straight from the production branch, and had been operating robustly for over a year. I quickly demonstrated that they were wrong.

This is a fairly typical scenario, where new recruits place too much confidence in their own knowledge and skills, despite what is often a domain knowledge deficiency, while simultaneously placing too little confidence in the team they are joining.

More often than not that quirky piece of code has a reason, and that seemingly less-than-optimal solution probably has a very good explanation.

Putting It Together - How To Fit In

• Always first presume that you're wrong. While developers do make mistakes, and as a new hire you should certainly assist others in catching and correcting mistakes, you should try to ensure that you're certain of your observation before proudly declaring your find. It is enormously damaging to your credibility when you cry wolf.
  
  
• Be discreet with constructive criticism. A developer is much more likely to be accept casual suggestions and quiet leading questions than they are if the same is emailed to All. Widening the audience is more likely to yield defensiveness and counterstrikes.
  
  The team is always considering what your motives are, and you will be called on it and exiled if you degrade the work of others for self-promotion.
  
  
• The best way to earn credibility and respect, and an elevated position in the pecking order, is through hard work and real results. Cheap, superficial substitutes -- like best practice emails sent to all, or superficial comments about how great it would be to implement some silver bullet -- won't yield the same effect, and are more easily neutralized.
  
  
• Actions speak louder than words. Simply talking about implementing a team blog, or a wiki, or a new source control mechanism, or a new technology, is cheap. Everyone knows that you're just trying to claim ownership of the idea when someone eventually actually does the hard work of doing it, and they'll detest you for it.
  
  If you want to propose something, put some elbow grease behind it. For instance, demonstrate the foundations of a team blog, including preliminary usage guidelines, and a demonstration of all of the supporting technologies. This doesn't guarantee that the initiative will fly, and the effort might be for naught, but the team will identify that it's actual motiviation and effort behind it, rather than an attempt at some easy points.
  
  
• Not every application is a high-volume e-commerce site. Just because that's the most common best-practices subject doesn't mean that it's even remotely the best design philosophies for the group you're joining.

Conclusion

Despite all of the selfish and illogical traits described above, many teams are staffed with confident developers who are ready and willing to accept new hires into the fold, and who welcome and respond to criticism and doubt fairly and respectfully. Similarly, many new hires are selflessly trying to contribute their knowledge, and their contributions often are founded on nothing but good intentions. When they are jockeying for position, it's often to earn the respect and trust of their peers rather than to impress the boss.

Having said that, the development community is (at least thus far...unless the robots take over) full of humans, and humans are imperfect. Often we try to boost our own relative position by pushing down others, and we stake out domains that we defend against all logic. By nature most humans are prone to resisting change. New hires need to keep these realities in mind, not only to identify and minimize their own weaknesses, but also to identify the same in others and to adapt to the greatest extent possible, working to integrate without become life-long foes of their coworkers.

Tagged: [Software Development], [Programming], [Software-Development]