[The static location of this piece can be found at this address]
For the past while I've been using Visual Studio Team Edition for Software Developers, one of its benefits over the Professional Edition being the inclusion of static code analysis functionality right in the IDE.
This functionality comes via the FxCop codeset, which is an excellent -- albeit unpolished -- freely available tool for analyzing the probable code quality of Intermediate Language assemblies, testing code to ensure compliance with naming standards, best practices, and highlighting areas of code that are suspect. While it's less than pleasant starting FxCop analysis from scratch on long existing project -- to be met with hundreds upon hundreds of error messages -- it's a painless process if you add it to your quality checks early on.
The standalone FxCop is largely the same as the VSTE version, and in some ways is superior. For instance that it retains the ability to actually pass configuration settings to rules, rather than accepting whatever the defaults for the rule are.
One of the few differences between the standalone application and the VSTE-included version are the addition of several new maintenance checks in the Team Edition code, one of the most useful being the cyclomatic complexity checks. Cyclomatic complexity, for those who haven't come across it before, is often used to roughly gauge the complexity of a piece of code, to determine likely candidates for refactoring, and to identify what will likely become a maintenance problem in the future. Finding the most complex pieces of code often brings you to the buggiest code as well.
Given that I still use FxCop, both the .NET 1.1 and .NET 2.0 versions (not least because the integrated version offers no ability to configure settings for rules, instead only allowing you to wholesale enable or disable. This eliminates the ability to set thresholds for tests such as the cyclomatic complexity rules), the lack of consistency between the two versions was an annoying gap.
So I implemented a simple cyclomatic counting rule for the standalone FxCop. While in there, I added checks for statement count (the number of intermediate language "statements", which can be indicative of overly complex methods), and callout count (e.g. callouts to other methods, again which can be an indicator of overly complex/convoluted methods).
As one added benefit, I added the ability to log all of these metrics to an SQL-capable OleDB destination (e.g. SQL Server, Access, etc). If you configured an OLEDB connection string, as detailed below, you can do data analysis after a run to create pretty reports of the complexity distributions of your projects, and so on.
yafla FxCop
Rules for .NET 1.1 (e.g. FxCop 1.32)
yafla FxCop
Rules for .NET 2.0 (e.g. FxCop 1.35)
Like any tool of this type, there is only a moderate correlation between the metrics measured and actual code quality or maintainability: It is entirely possible that the optimal implementation is a highly-complex, lengthy method. This tool only provides guidance, helping to determine which code should get a complexity analysis, however from there experience and good judgement have to be applied to determine if it's really a fault. If you're using the .NET 2.0 version of FxCop, make use of the SuppressMessage attribute on methods that are necessarily highly complex.
Drop yaflaRules.dll in your FxCop Rules subdirectory (e.g. C:\\program files\\Microsoft FxCop 1.32\\Rules).
If you want more advanced settings, configure FxCop with your targets and selected rules and then save the project file. Open the newly created .FxCop file in an editor (for instance notepad) and find the <Settings /> element. Expand it to an opening and closing tag (e.g. <Settings></Settings>), and between it add
<Rule TypeName="MethodComplexity"></Rule>
Between the Rule element add any of the following entries as Name attributes of an Entry element (as exampled following) -
Connection String - an OleDb connection string
determining where it will log metrics. e.g.
Provider=SQLNCLI;Server=(local);Database=Analysis;Trusted_Connection=yes;
Target Table - The target table for metric
logging. Default -
MethodComplexity
Cyclomatic Critical
Error - Level at which a critical error is
triggered. Default - 60
Cyclomatic Error - Level at which an error is
triggered. Default - 50
Cyclomatic Critical Warning - Level at which a
critical warning is triggered. Default - 45
Cyclomatic Warning - Level at which a warning is
triggered. Default - 40
Cyclomatic Information - Level at which an
infromation event is triggered. Default - 20
Cyclomatic Recommended - Recommended level.
Default - 20
Statements Critical Error - Statement count at
which a critical error is triggered. Default - 500
Statements Error - Statement count at which an
error is triggered. Default - 350
Statements Critical Warning - Statement count at
which a critical warning is triggered. Default - 250
Statements Warning - Statement count at which a
warning is triggered. Default - 200
Statements Information - Statement count at which
an information event is triggered. Default - 150
Statements Recommended - Recommended maximum
statement count per method. Default - 100
Callouts Critical Error - Callout count at which a
critical error is triggered. Default - 100
Callouts Error - Callout count at which an error
is triggered. Default - 75
Callouts Critical Warning - Callout count at which
a critical warning is triggered. Default - 50
Callouts Warning - Callout count at which a
warning is triggered. Default - 40
Callouts Information - Callout count at which an
information event is triggered. Default - 30
Callouts Recommended - Recommended maximum callout
count per method. Default - 30
For instance, you might end up with a <Settings> element that looks like the following:
<Settings><Rule TypeName="MethodComplexity"><Entry Name="Connection String">Provider=SQLNCLI;Server=(local);Database=Analysis;Trusted_Connection=yes;</Entry><Entry Name="Callouts Warning">100</Entry><Entry Name="Cyclomatic Critical Warning">500</Entry></Rule></Settings>
If you opt to take advantage of metrics logging, the destination table (which will be default will be MethodComplexity, unless overridden with the Target Table name entry) requires the following columns:
ContainingType - text (e.g.
nvarchar(255))
MethodName - text (e.g. nvarchar(255))
Cyclomatic - int
Statements - int
Callouts - int
e.g.
CREATE TABLE [dbo].[MethodComplexity](
[ContainingType] [nvarchar](255) COLLATE
SQL_Latin1_General_CP1_CI_AS NOT NULL,
[MethodName] [nvarchar](255) COLLATE
SQL_Latin1_General_CP1_CI_AS NOT NULL,
[Cyclomatic] [int] NOT NULL,
[Statements] [int] NOT NULL,
[Callouts] [int] NOT NULL
) ON [PRIMARY]
Hopefully someone finds this interesting. It scratched my itch.
While it's evident that Microsoft is staffed with a lot of top-notch people, history has empirically demonstrated that they have quite a few dregs as well: Just recall how disastrously the whole .NET thing was handled circa-2000.
For those who forgot, suddenly every product (including those finished or on the verge of being finished) became a part of the .NET vision, even if they had absolutely no interaction with the .NET technology stack: Windows Server.NET, Exchange.NET, Messenger.NET, SQL Server -- all a part of the .NET generation -- just as Microsoft declared everything in the generation before a part of the DNA vision (I still hear developers talking about "Microsoft DNA", not really sure what they're talking about).
As a developer who was heavily involved with the betas of what we call .NET today - a runtime and a framework, and the associated tools, for building next generation solutions - I really had no idea what .NET was in Microsoft parlance. Just as ActiveX got muddled into a meaningless term, .NET was being hijacked to basically mean "buy whatever is new or coming out soon".
Eventually that insanity stopped, and .NET collapsed down to a sortof virtual-machine runtime, a framework, and a set of tools. .NET 1.0 was one runtime, one framework, and Visual Studio.NET 2002. .NET 1.1 was a new runtime, a new framework, and Visual Studio.NET 2003. .NET 2.0 was a new runtime, an expanded framework, and Visual Studio 2005 (note the dropping of .NET on the naming, given that Visual Studio, as always, also makes non-.NET applications). There are countless assemblies and extension libraries available targeting each of them, and of course I can make libraries tomorrow that target .NET 1.0, .NET 1.1, or .NET 2.0, and it doesn't magically evolve them into .NET 3.0.
Well it looks like Microsoft is at it again. They've decided that Vista's technology platform, WinFX (which will be partially backported), is so great that it can't be just a set of assemblies or systems that the .NET runtime interacts with. No, it must be .NET 3.0! So now if you have the .NET 2.0 runtime, the .NET 2.0 Framework, targeting it with Visual Studio 2005, and you add in the WinFX framework...voila, you have .NET 3.0.
Insanity. Absolute, unbelievable insanity. Perhaps there's some amazing explanation -- for instance that their April Fools project ran a little long, and they just got the output out -- but I suspect it is just more of the same that we saw circa-2000. Some short-term euphoria over a gonna-be-released-soon project has them screwing with the terminology yet again.
Already the boards are full of "So....does this mean WinFX comes with LINQ?" (LINQ is one of the technologies promised for the next real wave of .NET)
Way back in junior high I had a good friend who was a huge fan of military aircraft.
His bedroom walls were covered with huge, hard to procure and often expensive posters of these deadly devices. His desk featured an actual (albeit non-functional) 20mm shell, of the variety used in the depleted-uranium spewing gatling gun.
His favourite military fighter jet happened to be the F-15 Eagle.
Feeling a little left out, I started pouring over his resources, carefully reading his encyclopedia's of fighter aircraft, absorbing all of their attributes. I decided that my favourite fighter jet was the F-14 Tomcat: Clearly its ability to land on carriers, its swing-wing engineering, and the long range phoenix missiles it supported, made it the superior aircraft.
There was no way the F-15 Eagle compared, I argued. The F-14 Tomcat was obviously the choice of those in the know. The enlightened ones, if you will.
Yet the reality -- and I think my friend Brian always knew it -- is that I chose the F-14 primarily because it wasn't the F-15. After picking a natural alternative, I started building layers and layers of justifications for my decision.
I see the same sort of thing fairly typically in software development: Big up front design versus agile designs; Getters/Setters versus fields; namespace naming guidelines of type A or type B; variable naming standards; stored procedure naming standards (or the religious "stored procedure versus dynamic SQL" argument that rages on in teams across the lands); the sorts of types to use for primary keys; the languages and platforms to choose; whether or not to use XML, and what to use it for.
So many times, it seems, people choose their positions based not on actual analysis and honest beliefs, but rather because they're countering someone else in their team -- especially when attempting to undermine authority, actual or perceived -- or they battling someone else in their organization (that dastardly team in Sector G that's trying to get kudos by setting the development guidelines!), or they're deriding someone in the industry.
Often They're just trying to be different and difficult, and the beauty of software development is that there are many, many right ways to do it, and it's easy to find allies in discussion groups to assure one that everyone else is idiots, and their new position is the One True Way.
It's easy to appeal to authority, given that there's some big name or organization that, in some form, promotes just about every software development practice and standard imaginable (Microsoft is a particularly good example of this, as throughout the organization they follow so many standards and practices, that one can easily find an example conforming with their dogma, using it as an example that it's the "Microsoft way", ignoring the many exceptions).
Of course all of this doesn't preclude disagreement on standards and processes and techniques -- people often truly disagree because they legitimately and rationally believe something different. In a full of intelligent, self-directed professionals, such disparate beliefs and conclusions can be enormously beneficial. The problem is when interpersonal issues materialize as technical disagreements.
"That design might work for a stateful desktop app, but it isn't appropriate for the stateless web."
"O/RM isn't appropriate for stateless environments like HTTP!"
"This component wasn't made for the stateless environment of HTTP!"
"...but HTTP is stateless!"
If you've done any sort of web development, you've probably heard proclamations like these. You may have even made them yourself.
But what do they really mean? Do they add any value to the conversation?
Stateless refers to an architecture where each HTTP request is fundamentally detached from requests that came before, and unrelated to requests that will follow.
In a stateless world, the browser initiates a TCP connection on port 80 - traditionally, or port 443 if it's a secure connection - and then sends some basic commands, such as the desired document (e.g /images/coolpicture.jpg), along with this-request preferences like the user's desired language.
With no prior information about the caller - acting only on the newly generated information in the request (e.g. the document requested, along with user submitted form values) - the server sends the results.
> GET /images/coolpicture.jpg
< the binary data for /images/coolpicture.jpg..
After the single request is serviced, the connection is torn down in this stateless scenario. The desired goal was to service each request as quickly as possibly, freeing the resource-heavy, finite-quantity connection to service other callers.
Maximum output with minimum resources.
This served the early web very well. Mirsky's Worst of the Web could be served out to thousands of anonymous consumers with gusto on minimal hardware, fulfilling the liberal information sharing origins of HTTP.
For a historic analogy, think of the 411 telephone service - you dial the number and establish the connection. You tell the operator the person whose number you require, and they provide a number in response. The call is disconnected, freeing the line and the operator for the next caller.
This is stateless in that the service relies upon no contextual information preceding the call to provide the service, allowing a small number of operators and connections to handle a large number of lookup requests, needing no resources beyond a simple phone book.
A stateful 411, on the other hand, would be one where you called 411 and left the phone off the hook, maintaining the connection for perhaps days at a time. With each number lookup request, they would try to interpret what you really mean based upon the requests that came before.
"Earlier you asked for a bait store on Main street, and now you're looking for a tackle store. I'm going to guess that you probably want one on or near Main street. The number is..."
Such a stateful connection wouldn't even require you to maintain the call - they could just pull up your records based upon the calling phone number, immediately having the history of your interactions to draw from in a stateful manner, regardless of the transience of the individual call.
The stateless definition of HTTP was used to contrast with existing services like telnet and FTP, where a TCP connection (itself a stateful protocol) was made, after which a state was maintained and modified from command to command -- whether you were logged in, what directory you were in, what application was running, and so on.
The state was alive and changing until the connection was dropped, with a block of server resources dedicated to keeping alive a world just for you.
That design worked for those services because connections were generally "higher value" per request - a long running file transfer that couldn't serve many clients anyways, as a function of the large number of bytes per request; a professor running some batch jobs; etc.
Most readers will know that almost all websites these days appear to be stateful.
You log on. It presents data that is specific to you, using preferences that are individual to you. As you do things, the environment changes and adapts, incorporating your interactions into following requests.
This isn't just an illusion, or a bastardization of the web: THESE WEBSITES ARE STATEFUL.
So how did the web sneak up and become stateful on everyone? Well, generally via the magic of cookies (alternately via URL-appended session identifiers to simulate cookies), an addition to the HTTP protocol that was first implemented by Netscape back in 1995.
A session cookie is often nothing more than a unique identifier (preferably with enough entropy that users can't guess each other session identifiers, for instance a randomly generated GUID), passed to the server on each request, allowing the web server to tie requests together, building a set of session data to provide state for a given client - The logon form changes the home page render changes the topic listing changes the calendar selector changes the news view, and so on, with each page having available a set of stateful information about the client, forming a sort of virtual "persistent connection" over many individual, seemingly isolated HTTP requests.
Ignoring the fact that in the modern world HTTP connections are reused (given that a client will often request dozens or more documents to build a single page - or in the case of Digg about 37,528 - it was found to be cheaper to just let the client reuse a built connection for multiple requests), often people differentiate HTTP from being "stateful" because it doesn't maintain a constant connection for the entire session.
Yet what is a connection? In this case it would be TCP, a "stateful" protocol. TCP is stateful in that it changes based upon what has happened before, and each packet for the duration of a connection relies upon those before them getting through okay.
You can establish a connection, let it sit for a while, and occasionally pass data back and forth.
TCP is stateful in contrast to IP (or its very light encapsulation, UDP), which is individual packets that live or die by themselves, with no consciousness of packets that came before, or those that will follow.
But wait, isn't it TCP/IP? TCP on top of IP?
Why yes, it is. TCP is fundamentally "IP with cookies", allowing it to maintain session state, tying many stateless packets together into a nice, clean stateful correspondence. This differs little from HTTP with cookies, a fundamentally stateful protocol when coupled in virtually any post-1996 implementation, where the idea of sessions and statefulness are the norm.
So why does everyone keep yabbering nonsense about HTTP being stateless (pedantically true, but practically irrelevant and entirely misleading)? Why do so many people talk about the web being stateless in the face of endless contradictory evidence?
I think it's just a cop out: People want to validate their crappy web apps - possibly due to laziness or a desire to migrate back to fat apps - so they clutch onto the justification that it's a fundamental limitation of the platform that limits their abilities, constrains their design or forces them into hackish implementations.
In reality, the web that we've been developing against for the past 10 years has allowed tremendous statefulness, including building up and maintaining enormous quantities of server-side state for every session (just like a fat app or a DCOM component): Just because that isn't appropriate for a very high volume, low value-per-transaction anonymous user website should in no way guide you in your implementation of a low user count, very high value-per-transaction vertical market web app.
You have the ability, and the mandate, to do what's right for the problem, and no one solution or dogma fits all web needs.
Joel Spolsky, the well-known blogger and ISV owner, kicked up quite a storm recently with his piece entitled Language Wars [for those following the `debate', yes, I'm late to the party on this. I make it a general standard to avoid responding to blogs on here -- the whole blog thing is entirely too recursive -- but some recent reactions to his piece pushed me to post].
The article leads off with some pragmatic wisdom, advising enterprise-y, low-risk type shops to use well-known and well-proven technology stacks -- solid advice that's hard to argue with -- yet he then ends the piece with a comment about an in-house, next-generation, super-duper language being used to develop FogCreek's premiere product, FogBugz.
The discord was so great that most readers presumed that the Wasabi thing was a joke, or alternately that the rest of the article was the joke (which would have been an awesome revelation). Much confusion ensued, to the point that Joel had to put up a post clarifying that he was actually serious about the Wasabi thing
Aside from the seeming hypocrisy, what really instantiated some JoelCritic<T> instances (via the BlogCriticFactory) were Joel's comments about Ruby, where he seemingly indicated that it wasn't ready for prime time.
...but for Serious Business Stuff you really must recognize that there just isn't a lot of experience in the world building big mission critical web systems in Ruby on Rails, and I'm really not sure that you won't hit scaling problems, or problems interfacing with some old legacy thingamabob, or problems finding programmers who can understand the code, or whatnot...
...I for one am scared of Ruby because (1) it displays a stunning antipathy towards Unicode and (2) it's known to be slow, so if you become The Next MySpace, you'll be buying 5 times as many boxes as the .NET guy down the hall.
I'm sure Joel anticipated the backlash. Perhaps it was even the motivation behind the posting: The resulting torrent of discussion brought quite a few visitors to his blog, and earned him a lot of inbound links, both of which have definitely helped with his new business ventures. No publicity is bad publicity, they say, especially if it's timed to coincide with the launch of a new job board (as an aside, Ruby, Wikipedia, OSX, Python, Lisp, and ERLang are all terrible! People with the letters J or P in their names are jerks!).
Ruby is still new enough, and with a small enough community, that many of its users double as evangelists -- think of the Amiga computer, the BeOS operating system, or any other contextually-superior alternative embraced by a small enough group that many feel an ego-intersection with the technology, motivated to defend and advocate it when the opportunity arises. Linux once had such an attack-dog core of rabid enthusiasts, though as the user base has grown, and it has become more pedestrian, you really have to target a Linux-niche (such as a little used distro) if you're aiming to stir up a hornet's nest.
That entire lead-up was just some context for the actual topic of this entry: So-called premature optimization.
A common response to Joel's complaint that Ruby is slow or resource inefficient is the frequently incanted declaration that such complaints are nothing but "premature optimization!"
I've seen the same deflection shield used to defend abhorrent database designs, convoluted, overly-abstracted class designs or message patterns, and virtually anything else where a realist might proactively ponder "but won't performance be a problem doing it like this?", only to yield the response "You know, premature optimization is a classic beginners mistake!"
If you don't want to be lumped in with beginners, the lesson goes, it's best to pretend that performance simply doesn't matter. We'll cross that bridge when we get to it.
Premature optimization is the root of all evil (or at least most of it) in programming.
Donald Knuth
I remember the early days: I once spent about 16 work hours optimizing a date munging function, increasing its performance from something like 2 million iterations per second to 4 million iterations. In the grand scheme of things, the performance difference was completely negligible, but from the perspective of artificial benchmarks it seemed like tremendous progress was being made.
That was premature optimization.
Indeed, anyone who's done time in the software development industry can identify with what Mr. Knuth was saying, probably having been involved with (or responsible for) project plans gone awry when efforts focused on highly-complex caching infrastructures, or ultra-optimizing some seldom used edge function.
Yet what is arguable, and situation specific, is deciding what qualifies as premature, versus what is simply proactive, predictive, professional performance prognostications.
NOT ALL PERFORMANCE CONSIDERATIONS ARE PREMATURE OPTIMIZATION!
While there is no doubt that there is such a thing as premature optimization -- it is an evil distraction that sidetracks many projects -- there are critical decisions made early in a project that can cripple the performance potential (both resource efficiency, and resource maximum), making later optimizations enormously expensive, if not impossible without an entire rewrite.
Whether it's heavily normalizing the database (or its nefarious doppelgänger, the classic database-within-the-database: "This single table can handle anything! Just put a comma separated array of serialized objects in each of the 256 varbinary(max) columns! Look at the flexibility! Query it? Don't you bother me with your premature optimizations!"), creating an application design that's incongruent with caching, or choosing an inefficient platform.
There are credible performance considerations that need to be addressed at the outset, and revisited as development proceeds. It is absolute insanity, and entirely irresponsible professionally, to simply stick one's head in the sand and hope that some magical virtual machine improvements or subcolumn indexing decomposition and querying technology will occur before deployment, or before the economics of scaling come into play.
And speaking of scaling, the canard that the horizontal-scalabilty intrinsic with most web apps (unless you really screwed up the design -- as many people do -- and made horizontal scalability impossible) makes the problem a nonissue is absurd: Perhaps if your project has a high transaction value then you have the luxury of adding more servers to serve a small number of clients, yet for most real-world projects adding resources is a big, big deal. And it isn't simply the cost of a low-end Dell 1850: Whether you're colocating or hosting in an expensively rigged corporate server room, the cost of each server is substantial.
You end up in the dilemma that you're financially (or physically) limited to a set quantity of resources, having to limit or scale-back the functionality provided to each user due to the inefficiencies caused by early decisions. "Sorry we can't implement that cool AJAX type-ahead lookups because the callbacks would kill our servers - we're already saturating them with our stack of inefficiency, so there's no overhead left."
I think the lackadaisical attitude towards efficiency is a result of experience derived from countless unvisited or seldom used web apps deployed across millions of PCs, colocated with equally as spartanly used peers. When a site sees a dozen visitors in a day, it's easy to declare that performance is a seeming nonissue nowadays - that it's only a concern for game programmers and nuclear modelling engineers. Then one day the page gets mentioned on Digg or Reddit or Slashdot or BobOnHardware and in that potential moment of glory the app falls over and dies, again and again.
None of this really has anything to do with Ruby. Personally I haven't used it beyond the tutorials, though I do know that it does very, very poorly on the standardized benchmarks. However it is distressing seeing so many people dismiss Joel's comments (or comments about Python, or ERlang, or XML, or any other technology) as premature optimization.
Why is it that "90% done" (and
its partner in crime - the ubiquitous "almost done!") is
the progress report for virtually any project, over virtually all
of its life-cycle?
Why has 90% become the fictional number of choice? Why not the more conservative 80%, or the bolder 95%? Given that it usually has little correlation with reality, they're just as real.
Projects should be reported as 87% done. Even when there's the ominous "we'll solve that problem when we get to it" task maliciously eyeballing you from later in the project plan, or the "it doesn't work and we have no clue why?" runtme reality, still say 87% with confidence and pride.
The well-known Hanlon's razor states-
Never attribute to malice that which can be adequately explained by stupidity.
While it's a seemingly pessimistic perspective on the capacity of one's fellow human, it is an undeniable truth that we often mistake carelessness, thoughtlessness, or outright ignorance as malicious intent.
Yet it's a more serene existence -- to the benefit of one's lifespan -- to simply assume that the person who dangerously cut you off on the road, for instance, is just a moron deserving a bit of sympathy, rather than considering him or her a roadway foe challenging you to a deadly battle of wills.
From a software development perspective, however, I think an inverted variation would serve the industry well.
Never blame others until adequately considering the possibility of your own (negligence | carelessness | stupidity).
As a general rule, the denizens of the software development profession -- it certainly isn't limited to this profession, but given that it's the general focus of this blog it's the one I comment upon -- have a tremendous capacity for assuming the worst of others, far before considering the unsavoury prospect that maybe -- just maybe -- it's actually their own mistake or lack of knowledge that's the cause of the issues they face.
It is far too common to cast a wide net of blame, declaring that Microsoft's products are screwing up, the documentation is all wrong, the server is malfunctioning (maybe because of cosmic ray particles toggling memory bits), the installation tool is a dud, and one's coworkers are surely idiots insidiously and maliciously changing code just to make one's brilliant code poetry fail to achieve its momentous glory.
After such hand-waving, blame-weaving dramatics, in most cases the developer realizes that they skipped an obvious step in the instructions, or they forgot to get latest of the entire branch, or they were copying the wrong file or looking at the wrong folder or running the wrong executable, or they were using the class entirely wrong, or they completely misunderstood how the operating system security system works, or they set a global setting a week back that completely changed how the application functions, or they ignored the email and documentation and group meeting detailing system changes, and so on.
They quietly retreat -- don't expect a retraction -- until they repeat the same mistake the next time something doesn't go exactly as imagined.
I've met these people in the industry. I've worked with these people. I've been one of these people.
I think we can all relate to situations where we've railed against a company, a product or a person, only to have the embarrassing realization that we were simply doing something dumb.
And it's not even that doing something dumb is noteworthy: We're humans, and we're bound to make mistakes. The problem is that we often don't even give a moment of time to even the possibility that we could be at fault, instead just assuming the worst of others.
It's far more beneficial to both productivity and team morale to have a little bit of self-doubt in these situations: Assume the worst of yourself before assuming the worst in others.
After hashing out this entry, I wondered why it wasn't appearing on the public blog. After berating various products and services, I remembered that I recently outsourced my DNS (for the reasons described here, using the service recommended by a reader), and forgot to add an entry for the FTP server. Whoops.
As a completely offtopic aside, one of the reasons I switched DNS providers was to have support for a domain SPF record. While it does nothing to stop the tide of pump-and-dump investment scam spams, at least it allows those recipients utilizing the service to immediately dump-bin those that claim to come from yafla (I get about 100 bounces a day, and who knows how more actually get through), knowing that the from: address was forged.