Prioritizing Solutions on Difficult Projects

Follow-up to Last Month's Column: The Ada programming language is slowly increasing its presence in the open source community, such as the 12 Ada presentations at a European open source conference last year (FOSDEM 2009 Home Page). There's over 1.5 million lines of Ada open source software in Debian Linux (Debian Home Page). Ada support is now available for the popular IDE, Eclipse (Eclipse Home Page).

Sometime before 1868, during the early years of Middle Eastern archeology, a group of Arabs found a 4-foot high monument stone with unknown writing on it. They offered to sell the stone to several different groups in order to secure a good price for it. During the negotiations, these men decided to divide the stone and sell the pieces. Some accounts say that it was a dispute over ownership; others say that they hoped to get a better price by selling fragments to multiple buyers. How do you break a stone? They used a simple solution. They heated it and then poured cold water on it.

It may not have occurred the these men that some things, particularly volcanic basalt, are not prone to split under extreme temperature changes: they tend to shatter into a nearly impossible jigsaw puzzle. Most—but not all—of the fragments were recovered and with the help of a rough copy of the original text made by a Captain Warren at an earlier time, the surviving pieces were reassembled and the artifact is currently on display at the Louve Museum in Paris (Home Page).

This stone, known as the Moabite Stone (or the Mesha Stele, after the Moabite king it mentions, Mesha Stele, Wikipedia) was one of the most important Middle Eastern discoveries in the past 200 years, providing irreplaceable archaeological information that affects the very foundations of Western Civilization. And this artifact, which had survived 2700 years, was nearly lost forever due to simple fire and water. And by the greedy men who thought they had a simple solution to their problem.

In the book "Software Creativity 2.0", veteran software pundit Robert L. Glass (Home Page) explores creative development under the strict discipline of a business setting. In Chapter 1.8, Mr. Glass looks at P. J. Plauger's (Home Page) concept of "The Falutin' Index". "Falutin'" is a slang term for something pompous or ostentatious (AlphaDictionary). Mr. Plauger uses the term to mean "complexity" as in a "high falutin' project" being a complex project. In Mr. Plauger's original article, he suggested that there are high falutin' projects, high falutin' programmers, and high falutin' solutions. Likewise, there were low falutin' (simple) programmers, problems, solutions and projects. If you assign a high falutin' programmer to a well-defined low falutin' problem, you could get an inappropriate, over-engineered solution. If you assign a low falutin' programmer to a well-defined high falutin' problem, you could get an inappropriate, under-engineered solution.

I remember inheriting a program at one company where the programmer needed to parse a data file. Wanting to learn yacc ("Yet Another Compiler Compiler", basis of GNU Bison, Wikipedia), he wrote a solution completely in yacc. When parsing problems arose after he left the company, it was faster to scrap the entire program and write a new, simple solution than to learn yacc and try to decypher his undocumented work. The original developer used a high falutin' solution to a low falutin' problem, costing the company time and money.

Mr. Glass quotes one of Mr. Plauger's stories. "I once saw an embedded system that essentially ran a glorified vending machine...It did the moral equivalent of counting coins, despensing goods, and making change...The proposed design modeled the data transform as half a dozen processes running under a commercial real-time operating system." The complex solution was scrapped for a simple one process program. (Page 48)

Mr. Plauger also points out opposite cases of low falutin' programmers on high falutin' projects. There are companies that hire large teams of cheap, underskilled developers and use code reviews, under-equiped quality assurance and project management staff to attempt to deliver complex projects. "You don't have to guess the outcome. You've either experienced this sort of fiasco personally, had a friend who did, or read about the financial fallout." (Page 49)

The world contains both high falutin' and low falutin' problems. The goal at the beginning of the project was to recognize whether a problem was complex or not and assign the right type of programmer and solution to save time and money.

The Falutin' Index is difficult to apply to real world projects. The story of the Moabite Stone is a reminder that it is often difficult to measure a problem's complexity, especially in software development where requirements are rarely known, understood, accurate or even stable, and it is only after the solution is put in place that you can evaluate its success or failure.

I've been assigned to many "death march" projects (Death March, Wikipedia) in my career—projects that are destined to fail even before they are started. These often occur when deadlines are imposed by management while not determining which of the requirements can be reasonably achieved by that deadline ("Facts and Fallacies of Software Engineering", Robert L. Glass). You know the ones where the boss gets off the phone with the client and says, "I told the client we already have this feature. So get your team together and build it before Tuesday." During one such project, I was assigned to the Linux administration team. The project had only been given about 1/3 of the time needed to make even a minimal solution (based on the actual time to complete it after the deadline had passed).

As I wrote in "The Diversi Effect" (Lone Coder November 2007), I wondered if there was some minimal preparation I could do to mitigate the most risk possible with the project. I could easily spend the entire time allotted to the project doing things to "do it right" ("Doing it Right or Doing it Wrong?", Lone Coder May 2008) but that would not help to get a functional solution by the goal date. I decided the one thing that needed to be done "right" to keep the project moving was the "push script".

A push script—that is, a shell script to take the software from the development server and copy changes out to the production cluster—took me about 5 hours to write and another couple of hours to fine-tune. The script could have simply copied some files around. But I wrote this script as solid as I could make it. For every file, it confirmed that the servers were up, checked the source control version, and confirmed that the file existed and that the name was unique (so there would be no ambiguity about what was to be copied). The script performed syntax checks if possible on the files to verify I wasn't copying broken code. The script copied the file, set all permissions, synchronized all the servers in the cluster, deleted any temporary files and recorded every change in a log file against the original request so the programmers and quality assurance people could see and verify everything. I tried to eliminate any possible source of error or miscommunication I could think of.

I was worried whether or not that 7 hours of development would result 7 hours of savings, but I didn't need to. In one week of the project alone, my script copied several hundred files without incident. As a result, there were never any problems associated with the software pushes and the developers and testers could focus exclusively on getting the project working. That 7 hours more than paid for itself.

In a death march project, this was a lucky guess rather than good design. Another person may have called the push script over-engineered because it was a small bit of programming of low visibility. But this was a case where careful prioritizing before getting into the thick of the project really paid off. The push script was the keystone on which all the other teams depended on to work together seamlessly.

A project can run completely off-track with an over-engineered (or under-engineered) solution. Even in death march projects, it's worth taking time—even one or two days—to determine the best way to tackle the project to avoid wasted effort and confusion. Benjamin Franklin (Wikipedia) once said "For the want of a nail, the shoe was lost; for the want of a shoe the horse was lost; and for the want of a horse the rider was lost, being overtaken and slain by the enemy, all for the want of care about a horseshoe nail." In most cases, you can't check every nail in the horseshoe and determining the best approach is a gamble. But you have to check what nails you can. Like the Moabite Stone, a person's dreams can shatter in a million pieces all due to mismatching priorities on a project.

January 22, 2010