Doing it Right or Doing it Wrong?

In virtually every company I've worked for, from a pair of web developers in a small town to a multi-billion dollar company, they always say the same thing: "we have no money to do things right" or "we don't have time to come up with a good solution". These are really the same thing. If small companies worry about the bottom line because their finances are limited, and big companies cut corners to complete, don't all businesses do things wrong on purpose, and then try to justify that wrong is right?

"Being right by being wrong" is not a good way to run a business. Rather, it's a sign of psychological delusion.

A good example is the failure of Ruby on Rails. Rails promised rapid development of web sites but it soon became apparent that Rails' main delivery was FUD (Fear, Uncertainty and Doubt). Sites that relied on Rails discovered its limitations for scaling up to serious commercial applications. Hansson defended Rails in Wired magazine by saying it was deliberately kept simple to avoid frustrating developers...by leaving out design features to handle size and complexity, design concepts taught in software engineering classes. The result was a product that created quick demos that could never expand to handle a serious commercial websites. Following a philosophy that wrong was right, many companies that relied on Rails, according to the Wired magazine article, found that they had written themselves into dead-end web businesses.

Take a look at the heroes of science. People like Einstein or Turing are household names. They changed the lives of people. But many of these heroes had messed up personal lives, broken marriages or even suicide attempts. In 1980's British archeology series "Testament", the host compared scientists to saints of old and gods of religion, but what kind of example do they represent? Is "doing it right" in science nothing more than a blind obsession to failed ideologies that leads to self-destruction? (Compare this with my discussion of the ACM in The ACM Lost its Way).

You know what people mean by "being right":

• being cheap
• doggedly following a standard
• to have value to a peer group (that is, ego)
• to do what everybody else is doing
• doing it how you were taught in university
• in the case of Rails, ignoring reality

In the book "Slack", consultant Tom DeMarco discusses the common practice of matrix leadership--that is, having project managers who are separate from the departmental heads, so that each developer has more than one boss and had to work on multiple projects at a time. According to a study done by the Atlantic System Guild with hundreds of developers, developers lost nearly a hour a day through the costs of a matrix environment: the mechanics of moving to a new task, rework due to inopportune aborts, immersion times for think-intensive tasks, emotional frustration and loss of a focused team environment with a single goal. The study's conclusion was that software companies trying to be more efficient by hiring project managers usually lost productivity instead. But for most companies, appearing to be right was more important than actually being right.

In The Diversi Effect, I talked about how small, targetted optimizations in programming can be used to great effect, while excessively complicated optimizations can sometimes result in an inferior product. But to do targetted optimizations, you have to spend a little money to tell what needs to be optimized, and a little time to implement it, in order to reap great savings in time and money later on. The first step to being right is being smart and seeing the big picture.

I had a professor who often talked about "dipthong compression" (or DC). In English, there are many common pairs of characters, including "sh", "th" and "er". Dipthong compression is a dictionary compression algorithm that searches for pairs of characters and encodes them to save space. Is DC efficient? The average compression is about 70% of the original size (a 30% reduction). But it can be written from scratch in a couple of hours, is easy to understand, easy to debug, is very small, runs very fast and works acceptably on short English strings. To me, DC is smart because it is a Diversi effect solution: it is a simple solution that delivers well compared to what you invest into it. So I used dipthong compression to reduce the memory footprint in the latest version of my Tiny IDE for Anything.

To me, doing it right is delivering the best solution possible within the time, budget and reliability constraints.

Recently I was using Azureus, the popular Bit Torrent file sharing program on SourceForge. This Java application is very portable, has an automatic update system and has nice displays. On the other hand, it is a resource hog and if you get into the configuration screens, you need an IT certificate to adjust the settings. I've gotten into the habit of always running Azureus at a low priority with "nice -20" (-10 won't do it) to make sure it doesn't hog the processor and cripple my other work.

While downloading a large torrent last year, Azureus crashed when it ran out of file descriptors. It continued to put a large stress on my CPU so that, even when I attempted to log in as the administrator, the log in timed out because the password prompt took more than 60 seconds to appear. According to the Azureus wiki, this is not supposed to happen. In fact, there's no reason why it should happen in any well-built program. I tried going with ktorrent, the KDE Bit Torrent client available with OpenSuSE. No good. It, too, attempted to open more than 1000 files simultaneously when the download was started, and crashed.

Now many programmers would say, "Good grief! I could do better." Absolutely. The Bit Torrent protocol is actually pretty simple and a decent programmer could probably write a simple text-only Bit Torrent client in a week or two that could gracefully handle a large torrent.

But there's a difference between a decent programmer and a smart programmer.

A smart programmer knows that creating a working client is less than 10% of the work. 90% of the work is handling problems gracefully. What if the Internet connection goes down? What if the user deletes one of the files as it is being downloaded? What if another Bit Torrent client tries to bombard your program with bogus requests? Can the program recover without locking up the user's computer? How and when do you verify that the chunks you are seeding aren't corrupted? And, most of all, aren't there many bit torrent clients already available and perhaps there's an existing one that works better.

Just because you can do something doesn't mean that you should do something. I wasn't about to write yet another BitTorrent client. Instead, I edited "/etc/security/limits.conf" and raised the file descriptor limit from 1024 to 8192. I logged in and confirmed the new limits with the "ulimit" command. Azureus was up and running again...

Until it ran out of memory.

Oh, well.

If programmers judge themselves by how "right" their solutions are, then how intelligent is obsession, or working in isolation, or sacrificing the wrong things in the pursuit of science? Work skilled but also work smart.

May 19, 2008