When Web Scripts Appear to Scale

I was at the Post Office at the beginning of November and saw a Christmas tree and other winter holiday decorations on display. I mentioned that it was not even Remembrance Day (Wikipedia), Canada's day to remember war veterans, or American Thanksgiving (Wikipedia) yet.

"I know, " she said, apologetically. "Everyone knows it's wrong. But the order came down from the head office of Canada Post. We don't even get poppy box for Remembrance Day. But at the start of November, the Christmas tree is up...but I refused to plug in the lights!"

"Way to go! Fight the power!" I said.

At The Source (Home Page)--why did Circuit City (Home Page) rename Radio Shack in Canada, one of the most recognizable brand names, after a type of yogurt? (Yoplait Source Yogurt)--the salesmen were all dressed in red and green clothes and wishing people "Happy Holidays". There was no snow on the ground. Across the street at Wal-mart (Home Page) veterans were handing out poppies. Americans would wonder what was going on since they didn't have their November turkey dinner yet.

This, of course, is stupid. It looks stupid. Everyone knows it's stupid. But Canadians were always conformists, and nobody wants to be the first to admit that it's wrong. So we all pretend that everything is OK. Even though it isn't.

In "The Diversi Effect" ( Lone Coder), I mentioned how sometimes the simplest solution is the best. The key word here is "sometimes". Computer programmers often over-engineer solutions to simple problems. If you try to be simplistic about truly complex things you will get bad results. If you overlook required components of your project, then your solution will not work. A light bulb can be simplified by replacing the inert gas inside the bulb with plain air but then the light bulb burns out right away.

With web site development, most sites are very simple and use simple tools for rapid development, which is the way to go. Debugging is usually trivial and interdependence of components is low. Languages like PHP make it easy to integrate databases with a minimal amount of typing. Many PHP developers do not even know how to create a PHP object class, or if they do, they don't know the proper way to use it.

"But PHP is a dumb-ifying language," one multi-language computer expert said to me recently. "There's only one dumb way to do things. There's no concept of a real null. PHP people run across other languages like Java and discover a whole new world out there."

With a simple web site on one server, everything works as expected. Every function exists. The database is accessible. Backups and restarts are easy.

But as web sites become large, the requirements change. That PHP language that lets you type fast might now prevent you from creating large programs with complex interactions. The Content Management System (CMS) that made it easy to throw up a blog now gets in your way when you're trying to create more sophisticated, custom content. And when your one server grows to 500 or more, issues such as network delays, connectivity problems and maintaining a machine inventory come into play. So although many web developers are used to working at home on projects, few web developers have experience in dealing with the complexities of large web sites. And in most companies where the mantra is "Get 'Er Done, Son", there is a lot of pressure to ignore the requirement that all the subsystems "play nice" with each other. The web site becomes increasingly hard to maintain: like a house of cards, each new component can bring down the entire site in a cascading failure.

The 80-20 rule (or the Pareto principle, Wikipedia) says that 20% of a program does 80% of the work. And, of course, it also means that 80% of programming is dealing with rare 20% of situations, the exceptional and the abnormal. Brooks adds in "The Mythical Man-Month", "designing grand concepts is fun; finding nitty little bugs is just work. With any creative activity comes dreary hours of tedious, painstaking labor, and programming is no exception. next, one finds that debugging has a linear convergence, or worse, where one somehow expects a quadratic sort of approach to the end. So testing drags on and on, the last difficult bugs taking more time to find than the first."

For example, a poor article writer (or an overworked one) will write an article that says "here's what I did". The good article writer will try to break his work, document what went wrong, so the article says "if you see this, here's how to fix it". 80% of the writing process is to describe the uncommon and the exceptional, not what works. But this kind of quality and research takes time and effort.

I was reminded of the 80-20 rule when I ran into a programmer. Otis (not his real name) was an ambitious man who wanted to look like a top software developer, even though he wasn't one. He spent a lot of time hanging outside of managers' offices, talking about "his guys" on "his projects". When the managers weren't available, he hung around the manager's spouses (one of whom referred to Otis as her "stalker").

Otis consistently off-loaded his work on other people to make himself look more productive, to the anger of his teammates. He would brag about how fast he could design and build programs. Projects another developer would take months to design Otis built, tested and deployed in a few weeks. He didn't accomplish so much because of his skills. He did it by not following the 80-20 rule. He would design and build a mock-up or prototype, hand-off the work to another developer who would do all the testing, debugging and abnormal condition handling. By the rule, Otis never completed more than 20% of the work on any project.

Otis wasn't a good programmer. He was undisciplined, impatient, unable to focus on a task, research it thoroughly, and see it through to completion. He took no responsibility for the poor quality of his work, never tested anything thoroughly. People who relied on Otis' work would learn the hard way that what they paid for was not what they received. "But," said one of his coworkers, "he's good at what he does. He's good at appearing confident and competent."

Otis talked to me about his new project, a TCP/IP daemon written in PHP that received socket connections and performed some simple analysis. Most people would use C++ but Otis, who knew PHP, knew he could prototype something quickly. He was very proud of his work. The software was very small and ran very fast, little more than a series of system calls. He planned to roll out his project on servers handling thousands of connections in a real-world setting.

Networked applications are more prone to failure than standard programs. So I asked him about how he handled the abnormal cases, 80% of the work of building such a daemon.

But Otis explained that he was such a genius that he didn't need to do thorough testing or plan for the types of problems network applications have to face. The fact that his simple program ran very fast showed his skill and in the small load-testing environment he used, it worked very well under moderate, predictable traffic. I must have been jealous of his skills, he reasoned--that's why I questioned his work.

As Brooks writes, "Because [computers] are tractable, we expect few difficulties in implementation; hence our pervasive optimism. Because our ideas are faulty, we have bugs; hence our optimism is unjustified."

In Wired magazine, there was a investigation of the Osprey, the U.S. military helicopter that had blades that tilted forward so it could fly like an airplane ("Saving the Pentagon's Killer Chopper-Plane"). But the Osprey was hugely over-budget and had resulted in the injury and death. Investigation showed that the problems were linked with known bugs that we're never fixed. People kept pushing the Ospery project forward so the engineers couldn't repair fatal errors. Like Otis, the managers looked good with the 20% of the work that was done while neglecting the 80% required to make the Ospery a practical vehicle.

Soon I began hearing from Otis of problems with his PHP daemon. If it was started under high loads, it became confused. If there was a sudden surge in traffic, it hammered his client's database like a denial-of service attack. And when attached to the rest of his client's system, if it failed it brought down the client's entire web site.

But Otis shrugged when I asked him if he learned anything. In his opinion, there were always a few bumps on the road to success. He walked off, with an exaggerated strut, sucking on his water bottle, insulting the female developers.

Meanwhile, the next week at the post office, the clerk had all the lights turned on the Christmas tree. "Don't say anything, " she said. "I do as I'm told...even if it's wrong." Then she told me of her trip the supermarket. They were playing Christmas carols. A man came in the door, listened, and swore. Everyone laughed. Like the pink elephant in the center of the room, no one wanted to admit that something was wrong. Nobody wanted to look stupid. But they all knew the elephant was there, right in front of their nose.

November 23, 2008