PegaSoft Canada White Paper
Draft - 2003
By Ken O. Burtch
Credentials doesn’t mean anything if the software doesn’t hold up in day-to-day use.
Open Source is not a cure-all. For one thing, it doesn't cure laziness. Open Source, at it's root, means, "other people will do my debugging." Some people (following the 80-20 rule) do the 20% of the project that's fun then release the project as "open source" hoping that others will do the mundane 80%. Do be surprised if no one takes up this challenge.
Open Source often means software developed "by committee". As a friend of mine likes to say, "If a committee arranges lunch, everyone will be eating hot dogs." Open source is a strong tool when everybody knows what the goals are, such as duplicating existing functionality.
The problem of the "Black Hole of Complexity" has been steadily growing for the past two decades. Consider the Pentium processor line. Through pipelining, cashing, branch prediction and heuristics, Pentium class processors succeed in making lousy programmers look good. But it also succeeds in making good programmers look mediocre. The chip is so complex that it's nearly impossible to write a predictably optimal program.
Complexity has also made obsolete the idea of a "guru". Systems have so many different competing standards, developers and projects that no one can truly be called a Linux guru. No one is truly a master of all things related to Linux because it is humanly impossible to do so.
In "Rebirth of the Computer Industry", Howard Lawson speculates that open source encourages what he calls the "Black Hole of Complexity". Like a Mandelbrot Set, the larger the system, the worse the details. In open source, with each person pushing his own agenda, nobody takes responsibility for simplifying Linux. Instead, companies use the complexity of Linux in order to sell support, making a profit at the user's expense. [Rebirth, pg. 26]
Given enough time, even Linux will spiral out of control at the hands of open source developers, leading Raymonds assertion that open source reduces errors in the dust. Given a finite number of programmers with a finite amount of time and projects that grow infinitely complex, it is (theoretical) that all projects will eventually spiral out of control.
Laziness is not a reason for open source. 80-20 rule suggests that for every 2 open source projects a person initiates, they must participate in 8 others for open source to work.
Bill Scott of StoreReport.com says "Giving up control [of a department] to a trusted business partner makes a lot more sense than turning your business over to a staff of employees who may not have your best interest at heart. When you consider that the greatest security issue is 'sabotage by employees', why not eliminate it entirely by eliminating employees? Computers replace people." He makes his living bu FUD, but it's justified: how many companies are designed to run their employees into saboteurs? Can you be certain that the "trusted" partner will not also turn against you if you screw him over?
As long as their been “hackers”, there’s been people wanting to imitate them. There’s a certain appeal to being the “Lone Ranger” jumping on to solve problems with some technical silver bullets then riding off into the sunset. The first bout true hackers had was with the so-called hackers breaking into computer systems. These hackers were summarily dubbed “crackers”, ones who tried to earn the title of hacker by cheating to get it.
There is a thin line between true genius and flat out determination. When I was in high school, the smartest guy in class was a guy who went home at nights and spent all night studying. He pulled in the highest grades through sheer determination. I could never compete with that because, although I did my homework, I had other priorities as well. Crackers recognized this principle. If your not a genius, you can appear to be a genius through hard work hammering away at a system.
Or as another example, I’m always suspicious when people get 90% or higher in university level courses. They are applauded as geniuses, but in my experience, most students landing near perfect scores do not do so because of their affinity to the subject, but because they’ve already covered the material somewhere else. Making any course into a refresher course will boost the mark.
As a result, there are far more hackers claiming to be hackers than who really are. And people with a true affinity to computers are liable to seem slightly less heroic than the fakes who try to emulate the hacker myths. Any fake hacker is no better than a cracker trying to claim something he hasn’t earned, and in the process discrediting and slandering the reality.
Since false hackers are such a common occurrence, I’ll speak a bit about them here.
They have been books and papers written on this subject. Here’s a few personal observations.
1. False hackers boast about their knowledge. They try to show themselves as superior by criticizing those who question their knowledge, or use trivia to make themselves feel that they are superior in knowledge.
True hackers believe don’t boast about they’re knowledge. They don’t brag, defend themselves or try to show they are superior to other people.
False hackers may make idle boasts. Once I had a run in with a false hacker who kept sending me flaming emails because I wouldn’t agree with his opinion of himself that he was a expert. Finally, he boasted that my web site was so lame he could write it in a day. Fed up with him, I emailed his users group mailing list a short challenge: if he could duplicate the functionality of my web site in 3 days, I’d pay him cash. Did he demonstrate his abilities? Of course, not. Unable to fulfill his rash boasting, he posted a long flame full of name calling to the user group.
In the Bible, Paul the apostle questions the value of credentials and people who boast about themselves. Instead, Paul says that a man’s worth is measured by what he does and the usefulness of his words. Credentials, claims and boasts, if they don’t back that up, mean nothing. [2 Cor 12]
2. False hackers believe someone else will do their work. In the Batman comics, Bruce Wayne dons a cape and battles crime in the night. Who keeps his stuff running while he’s away? Alfred. If Batman was a Disguised Hacker, Alfred would get no credit and would expect to clean up Batman’s dirty little secrets.
Good thing Batman is True.
A real hacker knows about the 80-20 rule. He never boasts about getting prototypes up and running.
3. False hackers steal. The only good way to do open source development is to steal cycles from my employer.
While volunteering with the Canadian Linux User Exchange at Comdex/Toronto one year, a Linux company put out a box marked with a sign “Donations to Support Linux”. Convention goers tossed coins into the box in order to support the Linux cause. When the convention was over, the company used the money to buy beer. It would have been better if they had given the money to Microsoft.
Whether your stealing cycles or stealing passwords, you’re still taking something that doesn’t belong to you. You’re being a cracker. True hackers want to give, not take away. True hackers never steal.
4. False hackers deny moral codes or religion. Computer expertise, per se, has nothing to do with religion, so why do false hackers claim with pride to be atheists, or, at least, Zen?
Hackers are not anti-establishment (or “anti-suit”) because they have nothing better to do.
5. False hackers equates “suits” professionalism. Lack of showers or hygiene. No agendas or meetings.
False hackers are against showers because it puts people on the defensive. False hackers are against agendas because giving people something to think about means that the false hacker no longer controls the information. False hackers are against meetings because it means having to allow others to express their opinions. He who speaks loudest sways the crowd, and anarchy is the perfect vehicle for that.
True hackers know that professionalism, per se, is not bad. The abuse of professionalism, the mendacity of North American companies, and the control of groups of people to fulfill personal agendas is bad. Missing a shower because you saved a billion dollars of people’s life savings by rescuing a system is good. People being upset with you because you did a good job is good. Missing a shower and having people upset as a result is bad.
6. False hackers believe that you don’t need to understand other people.
Understanding the enemy.
False hackers are dangerous in any team. They are generally malicious individuals who are out to destroy your efforts. Rather, they are self-interested individuals who are willing to derail a project for their own self-interests. It's never a good idea to say, “Oh, he's just a prima donna” or “He's just a maverick”. These individuals have psychological issues that need to be sorted out—they are not normal, well-balanced nor do they have healthy personalities.
The Prima Donna
A prima donna is a developer who is unskilled or skilled at a small area. They have invested their self-esteem in their career and feel that any questioning of their abilities is a personal attack on them. A prima donna will try to exaggerate their abilities and other developers who they are afraid will expose their limitations.
The irony is, as I said in the last section, there are few true hackers in Linux because the field is so large. Most people are experts at a subset of Linux and have no problem with that. A prima donna is unwilling to accept this and will sacrifice productive team members in order to preserve the illusion that he knows everything, even if it means sacrificing the project to do so.
Prima donnas are difficult to deal with because any suggestion that they don't know everything will be met with resistance. Sometimes a team leader's only recourse is to isolate them from the rest of the team.
The Maverick
The maverick is a developer who has skills but who doesn't like producing good quality work. They know that when a project is complete and does what it should then the need for them is over. The longer they can stretch out a project and the more they can lock out other team members, the more they are needed. They will write complicated, difficult to read code and comment nothing so they will be kept on staff to maintain it.
The maverick is unable to see that if good work is done that they will be kept around. Their fear of the future causes the quality of their work to degrade, increases maintenance costs and makes it difficult to interface their work with the rest of their team. Although their work looks good on the surface and they have programming skill, having a maverick on your team is no better than hiring someone with lesser skills.
Don't let a maverick railroad you. To get the best work out of him, insist on code reviews and proper documentation.
The Gentleman Thief
Polite and friendly. Always smiles and laughs at your jokes. The gentleman thief is an opportunists waiting to steal away the team your developed or the project your landed. Saying things like “Nothing personal; it's just business”, the gentleman thief politely steals your hard work.
The gentleman thief acts out of greed and pride. It's a game to dup you and the sense of getting results he didn't earn is as important as the money. In his mind, the one who gets more results for less work is the smartest man. In reality, the gentleman thief is not different than someone who breaks into your house and steals your TV and thinks he's a smart shopper—he's nothing more than a leach that thinks the world owes him a living.
Don't allow the gentleman thief to seduce you. Grade them on their performance, not their talk, and let them know that leaching will not get them ahead.
The Fisherman
Searching for information good or bad, the fisherman is a developer who has little or know programming knowledge. He hangs around the water cooler or after work, chatting with people on the team while searching for information or planting seeds of discord. They might say “So how would you tackle this kind of problem” and then present your solution to the boss as his own. They might say “Jim seemed to have some trouble yesterday. He doesn't really seem to know what he's doing, does he?” and suggest that he should be the team leader instead of Jim.
A social engineer skilled at manipulating people, the fisherman seems keenly interested in you and what you have to say. However, it's just a ruse to cut-down team members and to steal information to make himself look productive and keep himself on the payroll. A fisherman will undermine the trust on the team and will not hesitate to push people out of the team if it means keeping himself around.
Like the gentleman thief, tell the fisherman that gossip is not appropriate behaviour and that he will have to do real work to get real rewards. Build up trust in your team so nobody will take the fisherman seriously.
The Right Guy
The right guy always wants to do the right thing. By “right” he means that the team should follow his personal interests and not what's in the best interests of the project. Many university students naively believe that what they are taught by their know-it-all profs is the whole story: in school, they work on toy problems, completely defined problem domains, and don't have to deal with the real-world issues of personality conflicts, client management or project costs. They demand that things are done their way because they don't understand that there may be other, better alternatives.
The right guy is an individualist driven by a need to prove himself worthy by being right. He wants to be “right” to hide his ignorance. He will avoid working with team members, discussing alternatives or looking at the hidden costs. He will threaten to quit the project rather than learning something new. He is a force that fragments a team. In the army, new recruits who took officer training before becoming cadets are often subjected to the most strenuous training to ensure that they realize they are part of a team and no cadet is more important than another.
Sometimes right guys try to take on the entire team's responsibility, trying to be right about everything, and burn themselves out—dragging down the project in flames.
The Basement Developer
Lurking in the basement of a building or a back corner, the basement developer is terrified that his skills aren't good enough. He may be a truly skilled developer but he's worried that his failures will humiliate him in public. He sticks with obsolete technology, well-known tools, and undertakes low-profile projects because they are “safe”. He's the guy running Linux kernel 1.0 when the standard is kernel 99.9 because he “likes” 1.0—in reality, he's terrified of how he will look if he trades up.
A basement developer suffers from low-self esteem. In a team, he will warn people away from new tools that cut costs, will exaggerate the risks involved in doing things the “unsafe, new fangled” way. He may even leave a project if he his forced to learn new technologies. He requires special hand-holding and reassurance that the new technology is safe but his programming skills can be a valuable team asset.
In call these cases, these false hackers are not “naturally” disposed to be who they are. They are destructive team elements with personality disorders. Don't let their delusionary views of the world lead your team on a course away from your real objectives. Let them know that you're doing real work and that they are jeopardizing their share in the rewards by their behaviour.
Companies and organizations often hire hackers believing them to be the real thing. The problems of hiring was already talked about above. It’s easy for a false hacker to slip
As a rough rule of them, the 127 Rule suggests the 9 out of 10 people claiming to be gurus are, in fact, fake gurus in disguise. False hackers fall into the “2” or “7” category of the 127 Rule. They are never “1”’s because they are not heroes.
False hackers are experts on false pretenses and are often insecure. They will want to gain control because they will use that power to discredit or harass those who question them. Unless you stand up to them, they will attempt to control your group through sheer force.[Open Source]
What about the false hacker to claimed he could duplicate my web site in a single day? He’s still a high-ranking member of his user group.
Dealing with false hackers is difficult because their life is a empty façade, even when everybody knows it but them. They want to feel special and superior, and attacking their credibility is attacking their religion or the foundation of their life. They live in a world of mendacity.
Remember that Microsoft is a company designed and run by hackers.
Open source development has nothing to do with false hackers.
"Basically, this is the way the economy works. I do a service for you, and you pay me, even if you claim you didn't want the service and that I "ruined" something of yours." [LDT]
OpenSource.org has a poor defence of open source called “The Business Case for Open Source”. This is a FUD document containing many of the classic FUD elements: clear statements (“if that’s too abstract for you”), Too Good to Be True statements (“smaller shops can handle larger projects”), threats (“kill your business”), false dichotomies and so forth.
From the point of view of software development, the paper makes 4 principle claims.
First, they claim that development becomes faster because more programmers work on the project. This isn’t true because of the 80-20 rule. For open source to work, for every 2 projects that a team makes open source, they must contribute to 8 projects sponsored by other teams. The average development speed is unchanged.
According to one article, the open source development of Netscape 6 actually slowed development. [Reference “case for open source” folder]
Second, they claim production overhead is lowered because of free outsourcing. This is also incorrect for the same reason as the last claim. By the 80-20 rule, people must donate their time to your project, but 80% of your time (which you must pay for) must be donated (for free) to their projects. The upshot is that the development costs are redistributed, not reduced.
Third, they claim broader markets exist as people provide free ports of your software. This is unlikely due to the 127 rule. The problem is that nobody wants to spend their time and money porting your software unless they have no other recourse. There have been many requests to port some of my projects. When I say, “It’s open source…go do it”, they ask if I think they are a “sucker”. “Free” ports are not possible for the majority of open source projects.
Finally, they claim that open source serves to secure mind-share. This argument is a false dichotomy: closed source software can also be free for download. Commercial software can have free demos. There’s nothing uniquely special about open source downloads that makes it any more accessible for trials and testing.
So open source isn’t cheaper, faster more portable or more effective at marketing itself. And FUD is alive and well in the open source community as it is in the closed source community.
A common argument for open source is the value of open source. Some have argued that the success of open source is assured and inevitable.
In “Shoulders of Giants”, Con Zymaris compares open source to the scientific method. He says that “science is a process of verifying or culling hypotheses and is in essence an open and self-correcting system.” [SOG] Peer review allows science to correct mistakes, and open source likewise allows source code to be reviewed by peers.
But science has often failed because of peer pressure and a lack of accountability. The idea that the earth revolved around the sun was first proposed before the time of Christ, but the world had to wait 1500 years before the idea was fashionable again in the time of Copernicus [cosmos]. Peer review can work against advancement and self-correction.
Computer science has had its share of sacrificial lambs, including the goto statement and the failure of neural nets.
There is a fundamental difference between the scientific method and programming: science believes that the universe is controlled by predictable laws and that these laws are consistent, repeatable and knowable. No matter where scientists are, the facts they pursue are concrete. But programming is as much an art as a science. Users don’t always know what they want and certainly not with any scientific accuracy. Programmers may have conflicting visions of the end goal. Optimizing one part of a complex application may cause another part to operate poorly. Peer review is much more subjective in programming that in science.
If we stopped here, then we’d be making another sacrificial lamb. But NASA has turned peer review into an art. There source code is not only reviewed by errors are logged and anticipated. If an error is highly probable and it is not found, the programmers are sent back to their code to find it.
If peer review is useless for goal settings and major changes in project direction and fixing major design bugs, what is it good for? The answer is “obvious”: peer review is best as discovering obvious bugs such as typos and basic logic bugs. Different people have different skills for the detection of errors. A C programming who has a talent for equal sign bugs is more likely to detect equal sign mistakes than one who is not.
Ironically, obvious bugs are often introduced into a project by developing the project with the wrong tools. The foremost open source language is C but C is notorious for “obvious” bugs. A programmer can eliminate many obvious bugs simply by using a better language. [So is peer review a benefit to open source or a kludge to make C a viable development language? The issue gets muddled.]
For small projects, using appropriate development tools increases the effectiveness of a small number of peer reviewers. For larger projects, it makes the peer reviews more effective.
If you are using peer review to catch bugs introduced by C, PERL or Python, perhaps you need to use better languages rather than have more peer review. It may be more efficient method of isolating errors.
Another way to make better use of peer review is to chose project more suited to it. Peer review is less effective when there is not a well-understood common goal. Projects like Linux and Freeciv are based on software that programmers spend many hours with, learning not only the look, but the weaknesses and the feel. If Freeciv doesn’t fee right, a Freeciv developer can run Civilization II and compare it. If Linux doesn’t feel right, a kernel developer can run programming experiments on a commercial UNIX. Peer review is always more effective at copying existing software than for creating new, groundbreaking technology that is being “felt out” as development goes along.
The need to feel valued is a universal human desire. We want to feel like we’re an important contributor, that our lives meant something. North American companies consider people to be “human resources”, quantities with no intrinsic value or individual skills. Computer personnel are sometimes expected to contribute to programs that can make-or-break a business deal or an entire company without any recognition or thanks. As workers with access to privileged data, corporate developers can find themselves with a job without any warning for “security reasons”. It’s no wonder many programmers feel mistreated and undervalued. Nobody loves a resource.
Driven by a need to feel appreciated and to have a person connection on a project, some programmers turn open source. By opening their source code up to peer review, they vindicate their talent and demonstrate their worth despite the attitude of corporation. They sit by their email boxes waiting for the accolades. And accolades are always a good thing.
However, open source praise can be very rare. If you release a project quietly, few will notice it. You need self-esteem and self-confidence enough to market and promote it, to be your own advocate. If you attract enough people, you will begin to get feedback but the 127 rule suggests that the majority of feedback will be neutral or negative. People will tell you your design is all wrong. They will say that features are missing and, instead of contributing, they will delete your source code. But if you can survive this painful and lengthy process, praise will eventually come.
Peer recognition is a fickle thing. Bad projects can be praised. God projects can be dismissed. Popular projects tend to be favoured over quiet superior work. The praise of faulty human beings is uncertain at best. If the only reason you are interested in open source is pure recognition, perhaps it is religion you should be pursuing instead.
If you are an open source contributor, be sure to send a few words of encouragement along with your criticisms. They will never be wasted.
With peer review, it seems reasonable to assume that open source software can be more reliable and secure. Many people investigating the source code, people with different skills, should create a more reliable product than a small team can implement and test alone. Users don’t have to wait for a small team to make fixes: they can patch the project themselves and contribute the fix back to the community. More reliable software is more trustworthy when handling sensitive data or fending off malicious intruders.
We’ve already seen that peer review is most effective for certain kinds of bugs in certain kinds of projects. If a project is innovative, or unpopular, or contains complex bugs, peer review will be of limited value.
A survey of major security advisories at CERT, conducted by me, showed that the vast majority of security alerts were for buffer problems (that is, array mistakes). Some programming languages (notably C-based languages) are prone to these sorts of errors. Peer review can catch many of these mistakes, but if security was an important consideration in open source projects, why are so many open source projects undertaken in C-based languages? It’s the wrong tool.
It’s also arguable that peer review is primarily aimed at fixing broken functionality, not at approaching a project form the point of view of a malicious hacker. Since security is a consequence, not a goal, security bugs can be missed or ignored. However, once a bug is announced for a popular project, someone usually contributes a fix quickly.
From a user point of view, simply fixing the bugs isn’t enough: the software must be replaced on their system. Open source offers no advantage over closed source for actually updating software. In fact, malicious programmers have faster access to security loophole information. Unless user continually update their software, creating a moving target, open source offers has no advantage. In fact, it may be a sign of slow binary rollouts.
This was shown with the 2002 Linux Slacker virus that exploited an announced hole in OpenSSL. The virus itself was open source and relied upon the Linux development tools to be installed and accessible.
In “Shoulders of Giants”, a paper on the superiority of open source, Con Zymaris claims that open source development is not only viable but that it will make closed source development obsolete. But is this a reasonable prediction?
Zymaris argues his point on a number of grounds, including:
peer review works for the scientific method. But we have seen that that is not always true and that the scientific method has fundamentally different goals than software development
peer recognition will cause programmers to abandon closed source. But we have seen that praise is fickle and difficult to obtain, even in open source
open source software is extensible. But many don’t want to extend software and may even become a project’s enemy
open source is more secure. Patching functionally bugs and language weaknesses improves reliability, but weaknesses are easier to exploit and there’s no easy way to install patches.
In none of these is open source’s dominance guaranteed.
Zymaris goes on to describe what he calls the “Tom Bombadil” factor-his belief that open source software is invulnerable to monopolies and corporate corruption. “All known cases,” says Zymaris, “have shown that [a creation of a defacto standard], eventually, becomes corrupt with the power they have to wield.” [SOG] This is, of course, an overstatement, but corporate corruption is a reality and the reasons for it were discussed earlier.
In “Slack”, Tom DeMarco notes that the first instinct of many is to try to solve problems by altering process. In this case, Zymaris believes that an ethical problem can be combated by the way we produce projects and services. Methods cannot change the human heart.
The Canadian securities industries is supposed to be a self-regulating industry, much like open source development. It didn’t stop Bre-X, a fake gold mining company, from stealing millions of dollars. The legal protection of licenses like GPL doesn’t prevent source code abuse. And the discussion of the 127 rule shows how a few corrupt individuals can control the many.
There’s no doubt that self-regulation is helpful provided that 2’s aren’t in power. But if people rely on a process to protect their rights, 2’s will find a way around the process or will dominate it. The everything is lost. Open source doesn’t ensure uncorruptability.
Based on the evidence, open source is a viable software model, but there’s no evidence that it will dominate over closed source development.
"Research has shown that the more people present when humans commit violence, the more likely anyone is to come to the rescue." [IA]
People want to contribute to open source after it's finished. They don't want to design it. They don't want to fix it. They want to port it or add additional functionality to a fully working project.