The Business Shell Evolution Blues

Where did all the onion bagels go?

Onion bagels used to be the most popular flavour of bagel, at least around here. There were onion bagels in bags, onion bagels in the bakery section of supermarkets. Every store carried them. Tim Hortons carried them. Not anymore. According to a Sobeys bakery clerk, the problem is many of the bagel and bakery companies merged together, and they are the suppliers to supermarket chains, both in pre-packaged bags and bagel mixes for the bakery section. For some reason, these mega-bagel companies each decided to reduce their product lines, and all of them simultaneously discontinued the most popular bagel flavours. The best-selling onion bagel no longer exists.

If all stores buy from you, you can tell people what they will eat.

I used to work for the Mackenzie Financial Corporation, a North American mutual fund company. Every team has a project that nobody wants to work on. The anchor around our necks was the M.O.R.E. reports. Originally this was a Korn shell script to print Excel spreadsheets on a UNIX printer. Over the years, the M.O.R.E. reports kept growing: printing more complicated reports, publishing them as HTML, formatting them and controlling access rights. A simple shell script grew into dozens of large, complex, interconnected scripts with meaningless variable names and no documentation. Work on the M.O.R.E. reports was always assigned to the newest guy on the team since nobody wanted to touch them.

As the new guy, I got to work on the M.O.R.E. reports. And as I worked on them, I realized the problem with the reports was that they hadn't evolved properly. When it was a shell script to print spreadsheets, it worked well. But shell scripts don't work the same as most modern languages: they have their own logic and way of thinking. The larger the M.O.R.E. reports grew, the more difficult it would be to convert it to a more appropriate platform. It was a problem of software evolution.

So I thought about this. The original Bourne shell was limited for a couple of reasons. First, it had to be small to fit into the limited resources of early UNIX machines. Second, it had to make typing commands as simple as possible, and with few keystrokes. With modern machines, the resource limit was no longer a problem, but typing commands was still an issue. And the Bourne shell had become a standard on all UNIX-based machines.

I created the original Business Shell (BUSH) as an experiment to see if a different language could be used while maintaining the Bourne shell command structure. I wanted to replace "for" and "if" while keeping the standard "ls -l". For a language, I needed something that had the features and reliability to work at a financial company while being widely available and portable, able to allow the scripts to migrate and evolve, solving the problem of the M.O.R.E. reports. Similar attempts had been made before, such as the C shell (csh, Wikipedia), but C has many well-known problems. I needed something more reliable. I chose a subset of GCC Ada which I called AdaScript. In BUSH, things like for loops and variable declarations would adhere to the ISO Ada standard which would allow them to be compiled in GCC or other Ada tools, while the commands would follow the Bourne shell rules.

To make the script evolution easier, BUSH operated in two modes. In the normal mode, Bourne shell commands were allowed. This made making scripts (and using the command prompt) easy. In the more restrictive mode (pragma ada_95), the shell commands had to resemble function calls like "ls ("-l"). As a project grew larger, the developers could enforce more Ada compatibility, making the final port to GCC Ada easier. The shell features could be gradually phased out instead of waiting for a costly total rewrite.

BUSH grew over a lot over the last six years. However, I haven't released a new version of BUSH in over a year. You may wonder what the hold up is. Ironically, it's the evolution of BUSH. What started as an experiment in software evolution is now a true shell, larger and more powerful than the original Bourne shell. It has hundreds of built-in functions, a debugger and database client features. These are all good things but they are way beyond the original experiment. In order to move forward, the infrastructure of BUSH must grow to accomodate new features. Here are some of the things holding BUSH down:

• Windows version: Ada is a financial, scientific, aerospace and military language. Most of these areas are heavily invested in Microsoft products and they have no use for a Linux-based shell. The most common request is for a Windows version of BUSH. First, I'm not a Windows programmer. Second, shells are tightly coupled to an operating system. Although I've been working to isolate the operating system dependencies to make it easier to port BUSH to Windows, Windows is fundamentally different from UNIX. It would take a skilled Ada/Windows developer a couple of months of full-time work to port BUSH to Windows. I haven't been able to find anybody to commit to that.

• BUSH RPM: For better exposure, BUSH needs to be distributed in the RPM format. RPM is the standard package format and BUSH could be included with Fedora, OpenSuSE and other RPM-based Linux distributions if there was simply an RPM of it. Well, RPM's are not easy create. I did work on an RPM of BUSH but it failed. I plan to return to this problem on the next version of BUSH.

• Exception Handling: For reliable error recovery, BUSH needs exception handling, but it will involve a major rewrite of the language parser.

• Dynamic Modules: For large projects, BUSH needs to have user-defined packages/modules that can be loaded when the program runs. Here, again, more major rewrites, this time it's the symbol table.

• Objects: All the cool interpreted languages have objects. But objects encapsulate other things, meaning that I would have to complete the rest of BUSH's features before adding objects. So objects have always been something to be added later.

• Web Sessions: To be useful with large web sites, BUSH needs session variables. These also require a lot of work, especially to ensure they are handled in a simple, secure way. I don't want to repeat the mistakes of PHP sessions.

• The Dubyah Factor: As many Canadians have pointed out, "BUSH" is not a good acronym due to the unpopularity of a certain American president in the international community. Of course, I wasn't even considering that particular Bush when I came up with the name. But after six years, is it a good idea to rename a project? Perhaps.

BUSH wasn't the first shell that attempted to deal with the evolution problem of shell scripts. Java developers have recently released a Bean shell (bsh) which has, in my opinion, many of the same failings of the C shell. The fact remains that Ada is not a popular language. It was a reasonable choice for Mackenzie Financial but it is not reasonable for some shop developing in Java to switch to Ada (even if one thought that Ada was a superior technology). Like the onion bagel, we conform to the limitations of the dominant standards, no matter who sets them.

When will the next version of BUSH appear? I'm not sure. Most of the priority features are simply too large to be done quickly by one person. I'm half-finished adding dynamic modules but I'm getting older and I don't have the free time that I used to. If Linux had taken off in industries where Ada was popular, I might have been able to get financial support to work on BUSH. That seems unlikely now.

I was speaking to an open source developer in the Toronto area. He had a new, clever idea that, like BUSH, was a new way of thinking and could be a great advantage to people. But, also like BUSH, it went against established conventions. He quit his job to work on his project. It's now his life's work, his proof that he lived in this world. I didn't want to discourage him because true innovators are rare, but I asked him how people would use his project. He didn't know. It wasn't important. His project was something that he just needed to do.

Perhaps that's the benefit of open source: if the industries ever change, the BUSH source code will still be around. But BUSH isn't the proof of my existence. Hey, it's only software.

As for Mackenzie Financial, I can't help but wonder what would have happened to the financial community if a technology like BUSH would have been developed and used. What kind of cost savings could have been made? What kind of computer failures could have been averted? It's fun to think about, but I guess I'll never know.

April 21, 2007