A Server by Any Other Name

Grocery stores create millions of cash register receipts every year. These paper slips list customer purchases, usually in a layout difficult to read. They are also used to solve disputes and track expenses, so they include details like the date and the store location. You would think that, after years of creating receipts, their problems and requirements would be well-known. Some receipts are better than others. There ought to be standards or conventions for content and layout. They should be easy to understand, with directions on how to resolve disagreements, and be designed to handle common, real-world problems like torn paper or faded ink. However, that doesn't seem to be the case.

I took at look at receipts from several grocery stores from last year, looking at what appeared on them besides the items purchased:

An asterisk indicates success. The best receipt was from Price Chopper, the worst from Metro, but none of the receipts were designed to meet what I would consider to be minimum requirements. Let's look at the Metro receipt:

• Metro has redundant date/store info printed at both the top and the bottom of the receipt, making it easier to piece together a torn receipt. But it only gets half marks because the information printed in the same order on both lines (when ink runs low, both lines will lose the same information).
• Metro's total/balance appears to be labeled "tax" because of a bad layout.
• Metro has some dates in dd/mm/yy format and some dates yy/mm/dd format, making the dates difficult to read. All companies used six digit dates without identifing the component order, making dates like "10/11/10" ambiguous.
• Metro gets half marks for customer service because, although it lists a 1-800 customer service number, it doesn't list the current store manager. Store managers can change.

Some may say this is a trival example: I mean, come on, it's just cash register receipts. Most people throw them out when they get home, and angry shoppers can just go shop somewhere else. Dealing with so many shoppers every day, surely all the cash register receipts above are good enough and it's not worth the money or time to design readable and usable receipts.

Perhaps.

What about choosing a hostname for a computer. This is another problem receiving little time or attention from computer professionals. When the install program displays "Hostname?", a sys admin might spend all of three seconds before typing "anubis" (Wikipedia) and clicking next. Mythological references, cartoon characters, animal names, and allusions to science fiction or fantasy literature are the most popular hostname categories (Web Hosting Top.) Some people do even less, using names like "production1" or even using random numbers like "a14232". I mean, come on, it's just a hostname. Is it worth money or time to decide what makes a good one?

Even with a small number of servers, some hostnames are superior than others. In fact, there is an Internet standards document (RFC 1178 - FAQs.org) that provides guidelines for choosing hostnames, including:

• Avoid confusing or ambiguous terms. (e.g. don't call a server "up" or use a name that is also a domain name)
• Don't make names too general (e.g. don't call a server "server" if you will have a second server in the future)
• Keep names brief so they're easy to type (e.g. "db1" is preferable to "new-database-server-running-mysql-1-installed-by-fred").
• Avoid random strings as typos are hard to detect (e.g. names like "a14232" and "a14322" may look the same if you rushed).
• Don't use silly or alternate spellings (e.g. you don't want to be explaining why a name is funny while the boss is threating to fire someone).
• Use lower-case for consistency.

Small server environments (with a dozen machines or less) can function well following these guidelines. Where there are few servers, everybody can remember what each server does. They will know that "anubis" is a database server, for example.

With larger data centers (with dozens or hundreds of machines), people cannot remember what each server does, let alone come up with unused names easily. When the server "anubis" fails, the only thing the first person to respond knowns is a hostname. You need an idea of what the server is for and which people on your team you need to contact to investigate the problem. In the worst case, when the president of the company calls you to find out if the failed server is affecting customers, you don't want to answer "I don't know".

The challenge in coming up with a good hostname practice is to balance meaning with security and permanence. Hostnames should convey information about what to do if the server fails while not revealing confidential information and not forcing the sys admins to rename the machine after minor changes. Changing hostnames is disruptive. Information like serial numbers could be abused (or, at least, is useless since someone typically will have to travel to the data center anyway to examine the server before calling the manufacturer for support).

The hostname needs to identify two critical pieces of information:

• The environment that it runs in to determine priority.
• The general function so that the right people can be contacted.

So "prod-db-1" is an example of a good hostname. It identifies the computer as a database server supporting the production site. If this server fails, you know immediately that it affects the actual users and that a database administrator is needed to restore to the software. A server with hardware to support databases is likely to remain a database server in the current environment so the hostname will rarely change. The name contains no confidential information like server's serial number. This strikes a good balance between crisis recovery and information that is secure and stable for the life of the server.

Some organizations with servers with in many small data centers will include in the name the telephone extention of the team to contact if the server fails. In this case there are a large number of teams and the team location is more relevant for contacting people than the general function. Organizations with many subsidiaries or web sites might include this is in the name as well, assuming the hardware will likely remain with the subsidiary or a particular web site.

One drawback of this approach is that it doesn't identify what software (or "services") are provided by the computer. (Using a name like "anubis" doesn't either.) With large number of servers, you need a simple way to identify where software is located. This is done using hostname aliases (in /etc/hosts) or domain CNAME (or Canonical name) records in your local DNS server. For example, "prod-db-1" may have an alias "wordpress-db-1", indicating that the server has the database for your installation of Wordpress (Wordpress Home Page). When you configure Wordpress, use the alias name (not the hostname) to connect to the database. Unlike hostnames, these alias names are easy for sys admins to rename.

Using aliases for servers has other advantages:

• In the event of a failure of a server, the sys admins can easily move services around. If "prod-db-1" fails, causing the Wordpress database to be unavailable, the sys admins can load a backup copy of the Wordpress database onto another database server (such as "prod-db-2") and can redirect the "wordpress-db-1" alias to point toward "prod-db-2". This technique is also useful when servers are shut down for upgrades, or when buying new hardware and moving several pieces of software onto one, more powerful machine.
• For sites using multiple environments (such as development, unit testing, alpha or integration testing, beta or User Acceptance Testing testing, release staging, production, etc.), you can use aliases to identify which servers have which services without having to alter the software. For example, "wordpress-db-1" could refer to "prod-db-1" on production, "dev-db-1" on development, and so on. depending on the environment. As long as the software only uses the hostname aliases, the software can move seamlessly between environments. Don't put the environment name (e.g. "prod") in the service name as that would prevent software from being copied unaltered between environments.

• Having a good naming convention is helpful for using many large server methodologies, like Service Oriented Architecture (SOA).

What about other important information that could have been in the hostname or aliases? This information can now be put into the login banner file (usually /etc/motd). If there is a problem with the server but it is operational, simply logging into the server can provide these details. The banner can include:

• Make and model of the computer.
• The location of the server (such which data center and rack position).
• The operating system.

One method to manage this information is to add it as comments in the local DNS servers. A shell script can auto-generate the login banner files by doing a hostname lookup and extracting the information from the comments. If you are not using a local DNS, the information could be kept in a text file somewhere and use a format that makes it easy to search with grep. This puts all your detailed server information in one central place.

Choosing hostnames may seem as unimportant as the content of a cash register receipt, but good names are very important as the number of servers grow. A good hostname will make scaling up a data center much easier and will be a big help in a crisis. Who knows, it might even save your job.

May 22, 2010