Case Study: How to Create a Dell Linux Web Server

This article summarizes how to take a new or empty Dell server and create a working Linux web server, such as an Application Service Provider might use.

If you are familiar with the networking and installation basics of other systems such as Windows or Apple computers, you do not need a Linux expert to setup a basic installation. However, to do a complete, customized installation tuned for maximum performance, you will need experienced Linux people on hand to configure everything you need. Open source technologies allow software to be optimized for your particular hardware: to perform a complete installation, you will need to know how to navigate in the Linux operating system, how to build software under Linux (using make, tar, cat and other commands), and how to configure the Linux kernel parameters.

What Kind of Server Should I Purchase?

The main bottleneck in resources is your Internet connection. An 66 Mhz 80486 computer can act as a home firewall, and a 133 Mhz Pentium I web server can max-out a home high-speed Internet connection. Computers today have far more processing power than most connections can handle. If you are purchasing a standard business Internet connection, any new Dell server--even the low end models--should have more than enough speed to serve up most basic web sites. Your main interest in a server is redundant power supplies, RAID hard drives, and quality equipment that will give a high up-time.

If you have high-processing needs (like 3D-graphic modeling, heavy data sorting or artificial intelligence applications), you'll have to perform some tests to see exactly what kind of hardware you will need.

Typically, you will purchase two servers. The development server, usually located at your office, is where all software development takes place. A second identical server called a production server is the "real" server that contains the latest, stable copy of your web site or product. It is the production server that users access.

The goal is to set up the development and production servers as closely as possible. They should definitely have the same version of Linux. Ideally, they should also have the same hardware. If the two servers are identical, loading the product onto the production server should be easy and the risk of errors is minimal.

Why Does a Linux Installation Take Longer than a Microsoft Windows' Installation?

The basic Linux installation process is very similar to the way you install another operating system. However, installing server software is different from installing simple programs on your Windows desktop: you will need to configure the parameters for your databases, web server, etc. to meet your particular needs. You will need to do this whether you are using Linux, Windows or another operating system.

Open source technologies allow software to be optimized for your particular hardware. Unlike some software, which is built for a minimum system to run on as many machines as possible, Linux allows you to rebuild software, tailoring it for your hardware. Why run software optimized for a five year old processor when your server uses the latest chip? It would be better to rebuild the software for the actual hardware you are running. Of course, it takes longer to rebuild the software than doing a regular install, but taking this extra step will improve the performance of your server.

Servers can also include Dell-specific hardware. Dell has its own installation tools and you will have to take special steps to setup your Dell products. The Linux CD-ROM will not include everything you need.

Once you get your server configured, you will also want to run it for a week or two to test the installation, install backup software, and to consider anything you may have forgotten.

Required Tools

To perform a complete Linux installation on a Dell server, you will need to know how to use the following tools:

A Text Editor -- you will need to know how to use make changes to text files. If you are not doing an X Windows install (that is, if you won't have a desktop), you should be familiar with the standard vi text editor.

A Shell Session -- This is the Linux "DOS prompt" Start a shell session by opening a "terminal window" by right-clicking on the desktop and selecting the appropriate menu item. You can exit a shell session by typing "exit".

A Web Browser -- If you are not doing an X Windows install (that is, if you won't have a desktop), you can use the elinks ncurses-based version of the lynx text browser. Use the mouse to move and click links. Press ESC to get the menu bar. Red Hat now uses elinks in place of lynx.

Common Shell Session Commands

Here is a summary of basic commands used in a shell session during a software install.

• The cd command changes the current directory (folder). For example, cd /usr/src will change the current directory to /usr/src.

• The mv command moves or renames a file. For example, mv file1.txt file2.txt renames file1.txt to the name file2.txt.

• The cp command copies a file. For example, cp file1.txt file2.txt creates a copy of file.txt called file2.txt.

• The rm command deletes (removes) a file. For example, rm file1.txt will delete file1.txt. Red Hat is set by default to prompt you to verify that you really want to remove the file--there is no trash can or recycle bin to retrieve a file once it has been accidentally removed.

• The ln command creates a shortcut (a link) to another file. It is usually followed by the -s (symbolic link) switch.

• The chown command changes the owner of a file. Under Linux, only the owner of a file can delete the file. For example, chown root:nogroup file.txt will change the owner of the file to root (the administrator), and the group owner for the file is nogroup, the default for no particular group.

• The chmod command changes the access rights to a file. Access rights are complicated affair: see the Linux documentation for specific combinations of permissions.

• The fgrep command filters output, return lines that match specific criteria. The fgrep command is used to find a specific word or phrase in the output of a command. Use fgrep -rsn to recursively search directories for a file containing a word or phrase. For example, fgrep -rsn 192.168.1.2 will search beneath the current directory for files containing the IP number 192.168.1.2.

• The rpm command to install and remove software. Red Hat Package Manager (.rpm) files are standard file format for installing programs on Red Hat Linux. These contain only a basic version of the program (no source code). rpm -i package.rpm will install a package called package.rpm. rpm -e package.rpm will remove the package, if no other packages depend on it. To get a list of all the packages installed, use rpm -q --all. To look for a specific file, the easiest way is to grep the output.

There may be additional commands you will need to know, but these are the basic commands.

Tools Required to Build Software

Tuning will require a Linux developer. You'll need a developer because you will be rebuilding software in these steps.

The tar command creates compressed tape archive files (.tgz or .tbz). Tar files (or "tarballs") are the usual way of distributing Linux source code. Use tar xfvz package.tgz to unpack .tgz files or tar xfvj package.tbz to unpack .tbz files.

The basic steps building Linux software from source code is:

• cd package -- move to the package source code directory

• ./configure -- configure the software for your system

• make -- build the software

• make install -- install it

However, additional steps are often necessary. Always read any README or INSTALL files included with the source code and follow the instructions contained therein.

Step 1: Record Hardware Parameters

The first step to install Linux on your Dell server has nothing to do with software installation at all. Examine the server and record all the model and service information. You will need this information in order to get support for your computer or to look up help on the Internet. It is usually easier to get this information before connecting up the computer or moving it to a new location.

From the shipping document, determine the hardware configuration. After you've completed the basic install, use the /proc file system and the ispci command to check the chip sets and other information (this will be discussed later). You will need this information for choose replacement parts and to optimize or rebuild the Linux kernel.

If you are co-locating the server with a service provider, print out and tape this information to the server. Include contact information so the company can get a hold of your in the event of an emergency.

Step 2: Start or Finish Red Hat Installation

The next step is to start the Red Hat Enterprise installation. If the machine is new from Dell, the install will have been started for you but you must complete it. If you're reinstalling Red Hat, you will have to start the install from the beginning.

Red Hat installation by Dell is a "typical" Red Hat install.

1. Create two starting users (you can create the rest later)

   1. When prompted, create a user called "root". This is the system administration login and you will be using it a lot to complete the setup.

   2. When prompted, create an additional user called "webmaster". This is the user who will own the web site files.

2. Setup the Basic Networking. This should be the networking settings for the computer's current location, not the settings where the computer will be moved to when the install is complete.

3. Set the Security Level. Disable the firewall. In most cases, the server will be behind a separate firewall computer so firewalling is not necessary. If in doubt or if you know the server is going to be directly on the Internet, with no firewall protection, you should turn on firewalling to maximum.

4. Set the Date and Time. Use Network Time Protocol (NTP) and clock.redhat.com.

5. Customize the Software Packages.

   When Red Hat presents you with your desktop, you will have completed a "typical" (common setup) Red Hat install: Red Hat has installed a collection of basic programs that cover most applications. There will be a lot of software installed that you don't need, and additional useful programs are available on the Red Hat CD-ROMs. You will need to customize the software installed so that it matches your needs.

   Run add/remove packages from the main menu. Red Hat shows several software categories. To add/remove specific programs, click on the details button. If you are not sure what a program does, leave it alone or do a web search to find out more about it.

   In particular, remove any desktop software including office suites, graphics editing, music playing, etc. This is a server, not a desktop computer. Set the login options for a standard, non-graphical login (i.e., init default to level 3) using the Login Screen tool.

   If you intend to rebuild your core applications (discussed below), you can remove the older, unoptimized versions that come with Red Hat.

   Turn off any unnecessary services using Red Hat's services tool (or the setup command in a shell session).

Step 3: Install the Lastest Security Packages

If the installation software doesn't prompt you, click on the circular icon on the right hand side of the tool bar on the Linux desktop to start Red Hat's update manager. (You can also start it by typing "up2date" in a shell session.)

The first time you run the update manager, Red Hat will ask you to create an account for downloading updates. Each account, I believe, is for a client (not a computer).

System is registered but not yet "activated". Go to www.redhat.com/now to activate the product. Be prepared to enter your Red Hat Enterprise registration number.

Select the account you created. Run the update manager and install all the available updates, including all the kernel updates.

Reboot the computer to make sure all changes have taken effect. In the unlikely event that the computer does not give you a login prompt, you can still gain control of the computer to make changes. At the grub screen where you select a kernel, type "a" and add the word "single" to the list of kernel parameters. You will start the computer in Single User Mode, a maintenance mode. From here, make changes and use the "init" command to resume the startup (e.g. init 3 for a non-graphical server).

Setup secure shell passwords for your user accounts. Log in using "su - user" where user is the name of a user. Generate a new set of SSH keys using ssh-keygen -t rsa for each user (including root). Use the default save file (/home/user/.ssh/id_rsa) and an empty passphrase (your files will be protected by Linux passwords already).

Copy the corresponding public keys (/home/user/.ssh/id_rsa.pub on the new machine) to the authorized keys files on the computers you will be signing on from (i.e. /home/ken/.ssh/authorized_keys on the development machine, etc.). If you cut-and-paste with a mouse, make sure a line break is not added to the key at the end of the first line. Make sure that write permission for group on the authorized_keys file is turned off. If you can't connect with ssh, check the /var/log/secure log (on the new computer) for error messages.

Step 4: Install Dell's Software

A Dell server comes with additional Dell server management software called Open Manage (tm) Software (or OMSW). This soft is written in Java so that it will function both on Linux and Windows. Under Windows, there are short-cuts in the start menu to them. It has its own copy of the Java run-time environment is located under the /dell folder. The Dell guys didn't worry too much about security: the ownership and permissions are setup incorrectly.

The Dell software differentiates between management and server software. The server software is installed on the server and does some automatic tests for you. You'll need to install the management software on another computer so you can connect remotely, run tools and diagnose problems.

To add Dell's software, open a terminal/shell session and use the following commands:

$ cd /dell/omsw
$ ./start.sh

This will install Dell's Server Administrator, Diagnostic Administrator, etc. so they will be started whenever the server is restarted.

	...
	attatched scsi generic sg1 at scsi0

Type "exit" to leave the shell session.

Congratulations! You've finished installing and configuration Red Hat. The devices are configured, the security patches are installed, and the basic application software is installed.

Step 5: Download the Linux Kernel

Although you have a working version of Linux, Linux is "open source": this means you can download and rebuild your critical software, tailoring it to your particular hardware.

The core of the Linux operating system is called the kernel. The kernel is the program that manages all the devices and coordinates all the running software. By downloading and building a custom kernel, you increase performance, get the latest Linux security patches, speed up the boot process and free up system resources.

In the case of Red Hat Enterprise 3, it ships with a modified version of Linux 2.4.21. Red Hat has retrofitted this kernel with security patches from newer version of Linux. Ideally, try to get the latest Linux kernel to run on your computer.

1. First, download the newest Linux kernel from www.kernel.org. Place the file in /usr/src and untar the kernel sources in this directory.

2. Update the Build Tools.

   Check the Changes file and upgrade any software packages that are too old for the new kernel.

3. Configure the New Kernel

   Get the hardware information you gathered in Step 1

   Use the make menuconfig command (there are other variations that you can also use). Beware: there are many submenus. Include only the necessary features for your hardware. The new kernel will start faster and will be optmized for your processor. Wherever possible, compile the drivers into the kernel and don't use modules (M).

   If you have more than one processor, or if the processor is hyperthreaded (most modern ones), turn on SMP (symmetric multiprocessing). With hyperthreading, you computer will appear to have more processors than it actually has.

   Backup your .config options to a safe file in case you accidently erase it.

   make 2>&1 | tee kernel_build_output.txt
   make modules
   

   The tee command will save the output (including error messages) to the file kernel_build_output.txt.

4. Installing the Dell RAID/Ethernet Drivers. Dell drivers are installed using a command called dkms. Similar to RPM, DKMS (Dynamic Kernel Management System) is a set of shell scripts for unpacking, compiling and installing third-party kernel modules without placing them in the kernel source code directory. It is very dependent on how things are named and where they are located. It also has poor error recovery. For full information, check out Dell's online DKMS documentation. Basically, you will need the dkms match, dkms build and dkms install commands.

Step 6: Installing the New Kernel Using Grub

Compile and install your kernel modules:

make modules_install

"Unresolved symbols" indicates some things are modules and some are not but they depend on one another. Use make menuconfig to look for drivers based on the unresolved symbol names for items you missed that can be turned off or compiled in.

Install the kernel:

make install

Do not remove the original Red Hat kernel. You will need this kernel in case you install new hardware in order to boot the computer.

If for some reason you want to manually update the Grub bootloader, edit the grub.conf file. Run grub and (usually) type "root (hd0,0)" and then "setup (hd0)". Only do this if you know what you are doing (or you have time to redo the installation).

Restart the computer to try out your new kernel. If you are unable to boot, boot using the original kernel. If you are unable to do this, use the Red Hat installation CD and select rescue mode. This will mount your hard drives and you'll be able to make changes to your systems.

Examine /var/log/messages for module errors ("char major" errors means that /etc/modules.conf needs to be updated for aliases for the proper device drivers) or applications startup errors.

Step 7: Customizing Your Environment

1. Check for Errors during Startup

   If ntpd is failing, try adding a -d option to /etc/rc.d/init.d/ntpd

2. Setting up /etc/profile

   Set up the environment variables for your software here, as well as any shell session defaults. These include the cvs's CVSROOT and Java environment variables.

3. Setting up /etc/motd

   Create a "Message of the Day" file. This will be shown on a secure shell login and provides a nice way of identifying the machine you are logging into.

4. Setting up /etc/issue

   This message is shown on the main console. Create a login message with the company name, address and phone contact information for the service provider in case of an emergency.

Step 8: Rebuild Your Core Software

Download and rebuilt your core software. Use the GCC optimization settings for your machine. For a web server, rebuild your database and web server. Use the exact GCC optimization settings for your processor (e.g. -march=pentium4 -O3).

Review your configuration files for your core software. Tuning tips are often available on a web search. For example, if you are running a PostgreSQL database, you can greatly increase your memory usage and achieve better performance. Be prepared to increase the shared memory limits on the kernel as required in /etc/sysctl.conf (e.g. for 256 Megs, kernel.shmmax = 268435456 and kernel.shmall = 268435456).

Write a shell script to periodically check your web server and restart it if the server should crash, emailing the administrator to investigate the problem. You don't want angry phone calls from your clients when the server is unavailable. Schedule the script with the crontab command.

Step 9: Change Your Networking for Computer's Final Location

If you are moving the computer to another location, run the setup command and enter the networking information for final location. If you have several DNS nameservers, edit /etc/resolv.conf and add them there. If you need to change the hostname, edit /etc/sysconfig/networking/profiles/default/network and type the new hostname there.

Step 10: Reboot

Make sure that the computer restarts without any errors before you move it it its final location.