Live in southern Ontario? Attending dinner meetings is free.

PegaSoft's March Dinner Meeting

Date: March 21, 2006 at 7:00 pm
Location: Office Bar and Grill

Attendance

Ken B, Mel W, Ivan F
Scott E did not make it.
Evan L was unable to attend due to a scheduling conflict.

Meeting Business

In the interests of full disclosure, Ken is unemployed and is drawing money
from the PegaSoft account to pay for the PegaSoft web site costs.

Open Forum -- Industry Quote

The views are those of the participants.

"Quite simply, I wanted to examine this factually, using real customer
scenarios to test this hypothesis: Can Linux run on older hardware than
Windows? In many developing countries and public institutions, such as a
local library, they typically don't have deep technical staff, so they
need to use software without lots of modification and customization.
This is why our testing focused on installing modern distributions of
Linux and modern versions of Windows 'out of the box'-simply putting the
CD-ROM in and installing-on the legacy PC hardware in our lab. -- Bill
Hilf, director of Platform Technology Strategy at Microsoft

Mr. Hilf's tests were neither factual nor involved real customer scenarios
and his conclusions could be viewed in different ways.

Linux is a distribution containing thousands of software packages and
XP is an operating system only.  Some of Mr. Hilf's failures
refer to programs included in addition to Linux not Linux itself.

The idea that Linux installs itself on any hardware automatically is absurd.
Software cannot be installed on hardware that it was not designed for.
Windows XP can not be installed on an embedded device or a Power Mac using an
Intel Windows XP CD-ROM.  Since Mr. Hilf was installing XP on machines that were
beneath XP's advertised requirements, it suggests that XP is an older
technology than advertised and runs doesn't take advantage of modern hardware.

An operating system must be installed and configured by someone who is a
trained professional in issues like partitioning, virtual memory,
networking and how to run a setup tool.  He allegations that Linux needs
"deep technical staff" refers to one's ability to run the open source setup
tools ("configure" and "make").  [Would he claim that to double-click "setup"
under Windows XP is "deep" knowledge?--KB]

The Linux kernel can be optimized to one's hardware with a few keystrokes.
Unlike Linux, XP can never be optimized for one's hardware. [GenToo Linux
rebuilds all software on-the-fly for your hardware when you install--KB]

There is also the question of what is "usable".  Ken B relayed the story of
someone who installed Windows 98 and Microsoft Office on a 386: although
the hardware requirements were below what Windows wanted, the install was
successful, but the results were very slow.  Ivan F related that the beta
version of Windows Vista (Longhorn) is reported to be brutally slow on
modern hardware and would be unusable on older machines even if it could be
installed.

Despite Mr. Hilf's conclusions, Ivan F reports Debian Linux installs and
runs well on his 486.  Both Debian and Knoppix are designed to run on a
large selection of hardware.  Other Linux distributions have more modern
requirements because they are optimized to perform better.  Suppose these
Linux versions were not optimized for modern machines: Mr. Hilf could then
claim that Linux was slow.

A distribution could include optimized versions of programs for different
hardware but it would multiply the testing requirements and the size of
the distribution.  Ken B. related how the SuSE Linux 9 installer said his
machine is "impressive" and questioned whether or not he wanted to install
the Intel version on an AMD computer.  Both versions of SuSE Linux were
distributed separately.

The participants concluded that Mr. Hilf's true objective was to throw
suspicion away from Vista's bad performance, to hide XP's poor
optimization and to keep people from trying Linux by making Linux's install
process sound scary to people who don't know any better.  He also may have
been making a bid for XP on the $100 laptop project.

Red Hat Fedora Core 5 Released (with Mono and XEN) [Slashdot]

Ivan F related that XEN provides a VMWare-like ability to run more than
one operating system at a time.  In theory, XEN 3 will allow Linux and
Windows to run simultaneously and a user will be able to switch between
the desktops with a couple of keystrokes.  XEN requires on-chip virtualization
features provided with the latest Intel and AMD processors.

Ken B, as he related in his Lone Coder column, was frustrated with Fedora 4's
instability.  Installing upgrades and patches fixes one software bug but will
cause another.  Red Hat's own update tool never worked properly and Red Hat
never provided a fix.  These instabilities made it difficult for developers
to support Red Hat.  [The BUSH download poll shows Red Hat's market share
has taken a huge nosedive since Fedora--KB]

It was pointed out that Red Hat's intentions may have been to deliberately
leave bugs in to encourage open source developers to provide patches.

CentOS and White Box are two free Red Hat meta-distributions that are based on
with Red Hat Enterprise and are, therefore, fully tested.  Novell's
OpenSuSE 10 provides Mono and XEN and is fully tested and stable.

There was strong reasons for developers to move from Red Hat.

[Brian Sharwood] said Canada's largest municipal electrical utility,
which last year purchased Toronto's street light system for $60 million,
will likely install the necessary wireless transmitters and receivers
atop every fourth or fifth lamp post as a way to blanket the city with
coverage -- what the industry describes as "wireless mesh
networking." [BoingBoing]

Skipped due to lack of time.

PegaSoft Member Projects

Business Shell (BUSH) (Ken B)

No recent updates. Changes will be made shortly for using BUSH in the
Online Game Project.

PegaSoft Online Game Project (Ken B and Mel W)

To help new developers to the project, a draft specification is now in the
CVS repository.

Ken B and Mel had been discussing the messaging format.  Mel was confused by
Ken's emails so Ken went over the design logic of his latest proposed format.
Mel found it difficult to predict what features would be needed without a
working prototype of the game.  Ken agreed.

Ivan F recommended that Ken and Mel investigate dbus.

Mel demoed his new "proof of concept" game client which featured a
selectable first-person or overhead view.  The joystick controls changed
depending on the view.  The client was also being designed to allow players
to remap the controls.

Mel had updated his Python dummy server to allow multiple clients.  He
successfully logged in two different players who appeared on each other's
screens.

Ken installed the AdaXML packages available for Ada Core Technologies.
Documentation was sparse but he was able to integrate SAX XML parsing into the
server.  Mel said the client was using the older EXPAT technology (an ancestor
of SAX).  The new server implements the latest proposed message formats and
communicates over an asynchronous I/O channel.  He successfully ran simulations
for several basic server commands.

For next month, Ken and Mel would attempt to combine their work and get the
different components speaking to each other.  Ken would have to install XML
support on the development computer and install the latest version of the
server.  Mel will have to modify his dummy server to call the Ada server.  Ken
and Mel would try to meet face-to-face to test their results.

Discussion: Thousand of Clients per Server (Ken Burtch)

      * "Game Programming Gems" is a book series by Charles River Media
        
      * Features white papers from developers at ATI, Electronic Arts,
        Sony, Nintendo, etc.
        
      * Major sections include Mathematics, Physics, Artificial
        Intelligence, Graphics and Network Games  
        


      * 6.2 "Thousands of Clients per Server" by Adam Martin, Grex Games
        


      * the term "multiplayer" is deceptive because there are both
        different types and different ways of handling multiple players
        
      * these can be conceptually divided into different categories by
        the number of players
        
              * 2 players - one thread, split screen or shared keyboard
                
              * 50 players - one client per thread, networked off one
                home PC
                
              * 500 players - multiple clients per threads, fast
                Internet connection
                
              * 1000 players - non-blocking sockets, 100 clients per
                thread, optimize server configuration
                
              * 50,000 players - requires a cluster of co-operating fast
                servers
                
                      * Intel/AMD architecture requires special hardware
                        (fast motherboard bus)
                        


      Problems supporting many players
        
      * nondeterminism (being unable to read source code to understand
        what's going on)
        
              * scheduling algorithms and threads break up the step-by-step
                nature of algorithms making them difficult to follow
              * may be many possible paths to do one action

      * nontrivial failure modes (handling system component failures
        gracefully)
        
              * don't want a temporary glitch to disconnect player
                sessions
                
      * problems of scale (overhead and testing problems)
        
              * small overhead costs can have a large impact if multiplied
                thousands of times
        
              * testing as you gradually connect more clients
                
      * overload chain reaction
        
              * the more overloaded the server, the more synchronization
                and resource management issues
              * without overload handling, server will fail exponentially
                
      * human factors
        
              * clicking multiple times to send multiple requests when
                the system is overloaded, increasing the overload
                
      * fair scheduling
        
              * if 50 clients have a 10-500 ms response time, 10,000
                will have 10-100,000 ms (plus network delays)
              * a slower but more predictable response is better.


      Techniques to support many players
        
      * resource allocation could be trivial, pooling, pipelined or
        batched (performance/reliability)
        
              * trivial - N clients need N resources (easiest)
                
              * pooling - N clients use < N resources from a shared pool
                (high startup costs)
                
              * pipelined (staged) - N clients are queued to use < N
                fixed resources (most scalable)
                
              * batched - N clients use < N resource requests bundled
                into a single request for resources (high usage costs)
                
              * database connections are usually pooled, database
                queries are usually batched
                

      * handling non-determinism (reliability)
        
              * always check resource allocation to avoid corrupt data
                entering a system running 24-hours a day
                
              * restart the game server on a regular basis to ensure a
                known state
                
      * testing (reliability)
        
              * create a test suite that can run unattended for days at
                a time
                
              * create sets of test components that can be reorganized
                to simulate different scenarios

              * Mel mentioned TDD (test driven development) where you
                write the test suite as you develop source code
                
      * Asynchronous I/O (isolating bottlenecks)
        
              * a pool of threads/pipes to handle client requests (as
                opposed to one per client)
                
              * divide game server into separate programs, each with
                accessed with AI/O
                
              * use pools of message buffers to avoid dynamic allocation
                overhead
                
      * handling overloads
        
              * overloads: the game gets out of sync and human factors
                increase problem
                
              * first, stop actions that increase the load (such as a
                "back off" message sent to clients)
                
                      * KB: also, give priority to resource freeing
                        actions (e.g. player logouts)
                        
              * second, suspend or disconnect players with contributing
                to highest load (forced logouts of most active users)
                
              * KB: optional actions or creating messages that don't
                accumulate information
                
      * priority scheduling (handling load spikes)
        
              * Shortest Remaining Processing Time versus Fastest
                Connection First
                
              * For SRPT, create a crude estimation of time: cached
                response (+0), non-cached response (+2), and so forth;
                decremented to prevent starvation
                
              * For FCF, handle player with network Internet connection
                first
                
              * KB: good for spikes but don't solve long-term overloads
                because SRPT will devolve into round-robin scheduling.
                


      Application to PegaSoft's Online Game Project
        
      * although the game won't handle 50,000 people, some of these
        principles are already in use
        
      * game server
        
              * two subsystems with asynchronous I/O (proxy and server)
                
              * fixed-sized message buffer pools (to avoid dynamic
                allocation)
                
              * scalable through use of pipelines
                
              * on-the-fly reset capability (proxy can issue shutdown
                request)

              * ability to prefetch data that wasn't required yet to
                keep the network connection in use when the server is
                idle
                
      * proxy
        
              * will schedule incoming messages
                
              * will cache some responses
                
              * will discard duplicates (human factors)

              * can reset server on a regular basis
                
      * a comprehensive test suite cannot be done until the messaging
        system is fully operational

      Overall

              * the article was a good survey of load-handling approaches.
                However, it was not exhaustive: several techiques Ken would
                have mentioned are not present.
              * most of the suggestions apply to any high-load software, not
                just games
              * the article had some grammar and organizational problems that
                should have been caught by an editor

Next Dinner

The next dinner meeting is Tuesday, April 17, 2006.  Location to be announced.