How to Build a Supercomputer
from Scratch, in 10000 Easy Steps
When Sparky got to his last USAF job before retirement, he found
that
his new duty title
"Assistant Branch Chief" really meant "Glorified Gofer Boy" for some
bored PhD-type USAF civilians. These people really didn't want to
fool around with all this military-type stuff, they just wanted to
play with their pet projects.
Anyway, due to the heavy workload (sarcasm evident?) Sparky flailed
about for awhile (which is the same in every tech job Sparky's ever
been in), until he found a vacuum to fill.
A foreign researcher at Sparky's new place of employment was working
with Finite Difference Time Domain (FDTD) modeling of
radiofrequency (RF) absorption in living tissue. Say that five times
fast. The model of most research interest is the Visible
Human Project (VHP), since, with a sufficiently powerful
computational machine, one can predict RF effects on humans.
The default resolution of the VHP is 1mm
in three dimensions (colloquially known as the "1mm Man"). The
resultant data set is about 18 GB, which must reside in the memory of
the machine processing the data, or disk-swapping (much slower than RAM
access) will kill productivity on the process.
So this researcher was trying to run the model at a resolution
of 5mm in three dimensions (the "5mm Man", with a dataset of 720MB) and
trying to get meaningful data out of a two-processor Pentium 500 MHz PC
using Windows 98 for the Operating System (OS). This machine was no
match for
the job, and the output only allowed for generalized predictions of RF
tissue absorption/heating. In fact, just getting this machine to output
data was something of a task. The average run time for a simulation was
on the order of a month, if it didn't crash first, or output a
corrupted data file. Tortuous workarounds were required - this machine
was once moved to a different
building, still running, powered by two UPSes, one plugged into the
other, so that a month of runtime would not be lost.
Through discussions around the office, various serfs, Sparky
included, decided we needed a Beowulf
Parallel Computing Cluster to run the model for the VHP. And we
only had a
few problems to overcome; very little experience with Linux, zero
experience with parallel computing, and zero (none, nada) hardware.
One of us;

Mr Aldon, shown here putting together a
Beowulf node, had the Great Idea of latching onto some of the used
desktop computers various units in the USAF routinely turn in to Dr Mo. (Actually it's now Dr Ms,
which must be in keeping with the New! Improved! Kinder! Gentler!
Politically Correct Military - which makes an Old Fart like Sparky
wanna puke.) Due to Mr Aldon's ingenuity, we ended up with a pile of
Pentium 166 and 200 MHz Desktop Boxes, which we cobbled together using
Redhat 7.0 Linux for the OS, and came up with a
Beige Box
Breadrack Beowulf, a 12-node machine which actually produced
meaningful data (at the
3mm man resolution), albeit after much wailing and gnashing of teeth.
The runtime for a simulation on this machine was approximately 4 days,
which was a vast improvement over a one month runtime, and more
importantly; It
Didn't Crash.
Another problem immediately surfaced; with output files of 1 to 2GB, we
had no way to get the data into the Windows desktop machines the
researchers used to smursh data and write reports. At this time, DvD
recorders (at least those which we, with no budget, could afford) were
not on the market yet, and CD recording (only 650MB at this time) was
not a mature art, at least under Linux (or under Windows, for that
matter). We tried splitting the datasets up into separate files and
burning them onto multiple CDs, but the nature of the data required
even more wailing and gnashing of teeth from the researchers as they
tried to put the files back together again.
What we needed was a way to get the data from the Beowulf machine,
across the base network, right to the Windows desktop machines on the
researchers' desks. At this point in time (and most likely, still
today), running a Linux machine on a USAF Base network was(/is) a Big
No-No. This is due to the ignorance of Microsoft-Centric System
Administrators who know absolutely nothing about any OS but Windows.
Microsoft drills into their heads that Linux is not secure, and Windows
is. What a crock!!! As an example, Sparky has been running nothing but
Linux since 2001, and has no (zero, nada) anti-virus software on any
machine. Never been hacked or had a virus/trojan/bot/malware/etc
either. Try THAT with Windows! This Windows monopoly of government IT
systems represents a humongous waste of taxpayer money. Sparky could
climb up on a large soapbox at this point and continue ad nauseam, but
you get the idea.
Suffice it to say that this was a hostile environment in which to run a
Linux server. An html or ftp server was out of the question, and secure
shell (SSH) software
was command-line only during this time frame. The only protocol which
we
had at our disposal was SMB/CFIS (Network Neighborhood on your Windows
box) using a Linux box as a Samba server.
Remember the dual Pentium 500 MHz machine the model was originally
running on? Well Sparky turned it into a dandy Samba server and the
data semi-magically appeared in Network Neighborhood on the
researchers' desktop machines.
The box
next to the printer is the samba server which farms the data out to the
researchers. An interesting tidbit is the box at the upper right. That
is a Pentium 166 MHz machine with two NIC cards running a floppy (yes,
floppy) disc-based Coyote
Linux firewall. This was mandatory in order to keep the
Powers-That-Be on the base network from screwing up our server in the
event they were able to break through the firewall. And Oh Yes, they
tried. Although with their limited tools (i.e; Windows-Based) they
were never successful.
Along the way, we kept getting more powerful desktop machines (although
not as powerful as
we really needed) from Dr Mo, and we tried the
Naked
Breadrack Beowulf, a 24-node machine which produced quite a few
datasets over the course of several months,
but was even uglier than the Beige Box Breadrack Beowulf. In fact, its
ugliness was its undoing, which we found out The Hard Way. It seems
that
certain PhDs, especially those types who tend to be in charge, like
bright, shiny things, and this machine was Sofa King Ugly, that they
couldn't bear to show it to any of the MFWICs
who held the purse strings.
In an attempt to hide the offending contraption out of sight, such as
in
a closet (although one with plenty of ventilation) the Naked Breadrack Beowulf was morphed into the
Sanford and
Son Beowulf. Talk about ugly! But since the purpose was to make it
compact and
hideable, it would have been successful, except for a bastardization of
one of Einsteins maxims; "Things should be made as simple as
possible,but not simpler". As can be seen here;
this
thing was just Toofa King Compact! Many simulation runs were aborted
because someone bumped the machine and one of the nodes would reboot.
And yes, with all the components of 24 desktop machines in a space that
size, maintenance was a nightmare. So we had a new rule to live by;
Design for Maintainability! Also, the heat coming off of this rack was
amazing, and these were only Pentium 233 and 266MHz nodes!
It was a useful machine though, and enabled us to gain much-needed
experience with building, operating and maintaining a production
Beowulf machine.
Sparky had a rude awakening about this time; he had naively assumed
that the
research grant process depends solely on scientific merit. He found out
that
just ain't so - politics plays much too large a role in determining
which research projects get funded.
Somewhere amongst all this brew-ha-ha, and the datasets of meaningful
RF absorption values rolling out in production-line fashion, someone
who held purse strings began to notice, and a contract (with actual funds)
was granted to cousin Tony's company (Sparky and Tony didn't know they
were cousins at the time, so don't blame us) to actually build a
machine from the ground up to run the 1mm Man dataset. As noted
previously, this required the entire 18 GB model to reside in memory on
the machine.
Cousin Tony's company eventually delivered the three bay machine shown
here;
Although when it showed up, it looked like a bunch of boxes and
equipment - and no rack. Prior to delivery, Mr Aldon had been in
lengthy discussions with the base metal shop, wheedling and cajoling
them into fabricating metal plates to mount the individual nodes in the
rack which we purchased.
A closeup of one of the nodes;
Showing the fabricated plate and method of mounting the node hardware.
Mr Aldon obviously remembered The Good Old Days, when Quasar TV Sets
came with the "Works in a Drawer" for ease of maintenance. Remember
what we learned on the Sanford and
Son Beowulf?!? Design for Maintainability!!!
How's this for ease of maintenance???
Some detail of the interior of the rack, showing the Mambo HP switch to
connect the whole thing together;
As originally delivered, the machine had 72 nodes, made up of AMD
Athlon 750MHz
and 1GHz CPUs with 500MHz of RAM each. This gave a total system memory
of 36GB.
After several complaints from certain HMFICs
that the machines might be producing data, but they weren't bright and
shiny enough, having no banks of blinkee lights nor Star-Trek-esque
visual displays, Sparky scrounged up a
Red Light Special which signified that the
Beowulf room was indeed a Happening Place. This conferred a warm, fuzzy
feeling unto the Questers of Bright, Shiny Objects.
As some of you may know, if you can cheaply narrow down what you are
trying to find out through an experiment, then you have a savings
multiplier which allows you to refine the design of your test. This has
several effects - one is to make some tests cheaper, and another is to
allow for more complicated experiments, with greater probability of
success.
A short while after the 72-node system went online, the MFWICs
realized that all this screwing around we had been doing (with
essentially zero funding - except for the contract to Tony's company)
was actually enabling the researchers to predict heating effects on
living tissue, as verified by experiments that were ongoing. And by
some sort of funding
miracle they gave us more money to upgrade the system to 96 nodes.
One of the cool things about a job like this is you get to play with
lots of neat equipment that you don't have to buy!
Stuff like a box full of CPUs;
Or (at the time) about $5K worth of memory;
Or maybe a pile of disk drives;
Here's Mr Aldon loading the OS on (yet another) node;
And the finished add-on rack, with the additional 24 nodes installed
(not too sure of they were all running at this point);
It's tough to see in this picture, but the flourescent light grating
(with no light) above the rack is actually a vent into the
air-conditioned space above. Our facilities guy, Mr Frank, was
constantly engaged in heated negotiations with the Base Civil
Engineers. Apparently the Beowulf room required more AC capacity than
any other office space on the base. Keep that in mind, kiddies, when
you're planning your own parallel cluster to rip Dvds!!!
So what's the outcome of all this? The answer is - a bunch of numbers!
However, using graphics software, pretty pictures can be made from All
This Data. Here's an example;
What you see is the 1mm Man, with an RF source, about 915MHz,
illuminated from the front, not sure about the polarization. The red
colors represent more intense
heating of the tissue - just like looking at the WX radar.
The Researchers made much foofaraw from the results of this work, but
Sparky's just an engineer;
From: The
2002 Bioelectromagnetics Society Symposium
FINITE DIFFERENCE TIME DOMAIN (FDTD) MODELS PREDICT ORGAN
RESONANCES IN
THE 1-MM MAN MODEL. J.M. Ziriax, J.A. D' Andrea, W.D. Hurt, D.
Verrett, P.A. Mason, D. Hatcher, and D. Cox. Naval Health Research
Center Detachment, Brooks Air Force Base, Texas 78235, USA; Air Force
Research Laboratory, Directed Energy Bioeffects Division, Brooks Air
Force Base, Texas 78235, USA.
INTRODUCTION: Empirical dosimetry of even a single frequency
is time
consuming and labor intensive, and as such, is typically limited to
critical dosimetric data. Computer models can be used to explore a wide
range of frequencies and exposure conditions in anatomically realistic
models without the expense and effort of empirical measurements.
OBJECTIVES: Localized SARs vary dramatically with frequency and
exposure orientation. Here we report on an ongoing exploration of
localized and whole body SARs in our 1-mm human model across a range of
exposure frequencies and orientations.
METHODS: NHRC-DET and AFRL/HEDR
have jointly developed models of mouse, rat, goat, and rhesus monkey in
addition to the human model (Mason et al., 1995, 1999). The 1-mm man
model was developed from images provided by the National Library of
Medicine as part of the Visible Human Project
(http://www.nlm.nih.gov/research/visible/visible_human.html). Our FDTD
program, based on code originally developed by Kunz and Luebbers
(1993), was used to calculate SARs in the 1-mm man model. The
electrical properties of each of tissue were set according to data and
fits published by Gabriel (1996). The code has been parallelized using
the MPI message-passing library. The advantage of using the MPI is that
the code can run on parallel computer systems composed of networks of
computers. These may be networked workstations, or massively parallel
systems such as Linux-based Beowulf systems. These systems are easily
constructed of relatively inexpensive PC-hardware. The figure shows
sample results for whole body, skin and eye structures for 1700 to 4000
MHz in the PEKH orientation (RF source is to right of then man with a
vertical E-field).
Note: Graph not available yet.
CONCLUSIONS: In this frequency range, SARs for whole
body and skin change monotonically, while SARs for the eye structures
change non-monotonically. We will expand the graph to lower frequencies
where we have seen eye (900 MHz) and whole body (70 MHz) resonance in
coarser or head-only models. As the FDTD predicts SARs with the same
resolution as the anatomical
model, in this case in 1-mm volumes, then this resolution easily
exceeds that of typical empirical
methods. The thermal and biological
significance of these highly localized SARs will have to be determined
experimentally.
References;
Gabriel C. Compilation of the Dielectric Properties of Body Tissue at
RF and Microwave Frequencies,
Brooks AFB, TX: Armstrong Laboratory Report, AL/OE-TR-1996-0037, 1996.
Kunz, KS and Luebbers RJ. The Finite Difference Time Domain Method for
Electromagnetics, CRC
Press, Inc., Boca Raton, FL, 1993. P. A. Mason, et. al., Database
created from magnetic resonance images
of a Sprague-Dawley rat, rhesus monkey, and pigmy goats, FASEB J., 9:
434-440 1995
Mason PM, Ziriax JM, Hurt WD, Andrea JA. 3-Dimensional models for EMF
dosimetry. In Electricity
and Magnetism in Biology and Medicine edited by Bersani, Kluwer
Academic/Plenum Publishers, 1999.
The views expressed in this article are those of the authors and do not
necessarily reflect the official policy
or position of the Department of the Navy, Department of the Air Force,
Department of Defense, or the U.S.
Government.