This chapter describes the particularities about building a Beowulf cluster
using Debian GNU/Linux and FAI. For more information about the Beowulf concept
look at http://www.beowulf.org
.
The example of a Beowulf cluster consists of one master node and 25 clients. A big rack was assembled where all the cases were put into. A keyboard and a monitor were also put into the rack, which are most of the time connected to the master server. But since we have a very long cables for monitor and keyboard, they can also be connected to all nodes, if something has to be changed in BIOS or when looking for errors, when a node does not boot. Power supply is another topic you have to think about. Don't connect many nodes to one power cord and one outlet. Distribute them among several breakout boxes and outlets. And what about the heat emission ? A dozen nodes in a small room can create to much heat, so you will need an air condition. Will the power supplies of each node go to stand by mode or are all nodes are turned on simultaneously after a power failure ?
All computers are connected to a Fast Ethernet switch. The master node (or master server) is called nucleus. It has two network cards. One for the connection to the external Internet, one for the connection to the internal cluster network. If connected from the external Internet, it's called nucleus, but the cluster nodes accesses the master node with name atom00, which is a name for the second network interface. The master server is also the install server for the computing nodes. A local Debian mirror will be installed on the local harddisk. The home directories of all user accounts is also located on the master server. It will be exported via NFS to all computing nodes. NIS will be used to distribute account, host and printer information to all nodes.
All client nodes atom01 to atom25 are connected via the switch with the second interface card of the master node. They can only connect to the other nodes or the master, but can't communicate to any host outside their cluster network. So, all services (NTP, DNS, NIS, ...) must be available on the master server. I choose the class C network address 192.168.42.0 for building the local Beowulf cluster network. You can replace the subnet 42 with any other number you like. I you have more that 253 computing nodes, choose a class A network address (10.X.X.X).
In the phase of preparing the installation, you have to boot the first install client many time, until there's no fault in your configuration scripts. Therefore you should have physical access to the master server and one client node. If you have few space, connect both computers to a switch box, so one keyboard and monitor can be shared among both.
The master server will be installed by hand, cause it could be your first
computer installed with Debian. It you have already a Debian host running, you
can also install it via FAI. Create a partition on /files/scratch
for the local Debian mirror with more that 3GB space available.
Add following lines for the second network card to
/etc/network/interfaces
:
# Beowulf cluster connection iface eth1 inet static address 192.168.42.250 netmask 255.255.255.0 broadcast 192.168.42.255
Add the IP addresses for the client nodes. The FAI package has an example for
this /etc/hosts
:
# Beowulf nodes # atom00 is the master server 192.168.42.250 atom00 192.168.42.1 atom01 192.168.42.2 atom02
You can give the internal Beowulf network a name when you add this line to
/etc/networks
:
beowcluster 192.168.42.0
Activate the second network interface with: /etc/init.d/networking start.
Add a normal user account tom which is the person how edits the configuration space and manages the local Debian mirror:
# adduser tom # addgroup linuxadmin
This user should also be in the group linuxadmin. So, add a line to
/etc/group
:
linuxadmin:x:101:tom
To initialize the master server as NIS server call ypinit -m.
Then, copy the file netgroup
from the examples directory to
/etc
and edit other files there. Adjust access to the NIS
service.
# cat /etc/ypserv.securenets # Always allow access for localhost 255.0.0.0 127.0.0.0 # This line gives access to the Beowulf cluster 255.255.255.0 192.168.42.0
Rebuild the NIS maps: # cd /var/yp; make
Now user tom can create a local Debian mirror on
/files/scratch/
using mkdebmirror
. This will need
about 2.6 GB disk space for Debian 2.2 (aka potato). Export this directory to
the netgroup @faiclients read only.
Add following packages to the install server:
nucleus:/# apt-get install task-dns-server ntp tftp bootp nfs-kernel-server fai fai-kernels
Configure NTP so that the master server and all client nodes will have the same correct system time.
It's very important to use the internal network name atom00 for the
master sever (not the external name nucleus) in
/etc/bootptab
and /etc/fai.conf
. Replace the strings
FAISERVER with atom00 in /etc/bootptab
and uncomment
the following line in /etc/fai.conf
so the Beowulf nodes can use
the name for connecting their master server.
NFSROOT_ETC_HOSTS="192.168.42.250 atom00"
/etc/bootptab:
. . .failocal:\ :tc=.faiglobal:\ :sa=atom00:\ :ts=atom00:\ :T170="atom00:/usr/local/share/fai":\ :T171="sysinfo":\ :T172="verbose createvt sshd":\ :sm=255.255.255.0:\ :gw=192.168.42.250:\ :dn=beowulf.debian.org:\ :ds=192.168.42.250:\ :ys=atom00:yd=nisnucleus:\ :nt=atom00: . .
Uncomment the following line in /etc/inetd.conf
:
#bootps dgram udp wait root /usr/sbin/bootpd bootpd -i -t 120
and restart the inetd daemon. The user tom should have permission to create and the symlinks for booting via network card, so change the group and add some utilities.
# chgrp -R linuxadmin /boot/fai; chmod -R g+rwx /boot/fai # cp /usr/share/fai/utils/* /usr/local/bin
Now, the user tom can create a symlink in /boot/fai
using
>tlink atom_install atom01
to boot the first client node for the first time. Then start to adjust the
configuration for your client nodes. Don't forget to build the kernel for the
cluster nodes using make-kpkg(8)
.
Following tools for a Beowulf cluster now are available in
/usr/local/bin
:
/usr/share/doc/fai/examples/etc/netgroup
for an example.
all_hosts
to get the list of all hosts up. You can also use the
dsh(1)
command (dancer's shell, or distributed shell).
Wake on LAN is a very nice feature to power on a computer without having physical access to it. By sending a special ethernet paket to the network card, the computer will be turned on. Following things have to be done, to use the wake on LAN (WOL) feature.
http://www.uow.edu.au/
~andrewm/linux/#3c59x-bc
There's a little problem to enable the wake on LAN feature with a 2.2.19 kernel
and a 3Com 3C905C network card. You have to use a patched 3c59x driver. But I
managed it to get work. Download the file
poll-ioctl-2.2.18-pre16.c.gz
and copy it to kernel sources to
drivers/net/3c59x.c
. Then make a new kernel package and install
this new kernel. To wake up a computer use the tool etherwake (in woody) or
get the single C source file. For more information look at
http://www.scyld.com/
expert/wake-on-lan.html
.
lange@informatik.uni-koeln.de