The AnandTech Linux XBOX PC Experiment
by Kristopher Kubicki on November 10, 2004 4:00 PM EST- Posted in
- Linux
A Beowulf Cluster
So far, we have played around a little bit with the idea of a stand-alone XBOX doing some neat things. But what if we want to actually make a high availability processing cluster across all of our Linux machines at once? This is the murkier world of XBOX PCs, distributed computing. There are a lot of really good documents detailing how to set up a secure, robust and stable Beowulf Cluster, but this isn't one of them. We only want to benchmark 8 XBOXes in parallel operation.We started by finding a good location for our XBOX cluster to run. A few 2x4 wood boards stacked together on the lab floor were good enough for us, apparently. Just to get an idea of how big our cluster was going to be, we stacked the XBOXes up in two columns of four as you can see below. It is a good idea to have plenty of room on all 4 sides of your cluster so that you can easily check cabling and such.
Click to enlarge.
A 16 port gigabit switch provided us with network connectivity. Unfortunately, we are not actually using gigabit connectivity since the XBOX network devices only operate at 100Mbps. A 100Mbps switch usually runs for about $50, although it isn't wise to go too cheap on the largest bottleneck in the cluster. We crimped our own cabling as you can see in the image below. Even though we are just hacking a system together, keeping the cabling organized is very important since diagnosing a bad system will be nearly impossible if wires are all over the place.
The next step was actually placing the chip in each machine and removing the hard drive so that we could image it. Modifying each of the XBOXes took some time. It took us a few hours to get all of the mod chips in correctly and start up the SmartXX BIOS. By default, SmartXX does not boot to Linux, so we had to tell the BIOS after each chip was inserted.
Click to enlarge.
First, we installed Linux on the master XBOX machine. This machine will eventually become the master of the cluster, but we will use it as a template for the other machines before we clone the hard drive onto the other XBOXes. All of the programs that we planned on using, such as distcc, djohn, etc., were installed on this template machine.
We will need to identify each of our XBOXes in the hosts file. This will allow us to identify each machine without doing a DNS lookup on its alias. Our hosts file looks like this:
| master# cat hosts | |
| ::1 | localhost |
| 127.0.0.1 | localhost |
| 192.168.1.10 | master |
| 192.168.1.11 | slave1 |
| 192.168.1.12 | slave2 |
| 192.168.1.13 | slave3 |
| ... | |
By using a hard drive blaster, we able to image all of our other XBOX hard drives in about an hour. Using Ghost or another drive imaging utility will work as well, but you can only do one or two drives at a time instead. If you are really desperate (or cheap), installing Linux on each machine manually works, but leaves the most room for error. Right now, we have eight XBOXes that are exactly identical in every respect. We need to designate one of these to be the master, so the first machine is booted up for a little bit of configuration. Dhcpd, the program that will assign IP addresses to each of our slave machines, was configured as such:
# dhcpd.conf
#
option domain-name "cluster";
option domain-name-servers 192.168.1.200;
option subnet-mask 255.255.255.0;
default-lease-time 3600;
max-lease-time 86400;
ddns-update-style none;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.11 192.168.1.17;
option routers 192.168.1.1;
}
We really don't care which IP gets assigned to which machine; although, if we boot each XBOX in sequential order on the rack, their IP address will line up with their position in the boot sequence.
Facebook
Twitter
RSS
30 Comments
View All Comments
TimPope - Thursday, May 12, 2005 - link
not bad information but i would have liked to see some kind of real world performance using openmosix.. a single x box on its own as a pc is slow but stick 2-4 together using open mosix could make a reasonably good machine and still be pretty cheapHalz - Wednesday, November 17, 2004 - link
The rule followed in the article for the -j option, "number of proccessors + 1", overlooked the logical proccessors of the Xeon's Hyperthreading.. -j should have then been something around 5 instead of 3Halz - Wednesday, November 17, 2004 - link
Simply compiling on the Opteron and Xeon with the same number of threads as the full cluster would have illustraighted a difference.More testing should have gone into finding how many threads was the ideal number for the given platforms.
artifex - Saturday, November 13, 2004 - link
Aikouka, can't you just use one of those "HD Loader" type programs WITHOUT a modchip?I'd be all for modding my PS/2 if I thought I could actually do something useful with it, like stream audio/video from a PC or a ReplayTV or something.
KristopherKubicki - Saturday, November 13, 2004 - link
Halz: what should it have been?Kristopher
Aikouka - Thursday, November 11, 2004 - link
23, yes, you can still do just about anything. I know with the software mod that I use, I've been having problems getting the original MS Dash to load up, but I've gotten around that using other programs for the original dashboard's functionality (dvd etc).You know, you can also replace the HDD with just a software mod, and it's not that hard. So, if you don't want to hardware mod and want more space, you can still put in a bigger HDD. As much as some people don't like the XBOX, in my opinion, it's probably the best console to mod.
24, 2) Modchips also allow hdd loading if you have the PS2 HDD (using HDDLoader.) Also, it lets the warez'ers download and play games on the PS2 that they don't really own.
artifex - Thursday, November 11, 2004 - link
1) what we really need is a usb-based tv tuner that actually works. That would be excellent for adding functionality both to XBoxen as cheap PVRs (though I'd still just use XBMC to stream from my ReplayTV, most of the time), but also would be great for iMacs. I'm sure if someone came up with a decent open architecture design, the community would come up with drivers for both types of systems.2) what are modchips for PS2s useful for, other than playing import games? Especially with the new PS2s having no drive (is there still a header on the new board style to add one back?)
3) did I miss the obligatory dnetc test? You gotta do that, you know.
Booty - Thursday, November 11, 2004 - link
I don't even own an Xbox, but reading this article has me reaching for my wallet...But first, I want to get this straight - I can mod the Xbox and still use XLink, right? I doubt I'd get a Live subscription anyway, but it'd be nice to have that option possible.
Ideally I'd like to throw a bigger hard drive in there and then run XBMC, without losing the normal XBox capabilities.
So if I can do that, I'm goin' to the store this weekend... :)
Halz - Thursday, November 11, 2004 - link
The compile options for the Opteron and Xeon were starving the CPUs; the number of jobs (-j) was no where near optimal.Halz - Thursday, November 11, 2004 - link