The AnandTech Linux XBOX PC Experiment
by Kristopher Kubicki on November 10, 2004 4:00 PM EST- Posted in
- Linux
A Beowulf Cluster
So far, we have played around a little bit with the idea of a stand-alone XBOX doing some neat things. But what if we want to actually make a high availability processing cluster across all of our Linux machines at once? This is the murkier world of XBOX PCs, distributed computing. There are a lot of really good documents detailing how to set up a secure, robust and stable Beowulf Cluster, but this isn't one of them. We only want to benchmark 8 XBOXes in parallel operation.We started by finding a good location for our XBOX cluster to run. A few 2x4 wood boards stacked together on the lab floor were good enough for us, apparently. Just to get an idea of how big our cluster was going to be, we stacked the XBOXes up in two columns of four as you can see below. It is a good idea to have plenty of room on all 4 sides of your cluster so that you can easily check cabling and such.
Click to enlarge.
A 16 port gigabit switch provided us with network connectivity. Unfortunately, we are not actually using gigabit connectivity since the XBOX network devices only operate at 100Mbps. A 100Mbps switch usually runs for about $50, although it isn't wise to go too cheap on the largest bottleneck in the cluster. We crimped our own cabling as you can see in the image below. Even though we are just hacking a system together, keeping the cabling organized is very important since diagnosing a bad system will be nearly impossible if wires are all over the place.
The next step was actually placing the chip in each machine and removing the hard drive so that we could image it. Modifying each of the XBOXes took some time. It took us a few hours to get all of the mod chips in correctly and start up the SmartXX BIOS. By default, SmartXX does not boot to Linux, so we had to tell the BIOS after each chip was inserted.
Click to enlarge.
First, we installed Linux on the master XBOX machine. This machine will eventually become the master of the cluster, but we will use it as a template for the other machines before we clone the hard drive onto the other XBOXes. All of the programs that we planned on using, such as distcc, djohn, etc., were installed on this template machine.
We will need to identify each of our XBOXes in the hosts file. This will allow us to identify each machine without doing a DNS lookup on its alias. Our hosts file looks like this:
| master# cat hosts | |
| ::1 | localhost |
| 127.0.0.1 | localhost |
| 192.168.1.10 | master |
| 192.168.1.11 | slave1 |
| 192.168.1.12 | slave2 |
| 192.168.1.13 | slave3 |
| ... | |
By using a hard drive blaster, we able to image all of our other XBOX hard drives in about an hour. Using Ghost or another drive imaging utility will work as well, but you can only do one or two drives at a time instead. If you are really desperate (or cheap), installing Linux on each machine manually works, but leaves the most room for error. Right now, we have eight XBOXes that are exactly identical in every respect. We need to designate one of these to be the master, so the first machine is booted up for a little bit of configuration. Dhcpd, the program that will assign IP addresses to each of our slave machines, was configured as such:
# dhcpd.conf
#
option domain-name "cluster";
option domain-name-servers 192.168.1.200;
option subnet-mask 255.255.255.0;
default-lease-time 3600;
max-lease-time 86400;
ddns-update-style none;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.11 192.168.1.17;
option routers 192.168.1.1;
}
We really don't care which IP gets assigned to which machine; although, if we boot each XBOX in sequential order on the rack, their IP address will line up with their position in the boot sequence.
Facebook
Twitter
RSS
30 Comments
View All Comments
Nepsir - Thursday, November 11, 2004 - link
I was just wondering if you will do a test with mental ray on the cluster (I think you mentioned something about Mental Ray in the article). It would be really interesting to see the results of that since me and a couple of friends got modded X-Boxes and I have a 3dsMax6 license (Mental Ray is included).gibson042 - Thursday, November 11, 2004 - link
You might want to look at the "mini-cluster" project (http://www.mini-itx.com/projects/cluster/) when planning/designing/building your EPIA cluster. It's a 12-node (originally 6) cluster developed by Glen Gardner, and appears to be the same size or a little smaller than a stack of an equal number of XBOXes. Of course, the frame is custom built and the hardware is exposed to the world.Aikouka - Thursday, November 11, 2004 - link
I can say that I love having a modded xbox, because the capabilities are so much greater. As discussed earlier, a software mod can be a great way to do it (although it usually requires a game unless you want to take your HDD out, which is NOT a fun method unless you love swapping IDE cables.)It's also good to note that you should be careful when modding these or even make a backup of the hdd on your system, because things can go wrong, and if you like to fiddle with things, then your chances will be a lot higher. I know at one point I was stuck in error 16 and had to pull the HDD out and unlock it using the IDE swap method.
Are there plans for an article on modding something such as the PS2? I just recently tried a no-solder modchip in the PS2, and it doesn't seem to be going along too well. Maybe it's time for a flip-top.
Scarceas - Thursday, November 11, 2004 - link
"We crimped our own cabling as you can see in the image below."I'm still looking for the image, I know its there somewhere!
bleugh - Thursday, November 11, 2004 - link
If anyone wants their xbox motherboards upgraded to 128 megabytes of memory, I can do itI'm UK based and use BRAND NEW samsung memory chips
I can upgrade your existing motherboards, sell you ready upgraded boards, or even just the chips!
if one of the anandtech team wants their xboxes upgraded i'll happily do it at cost, please contact me to discuss
Dean
LotoBak - Thursday, November 11, 2004 - link
First, XBox live banning....Basically this is how it used to work. When your on live it does a hash of your bios. A mod replaces your bios. If you bios is not the retail ms bios the serial (eeprom) is blacklisted. NOW since halo2 has been released it appears that MS has changed a while bunch of shit on us. Preliminary information indicates that MS is banning by hard drive serial+model numbers, xbox serial, and flagging the gamertag as 'potential pirate'. Thus to unban the xbox both a new hdd and eeprom are needed. Then you must cancel and create a new xbox live account. There are more details about how it all works but bottom line is we dont know how it works now. They can throw new stuff at us anytime now. Rumors are flying that hdd capacities are being scanned as well. For all intents and perposes xbox + modded xbox's do not mix.
Yes there is a PS1 emu for xbox. That said xbox DVD drives do not read cd-r (or retail ps1 games) Therefor all games must be transfered to the xbox hdd in bin/cue or iso format
When comparing mods remember, the mod contains no modded bios stuff. They are useless in them selves. You will need to aquire a modded bios for your xbox (online, free). What this means is that every mod can contain the same bios's. Therefor they all contain the same base features (mostly booting unsigned code). Hardware features and support is where you want to base your decision on. If you want cheap + no manufacterer support there are 10$ solutions around for you(enabled/disables with power button). If you want support with a mod with a few extra hardware features (2 bios's, external switch) that'll cost around 30-40. Fancy Shamcy lcd support, 8 bios banks, blah blah blah fancy external thing thoes are up to 75ish. But in the end they all allow you to run the same software on the xbox.
if you want to learn read some beginner tutorials at http://tutorials.xbox-scene.com
Omega215D - Thursday, November 11, 2004 - link
After seeing Halo 2, it makes me wonder if they really need Pixel Shader 2.0. Its pretty cool that I can play old games on the XBOx through emulators though but i wanna see when the next XBOX kicks off before spending $200.ViRGE - Wednesday, November 10, 2004 - link
#12, probably not. The Xbox's GPU is a GeForce 4 class GPU - the lack of Pixel Shader 2.0 features can really be limiting.euph - Wednesday, November 10, 2004 - link
is there any way to use the gpu for the pov-ray rendering?ukDave - Wednesday, November 10, 2004 - link
Little quote that sums up the SmartXX XBL saga:Q: just bought a Smart XX mod and I still don't know if i'll be able to play xbox live so please somebody tell me if i'll be able to play it.
A: There is no easy way to answer this question. First of all, it all depends on circumstance. Have you upgraded your hard drive? Has your Xbox ever been banned before? And, for newbs, are you thinking about playing on live while using a bios from the chip.
If you answered yes to any of these questions you've won a lifetime ban from Microsoft!
Sleep time :) And i don't even use XBL :D