Greetings all, I am proud to announce that my ultimate home file server is now up and running (besides a defective hard drive, but the replacement is on the way, and when it arrives I will rebuild the array and complete my testing).
Now written here, preserved for posterity, are the complete instructions, hardware and software recommendations, and steps necessary to make your own ultimate file server for under $2300! First, the complete hardware list:
- SUPERMICRO CSE-833T-R760 Beige Steel 3U Rackmount Server Case with 760W Triple-Redundant Power Supply
- Intel BOXDX48BT2 LGA 775 Intel X48 ATX DDR3 Intel Motherboard
- 2GB or 4GB of 1600Mhz DDR3 RAM (I suggest 2GB as when I tried 4GB it was very unstable with this mobo)
- Any Core 2 duo processor (but feel free to go all-out if you really want…)
- 8x 1TB 32MB cache SATAII Hard drives – if you don’t need as much space, feel free to save some money and go cheaper.
- 1x IDE hard drive – use one you have lying around, or get a new one. The smallest I could find was a 40GB drive for $25 on pricewatch.
- SIIG SATAII PCIe Controller – even though it doesn’t say, some internet research proves it does work on linux.
- You probably want a video card for troubleshooting – I would have been lost without one (sadly). They just don’t make quality motherboards with good serial consoles these days.
- You probably want a CPU fan, as the stock CPU fan for intel core 2 duos is QUITE disappointing in my experience.
- You may want a SATA CDRom drive to use just for installation – the IDE is really picky about detecting hard drives and optical drives at the same time on boot.
- You may want a power supply extension cable (24-pin, from pwr supply to mobo)
- You may want some fan extension cables, this case has a ton of fans but the cables are short.
I know for a fact the above-mentioned hardware works, with Debian Lenny, as I am about to describe. With reasonable effort (not counting DOA parts… =| ), I now have my file server up and running. I can’t promise similar results, but let’s just say, it would have saved me a TON of time and money if someone had posted this list for me.
Ain’t she a beaut? Complete with 8 hot-swapable units of goodness. See the rest of the pics at the bottom.
First, you need to assemble everything. Open up the case, plop in the motherboard, screw it in (be sure to put in all the risers, they are necessary to support the expansion cards when you plug them in so the board doesn’t break. Hook up the power supply, motherboard, memory, main OS hard drive, and video card. If you have an optical drive, slap that in there, otherwise be prepared to network boot, USB boot, whatever.
Next, install your OS. Anyone who knows me knows Debian is the only way to go! Do a minimal install, you don’t want an X server lying around or anything. Apt-get install your usual necessities (zsh, screen, vim, sudo, less, etc). Next you will want to apt-get install “mdadm” and “lvm2″, and “nfs-kernel”, and probably “rsync” as well. Once you are sure your hardware is working, shut down, and go ahead and install the extra 8 drives by placing them in the hot-swap bays, then running SATA cables from the backplane to your motherboard. Since it is really tough to find a motherboard with 8 SATAII ports, all 8 of which are usable in Linux, I decided to compromise and just go for one with 6 ports, and buy a PCIe controller card. If you wanted your OS drive to be sata, or have some hot spares somewhere, feel free to buy additional controller cards, but this case only holds 8 hot-swap drives plus 2 5.25in bays.
I got these directions from LVM2 and Software RAID in Linux. Without these directions it would have been quite a bit more time consuming to figure out. The directions are a little old (and not debian specific), however, so best follow these. First, note that mdadm by default will scan all your devices (see: /etc/mdadm/mdadm.conf. Assuming all your drives show up and seem functional, go ahead and create partitions on them, then link them all in an array.
# for i in sda sdb sdc sde sdf sdg sdh sdi; do sudo fdisk /dev/$i; done
Each time fdisk runs, do the following. Press “n” to add a new partition, “p” for primary, then press enter twice to accept the defaults (use the whole disk). Then press “t” to change the partition ID to “8e” for “Linux LVM”. Next press “w” to write the new partition table and exit. After you have done this to all 8 disks…
# mdadm --create /dev/md0 --chunk=64 --level=raid6 --raid-devices=8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 ...etc
At this point it will create the raid and start syncing the disks. You can do this to check the status.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
4883799680 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]
[==>..................] resync = 10.2% (100386816/976759936) finish=640.8min speed=22789K/sec
unused devices:
As you can see, it takes some time, but eventually you see this:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
4883799680 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]
unused devices: none
Now things are ready for you to set up LVM. I am writing this part after the fact, and in my set-up I decided not to do LVM. This may or may not be a bright idea. If you wanted to set it up, it’d go something like the steps below. Also, I may have forgotten some of the steps, so please let me know if I am missing anything or if you believe any of this stuff is wrong.
# pvcreate /dev/md/md0 # vgcreate fileserver /dev/md/md0 # sudo lvcreate -l 1192333 -n fileserverlv fileserver
I think the number after the “-l” was the size I got from the output of the vgcreate command. Basically, you are creating a physical volume (which is just composed of the array), then a volume group which consists of that physical volume, then a logical volume in that volume group. What does all of these levels of abstraction buy you? You can resize the array, or the partitions (logical volumes) you create on the array, at will. I just created one large partition for now, but later, I could break it up if I wanted to. Also, if I replace all the drives, one by one, with larger ones, I can grow the physical volumes then grow the logical volumes which lives on them. Yeh, it’s pretty hot. At this point, you should have a device for the logical volume you can use (in the example above, /dev/fileserverlv I think, or maybe /dev/mapper/fileserverlv).
Next you create the filesystem. Since the chunk size I chose was 64kb, I wanted something which evenly divided that, and I wanted the index sizes to be appropriate for what is, basically, an enormous drive by most standards.
# sudo mkfs.ext3 -b 4096 -m2 -T largefile -n /dev/md/md0
Now the drive is ready to mount
# sudo mount /dev/md/md0 -t ext3 fileserver
Finally, get NFS set up.
# sudo apt-get install nfs-kernel # sudo vim /etc/exports
You probably want something like this:
cmyers@petabyte.cmyers.org:1 (3:47:18)] cat /etc/exports # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). /media/fileserver 10.0.0.0/255.0.0.0(rw,sync,no_root_squash,no_subtree_check)
The above example lets every host on the 10.0.0.0 network access the drive, read-write, and allows root users to get root access. Go ahead and start NFS next:
# sudo /etc/init.d/nfs-kernel-server restart
You probably want to copy over some stuff – here is the rsync command I used to grab stuff from the old server – be warned, I had to install rsync, then I had to muck with things to make sure the versions of rsync were the same – different versions are not always compatible, so if you have an old file server which is a couple years old, it probably has an older version of rsync.
# rsync -aAX --progress:/path/to/source :/path/to/second/source [...etc....] .
And there you have it! Your new fileserver is ready to go!
cmyers@petabyte.cmyers.org:1 (3:44:42)] df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 12G 902M 11G 8% / tmpfs 1007M 0 1007M 0% /lib/init/rw udev 10M 148K 9.9M 2% /dev tmpfs 1007M 0 1007M 0% /dev/shm /dev/md0 5.5T 521G 4.9T 10% /media/fileserver cmyers@petabyte.cmyers.org:1 (3:44:43)]
The inside of this case is so beautiful, I wanted to hang it on my wall next to a picture of Liz Hurley.
You can see the board layout pretty well in this shot. You can also see the totally sweet removable fans, and the ridiculous heatsink I picked up to replace the disappointing stock one that came with the CPU.
This shot shows off the SATA backplane. It’s a tight fit, but you can remove the fans to make it easier to work int here while you are plugging things in, as you can see in the picture.
