Carl the Zealot

Greetings all, I am proud to announce that my ultimate home file server is now up and running (besides a defective hard drive, but the replacement is on the way, and when it arrives I will rebuild the array and complete my testing).

Now written here, preserved for posterity, are the complete instructions, hardware and software recommendations, and steps necessary to make your own ultimate file server for under $2300! First, the complete hardware list:

• SUPERMICRO CSE-833T-R760 Beige Steel 3U Rackmount Server Case with 760W Triple-Redundant Power Supply
• Intel BOXDX48BT2 LGA 775 Intel X48 ATX DDR3 Intel Motherboard
• 2GB or 4GB of 1600Mhz DDR3 RAM (I suggest 2GB as when I tried 4GB it was very unstable with this mobo)
• Any Core 2 duo processor (but feel free to go all-out if you really want…)
• 8x 1TB 32MB cache SATAII Hard drives – if you don’t need as much space, feel free to save some money and go cheaper.
• 1x IDE hard drive – use one you have lying around, or get a new one. The smallest I could find was a 40GB drive for $25 on pricewatch.
• SIIG SATAII PCIe Controller – even though it doesn’t say, some internet research proves it does work on linux.
• You probably want a video card for troubleshooting – I would have been lost without one (sadly). They just don’t make quality motherboards with good serial consoles these days.
• You probably want a CPU fan, as the stock CPU fan for intel core 2 duos is QUITE disappointing in my experience.
• You may want a SATA CDRom drive to use just for installation – the IDE is really picky about detecting hard drives and optical drives at the same time on boot.
• You may want a power supply extension cable (24-pin, from pwr supply to mobo)
• You may want some fan extension cables, this case has a ton of fans but the cables are short.

I know for a fact the above-mentioned hardware works, with Debian Lenny, as I am about to describe. With reasonable effort (not counting DOA parts… =| ), I now have my file server up and running. I can’t promise similar results, but let’s just say, it would have saved me a TON of time and money if someone had posted this list for me.

Ain’t she a beaut? Complete with 8 hot-swapable units of goodness. See the rest of the pics at the bottom.

First, you need to assemble everything. Open up the case, plop in the motherboard, screw it in (be sure to put in all the risers, they are necessary to support the expansion cards when you plug them in so the board doesn’t break. Hook up the power supply, motherboard, memory, main OS hard drive, and video card. If you have an optical drive, slap that in there, otherwise be prepared to network boot, USB boot, whatever.

Next, install your OS. Anyone who knows me knows Debian is the only way to go! Do a minimal install, you don’t want an X server lying around or anything. Apt-get install your usual necessities (zsh, screen, vim, sudo, less, etc). Next you will want to apt-get install “mdadm” and “lvm2″, and “nfs-kernel”, and probably “rsync” as well. Once you are sure your hardware is working, shut down, and go ahead and install the extra 8 drives by placing them in the hot-swap bays, then running SATA cables from the backplane to your motherboard. Since it is really tough to find a motherboard with 8 SATAII ports, all 8 of which are usable in Linux, I decided to compromise and just go for one with 6 ports, and buy a PCIe controller card. If you wanted your OS drive to be sata, or have some hot spares somewhere, feel free to buy additional controller cards, but this case only holds 8 hot-swap drives plus 2 5.25in bays.

I got these directions from LVM2 and Software RAID in Linux. Without these directions it would have been quite a bit more time consuming to figure out. The directions are a little old (and not debian specific), however, so best follow these. First, note that mdadm by default will scan all your devices (see: /etc/mdadm/mdadm.conf. Assuming all your drives show up and seem functional, go ahead and create partitions on them, then link them all in an array.

# for i in sda sdb sdc sde sdf sdg sdh sdi; do sudo fdisk /dev/$i; done

Each time fdisk runs, do the following. Press “n” to add a new partition, “p” for primary, then press enter twice to accept the defaults (use the whole disk). Then press “t” to change the partition ID to “8e” for “Linux LVM”. Next press “w” to write the new partition table and exit. After you have done this to all 8 disks…

# mdadm --create /dev/md0 --chunk=64 --level=raid6 --raid-devices=8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 ...etc

At this point it will create the raid and start syncing the disks. You can do this to check the status.

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
      4883799680 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]
      [==>..................]  resync = 10.2% (100386816/976759936) finish=640.8min speed=22789K/sec

unused devices: 

As you can see, it takes some time, but eventually you see this:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0]
      4883799680 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU]

unused devices: none

Now things are ready for you to set up LVM. I am writing this part after the fact, and in my set-up I decided not to do LVM. This may or may not be a bright idea. If you wanted to set it up, it’d go something like the steps below. Also, I may have forgotten some of the steps, so please let me know if I am missing anything or if you believe any of this stuff is wrong.

# pvcreate /dev/md/md0
# vgcreate fileserver /dev/md/md0
# sudo lvcreate -l 1192333 -n fileserverlv fileserver

I think the number after the “-l” was the size I got from the output of the vgcreate command. Basically, you are creating a physical volume (which is just composed of the array), then a volume group which consists of that physical volume, then a logical volume in that volume group. What does all of these levels of abstraction buy you? You can resize the array, or the partitions (logical volumes) you create on the array, at will. I just created one large partition for now, but later, I could break it up if I wanted to. Also, if I replace all the drives, one by one, with larger ones, I can grow the physical volumes then grow the logical volumes which lives on them. Yeh, it’s pretty hot. At this point, you should have a device for the logical volume you can use (in the example above, /dev/fileserverlv I think, or maybe /dev/mapper/fileserverlv).

Next you create the filesystem. Since the chunk size I chose was 64kb, I wanted something which evenly divided that, and I wanted the index sizes to be appropriate for what is, basically, an enormous drive by most standards.

# sudo mkfs.ext3 -b 4096 -m2 -T largefile -n /dev/md/md0

Now the drive is ready to mount

# sudo mount /dev/md/md0 -t ext3 fileserver

Finally, get NFS set up.

# sudo apt-get install nfs-kernel
# sudo vim /etc/exports

You probably want something like this:

cmyers@petabyte.cmyers.org:1 (3:47:18)] cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
/media/fileserver 10.0.0.0/255.0.0.0(rw,sync,no_root_squash,no_subtree_check)

The above example lets every host on the 10.0.0.0 network access the drive, read-write, and allows root users to get root access. Go ahead and start NFS next:

# sudo /etc/init.d/nfs-kernel-server restart

You probably want to copy over some stuff – here is the rsync command I used to grab stuff from the old server – be warned, I had to install rsync, then I had to muck with things to make sure the versions of rsync were the same – different versions are not always compatible, so if you have an old file server which is a couple years old, it probably has an older version of rsync.

# rsync -aAX --progress :/path/to/source :/path/to/second/source [...etc....] .

And there you have it! Your new fileserver is ready to go!

cmyers@petabyte.cmyers.org:1 (3:44:42)] df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              12G  902M   11G   8% /
tmpfs                1007M     0 1007M   0% /lib/init/rw
udev                   10M  148K  9.9M   2% /dev
tmpfs                1007M     0 1007M   0% /dev/shm
/dev/md0              5.5T  521G  4.9T  10% /media/fileserver
cmyers@petabyte.cmyers.org:1 (3:44:43)]

The inside of this case is so beautiful, I wanted to hang it on my wall next to a picture of Liz Hurley.

You can see the board layout pretty well in this shot. You can also see the totally sweet removable fans, and the ridiculous heatsink I picked up to replace the disappointing stock one that came with the CPU.

This shot shows off the SATA backplane. It’s a tight fit, but you can remove the fans to make it easier to work int here while you are plugging things in, as you can see in the picture.

Well, I decided to head home a little early on Friday, I wasn’t getting much done anyways at work. As the bus tried to turn onto Pine, we saw a police vehicle blocking the way. The driver opened the door and asked what was going on, and the officer explained there was a pretty big wreck up ahead. We got off the bus and I started walking home to see police cars, fire trucks, and indeed a very bad wreck up ahead.

The ambulances, if there were any injuries, had already left, so I figured it had happened at least 10-20 minutes before I got there (around 4:50). I grabbed my camera and snapped a few photos.

View of the full scene.

Closer view – one of my better photographs, IMO.

Another close up.

Obviously, I didn’t want to get in anyone’s way so I took all my photographs from across the street, but it was pretty obvious what happened here. A taxi plowed straight into a white BMW. It seems likely someone was speeding, and someone else wasn’t paying attention. The taxi hit the BMW with enough force to drive it off hte road and onto the sidewalk, where it hit a telephone pole. I wouldn’t make any guesses about fault or what happened beyond that, however, since I didn’t actually see it, nor was I able to ask anyone what happened.

I work for Amazon.com. They are not your “average” software company. And so, I often wondered what it is like to struggle to find work straight out of college, just barely get that job, then work your ass off to get the requisite experience to get a job you don’t hate. This blog paints a pretty sad picture:

http://skepticalmethodologist.wordpress.com/2008/08/02/new-hire-cannon-fodder/

Wow, I haven’t seen something I disagreed with so completely since the windows 95 manual stated “Now windows is easier and more stable than ever!”. If this is how companies that aren’t Amazon operate, than I sure am glad to be where I am. Where this author is most wrong, however, is dragging the Amazons and the Googles and the Microsofts of the world into it. How the fuck does HE know? I don’t see any mention of him working for Microsoft of Google. He takes some article written SPECIFICALLY about YCombinator – a VC startup – and starts making all sorts of ludicrous conclusions about Microsoft and Google:

It probably began with Microsoft, but you can see its effect at Apple (notably Steve Jobs’ notorious temper) and I’m sure it’s at Google too (except it’s far more sinister there.) If you have recently graduated from college with a CS degree, congratulations, your stock options are just behind that Machine Gun nest.

Did Microsoft or Google even start with VC? I thought Bill Gates and Paul Allen started Microsoft by stealing from Xerox with their own sweat and blood. And Google started out at Stanford with a PhD thesis. Why is it “more sinister” at Google? Because their employees are even HAPPIER, and better compensated? Because they get free food?

The author of this blog suggests that companies hire all these “young, naive” programmers so they can make mistake after mistake, then suggests they shouldn’t try so hard, shouldn’t work so many hours, and should basically give up, coast through life, and be mediocre. How absurd! You don’t get a job at Microsoft, or Google, or Amazon, by being mediocre. The hiring bar is insanely high. This author either has some sour grapes over some startup they worked for, or they are completely making shit up. And as for the difference between startups and Microsoft or Google or Amazon – you can’t compare those inexperienced know-nothing startups who have only proved they have the strength to weather several months to a REAL software company.

Let me put it this way – Amazon was started in 1996 by a brilliant business man with a vision and a garage with some empty shelves where he hoped products would go someday. Now Amazon is a 20 billion dollar multinational company. It, unlike so many, survived the “dot com bust”. The people who founded the companies this author is referring to were “still in diapers” the year Amazon first made a profit.

Maybe I am just biased for Amazon, but I love working for Amazon. Sure, Amazon pays me “pennies” – over 8,000,000 of them a year, plus an excellent medical plan, plus I got a signing bonus, plus stock units worth an amount I’m not really supposed to discuss (let’s just say it’s more than you think). Add to that the fact that my work is interesting, challenging, and rewarding, and it’s a no-brainer. Add to THAT the fact that I could gain valuable experience here that I can get nowhere else, having worked for Amazon is a “get an interview free” card with just about any software company on the planet, my manager constantly talks about my career advancement and it is clear the company cares deeply about my future, and it’s a slam dunk.

So what hours do I *really* work? Am I just a brainwashed, 80-hour a week code monkey? Not at all. Amazon doesn’t care when you get to work, and Amazon doesn’t care when you leave. Amazon cares what you accomplish. So you know what? I’m not as smart as my peers. At Amazon, you are surrounded by brilliant people. Next to them, I feel inadequate at times – but that encourages me to work hard, and do my best, and it has made me a better engineer. So I work an average of 50-55 hours a week. I do this by choice, because I want that promotion, because some day I want to be able to take it easy, but today is not that day. When I do take it easy, it will be because I am talented and experienced enough to get my work done in 40 hours a week.

On that subject – the author is dead wrong about the ludicrous picture he paints of engineers:

Junior designers see lack of sleep as a ‘badge of honor’, they see long hours as proof of their worth.

That is, quite simply, the dumbest thing I have ever heard. What I see as a badge of honor is the guy that sits next to me, who I notice has a source control submit of *real* code every day, sends out code reviews twice as often as I do, and the guy leaves an hour earlier than I do, and arrives an hour later than I do, every god damned day. That guy is a fucking genius. In less time than me, he does twice as much. THAT is a badge of honor. THAT is an engineer I look up to, and hope to emulate some day.

So you can say what you will about “poor CS graduates” and the “mean startups” that abuse them, but from my perspective, that’s the exception, not the rule. If you want to work for a video game company, welcome to the club – so do 60,000 other CS graduates – and some of them are willing to pull 65 hour weeks if that’s what it takes. If you want to work for a startup that might mean retirement at age 29, well, so do 60,000 other CS graduates, so maybe you have to be willing to work a little harder. But if you are talented enough, you have plenty of “reasonable” options – for example, Amazon is hiring =P