The Angry Dome

I've been trying to enhance my mobile lifestyle lately. Or something.

Basically, as much as I love my MacBook Pro (oh yeah, guess I didn't blog that yet), I've really been hoping for something more portable that I could keep on my person at all times - you know, to handle any trouble that comes up whilst on call. That's the excuse, anyway - I also wanted a device that had a real web browser, a real mail client, GPS, wifi, and good media playback capabilities...

I was really, REALLY hoping that the "something" would be a phone, so I could ditch my existing unit and just replace it with a single device. The Nokia N95 comes really damn close, but its price tag is astronomical. Not only that, but it lacks compatibility with American 3G networks, which I really, really want in my next phone device (since we actually have 3G here...).

So, I went a different route - keep my aging W600i (which is a nice device, but old technology - at least it does EDGE) and "compliment" it with another gizmo for my mobile computing needs. Then, I'll pick up a cheap 3G phone (when such a thing exists) and ultimately plan on re-converging back to a single gadget that meets all my needs (once such a thing comes to exist).

The main advantage in this plan is that I'm now dealing with separate, less expensive components that I won't care as much about when they break or become obsolete to be replaced by my dream device.

Anyway, the N800 - it's pretty sweet. Packing a 300-ish Mhz ARM processor and a gorgeous display, it pegs media playback. Sure, I have to transcode stuff to a format that it's happy with, but that's fairly trivial with mencoder and once I've done so it works like a champ.

The browser is nice, too. It's got Opera - real Opera, not some crappy mini opera - and some oldish version of Flash. It can play stuff on youtube, even (if a bit choppily).

Wifi works great - good range, good speed. It's got bluetooth, and you can use your phone for connectivity in a pinch (and that EDGE connection, as much as it sucks, can be a really nice thing if you're desperate).

Where the n800 really wins, though, is hackability.

The device, and the Nokia 770 before it, runs a customized version of Debian and a slick Nokia-designed UI. Together, the platform is known as "Maemo," and it has all of the goodies you'd hope for from a real Linux device. Tons of applications have been ported to it, including xterm (a must), ssh (another must), openvpn (REALLY nice to have!), minimo (stripped down Mozilla browser), and... well, a crapload of other stuff.

There are also several apps unique to the device which specifically extend its functionality. There's this thing called maemo Mapper, which lets you use google maps in conjunction with a bluetooth GPS device to get live navigation. There's an FM radio app, there's an RSS reader, there's a weather app...

Basically, the N800 out of the box is exactly what it says it is - a web browsing appliance. But once you start tacking on the extra software, it becomes a full fledged mobile Linux computer. At about the size of a PDA (but with a MUCH nicer display), it's almost exactly what I was looking for.

Oh - and if you plan to ssh from the device, you can even hook up a portable bluetooth keyboard and make it a fully functional terminal. Sexy, huh?

Last time I described managing RAID pools with ZFS. This time, I'm going to describe what volume management is - and what it's like on Linux - as a way to compare a traditional model to that of ZFS (coming in part 2!).

Historically, with most operating systems, the notion of a "volume" was tied to the partition layout of a physical disk or RAID device. For example, an administrator could create 4 partitions on a single drive, and allocate different amounts of storage to each partition based on its potential use. He would then typically create a filesystem on each partition and determine a logical representation for that filesystem (in UNIX or Linux, this would be a location in the filesystem hierarchy - in DOS this would be an additional "drive letter").

If he wanted to have RAID configured, he could either use RAID on the entire drive (common with hardware RAID) and partition the resulting device (which commonly appears to the OS as a single, unpartitioned drive), or he could partition the drives and then configure RAID across the partitions (common with software RAID).

Partitioning is clearly useful, but it has some severe limitations. Once created, a partition scheme is generally difficult to modify. Growing and shrinking partitions (and their associated filesystems) is usually cumbersome, as is adding additional partitions to an existing layout. You could typically never have a filesystem that was larger than a single physical disk or RAID group. Basically, if you didn't accurately predict your needs initially, you could have a lot of work to do later.

Enter the volume manager. Most modern operating systems include tools that allow the administrator to add an additional abstraction layer on top of the partitioning scheme. Linux's LVM, for example, takes one or more block devices (in LVM terms these are "physical volumes" which generally equate to RAID devices, partitions, or entire drives) and allows the administrator to create "volume groups" which contain them. These volume groups can then be subdivided into "logical volumes" in whatever configuration an administrator desires, and on those logical volumes the administrator can create filesystems. To the administrator, these resulting logical volumes are functionally equivalent to partitions.

Volume managers, however, transcend many of the limitations of partitioning - they allow multiple devices to logically appear as a single large device, thus allowing filesystems to expand across physical boundaries; they allow for greater flexibility in resizing existing volumes on the fly; they allow additional devices to be added to an existing configuration on the fly; and they provide a mechanism by which other "useful things" can happen (such as snapshotting, encryption, or compression). However, there are still limitations - for example, LVM does not actually control the filesystem itself, only the logical volume on which it is placed. So, if you resize a logical volume, you still have to resize the filesystem that resides upon it, which may still require a reboot depending on the filesystem.

Now, if you want to combine all of this in Linux (to achieve RAID/LVM backed storage), you would essentially do the following:

1) Install physical disks
2) Partition these disks (fdisk)
3) Add these partitions to md RAID groups (mdadm)
4) Add those md RAID groups to volume groups using LVM tools (pvcreate, vgcreate)
5) Subdivide those volume groups into logical volumes using LVM tools (lvcreate)
6) Create a filesystem on each of those logical volumes (mkfs)
7) Map these filesystems to locations on the filesystem hierarchy (modify /etc/fstab and run mount)

Should he need to increase the size of a filesystem later, he must:

1) Modify the size of the logical volume (lvextend)
2) Modify the size of the filesystem (resize2fs)

To add a new filesystem, he must:

1) Create a new logical volume (lvcreate)
2) Create the new filesystem (mkfs)
3) Add the appropriate mount point (/etc/fstab and run mount)

Does this stuff work? Sure, and it works pretty well. However, from my perspective, it's all a bit cumbersome. But in't there a better way to administer storage?

Of course there is - and that way is ZFS. Next time, I'll tell you why.

So, you know, this game has been getting a lot of hype lately. I dismissed it out of hand a long time ago due to the horrific late-90's era graphics and the general consensus that it was a flaming pile, but having mostly burned out on WoW and not finding much else that runs on my Mac I decided to be fair and give it a shot.

First thoughts - the graphics look vaguely EQ-ish, and the performance is atrocious. Character models have a bit more detail than the environment, but are still crap by today's standards.

I can forgive some of this, though, given the nature of the product. Second Life is more of a framework than an actual game - it's basically a way to create, script, and interact with virtual objects.

From a technological standpoint, the backend is intriguing - the world runs on a cluster of Debian boxes, and supposedly they're planning to move to mono. The client is open source and cross platform (which is pretty much the only reason I even tried the game), and the protocol is documented (if not originally openly documented by Linden, it was reverse engineered).

In some ways I do want to like this project, and it's unfortunate then that I really can't. Despite the interesting design philosophy, when it comes to actual interaction with the world two thoughts come to mind - the fist being that all you can do in SL is explore user generated content, and the second being that user generated content sucks.

At the end of the day, SL feels like a pale clone of Stephenson's Metaverse, but unlike that fictional virtual world SL seems to exist just because it can exist, and not because it brings anything of value with it.

Stephenson's vision represents a future where the virtual world has as much to offer as the real world, where you could really lose yourself in an alternate reality that feels as real as your own but allows you far more freedom than you could ever really have. Above and beyond that, though, the Metaverse was the legitimate medium by which information was exchanged in useful ways, allowing you to experience knowledge visually - it is not only a virtual playground, but it's also a tool.

In contrast, SL may give you the freedom to create whatever you want, but it'll invariably suck due to the limitations of the engine, and it'll have no value over its real world or web counterparts which are either much more engaging or much easier to use. There's nothing to even do in Second Life, unless you like to fly around in a low poly universe with furries and cross dressers, and then click on a ball to start performing some action. It doesn't even have a legitimate reason for existing, like the Metaverse does - we don't even have tools to create content like what Stephenson envisioned, and what we do have (generally video, audio, text, and images) is just not suited for a free-form 3d environment.

So while SL is an interesting experiment, it's not really a game, and it doesn't have any real value as a way of sharing knowledge. Sorry guys - I guess the Metaverse will just have to wait.

UPDATE 12/12/2010:

I've been fighting a major performance issue with NFS over ZFS for a while now. It hasn't been a deal breaker or anything - everything I want to do over NFS has been working adequately - but I've noticed that my performance is subpar.

I didn't realize just *how* subpar until I started dealing with small files. Large files were generally OK - writing/reading a single large file via NFS would be about 80% as fast as with local storage. Since most of my mythtv operations are, obviously, with large files, this wasn't a big deal.

I noticed the problem with small files when I started migrating home directories to ZFS. Copying over the 85 MB ~mythtv directory took, according to time, 3m37.114s! What the hell is that about?

I then performed the same operation with rsync over ssh, which gave me a more reasonable result of 0m17.847s. That's almost 12 times faster!

I googled around a bit - well, more like a lot - and finally found out about zil_disable. Turns out that by adding 'set zfs:zil_disable=1' to /etc/system you can *dramatically* improve NFS performance on ZFS.

This is vaguely similar to turning on 'async' for Linux's NFSv3 server, but it impacts *all* filesystem operations, not only NFS. Effectively, NFS clients frequently request commit operations which inform the server to immediately write changes to stable storage. This happens at least on every file close over NFS, and the server itself forces a write to storage whenever metadata is modified. You can imagine then why there's such a huge impact when dealing with thousands of files - each single operation requires a flush of the cache and a write to disk.

After disabling the zil, I re-ran the same copy operation and achieved dramatically improved results:

mingus ~ # time cp -r ~mythtv /nfs/pvr/test/

real 0m18.265s
user 0m0.076s
sys 0m1.740s

Now that's more like it!

I've also noticed a moderate improvement with large files - both over NFS *and* locally.

I guess I'm actually a little surprised that the base configuration of the system would result in such horrid performance. I'm willing to make some trade-offs of performance for data consistency, but reading the blog mentioned above the risks seem tiny compared to the benefits of an order of magnitude performance increase.

I've been working for a while to migrate all of my data to my raidz box off of my mythtv box itself - which I finally completed this week.  I've been planning to rebuild the system using software RAID, so getting all of the video data off made things much simpler (instead of the ultracrappy LVM setup I had previously - basically, I had appended 2 400 GB drives together as a workaround for my limited storage).

Once everything had been rsync'd over to my Solaris box, the process of actually rebuilding and moving to software RAID was pretty painless. I'd made the rsync backup using an LVM snapshot when the system was live, which worked really well. Rebuilding the system using a gentoo live CD only took about an hour, and everything is working fine.

I found a post in some other language describing what you need to do to enable wiviz on OpenWRT .9.

Basically:

- Add backports to /etc/ipkg.conf
- ipkg install wiviz wl
- wl monitor 1
- Hit your router's web server at http://192.168.0.1/wiviz/wiviz.html

I've been running OpenWRT on a Linksys WRTSL54GS for a few months now, and I've been incredibly happy with the results. One of the main attractions was the ability to run OpenVPN on one of these little embedded devices, which pretty much eliminated the need for a full-fledged Linux or Solaris box as a router.

Now, the only real disappointment I had with the setup was a lack of fine-grained monitoring that I'd been used to with NTOP. After poking around a bit, I learned that ntop could support NetFlows sent from other device via a plugin, and that the OpenWRT device could send NetFlows with fprobe. This means I can use my MythTV box (for example) to run ntop on, and still receive all the data from the linksys.

On the OpenWRT side of things, setup is really easy:

root@openwrt:~# ipkg install fprobe
root@openwrt:~# fprobe -ibr0 192.168.1.6:2055

Where '192.168.1.6' is replaced with the IP of your ntop box. If you want to automatically start sending Flows every time the router boots up, you can add this to an init script.

Assuming you already have ntop up and running, adding a NetFlow is pretty simple there too. Navigate to the configuration options for the NetFlow plugin and fill in the relevant information. Activate the NetFlow plugin, and then you can view the statistics by selecting "Switch NIC" from the "Admin" menu in ntop. Once you do so, you'll start seeing what the router sees, which depending on your network may be, well, everything that's going on. At the very least you'll be able to track all connections from your internal hosts to hosts outside of your network, as well as all wireless traffic using the device.

This is the first of several posts in which I will explore some of the features of ZFS.

Sun has, traditionally, usually relied on software RAID instead of hardware RAID. It's a matter of some question as to whether this is really the best option - in most x86 servers, hardware RAID seems to be highly desirable - but I've often wondered just why that is.

There are some theoretical performance gains to be had by moving to hardware RAID (at least, over a traditional software RAID, such as Linux's raidtools or Sun's Solstice Disk Suite), but I've never even observed them personally. Furthermore, with hardware RAID, if your whole box or RAID controller fails, recovering data from your array could be a huge hassle - hardware RAID can easily lock you in to a chipset or vendor, whereas software RAID is completely portable.

My own opinion is that hardware RAID is popular simply because it's "easy" - you can order a box from your vendor, select a RAID level, and you never even have to think about it. Your OS will see the RAID arrays as standard block devices, and everything will just work - whereas software RAID requires the sysadmin to configure things properly, and traditionally has been somewhat annoying to configure.

Enter ZFS and raidz.

First of all, ZFS is insanely easy to manage. For example, to create my RAID5 setup, I had to issue a single command:

jnthornh@coltrane ~ $ sudo zpool create vault raidz c0d0 c1d0 c2d0 c4d0 c5d0
jnthornh@coltrane ~ $ zpool status vault
pool: vault
state: ONLINE
scrub: resilver completed with 0 errors on Sat Apr 21 13:16:18 2007
config:
NAME STATE READ WRITE CKSUM
vault ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c4d0 ONLINE 0 0 0
c5d0 ONLINE 0 0 0
c0d0 ONLINE 0 0 0
c1d0 ONLINE 0 0 0
c2d0 ONLINE 0 0 0

errors: No known data errors

Easy enough, right?  Furthermore, my pool - in this case "vault" -  becomes immediately available and mounted as /vault.  I can instantly use the array without any further configuration.

This is, from my perspective, completely awesome.  No hassle dealing with PERC interfaces trying to set things up, no dealing with Linux RAID tools or solstice disk suite - it's all done.  It just works.

Of course, raidz provides other advantages - like on the fly checksumming and guaranteed data integrity even in the case of a power failure.  Traditional RAID 5 setups, should they use power during a write, can end up in an inconsistent state as some drives may have been written to and some had not.  Not an issue with raidz.

This is all quite neat, but the big question is this - what are your recovery options?

In order to test this out, I unplugged one of my drives and rebooted the box.  The volume came back up properly, read/write - but zpool told me it was in a degrated state.  I copied some files around, plugged the drive back in, rebooted, and then 'resilvered' the array (which took only minutes and a single command) - and everything was back to normal.  This is what you expect from RAID5, so this isn't really a surprise, but it's good to verify that things are doing what you expect them to.

Now, the real challenge - what happens if my OS is hosed?  I decided on a 6 disk setup, 1 crappy IDE drive for the OS and 5 drives for the raidz.  I would have liked to use raidz for everything, however a raidz root or /usr is still not supported by solaris, so I've got them on their own drive.  I decided to simply re-jumpstart on another crappy IDE drive should my existing one fail, a process that should only take a half an hour or so.

So, I tested it.  I rebooted and reloaded the OS (up to Solaris Express while I was at it - I wanted ISCSI target mode which didn't make it into 11/06) from my jumpstart server.  I logged in, and my raidz didn't show up - hmm, bummer.  After a quick 'man zfs' I discovered what I had to do - 'zfs -f import vault'.  Like magic, my zfs pool appeared, and was even mounted at the same mount points and shared via NFS with the same options.  That means if I lose the IDE drive, I should be able to fully recover in under an hour.

Add to all this the fact that raidz actually *outperforms* most hardware solutions (according to sun, anyway, there are optimizations to read/write that the OS can do that hardware controllers can't) and you've got a winner.  Quick performance benchmarks showed my raidz spanking single SATA drive performance for reads and about equaling it for writes in my case - which is really all I could ask for.

Gentlemen, behold!

jnthornh@coltrane ~ $ zfs list vault
NAME USED AVAIL REFER MOUNTPOINT
vault 619G 1.18T 48.7K /vault
jnthornh@coltrane ~ $ df -h /vault
Filesystem size used avail capacity Mounted on
vault 1.8T 48K 1.2T 1% /vault

I had a few delays, and some things didn't go as smoothly as I might have hoped, but my home file server now lives and is backing my MythTV box.

I've started sketching out some documentation on my wiki. I won't go into all of the details in this post - I plan to add more information later - but my biggest delays weren't even related to the storage box itself. In fact, I stalled for weeks just because I was too lazy to crawl around under the house and run some cabling.

I did take a good chunk of time to hammer out some failure recovery stuff once everything was working. The totally awesome news - if I reboot with a drive missing, everything works. If I re-jumpstart the box and do a 'zfs import', the raidz pool magically comes back (and is even mounted for me).

ZFS really does seem like magic if you're used to dealing with normal filesystems - more on that later.

Behold my awesome storage capacity.