The Angry Dome

Yes, Gentoo Linux sucks.

There are a couple of huge reasons why it sucks:

The sheer time it takes to compile everything means you've got a lot of wasted cycles on your hands, even when everything works.

It requires you to waste time with ultimately mundane configuration decisions that are handled automatically and competently by most other distributions.

Portage is broken, at least for several key packages I was trying to use. Admittedly I was setting up a system with fairly non-standard hardware with the 2.6 kernel, but packages for what I wanted were in portage - they just didn't work. I was having to compile a bunch of crap from the original tarballs - I thought Gentoo was going to do all this for me, hmmmmm?

Gentoo is of no value in a production environment and of marginal value anywhere else. The only uses I see for this distribution are for tinkerers and people who just want to learn what makes Linux tick. Otherwise, you're just creating more work and wasted time for yourself.

The other day, I heard an interesting snippet of an interview on Fresh Air with P.J. O'Rourke, a conservative journalist and satirist working for the Atlantic Monthly.

O'Rourke brought up a point that I've been aware of for many years - that is, that a vast majority of the political commentary coming from the American media is so blindly polarized that it has no chance of convincing anybody of anything. O'Rourke's primary examples of this effect are Rush Limbaugh and Al Franken, who he argues are essentially two sides of the same coin.

I might take some issue with that comparison - I find Franken to be substantially more humorous and a bit more concerned with facts than Limbaugh - but O'Rourke's main point, that neither man is likely to convince anybody who doesn't already agree with him of anything, is spot-on.

I've concluded that there are, essentially, two types of politically concerned citizens in this country - let's call them type A and type B. The type A voter establishes a strong set of beliefs, latches onto a party, and refuses to examine its failings - he sticks with that party unflaggingly, voraciously consuming every piece of criticism against his "enemies," and blindly lauding his chosen allies. A type B voter may hold equally strong beliefs, but he is very much interested in objectively examining the reality of the process instead of strictly alligning with one party or ideology.

It could be argued that the incendiary and highly polarized rhetoric produced by the likes of Michael Moore, Al Franken, Bill O'Rielly, and Rush Limbaugh serves to validate the former behavior. Among these individuals there really are no shades of grey - they all paint unabashedly negative pictures of their opponents, frequently characterizing them as at least incompetent and at worst pure evil.

Now, consider this for a second - if you're a relatively moderate person, who holds many views in line with the Republican party, who is currently leaning towards voting for Bush, how would you likely react when people such as Michael Moore paint Bush as completely incompetent and borderline evil? Would you suddenly "see the light" and vote for a Democrat? Would you become an instant fan of Moore, believing everything he presents?

Probably not. The pundits don't realize (or don't care?) that their rhetoric makes the citizens who identify with the politicians being criticized idiots by association - if a person likes Bush, and somebody calls Bush an idiot, that person will probably take that personally and will stop paying attention to whatever that pundit is saying.

These media polarizers love nothing more than to preach to the people who already agree with them. Moore's films are made for people he agrees with - he makes only token attempts to present both sides of an argument, so somebody holding different views going into such a film will see it only as propoganda, disregarding whatever actual truth it contains. The same holds true for O'Rielly, Limbaugh, you name it.

All of this plays right into the hands of the Type A voter. Type A voters already believe that politician X is bad, so when pundits kick it up a notch and tell us how absolutely evil politician X really is, the type A voter eats it up. Yes, politician X is evil. Politician X couldn't possibly have any good ideas. Any viewpoint that Politician X espouses must obviously be wrong, and anybody who agrees with Politican X must also be wrong.

These talking heads drive a wedge squarely through the voting population. They also reinforce the 2-party system - when the opposition is portrayed as evil incarnate, type A voters feel obligated to prevent a doom and gloom scenario by voting for whichever of the strong established parties could stop the horrible alternative from destroying the country. The real issues that are rotting both parties from the core (primarilly corporate ownership of the political process) lay hidden, and are neatly ignored by a vast majority of voters.

Type B voters, on the other hand, like to have our viewpoints challenged. They like to hear why people think they're wrong, so they have an opportunity to consider their own positions and strengthen them - or, if presented with a good enough argument to the contrary, abandon them. They allow for the possibility that what they believe may not be right, and they like to know why people agree or disagree with them so they can be sure they're making the right decision.

The old expression "you catch more flies with honey than with vinegar" comes to mind. If you actually want to convince somebody of something, you can't just go around telling them how wrong they are - you have to present both viewpoints as reasonable, but clearly explain using facts why you believe a particular option is preferable. If you just tell a person he's wrong, without taking the time to understand why he believes as he does, you only look like a zealot who will never convince anybody of anything.

I spent most of the day trying to figure out why some machines at work had stopped showing up in the SMB browse list after I added them to our samba domain. The answer? They had apostrophes in their "Computer Description" fields.

What a fun bug.

It sometimes bothers me that my email solution at home is far better than the one I use at work.

At the office, we're shackled by this bizarre need for Outlook calendar synchronization. My supervisor is under the impression that the users would never accept a calendar program other than Outlook - I can't really understand this, but the demand was made, and Outlook calendars it was.

Of course, Exchange + Win 2k3 Server + 50 CALs is incredibly expensive, far beyond the company's budget. So we ended up with SuSE OpenExchange 4.1, which we picked up on the cheap ($900).

OpenExchange is a pretty solid product, but the "Outlook Synchronization" is far from stellar. The system has a pretty nice (but closed source) webmail/calendar interface, and comes stock with webdav, spamassassin, postfix, cyrus, and several other useful packages to run out of the box. User accounts are stored in LDAP (ah, my good friend LDAP), address books and schedules are stored in Postgres. The "synchronization" with Outlook is accomplished via a client-side webdav connector, which uses DAV to synchronize a user's address book and contact list periodically.

The server that runs OpenExchange is a dual P2-600 with 512 MB of RAM, using 3 drive RAID5. I thought the system would be plenty beefy for a mailserver (it wasn't like I could convince them to get better anyway), but I was surprised to see consistantly high CPU usage and memory leaks that bit deep into the swap space. By my estimates, if I leave the box running for 3 months it will require a reboot.

The big problem with OpenExchange is that it's more of an appliance than a real system. It's designed to be administered almost exclusively via the web interface and YaST2, neither of which really give me the raw flexibility I desire. OpenExchange also has a per-user licensing scheme that reminds me of Microsoft - you have unlimited IMAP/POP connections, but for $900 the groupware component only allows for 10 concurrent connections. It also suffers from feature creep - the system does way more than it needs to for 99% of the users (it can even act as a Samba 2 PDC - who the hell would want that at this point?).

I feel like the OpenExchange box is some Linux bastard child, and Outlook is evil incarnate, but I'm simply unable to do anything about either situation.

At any rate, at home I'm using a much preferable solution - exim, amavis, spamassassin, clamav, Courier IMAP, and squirrelmail.

Spamassassin and clamav work swimmingly, flagging all of the garbage mail and punting it back to exim via amavis. Exim delivers it on to the user, but a few lines in my ~/.forward file on the server enabled me to dump viruses/spam/etc directly into special IMAP mailboxes. Aside from the obvious advantages for organization, this gives you the chance to use sa-learn on your Maildir folders, which increases the effectiveness of spamassassin.

I then normally check my mail via the exceptional Mozilla Thunderbird, connecting to Courier IMAP over SSL and using smtp-auth with TLS for sending remotely. In the event that I don't have access to Thunderbird, I use Squirrelmail as a completely unimpressive but totally functional webmail client over http SSL.

This stuff all works so wondefully, I'm only left wishing I could use it at work. Ah well.

In the past, I've always run Red Hat Linux on my boxes at home. Sure, it had its quirks (RPM dependency nightmares, RH-specific config files, etc), but RHL was always rock solid, reasonably current, and cheap. Although I made a point to at least try other distributions, RHL was the measuring stick - and everything else just came up short.

Then came last year's bombshell - RHL was no more, and even RHL 9 was to be EOL'd in early '04. What's a guy with RHL on everything to do?

Out of desperation, I upgraded my EOL'd RHL boxes to Fedora Core, Red Hat's new "community driven" Linux distribution, hoping it would be a good interim solution that wouldn't involve a complete rebuild. I was reasonably satisfied with Fedora Core 1 at home, so when I took my current job last November I upgraded the EOL'd RHL systems at work as well. Unfortunately, given some reported problems with Core 2, combined with the rapid pace of changes and the semianual forced upgrades, I knew I'd need a better solution for the long haul.

I initially assumed I would end up with the same distribution for all of these former RHL machines. I hashed out the following lists of requirements:

Work:
1) Proven stability
2) Software availability
3) Ease of administration
4) Feature freeze - not a moving target
5) Longevity (long-term patch availability without forced major version upgrades)
6) Inexpensive

Home:
1) Stability
2) Software availability
3) Ease of administration
4) Longevity
5) Inexpensive
6) Flexibility

So the search began.

I'll start off by saying that every Linux distribution I've tried has at least one incredibly annoying aspect that keeps it from being perfect for me. RHEL updates cost far too much, Fedora Core is bleeding edge and has major version upgrades far too often, SuSE also suffers from frequent version upgrades and is a configuration nightmare due to the monolithic YaST, gentoo is too bleeding-edge and compiling patches is annoying on slower systems, etc.

After trying every flavor of Linux known to man, I finally narrowed down my options to a few finalists. FreeBSD (no, it's not Linux, but from a functionality standpoint it is similar), CentOS Linux, White Box Enterprise Linux, and Debian GNU/Linux.

FreeBSD 4 certainly met my requirements for stability, longevity, software availability (via ports), and price (free). But what about administering a FreeBSD system? I know vi like the back of my hand and I'm familiar with generic Unix concepts, but even though I've used other unices my primary system has always been Linux. From a Linux user's standpoint, the basics of FreeBSD look pretty darn similar, but as time goes on you notice that things aren't quite right. You start finding out that the --monkey option only works on the GNU version of foo, that the soandso.conf is in a different location and of a different format, and that you end up hitting man pages for commands you thought you knew by heart. I found myself wishing that the systems were either a little more similar or a little less, as I constantly ran into minor differences trying to do familiar tasks. The package management utilities for ports are spartan, and though I was able to accomplish what I needed to do, I found tasks taking twice as long as I expected them to.

For me, FreeBSD loses, but I can't help but thinking of the old breakup line: "it's not you, it's me." Package management could stand to be a bit more robust, but I imagine an experienced FreeBSD administrator could do fine with what's available. I may revisit FreeBSD when I have more time.

Now, on to the attack of the (RHEL 3) clones. White Box and CentOS are both functionally identical, repackaged-to-be-free versions of Red Hat's flagship offering, Red Hat Enterprise Linux 3. Running through my lists, RHEL 3 has everything except for price - there's a 5 year guaranteed lifespan, no constant upgrading, easy administration, and a thoroughly tested and promptly patched system with a stable set of core software. Lots of 3rd party binary packages are available in RPM format, and the chances are that you can find rpms for pretty much everything (RHEL 3 is similar to RHL 9, so most RHL 9 compatible packages should work just fine). The price meant that "real" RHEL was out of the question, so CentOS and White Box entered the picture. CentOS won over WBEL, as it's a project with multiple maintainers that seems to offer a higher probability of future support. Interestingly, it's trivial to switch a live system from either distribution to the other, as they both use yum/rpm for package management - so if the project you opt for dies and the other lives, you'll still be able to update your systems. The one achilles heal of both products - security patches come from recompiled RHEL3 srpms. Even though Red Hat is required by the GPL to provide the source code of these patches, there is nothing that says they have to make it easy for us to get that code quickly. Should Red Hat decide to pull the plug, the CentOS community will be left scrambling for a way to keep systems up to date. I imagine CentOS would become a proper "fork" at that point, with user-packaged rpms instead of recompiled RHEL srpms.

On to Debian. In a lot of ways, Debian is (to me) the Holy Grail of Linux, with a couple of glaring weaknesses that prevent me from using it on the systems at work. Debian is a joy to administer (thanks to dpkg, apt, and some other inventive solutions such as the alternates system), has virtually every useful software package available via apt, is completely free, is proven rock-solid stable with timely security fixes, includes great documentation, and has massive community-driven support.

The problem with Debian is that you can't have everything at once.

It's important to understand that Debian comes in several main flavors - Stable (Woody), Testing (Sarge), and Unstable (Sid).

Debian Stable (Woody) is the "official," frozen version of Debian, released "when it's done" and only patched to fix security issues and bugs. Woody provides one of the most rock-solid, guaranteed-to-work Linux environments out there. The problem, as you may know, is that Woody has virtually no new versions of software from the past 2 years - in fact, the default Woody kernel is still 2.2. Ouch!

There's a not-so-fine line between being "stable" and being "archaic," and unfortunately Woody crosses over to the wrong side of that line. Woody isn't exactly useless - if it has the software you need, you can set it up, automate updates, and be fairly sure you'll never have any problems with the box until the hardware fails. You can force Woody into the 21st century by pinning packages from testing/unstable or using unsupported backports - but if you install a bunch of unstable, unsupported software on Woody you have to ask yourself just how "stable" your Frankenstein system will really be.

Then there's Unstable/Testing. Sid and Sarge are closely related - newly modified packages show up in Unstable first, and only after they're proven to be fairly solid will they work into Testing. For this reason Testing often has missing/outdated packages, and is really only useful if you're willing to pull required packages from Unstable.

Debian Unstable/Testing can be bleeding edge, but it doesn't have to be. You can stick with Apache 1.3 and the 2.4 kernel, or you can easily throw on 2.0 and 2.6. Unstable/Testing includes virtually every free software package for Linux, easily installable via aptitude. I wouldn't be surprised if you could just "aptitude install the_kitchen_sink." It has an amazingly powerful package management frontend in aptitude, has a wonderful configuration utility in debconf, and has countless other little things that make it one of the cleanest, most enjoyable systems I've ever used.

The problem, of course, is that it's a moving target with questionable stability. You can mitigate some of the potential problems by selecting more stable versions of software, but in the end you can't escape the fact that Unstable/Testing is a work in progress and has absolutely no guarantees.

For work I decided on CentOS and didn't look back - its solidity and longevity made it the ideal choice for predictable systems designed to fill specific roles for years to come. Although I absolutely fell in love with Debian, Woody was useless to me without tons of backports and Sarge/Sid were just too unpredictable for production use. The only extra software I required for CentOS was ClamAV, which is available in handy RPM format and easily updatable via a 3rd-party yum repository. No fuss, no muss, statically configured systems designed to keep trucking without constant tinkering.

For home, I started to rethink my priorities. Just what was this box going to do, anyway? Flexibility seemed to really come into play, as the role for the system is very vaguely defined and subject to change frequently. Debian's massive vaults of packages started to look really attractive, and since (unlike work) users aren't depending on 100% reliability I can sacrifice some stability to gain versatility. I ultimately decided on Debian Unstable, which has continued to surprise me with its quality. I have yet to see any stability issues, and the system is a joy to work with.

Ultimately, I'm actually thankful to Red Hat for EOL'ing their venerable Red Hat Linux series of products. This has facilitated the need for the RHEL clone projects (CentOS/WBEL) which fill certain needs better than RHL, are 100% free, and never would have existed had RHL been continued. It also led me personally to re-examine Debian, which I've found to be a better solution for my personal system.

Something else of note - if you still have RHL 7/8 (maybe even 9?) machines and are trying to decide what to do with them, you can "yum up" a live system to CentOS/WBEL without having to rebuild.

If you want to enjoy all of the high-quality slashdot/somethingawful/INSANE eBay auction/message board links from IRC without the hassle of actually paying attention or grepping logs, I've added a link on the sidebar to the last 40 URLs my IRC client has spotted. The page is updated in realtime.

I finally set up email and this blog on armstrong. jeremy@etherized.com is now functional.

My eBay Powerbook scam saga is hosted here for your viewing pleasure. It's not funny, it's just sad, so don't go in expecting something like the incredibly amusing p-p-p-powerbook scam.

Everything seems to work. Awesome.