Computers, Science, Technology, Xen Virtualization, Hosting, Photography, The Internet, Geekdom And More

The Xen Dom-u Kernel And The Kitchen Sink

Posted on | April 28, 2010 | No Comments

Does anyone still like configuring and building their own Linux kernels to taste? I do. Call me a nerd, but I actually enjoy spending countless hours slimming down my kernel so it has just enough support for my hardware, with modular support for anything that I might be inclined to plug in. No, the performance gain is not appreciable, but I gain some memory in kernel space which comes in handy for working on oddball drivers.

Enter Xen paravirtualized dom-u kernels. These, quite frankly are the easiest in the world to build and configure once you have a suitably patched tree (if not yet using pv_ops from mainline). Either or, pv_ops or mainline, the process is the same. You are transported to a magical world where the front end of the Xen split drivers become all of the hardware that you will ever need to support for normal use. The rest of it, frankly is just file systems, security hooks, networking and whatever else may be needed if you intend to pass a PCI device to the domain.

Literally, its easier to start with a configuration that has everything selected as a module that can be selected as such. You then take out crap that you don’t / won’t ever need, like token ring, wireless, etc. Chuck the file systems that you (and anyone else using the guests) won’t ever want (e.g. minixfs) and select appropriate things to be built into the kernel to avoid needing an initrd. Ext3 and ext4 are good examples of things you’ll probably need to boot, which should be built in. Enable module support for everything netfilter has to offer (that could conceivably be useful in a dom-u) and then make sensible tweaks to processor options.

The point is, you can’t get into micro optimization (literally saving a few bytes on the drive in order to exclude a module like iSCSI or dm_multipath) when you are building a kernel that will be deployed on hundreds of virtual machines. On a Xen farm (especially a PV Xen farm) you want one kernel to maintain, period. This lets guests migrate around easily, helps you manage patches easily and helps keep support needs consistent. If you are a web host, your client should be delighted to find everything they would ever need just a modprobe away.

The last thing you want is a mix of kernels from dom-0 to dom-0, much less dom-u to dom-u on the same machine. That is just complete and total insanity. Yes, I have my own embarrassing deployment, but just one (which is where I learned my lesson years ago, and Xen was much more of a kernel hopping moving target in those days). Still, to this day, I keep encountering farms built out in-house with a menu of kernels that rivals Bob’s Big Boy.

If you are going to go through the trouble of rolling your own dom-u kernel, be sure to leave modular support for anything that could conceivably work within a paravirtualized dom-u. Trust me, someone will want it – you’ll feel really good instructing them on the ways of modprobe rather than firing up a new build, with yet another different kernel out in the wild to keep track of. What happens when another CVE comes out and you update all the kernels again? Can you really remember who had custom support for what?

One size fits all for Xen guests. Save the optimization for your desktop and physical servers. On a side note, I’m going to be pushing a new HG repository next week that contains nothing but kernel configurations, ¬†patches and scripts to apply them. A lot of people still seem to be having issues with pv_ops, so I decided to abstract ‘menuconfig’ a bit with some pre set ‘scenarios’. Its working well, I just want to polish it up before I push it.


Leave a Reply

  • Monkey Plus Typewriter
  • Stack Overflow

  • Me According To Ohloh

  • Meta