Computers, Science, Technology, Xen Virtualization, Hosting, Photography, The Internet, Geekdom And More

You May Have Xen – But How About Tofu?

Posted on | February 18, 2010 | 3 Comments

Enough procrastinating. I have been working on a Xen friendly operating system for several years, a lot of the time has been spent waiting to see what happened with pv_ops in mainline Linux. Two years later, we’re still waiting to see what happens to pv_ops in Mainline Linux. Everything else (such as the Xen API, Libvirt, low level Xen libs) have settled down to the point where they aren’t too much work to maintain. Who knows, we may even see a mascot this year!

Over the past few weeks, I have been quietly working (sometimes in lieu of sleeping) on my own solution to getting Xen into a solid, reliable distribution for web hosting / IAAS providers. The usefulness would of course not end in the e-hospitality industry, anyone who uses paravirtualized farms might like what I’m cooking up.

I’ve named it Gridnix, the first release will be named Tofu. Why tofu? Because I hate raw tofu, every time I eat it raw I say yuck and cook it with my own special sauce. That’s exactly the point of this operating system, to provide a stable base for developers to use as a building block when leveraging the power of the Xen hypervisor.Now, down to the nitty gritty. Tofu is going to ship as (gasp) a builder script designed to be on top of a minimal Ubuntu Jaunty or CentOS 5 OS. Why just a builder script? Because I can push useful stuff out the door much sooner if I don’t have to work on an installer right away. Additionally, I’m dealing with both Debian and Redhat package types, did I mention that I’m making this interesting? One of the biggest delays in getting from thinking about code to actually writing it was deciding on the base distro and package manager, considering that the Xen community is basically split in half when it comes to that.

Anyway, in a nutshell, here’s most of the features categorized by what people care about:

Package Manager:

On Ubuntu, is the same, apt. On CentOS its apt-rpm. Cue the rotten fruit headed my way at a high velocity. I need spot on dependencies, ease of upgrade (with spot on dependencies) and the ability to centralize repositories for both supported base distributions. I also need centralized documentation, or as least as much as is humanly possible. If apt-rpm is good enough for Dag, its good enough for me. A note from the Devil’s advocate, development on apt-rpm has slowed down quite a bit, however that alone doesn’t rule it out.

Additionally, I’m working on a way to shift gears so people can get later / more edgy builds to test stuff and roll back if needed on a per package basis. Remember those dependencies? Finally, I realize that most developers are going to want stuff that I just don’t have, or newer versions of stuff that I do have. Someone might want something I have, and I have the right version, but its not compiled with what they need. For this, I’m working on a way t o let you roll what you need from source and tell apt what you installed, so that nothing has to go outside of the scope of your package manager. It can go out of scope if you want it to, but its up to you. Can you see my paranoia regarding dependencies yet?

Hypervisor / Kernel:

Both Xen 3.4 and 4.0 will be offered, with your choice of a rebased OpenSUSE dom-0 / dom-u kernel, or (possibly) Jeremy’s pv_ops tree. Its very likely that Tofu will start out with 3.4, however don’t rule out a menu asking if you want one, the other or both. I have to see how time goes. Keep in mind that I have to maintain CVE patches too, so I don’t want to drown myself in everything that has ever been Xenified since 2.6.16. However, even if I fly  it with just a Frank-en-SUSE dom-0 kernel, you are still free to bump up to 4.0 and go with pv_ops. That’s why I discussed the package management first.

User Space:

You’ll have hooks for everything, familiar APIs, familiar libraries and a lot of custom stuff. Event notifications to programs you specify on dom-0 will learn of guest events (starting / stopping / crashing / migrating), you’ll have great insight into PV domains from dom-0 and you’ll have some cool new tools to play with. You’ll have a few new libraries that help abstract some of the more confusing low level stuff when it comes to Xen and you may even see some Xen friendly PHP extensions. You’ll get much newer versions of stuff than you are accustomed to having with leading enterprise operating systems and some new alpha stuff thrown in for you to try. Don’t worry, the alpha stuff is nothing critical, its just new stuff that you might not have played with yet. Userland is a tofu casserole, nice and crusty on the top with some surprises in the middle. Compiler, tool chain, perl, python, etc will be whatever version the authors of those projects call stable. Of course, again, you’ll be free to install what you like.


According to Havis Warfolds, people focused on security are self gratifying primates. I take it a little more seriously. SELinux on an as-is basis, I’m not yet sure about GRSecurity. Quite frankly, we come back to pv_ops and its fate in future merges with mainline as both  camps stay on the tip. Since we’re using Xenbus / Xenstore a little more than most, facilities exist to prevent / stop abuse. Default Xen user land  packages will ship with the significantly harder OCaml version of Xenstore. You’ll have triggers for IP/MAC protection via the same event triggers that I talked about above as well as what I think is a pretty hardened dom-0.

On the other hand, standards like PCI/DSS, HIPAA, HITECH and more depending on how productive congress is over the next few months do exist. I’m hoping to package dom-u templates to help meet them most of the way, but I doubt that will be done by the time I feed Tofu to the wolves that scrolled down just to see this section.


There is no spoon and there is no desktop. No X11, no Gnome, no KDE. Nadda, zilch, zero, zip. It will always be that way. This is a strictly server OS, written for people who don’t give a damn how cool you think it would be to have full screen Flash video on an Ubuntu desktop. This move alone lets me work with almost all completely free, unencumbered packages without worry or expectation of something better. I won’t even touch on how many complex bugs I’ll never have to read. If it makes you feel  better, there’s plenty of hooks to write your own web based control panel. I was planning one, there simply is no time to keep considering it. I suppose you could pull in KDE or Gnome, but I can’t vouch for the results. This is for people who like to read documentation and get to work on something better.


Doxygen for code, Asciidoc for everything else. This lets developers at least try to document their stuff as they go and provides the shortest pat of  going from wiki markup to man pages among other formats. I won’t hold up a release due to a buggy app, I’ll just pull that app  out. I will hold up a release due to incomplete or non-existent documentation, its important. Expect reasonably thorough documentation in English. Translators are of course welcome, as long as you don’t mind submitting your work for peer review. The people who wrote the programs that you are documenting usually won’t speak your language, they need some assurance that you translated more than transliterated, there is a difference. Its not that we’re easily offended by mistakes, but we do need to deal with the resulting bogus bug reports that are likely to follow. <em>This hammer breaks eggs" </em>is strikingly different from<em>This hammer Lays eggs.”

Release Cycle:

Hard to say at this point. One of the reasons I’m not going with Ubuntu Karmic is the new version of grub, when in fact, work is being done on a SQlite3 backed version of the old one.  I should be able to keep up with the Ubuntu LTS releases, its not exactly hard to keep up with RHEL. Beyond that, we’ll see. This is one of the reasons why I am so focused on package management and accurate dependencies.


IRC, Mailing lists and Bugzilla. I have no plans to offer paid support for a very alpha release. When its out, if you like it and want to advance it consider having one of your developers work on it out in the open. Likewise, if you disregard my best advice and use it in production, I’d appreciate your patches, even if they suck. At the least, it tells me what you want the OS to do for you by default.


An angry fruit salad. Its a mix of GPL2, GPL3 and 3 clause BSD. What’s new? The Gridnix stuff is 3 clause BSD when possible, depending upon what I link against. A choice in license is as strategic as it is fundamental. If you refuse to use anything but the AGPL3 and like something that’s in my code, take it, I’m not going to complain. You won’t be stealing my code, you’ll be taking advantage of the very unrestrictive license that governs it. The opposite also holds true. If you take the new tools, hack them into oblivion and make a kick ass appliance .. you aren’t obligated to give your clients the source code to your changes. I can only point out that everyone benefits when development happens out in the open, if you make your contributions free you stand a better shot at not losing them when your customers need to pull in updates, unless of course you want to handle that? In the end, with either case, its much easier to just send patches under the same license as the original work. Trust me, it will save you time and money.

As far as it goes, now, the only non-free bits might be kernel blobs. Tools exist to roll kernels that are devoid of those if you hardware will still work after doing so. As such, Gridnix (at least Tofu) is not a 100% free operating system, but at least the non-free stuff is contained in kernel space. I’m comfortable with that for a first release. Beyond the blobs, no non-free repositories will exist, nor will a ports tree that just wgets the stuff for you. Its a server OS, its really a no brainer. Its also not a solution in the box, just 66% of one.


Gridnix has four purposes (not in order):

  • Make paravirtualization simple and easy for IAAS providers
  • Make life easy for developers
  • Give cool new alpha stuff exposure and testing
  • Provide a rock solid base to build a Xen solution with

Tofu will make good on at least two of the above.

Release Date:

April 1, 2010, this is strategic. Failure to live up to expectations can be chalked up to a premeditated April fools prank. What kind of programmer would I be if I did not leave myself an out? :)

In all seriousness, most of what I discussed is nearly done. Its just a question of what will be done ‘enough’ by April 1. There’s a fine line between alpha and broken. I’m not looking for additional contributors beyond what I have, I can’t afford to get sidelined by new ideas at this point. I’m also not looking for new killer stuff to put in the mix, I really have my hands full. As such, code won’t see the light of day until the release date, I’ve self imposed a freeze beyond finishing what I plan to package.

Target Users:

Anyone using paravirtualized Xen guests. Once the security hooks go in, any enterprise that employs developers who need to take a project from sandbox to start on the same platform. Target alpha / beta testers include anyone who has worked with Xen / Linux and knows their way around C / Python and shell scripting. There will be a lot of low hanging fruit, if I was a great programmer I’d already be working for IBM :)

If you got this far, thank you for reading. I’ll avoid the urge to insert a gratuitous rickroll, however tempting it may be. :) Don’t expect the world, but it is looking quite good. I know I’ve talked about this before, but April 1 is a firm date.


3 Responses to “You May Have Xen – But How About Tofu?”

  1. Nik
    March 3rd, 2010 @ 12:34 pm

    Looking forward to following your progress on this. I’m currently using free XenServer from Citrix, but rapidly moving to my own platform. The lack of end user VPS control shells have held me back. xen-tools and xen-shell looks interesting though.

  2. tinkertim
    March 3rd, 2010 @ 5:23 pm


    That’s one of the goals. As far as the userland goes, I / we want to make Xen “just another part of Linux” including underprivileged users. I can’t say that we’ll have a new IAAS-centric API fully developed by the first release, but I can say the specification for it will be well underway and suitable for public comment.

    Like any other library / API of this nature, I’ll feel compelled to write some kind of a shell on top of it.

    I think in a few weeks we’ll have some code ready to share, shortly after that will be the alpha release. We’re not rushing, but we are in a hurry, if that makes sense? This is coming directly out of necessity, I need this stuff to use in production now.

  3. Cem
    March 10th, 2010 @ 12:41 am

    Yeah, i totally acknowledge that the citrix api is just a ‘pain in the ass’, without lubrifiants =)

    We’ve ‘hardly’ been able to interface xenserver, just to do the bare stuff like stop, reboot and console … could have been easier; but what can you expect from a ‘company’ oriented package ?

    Go Tim ! Go Tim ! Eat as much tofu as you can :)

Leave a Reply

  • Monkey Plus Typewriter
  • Stack Overflow

  • Me According To Ohloh

  • Meta