Posted on | October 3, 2007 | 1 Comment
Needing to fix something because someone else fixed something is just a part of being a system administrator. For years, I have tried various methods of controlling /etc to combat brain dead programs that modify things in /etc, as well as mindless admins who believe everything that some tutorial found via some search engine tells them.
My first offensive in this battle was using things like Subversion. If you think about /etc , its just a bunch of text files (just like source code). Logic says that you should be able to use a versioning system to control these files, easier said than done. The problems that I ran into were basic but stubborn:
- A centralized repository did not work for all things. Some computers adopted custom configurations (for clusters and such) but needed to continue to pull in changes that all nodes did, such as /etc/hosts.
- Other admins made lots of commitments, rolling back was difficult if needed
- Control panels (like C-Panel) delight in clobbering files in /etc sometimes these slipped in to other commits and broke all kinds of stuff on other nodes.
- The above drove me nuts.
The copy on write file system named ext3cow is not yet ‘officially’ production ready, however I can fully attest to its usefulness on file systems that don’t get ‘clobbered’ with lots of writes in high frequency. I still wanted some kind of version control for quick rollbacks, so I turned to Mercurial.
My setup was simple:
- Install ext3cow
- Install Mercurial
- Create /etc as a 1GB (should be plenty) ext3cow file system
- Go to /etc and type ‘hg init’
- Edit .hgignore to exclude things like passwd*, shadow* and others
- Type ‘hg commit -Am “Initial checkin of /etc”
- Edit .hg/hgrc and add this line :
style = null
This will help prevent someone mindless from exporting /etc via http. ‘null’ can be gibberish, it should cause the built in Mercurial web server to bail because it can’t find the theme named ‘null’ or whatever gibberish you write.
That’s it, your done. When you change something in /etc, do this :
- ss /etc
- cd /etc
- hg commit -Am “Changed foo to bar”
This gives you a combination punch to roll back major mistakes when things go wrong. Sometimes, other admins will change things, make commitments, realize the ‘uh-oh’, change again, commit again, and again, and again. Try rolling back (quickly) after such mistakes, good luck. With ext3cow, I just copy over the entire repo (which is now /etc) from a previous date in history, just prior to their first commitment, easy huh?
You’ll need to do a bit more work than I described to get ext3cow and Mercurial going, clear documentation is available on their respective web sites. The benefits of this are great, I can pull /etc centrally and merge on many nodes (say to pull in some new SNMP stuff), I can ask junior admins to send me patches prior to making changes, all kinds of nifty things.
Proper user setup and access to root via sudo (or whatever) is also important, I’m not touching on that here. Everyone who has root needs to be tracked, this is basic common sense. If you allow direct root logins to your stuff (to many people), this idea might not be for you
I spend hours every day fixing problems that were not caused by human error, the above combo helped me rule out quite a bit of human error, maybe give it a try