Everything stops for tea

An analogy for the successful sysadmin

I've only been a consultant for 18 months now, but you obtain a wide experience of many different sites in that short period of time. By the very nature of the job, most of the customers we serve have limited technical skills themselves - otherwise they wouldn't be paying us such significant sums and we'd all be back in the corporate data centre instead of lounging around some engagingly homemade offices in South Manchester. Occasionally we visit sites where the admins have excellent Unix abilities, but either don't have the resources to install the product, are smart enough not to want to touch it with a bargepole or, bizarrely, simply aren't trusted by their management to do the job. (The last are extremely upsetting for us, both because we sympathise and because we get treated like something that came with the packaging.)

But in most cases, we see people who either don't have the abilities or can't afford to hire permanent staff with the abilities. I have a great deal of sympathy for many of them. They know their limitations and are working hard to overcome them in a demanding role (...as their appraisal said last year). Unfortunately, a few recognise their limitations and are determined to cling to them as the only solid knowledge they possess. I could be much more specific here, but there's a risk that someone reading this might suffer a fatal attack of recognition, which would be bad for (our) business. Such folk are generally besieged by consultants who carry a larger stack of timesheets and invoices than they do manuals. (Not us, obviously. But those other consultants.) Why not? They're wide open to being ripped off and, worse, they don't seem to care. Ignorance is not a crime, but in the IT industry it's a useful source of profit. Wilful ignorance usually attracts a feeding frenzy, and sharks don't generally repeat the coastguards warning to arrogant surfers before they attack.

It would be unfair and inaccurate to make blanket statements about these sites (and sadly inappropriate to name them and offer a reward for their termination) because you can find them even in the professional environs of the City. Now that Windows NT has allowed every two shakes DOS jockey to labour under the pretence that they're a serious network administrator, the lost and the lame walk amongst us in ever increasing numbers. In fact, most universities have better managed and cleaner setups than the average, cash-rich corporation (of course, college admins can wear jeans, saunter in at 9:30pm, aren't taken round the back and shot for rebooting and are generally treated like free-spirited adults, which helps).

The Cuppa Char Analogy

Whether you're a competent administrator or not comes down to whether you understand what you're doing. Feh, obvious! But it doesn't really have much relevance to how many manuals you've read, at what age your parents taught you your first ls commands or how much money your manager lets you mis-spend, usually on training courses. In fact, being a good administrator is a lot like being good at making the tea.

The more I think about it, the more I like this analogy and the broader my amused expression becomes. Because if you're ever completely stuck and up against it, as one so often is when dealing with computers, the first thing you should do is get up and go make a cup of tea. And the ability to make a decent cuppa can make all the difference to the ideas you have while drinking it and the success of your subsequent actions. Also, system administrators spend a large amount of their time making tea - in marketing terms, it's a core competency. Don't leave it off your CV, and if anyone ever compliments you on your brew, put them down as a referee.

This doesn't apply to that American muck that people think is cool just because they can import a brand called Java and bore their colleagues to death with their sophistication, any more than it applies to programming languages. Coffee is only for when consciousness is desired above all else, including taste or the need to snog someone without making them feel ill. Like before or after an all-night upgrade. The civilised English world drinks tea, else it descends into violent anarchy and NT service packs.

Suppose you didn't actually understand how tea works, what it's about or why you make it like you do. Suppose you learnt to make tea simply by copying a set of movements that someone once showed you. Something like:

There, you've just filled the kettle. Of course, you don't know that. You're just following a set sequence of movements. You continue and, some time later, you probably arrive at a drinkable, if not noteworthy cup of tea. Unfortunately, you repeat these motions every time you want a cuppa, even if the default conditions don't exist. In various cases, the following may occur:

These are minor annoyances though. However, one day the cup is two inches to the right but you, being a dozy fuckwit, pour boiling water all over the worktop because you only know enough to tip the kettle over that same spot. Worse, the next day the entire worktop has collapsed and you pour scalding water all down your trousers. This comes as a painful shock. And you're not adaptable enough to make coffee or cocoa or even get a can from the machine instead.

It's the same with maintaining a multiprocessing server of any size. If you just go through the motions without understanding exactly what they're intended to achieve, you're running on borrowed time. One day, something on that machine will die/hang/report errors or otherwise do the one thing the salesman didn't promise, i.e. screw up. And you won't have the first clue about tackling it. Meanwhile, your organisation loses money and your boss is speaking to recruitment agencies while typing up your notice (slight exaggeration: most sites of this sort have duff admins precisely because management lets them get away with it or is too cheap to pay someone qualified). Then you end up getting a new job as a Novell admin in a dead-end office stuck in the early nineties, and you can't even do that properly because you didn't know what you were doing previously.

The true tea drinker understands that the water must be freshly boiled to draw the maximum flavour out of the dried leaves contained within the delicate perforated bag. They know that a drop of milk takes the edge off the taste, while two sugars adds a naughty sweetness to the result. Actually, if they really care, they'll take the time to use a pot (warmed, of course) and loose leaves, add the milk first and buy a recognised type. They like to start the day with English Breakfast to wake them up, but prefer the lighter flavour of Darjeeling in mid-afternoon. (Of course, there will always be some tosser who messes around with those stupid bits of wire that let them brew an individual cup at the expense of some fiddly washing up.)

Similarly, the smart admin will first have specified the right machine for the task, balancing disk, memory and processing requirements with the intended application(s). (They know that you can run more than one app on a single host, providing you use the right mix.) They aren't afraid of heterogeneous environments where applicable and are able to make these machines work together, rather than in isolation. And if one of them won't co-operate, they prefer something that will. Because they understand not merely what these platforms do but a little about how they do it, they know where to start looking when something goes wrong. In the first place, they can recognise that something has gone wrong (the equivalent of realising that your cuppa tastes weird because someone put salt in the sugar bowl). Armed with this knowledge, they're in a much better position to put it right or find out how to put it right.

A really keen tea drinker will probably take time out to swot up on the origins of their teas, learning about the growing fields of China and India and the appalling conditions in which the pickers work, researching drying methods and appreciating the various strains of plant that give rise to different leaves. Eventually, they may know enough to be able to grow their own (even if the present climate doesn't permit it). This is probably too much information for most people, who are happy just to enjoy a decent brew every day, but you have to admire their dedication and graft.

...But I'm not going to rhapsodise about Linux hackers here.

Will ye have a cup o' tea now?

That's a nice little pat analogy that may gently amuse some and cause others to groan and slap their foreheads sarcastically. But it makes it obvious why the movement drone is never going to last long, let alone accomplish much of value (like that fresh cup when you really need it - after installing Veritas, perhaps). Even a basic understanding (I know where I fit, and it isn't planting tea plants) of why you type these weird commands will help immeasurably.

One can always spot the sites where they really don't get the gist. While you work, someone sits over your shoulder and writes down everything you type in a note book covered in indecipherable scrawl, with headings like "Fixing backups" and "Clearing memory". You try to explain what you're doing as you do it, but they're not listening. They think they can retype the exact same sequence of commands (including the frequent pwd's I enter while working to make sure I don't change the wrong thing) the next time a vaguely similar error message appears. Or any error message. If they don't understand something/anything, they either use it recklessly (whether it's harmless or dicey) or avoid it altogether (usually when it would make life easier instead). Supposedly relevant questions miss the point entirely. Warnings go unheeded; it works now, so that's alright then. Thanks very much, send us the bill. Bye, see you next time!

Of course, sometimes the consultant is bluffing, doesn't know, can't explain it properly or doesn't care. But that's a different story...

24th April 1998

Big Bubbles (no troubles)