I have been a full-time Linux system administrator for more than a decade. This week, I lost my job because I am a full-time Linux system administrator. What happened?
For those outside my world, this is what a system administrator does: We manage server computer and networks. This means Internet sites, your computer system at work, and similar setups. The job dates back to the first time more than one person used a computer and someone needed to manage that.
That’s still the case, and there are many jobs for sysadmins. If you want to get one of those jobs, don’t worry.
However, I’ve been working in the world of leading edge startup technology companies, who write software themselves and also manage its use on the Internet. The trend here is toward something called DevOps (wikipedia article: DevOps). The short version of DevOps is: Software engineers take on the tasks traditionally done by “Ops” (system administrators) and largely automate them. It’s part of a general trend towards very fast product creation, quick response to change, and cost-cutting. (Look up “Lean startup” for more on this.)
Here’s how the whole setup works: You hire some young, energetic people. Make sure that they can pass technology skills tests. Even more so, make sure they are socially and ideologically suited to the environment. The engineers have to get along with each other and help each other out, and since most of them haven’t worked at normal jobs before, this isn’t a given. And most of all, they have to buy the local ideology, whether it’s “lean,” or “DevOps,” or “Agile.”
The work environment for these people is fast-moving and very disciplined. There are daily short meetings in the morning. Programmers almost always work in mutually accountable pairs. Everything is tracked: accomplishments, stumbling blocks, opinions. There’s a heavy emphasis on making new things and getting them “out the door” as quickly as possible. Dreaming at the desk, absent-minded professoring alone at the whiteboard? None of that.
Meanwhile, the job of the system administrator shrinks. Monitoring, software deployment, scaling the size of the systems up and down, a load of tasks are automated after a quick initial assessment. This is done by software engineers. New tools have accumulated decades of knowledge built into them. Other roles have been taken by services; there’s an entire ecosystem of companies who take away one piece of system administration and replace it with an easy-to-use service that attaches to your other easy-to-use services.
Aside from some holes in this fabric, the role of the system administrator in an organization like this has been reduced to high-level technical support. When engineers need to know something serious about the way operating systems work, or what a database server can do, the local unix subject matter expert is useful. Just not useful enough. It has become the Willy Loman profession.
Most of this is an extension of what system administrators have always done themselves. If you do something three times, automate it. Part of it is the result of the dot-com boom and the terrible laziness of its self-identified geniuses. If engineers are forced to work in an assembly-line environment while watching each other, people can’t horse around all day. None of that is unexpected.
The tiresome part for me is that the interesting jobs are going this way. This last gig was the best job I’d ever had. Everyone was smart, interesting things happened all day, and the company was accomplishing things I was personally proud of. There was a real team spirit and a feeling of involvement in something bigger.
Until I found out I wasn’t seen as useful, which is never a good time.
So my advice to you is: if you want to go into cool startups, you should either be a very rich founder, or a software engineer. Don’t go into operations.
And most of all, be young, very young, and inexpensive, and energetic. The startup world is necessarily cruel because it is built on the need of great returns on investment. If you are comfortable in a very interesting assembly line job that could be lucrative, this is your world. If you are someone with a store of knowledge, beware. You will be abstracted, automated, and discarded.
Be My Blog 
Sorry to read what happened. I feel I am in the same situation. I think I could even write the proposal up that would replace me (or replace my staff and keep me). Time for some re-invention of ourselves?
LikeLike
Don – Yeah, if I want to stay in this field it is time to re-evaluate the whole business.
LikeLike
I work at a place that would primarily be considered a “widget” company as opposed to a “service” company, so my outlook on this may be unique. We’re a profitable startup and we dislike most fresh-out-of-college guys. Sure, you can burn them out for cheap, but their experience is in writing code that is abandoned when the semester is over. It takes real experience to create maintainable code that’s going to run in an appliance in the field with lots of 9’s of uptime. You can’t just “oops, let’s deploy a hotfix to the middleware at 2am.” You release a software update today and the customers install it during their next maintenance window, two months from now.
I think we also have blurry lines between linux system administrators and IT. In IT there is one desktop-support guy, and the rest are dealing with infrastructure. Mostly it’s stuff like ensuring the LANs are correctly partitioned (we send around a lot of video), dealing with internet and satellite providers, dealing with building management over cooling and power circuits, and so on. It’s sort of a holistic-everything-adminstrator. Not so much “Linux” system administration beyond DHCP, NTP, and a giant RAID with minimal permission enforcement visible over SMB. A lot of what was Linux-specific admin stuff is now outsourced — Google for email, a web guy for the corporate website (which doesn’t need lots of low-level Linux fine-tuning; it’s just an off-the-shelf CMS with a custom skin on a virtual private server).
Based on talking to engineers and support people at Google and Amazon, I’m guessing that a lot of the Linux admin work is migrating toward companies providing servers as a service: cloud stuff like Amazon, Linode, and Rackspace; everything Google does; even Microsoft’s Azure has Linux servers now. Those companies worry about a lot of the messy details so you don’t have to — just write your Ruby app and deploy.
Things always seem down after “getting dumped.” You’re a smart guy with lots of experience. I expect this will be behind you soon and, with the benefit of time and hindsight, will seem like only a bump in the road.
LikeLike
«Programmers almost always work in mutually accountable pairs. Everything is tracked: accomplishments, stumbling blocks, opinions. There’s a heavy emphasis on making new things and getting them “out the door” as quickly as possible. Dreaming at the desk, absent-minded professoring alone at the whiteboard? None of that.»
Another thing that there’s no room for in that model: learning new languages/frameworks/platforms/etc. If you’re an employee at one of those frappunicoed up, nose-to-the-grindstone companies, you don’t get to decide to dick around for an hour or two a day, every day for a month, writing a hello world and then a toy app using some new technology, so you’ll eventually get the hang of it, and it may or may not be useful in the future– and you never can tell, until the future suddenly happens.
Or: maybe one guy goes and dicks around with some new framework, mulls it for a while, and then can tell other people it’s useful to learn, and I can help out, etc. And if that technology explodes, he’s the difference between a shop where NOBODY knows the new system, to where SOMEBODY knows the new system.
Or: somebody plays around with some smartphone from six years ago that nobody bought, but discovers that it has some great feature that maybe you could copy.
So you’re completely correct: “You hire some young, energetic people. Make sure that they can pass technology skills tests”. In other words, they’re walking in the door already having invested serious time (and not on your nickel) into learning whatever you think your company wants to do right now.
…But say the technology evaporates– if you’re running the company, you basically have to scream “everyone learn this new thing!! so you can quickly write impressive and stable pieces of software by a month from now!!– all while finishing our existing contracts!”. That’s very un-agile, so you just reboot the company (maybe or maybe not a new name) with a new crop of already-know-it workers, but the programmers don’t know eachother, or you, from Adam, and productivity will be extremely unimpressive, compared to if you had let the old crowd dick around a few hours a week and pick up a few new platforms/languages/etc.
Here’s an opinion that’s sort of tangential: Rands once said: if you have some guy there whose job has turned out to sit on one piece of COBOL that happens to be a bottleneck for the whole infrastructure and that’s all he knows, and has a DMV-like attitude toward, well, everything, still pulling a paycheck and still necessary, well, it’s the fault of you, Rands-advised manager, for not having done what you should now do: have him go take classes in new stuff, especially since the system in which the COBOL-O-TRON is crucial might get totally replaced at any time and with little warning…
But, with some quite motivated training, he might get to be guy who understands the new system better than anyone else.
(Conversely, losing him might be a bad idea, because, as you find out only once he’s gone, it so happens that he, without thinking much about it, is the only employee who knows the crucial order that the Big Servers have to be booted in, after a power outage, even though it theoretically shouldn’t make any difference. Or: he’s the only person in the company who could write at above a 5th-grade level!)
I don’t think Rands had advice about what to send this guy off to go learn. Maybe Rands just ment: wing it.
LikeLike
Position yourself as a consultant that can help maintain production cloud environments. That’s pretty hot.
Programming is increasingly a young man’s job. At 41, my options are narrowing, not expanding even with my skillset. Maybe it is time to get the band back together and go into rock. Bowie’s last album is awesome.
LikeLike