Building a Devops team, Part 2

This is a guest post by Brian Henerey, from Sony Computer Entertainment Europe

A few months ago I wrote about Building a Devops team here at Sony. I really enjoy leading a team, and have tremendous pride in those I’ve brought into Sony. When I was given additional head count recently, I was excited and went straight to work. When your screening process makes the front page of HackerNews however, you know you’re going to have to mix things up a bit. Here’s what was new:

Success criteria

Throughout this round of interviewing, I found myself asking a few questions about each candidate that became the ultimate success criteria:

  1. Can they do the job?
  2. Will they make this team better?
  3. Will we learn from them?
  4. Can they pick up new technologies and languages quickly?
  5. What support with they need from me and the team?

Self-evaluations

I like to measure things and be as objective as possible in my work. This is enormously hard to achieve in recruitment because it’s about people, not machines, bandwidth, latency, etc. After the first couple of candidates we found, I started asking people to complete self-evaluations. I ended up comparing, weighting and graphing people’s scores to see if it might be useful. I stole this evaluation form from Google’s interview process and added a few categories to tailor it to my team’s needs:

Self Evaluation Guide:

  • 10 - Wrote the book on it (there must be a book)
  • 9 - Could have written the book, but didn’t.
  • 8 - Deep understanding of corner cases and esoteric features.
  • 7 - Understanding and (appropriate) usage of most lesser known features.
  • 6 - Can develop large programs and deploy new systems from scratch.
  • 5 - Can develop/deploy larger programs/systems using all basic (w/o book) and more esoteric features (some w/ book, some without)
  • 4 - Can develop/deploy medium programs/systems using all basic (w/o book) and a few esoteric features (w/ book). Understands enough about internals to do nontrivial troubleshooting.
  • 3 - Can utilize basic features without much help, manage a small installation competently.
  • 2 - can write hello world without looking at a book, kind of figure out how a system works, if necessary.
  • 1 - Can read programs, make small changes to existing programs, or make adjustments to already installed systems, w/book handy.
  • 0 - No experience.

__ TCP/IP Networking

__ Unix/Linux internals

__ Unix/Linux Systems administration

__ Algorithms & Data Structures

__ C

__ C++

__ Java

__ Python

__ Perl

__ Ruby

__ Shell Scripting

__ SQL and/or Database Admin

__ Scripting language of your choice, not already mentioned: _______________

__ Build systems and/or Continuous Integration

__ Performance testing and/or Capacity Planning

__ Project Management and/or ITIL

__ Configuration Managment (Puppet, Chef, Cfengine, etc)

__ Support of Production/Highly Available environments

__ Amazon Web Services (or other ‘cloud’ technologies/services)

Comparing the self-evaluations

This chart was really useful in getting me thinking about what I wanted. Candidate-8 certainly scored a lot higher than Candidate-1, but does it actually mean I want that candidate more? For example, before I weighted the scores someone might be good at Ruby, Python, Perl, and Bash and have an usually high score. I don’t particularly need someone good at all of those as they’re so learnable. I ended up combining those languages into a single score where I gave an extra point or two for broad knowledge, but masked the original high totals.

The chart also kept me from being overly excited about a single skillset of a candidate. I’m hiring generalist polyglot programmers after all. A couple of times we found that if a candidate had gone too far down a specialist path it seemed to inhibit creative thinking.

It is important however to keep in mind two things with this chart. Some people over rate themselves, and others under rate themselves. I had one candidate tell me “If I don’t love myself, who will?”. This was pretty funny, but it also meant I was going to critique his skills against the high bar he set himself.

Remote technical test

So last time I blogged about hiring I gave away most of the details of the test, so we had to rethink how we were going to do this. In the end, we asked people to install Wordpress on an EC2 instance again, but we made the task considerably harder. There were about 5 goose-eggs in this test that people had to get through:

  1. Iptables was filtering incoming traffic to port 80 on the external NIC.
  2. A process was already running on port 80 and it need to be killed.
  3. This process was run inside an infinite loop so anytime it was killed, a new process would spawn.
  4. Apache configuration files were present which set the port to 81.
  5. No MySQL root password was provided.

We ultimately wanted to see enough familiarity with Linux and basic systems administration skills at their disposal. Using things such as netstat, telnet, pstree, and kill are pretty basic, but I only gave people 1 hour so there isn’t much time for googling.

After 2 candidates I decided that anyone who’s CV was good enough would have to complete the remote technical test before I’d do a phone interview. While I spent a few hours writing/testing the Chef cookbook that created the test instances, this test was a VERY effective screening tool. Candidate-3 wasn’t able to complete any of the problems above even though he rated himself a 4 in Linux Systems Administration which should mean “Understands enough about internals to do nontrivial troubleshooting.”

Not many people passed the remote technical test, which isn’t surprising as I was mostly drawing people from Development backgrounds that may not have much Systems Administration background.

In person interview rounds

  1. A member of my team would spend 30-60 minutes pair programming with a candidate. We would do simple things, usually in a language that the candidate didn’t know. If they don’t know Ruby, we’d do the Ruby Koans with them. This is mostly to see if this is a person we can spend a lot of close time with as a member of the team.

  2. We had a couple of fairly standard interviews as well, going over the person’s CV and talking to them about the role. This also included white board sessions of programming and algorithm questions.

Programming assignment

When I hired the two previous members of my team, they both went and taught themselves Ruby before they even had an interview because they knew it was important to the role. One of them created a demo application that calculates the number of stops between Tube stations. He included this with his application and I was suitably impressed. We decided we wanted to see that the candidates were interested in learning a new programming language and also see how quickly they could learn it. I think this is a LOT to ask a candidate since there is no guarantee we’re going to hire them. In the end I made this optional, and used it as a deciding factor if I was on the fence. If someone had opensource projects I would have accepted them instead.

The end is the beginning

I’ve filled my head count and now my real work begins supporting them and getting the best out of team.

Published on