I recall a few jobs ago I was managing the Systems Engineering team and trying to grow it into a group that could handle the daily onslaught of issues that would plague our service. We had recruiters working on sourcing folks & we felt like we had very high standards for who we wanted to hire. We talked to and turned away probably hundreds of folks it felt like (I got really good at interviews).

The problem: We were looking for Rockstars.

We needed people who could walk in and start to cleanup the mess without a lot of direction. They had to be able to script around broken code, sit in design meetings with engineering to build better applications & improve system performance, handle on-call, be self directed, design networks, get along with everyone, type fast, think fast, act fast and work long hours… the list went on. Every decision they made as an individual was likely to have a high impact on availability and so the thinking went, the higher quality team we built the higher our availability would be. My thinking has definitely changed since then…

Don’t get me wrong, these are all fantastic skills to find in a sysadmin and most of us would consider these requirements of a Senior level team member. The problem though is that when you find people with that level of experience they also are used to making the decisions and are often challenged to work in a collaborative way. Having a whole team of them can be a challenge. Decisions get made in isolation & individuals forge their own path rather than using common standards.

This is where I think there is a problem. Ignoring the reasons for my above scenario (Operations compensating for bad software is another post entirely) – there’s a more fundamental problem here.

So why is this a problem?

When I talk about “Rock Stars” I’m talking about very experienced & skilled team members that consistently shine above the rest. These folks normally end up as technical leads or at the very least are a team of one. The point is – they are often in positions where they are making many of the technical decisions, or are at least the final word on whether a decision is made. It’s an art to know how to do something because you’ve done it before, still allow other ideas to flow, and not condemn them because you know one way that works. This, in my opinion, differentiates some senior folks from others. Allowing new ways of thinking that challenge your own is hard – but that’s what is needed.

There are variety of problems with relying on Rock Stars:

• One person can never think of everything. A group will typically make better decisions on average than one person.
• Creating a pillar of knowledge in one person means when they leave that support is gone and very rarely is there compensating knowledge transfer & documentation.
• You don’t get a culture of experimentation and flow of new ideas when one person calls all the shots – or you are dependent on that persons willingness to experiment.
• One persons opinion can outweigh the data – this is a huge problem.

When you have a team of these folks, you split them up and have them all work with different teams – they are commandos. They go off and “get it done” – usually in a variety of different and sometimes conflicting ways. When they are done, only they understand why the decisions were made the way they were, and only they know all the details.

The other side of this coin is collaboration and process. In a collaborative environment you are sharing all the time & decisions in general are made by the group. An individual may propose a particular solution, but it’s up to the group to get comfortable with it and form consensus to move forward. When Rock Stars are present this becomes harder if they aren’t open to ideas. It’s possible, but is dependent on personalities and motives. Your senior members have to see their contributions as advisory & feel successful because they are contributing their experience and knowledge so that a group can make a better decision. The results can be amazing, but the team has to feel successful as a team & not because individuals are performing feats of strength.

Part of what we all love about being in Operations is the ability to try new things and see them work – the pride of ownership. When one person calls all the shots that doesn’t exist for anyone except them.

Building a team for collaboration

Building a good collaborative team starts with hiring. I think hiring very experienced individuals is good – but overrated (hopefully this isn’t a career limiting statement). You want to have someone who can guide you through common problems that are experienced in Web Operations to keep you from running ashore, but you also need that person to allow new ideas to flow. A Sr. team member that ask questions “If we do that, how would we handle xyz?” to prompt discussion is going to guide the conversation away from problems without putting too much weight behind their own ideas about how the problem can be solved. This also helps other team members talk through scenarios and understand more clearly why certain ideas aren’t ideal – everyone walks away understanding why you chose the path you did.

The second part of hiring is to find skilled SysAdmins who are more Jr. but are hungry to learn – these folks will be most likely to ask questions & will drive knowledge sharing. Your Jr. team members will also not be as likely to condemn a different way of doing something and should be challenging the status quo – “Why do we do this in this way?”. They need to understand the landscape before they can contribute to a decision, which means you spend some time educating them but the end result is that they understand the reasons for decisions as well as your senior members. Jr members should be cautious operators who are a little uncertain about their own knowledge of the system – this promotes bringing questions to the group.

Allowing experimentation and allowing Jr. members of the team to take ownership of decisions with guidance from the team cultivates more experience in those members and provides more consistently positive results. It’s also easier when you are hiring someone who may not have all the technical experience you need, but who has the aptitude to learn & has the right mindset (read: cautious, humble, communicative). Guided by a small number of senior team members who are watching out for pitfalls, but open to experimentation, and leveraging the power of collaboration and consensus you should get pretty good results & cultivate those Jr. guys into more senior members who can start to contribute more heavily over time.

This also makes it easier to cross-train and spread knowledge about how things work. Everyone needs to understand things to contribute to the conversations & leveraging things like rotating duties helps keep skills and knowledge sharp. Each team member will contribute something different when they work with a particular part of the system so you should develop a well rounded system where everyone has contributed in some way. You still have those Senior members guiding things – but they only help keep the boat upright and away from shore – they don’t dictate how every step of navigation occurs.

A magic side effect – trusting the data

One of the interesting things that happen when you work as a group to come to a decision is that you value one persons opinion less and you begin to put more weight on data. Individuals still have intuition and the value of that cannot be understated, but data should guide decisions whenever possible. When you have one person calling the shots it’s difficult to know if they are making decisions based on data or based on intuition which makes it difficult to question the decision. When a group makes the decision they are more likely to lean on the data to make a decision – this promotes testing and process that allows the entire group to become comfortable with the decision instead of just having trust in a person. If you don’t have the data to make a decision then you weigh your options honestly & either gather more data or make a gut decision… as a group.

When you have no choice

There is a time when organizations are small where you are better off with one or two really strong team members than with the above approach. You need to move fast and the value of consensus is lower than the value of just getting things done. This creates debt over time (technical & non-technical) but it’s a necessary evil in many cases. You need someone who can do a lot of different things and you don’t care about having a whole ops team who gets consensus as long as the business keeps running & features keep releasing.

This is ok for a while, but it puts the business at risk. When you rely on the performance of a small number of individuals then you are heavily impacted when those individuals don’t provide any longer – either because they quit, become disinterested, or move to another role. Unless you start building a culture of consensus & make it easier for Jr members of a team to contribute (which forces things like documentation, automation, standards) you will be at the mercy of your Systems Engineers. That’s not a place you want to be for long.

How do you shift from that small group to a collaborative team?

This is difficult, but often the writing is on the wall when it’s time to do something. You’ll probably start to feel like one person holds a bit too much knowledge about your system and that one person walking out the door would be a big problem. One good way I’ve found to help improve this situation is to bring in someone Jr. with the right attitude who is eager to learn, wants to make things easier for someone else, and is patient with someone who is used to owning everything. Start to try to ask the question “How would someone coming in here know how this is done?” and it’ll start exposing those areas where documentation & automation aren’t there.

As you start working on documentation & automation, and as new changes come along, turn those decisions into group discussions – make sure your Jr team members understand what is being done and why and encourage questions from them. They’re going to ask some naive questions, but they’ll probably also point out some different approaches that might work. Building a culture that supports those discussions is a big part of making this all better. You are going to have passionate discussions about topics people feel strongly about and this is OK as long as they are managed. When people care a lot about decisions it becomes difficult to separate emotion from the decision making process – this is where good leadership comes in to help guide the discussion.

And sometimes individuals aren’t happy with this approach. As much as it is not ideal, some folks do not want to do this. If you are committed to shifting an existing team then you may have to make some hard decisions in this case. Make those hard decisions – you will very likely be better off.

What does the end result look like?

It looks pretty awesome. When a problem arises, not only does everyone feel comfortable that they will be able to contribute but they promote the process so that they can get input from everyone else. After doing this for a while it becomes clear that better decisions come from the group. Most everyone can explain why a decision was made, and they hold each other accountable for maintaining a consistent process & outcome (design, testing, documentation, etc). There will be compromises of course, sometimes individuals aren’t sure the outcome was appropriate – but moving forward requires support from everyone. There are a variety of collaboration tools to make it easier to work through these conflicts and having some tools available is a huge help. I would advise anyone who is going to try to shift an existing team to take some courses on meeting facilitation & collaboration.

I wish I could give you the silver bullet to getting here, I don’t know what that is. It’s a combination of good hiring, consistent feedback that consensus is required, and open communication by all team members. Keeping a friendly atmosphere that still allows passionate discussion about what people believe is right is a difficult thing to foster – but it’s important.