Building a resilient data team

  • Team management
  • Data engineering
  • Process improvement
  • Onboarding
  • Expanded the team from 3 to 7 members to manage the increased data handling responsibilities
  • Created a cross-training framework to ensure all core functions were known by at least three team members
  • Maintained transparent progress tracking and weekly check-ins to guide onboarding and skill development

Managing a tenfold increase in data scope through strategic team expansion and cross-training

Context

My team’s client portfolio expanded from handling EHR data for 5 healthcare clients to also handling about 50 healthcare systems’ purchase order data. My data team initially consisted of an ETL-focused data engineer, and two analysts who worked on classification and data product development. We knew we’d need additional support.

My role

Given the rapid scope expansion of my team, I needed to find the right people to help us handle this growth quickly. And I needed to make sure we managed this change without any major gaps in service for our clients.

Growing up in the cave-rich, karst topography of the driftless region, I learned the spelunking rule of bringing three flashlights on any cave tour. Even with full preparedness, if one flashlight breaks, and your second has weak batteries, you should always be prepared with a third so you don’t get lost in the darkness.

As an analogy for a well-staffed team, I believe any core function of a team should be familiar to at least three team members so that if one goes on vacation, and another goes on parental leave, there’s always at least one person who can do the job. As well, multiple people knowing how to work on any core responsibility guarantees oversight for each piece, enables team collaboration, and gives the team a framework to share and expand their skills.

Solution

While recruiting an additional four team members, I worked with my existing team - as well as our partner data teams, our product team, and downstream engineering teams - to identify 17 key functions that we’d need to tackle as a team. These included elements such as managing ETL for ERP data pipelines, managing a particular classification model, and maintaining certain data quality scripts.

I created a spreadsheet with each of the items as a row, added team members names (as they were hired) as columns, and made this openly available to all our partner teams. Each week I’d check in with the growing data team to see who’d learned what, and added an X next to each component underneath the learner’s name, so we could see which items still needed people trained to reach the 3-flashlights mark. This directly guided onboarding tasks towards critical components of our systems.

Outcome

Ultimately, we were able to reach our goal and had successful and transparent cross-training that made the team much more resilient and able to handle our workload. As well, as each member of the team learned new processes, they were able to freshly and critically identify opportunities to improve and strengthen those elements of our data pipelines.

Insights

We were lucky in that I had budget approval to hire multiple new team members. By having conversations with my team early on, this list of critical team responsibilities helped us hone the hiring process to find people open to learning those skills in such an environment, and it greatly helped me give direction in what each new team member needed to learn during onboarding.

While it’s important to not have silos on the team - people who are the only holders of key knowledge - the onboarding / cross-training process can take a lot from those individuals while they try to ramp up their new teammates. It’s important to support them during this time by shifting other responsibilities off their plate as needed, and by communicating any expected short-term impacts to key stakeholders.

Many projects that have been supported by only one or two people for a while might not be in a state to be shared with new people, or to be maintained by a larger group. It’s important to set expectations that while potential improvements might be identified during the cross-training processes, team members should be constructive and not critical of the current state. It’s best to learn something well before trying to change it.

What I would do differently next time

I was aware that my team might have to greatly increase the scope of our responsibilities months before we actually took these on. The onboarding and cross-training needed to go fast, but I think we would have had more flexibility if the shift in responsibilities was more clear earlier on and if we had more runway to ramp up.

There are a bunch of processes that we were able to speed up and make more resilient by this change; but because we were rapidly in “fix the plane while it’s flying” mode, we didn’t track data for these beforehand, making it harder to quantify the impact of this process. In future such projects, I would spend a bit of effort at the outset to make sure we monitoring the impacts of our work.