Measuring developer productivity IRL: practical tips for platform engineers

“But how will we know?” is Abby Bangser, Principal Engineer at Syntasso and Team Topologies Advocate, favorite question. And she asks it often. How will we know if this tooling is effective? How will we know if our investments are helping the business? When embarking on a journey towards improving the developer experience, you’ll need a way to tell if what you’re doing is having the right impact.

How do you measure that impact? While developer experience has several benefits, the most sought after is improving productivity. The real problem is measuring that productivity in the development context. A stellar panel hosted by renowned tech journalist Jennifer Riggins set out to see how teams like Netflix and Spotify are addressing this issue.

In this post, I want to cover the main ideas that came from the discussion, starting with the nature of working as leaders in an engineering organization and capturing ideas on what to measure and what not to measure when assessing “productivity.”

{{subscribe-form}}

A “captive” audience

Platform teams and their customers have a unique relationship: developers are a captive audience. They are “forced” to accept the tooling and processes you impose on them—although in practice it doesn’t work like that.

This dynamic often leads to platform teams misunderstanding what a good developer experience looks like. They don’t dedicate enough resources to discover what developers actually need. It’s not uncommon to see platforms that set out to implement things that nobody ends up using even though it seemed like a good idea.

Developers are clever. They will jump over your golden paths and best practices if they’re not good for them and their goals. Thus, despite your best efforts, fragmentation keeps spreading through the organization, resulting in harder to maintain and scale software components.

Needs vs wants

Organizations tend to give their developers autonomy paired with accountability, but when fragmentation grows too much, most platform efforts are neither scalable nor cost effective. You’ll need to keep an eye on how far your customers are diverging form the expected path and come up with ideas to bring them back on track.

How do your bring back product teams who’ve gone astray? Discover features they can get “for free” when using your platform. You need to talk with them and see what they need, which is not necessarily what they want.

As Laura Tacho, CTO at DX, has witnessed in many engineering teams: the perceived frustration with a tool or process is not necessarily associated with the impact fixing it would do to make developers more productive. To differentiate wants from needs, you need to stay in the loop with your team: trying things, measuring, and adjusting quickly.

Measure what matters

It is tempting to measure what’s easiest to measure. Managers often focus on statistics around commits, pull requests, and timelines. However, focusing on activity can be counterproductive. Measuring what truly matters is key, but it’s usually more involved.

But when does something "matter"? Productivity can be very subjective. Kathryn Koehler, Director of Developer Productivity at Netflix, explains that the meaning of productivity depends on whether you want to prioritize acceleration or simplification. If you want to move fast, you’ll need to accept tech debt. If you want to refine your system, you’ll ship fewer features.

Understanding this distinction helps clarify what "productivity" means, enabling you to identify the metrics that truly matter.

‍

What to ask and how

Talking to your team is the first step towards improving any developer experience. How you do this effectively varies widely from one team to another. Interviewing members of your teams on their pain points can provide a lot of quantitative data. But due to scale constraints, surveys are the most common way of asking developers about what they think of their toolsets and workflows.

Helen Greul, Head of Engineering for Backstage at Spotify, shared how their developer surveys have an astonishing 90% response rate. But for most organizations, a 30% is already a high response rate.

When it comes to the questions asked on the survey, the panel experts explained that asking about sentiment and team priorities provides useful insights.

If you leave Netflix, would you be sad to leave our tooling?

In questionnaires, it’s generally advisable to avoid asking hypothetical questions and focus on the experience developers have at the moment with their tools.

Conclusion: transformation needs commitment

A question some leaders add at the end of their developer surveys is “Do you think something will meaningfully change as a result of this survey?” Understanding the level of trust developers have in the commitment from leadership is an important indicator of how effective a DX team is.

‍