AI Adoption: What to Measure First
In the previous post, we looked at how DORA recommends measuring the ROI of AI adoption.
But what if you don't know the revenue generated by your features or other business metrics? Yet you're still expected to show the effectiveness of AI adoption.
Let's bring the DORA approach down to the engineering team level.
I would split AI adoption into two phases: AI adoption itself and getting business benefits. These phases have different goals, metrics and outcomes.
It doesn't make much sense to measure business impact of AI until the team has actually gone through the adoption phase.
What to measure to understand the AI adoption state:
🔸 Active users. How many engineers actually use AI in their daily work?
🔸 Monthly usage per user. How actively do engineers use AI tools? Token consumption or API cost per engineer can be a good indicator. AI Gateway solutions such as LiteLLM can help collect these metrics.
🔸 AI-generated code ratio. What percentage of code is generated by AI? Important note: this metric measures only how widely AI is being used. It says nothing about whether AI is being used effectively or producing good results.
🔸 AI-assisted feature ratio. How many features are developed with AI assistance? The metric counts if AI was used in design, implementation, testing, documentation, or code review.
🔸 Engineering readiness. Is the engineering ecosystem ready for agents? Here I would use criteria similar to an Agent Readiness Framework: documentation, AGENTS.md, test coverage, CI stability, guardrails, and so on.
Giving developers AI licenses does not mean AI has been adopted.
Not after one month. Not after three. Not after five.
AI adoption happens at the team level. The goal of all these metrics is to understand whether the teams actually change the way they work. And it requires training people, overcoming resistance, changing engineering practices, and adapting the development process.
In the next post, I'll share my approach to measuring AI impact during the second phase using only tools that almost every engineering team already has.
#ai #engineering #ai4sdlc
In the previous post, we looked at how DORA recommends measuring the ROI of AI adoption.
But what if you don't know the revenue generated by your features or other business metrics? Yet you're still expected to show the effectiveness of AI adoption.
Let's bring the DORA approach down to the engineering team level.
I would split AI adoption into two phases: AI adoption itself and getting business benefits. These phases have different goals, metrics and outcomes.
It doesn't make much sense to measure business impact of AI until the team has actually gone through the adoption phase.
What to measure to understand the AI adoption state:
🔸 Active users. How many engineers actually use AI in their daily work?
🔸 Monthly usage per user. How actively do engineers use AI tools? Token consumption or API cost per engineer can be a good indicator. AI Gateway solutions such as LiteLLM can help collect these metrics.
🔸 AI-generated code ratio. What percentage of code is generated by AI? Important note: this metric measures only how widely AI is being used. It says nothing about whether AI is being used effectively or producing good results.
🔸 AI-assisted feature ratio. How many features are developed with AI assistance? The metric counts if AI was used in design, implementation, testing, documentation, or code review.
🔸 Engineering readiness. Is the engineering ecosystem ready for agents? Here I would use criteria similar to an Agent Readiness Framework: documentation, AGENTS.md, test coverage, CI stability, guardrails, and so on.
Giving developers AI licenses does not mean AI has been adopted.
Not after one month. Not after three. Not after five.
AI adoption happens at the team level. The goal of all these metrics is to understand whether the teams actually change the way they work. And it requires training people, overcoming resistance, changing engineering practices, and adapting the development process.
In the next post, I'll share my approach to measuring AI impact during the second phase using only tools that almost every engineering team already has.
#ai #engineering #ai4sdlc