Measuring what a redesign actually changed

Canadian Institute for Health Information, 2026

CIHI redesigned their public data tool that health system advisors, journalists, and patient advocates used to find pre-analyzed healthcare data across 13 Canadian provinces and territories. They migrated it to their main site, rebuilt the interface, and rearchitected the content structure.

The question was simple: did it actually work?

This is critical for CIHI's leadership; the measurement is meant to justify the modernization investment across the organization's highest-priority program.

How I set it up

In 2021, I ran foundational discovery research to understand key usability obstacles impacting users' experiences of the legacy product. 15 user interviews and 10 stakeholder interviews shaped the initial target model — the design vision for the redesign. A key insight was that future tools needed to better support the contextual information describing the data displayed in the tool: "How is it calculated? How are variables defined?"

Years of work went into the redesign. Now, I needed to measure whether the implementation matched that vision. I ran two parallel unmoderated usability tests in Loop11: one for the legacy product, one for the redesign. Each had an identical protocol. 8 tasks built around the validated user needs from Discovery. I measured and compared task success, abandonment, time on task, and ease ratings.

The catch: CIHI had already retired the legacy product, so UAT was the only option. The study required us to recruit internal staff with data analyst expertise as proxy users. Despite not having a 1-to-1 comparison in end-users' mental models, the findings could point to universal issues that would be issues for anyone, such as hidden information and restrictive task flows.

What I found

The redesign showed that overall task success increased by 20%. On methodology-finding tasks, it lifted 395%.

Originally, the methodology had lived in a secondary drawer in the legacy product. Users missed it constantly. The redesign gave it dedicated, visible space — this was an integral structural change leading to massive success.

What changed

I presented findings several times to various partners in the redesign team, product owners, and leadership.

The overall improvements were appreciated. They validated the hard work of dedicated initiatives.

Yet, the study also found some tasks scored lower. The problems were tied to known issues that needed to be deprioritized in the backlog. So, poor scores were anticipated, but the study quantified their impact on real users and ultimately moved them higher on the roadmap.

What I carry forward

If the old system is gone, sometimes modernization work means you can't test against production. You work with what you have: UAT environments and proxy users. The constraint becomes a research design point in a tight protocol given what's actually available. You name the boundaries and frame findings so the team can be confident to act on them immediately.

This work shapes how I think about research and strategy. Planning for benchmarking doesn't always happen upfront. This constrains the team's ability to define success metrics early, so future research and strategy can align from the start. That's the difference between research that informs products and research that influences products.