Global Navigation Research
PagerDuty · 2020 · Principal User Researcher
Methods: Card sorting · Concept testing · Tree testing · A/B testing ·
Scale: 1.3M+ platform users · 15,000-user A/B test · 37-participant tree test
The Problem
PagerDuty had grown fast. What started as an incident alerting tool had expanded into a multi-product operations platform — incident management, process automation, AIOps, service management — and the navigation hadn’t kept pace. Product teams had bolted on new surfaces over time, each making local decisions about where things should live. By 2020, the result was a navigation structure that made internal sense to the teams who built it but didn’t correspond to how any user actually experienced the product.
The stakes were real. Navigation isn’t cosmetic in an enterprise platform — it’s the thing that tells users what the product believes about them. Get it wrong and users can’t find their work. Get it right and the platform becomes intuitive in a way that drives adoption, conversion, and retention.
Research Approach
I started where the data already was: a full audit of the existing navigation structure, combined with stakeholder interviews with PMs, designers, and engineers across the platform. I treated these internal stakeholders as research participants — not just source material, but people with their own mental models about what the product was and who it was for. That decision mattered. It surfaced something that wasn’t in any analytics dashboard: the organization itself had three competing theories of how the platform should be organized.
From there, I ran open card sorts to understand how users actually grouped the platform’s features, followed by closed card sorts to test three structural hypotheses that had emerged from the audit and stakeholder work: a role-based model (organizing by who you are), a Jobs-to-be-Done model (organizing by what you’re trying to accomplish), and an all-access model (flat, everything visible). Each represented a coherent theory. Each was also incomplete.
After 25 card sorts, I made a call that turned out to be important: I stopped. The quantitative data was showing clustering patterns, but it wasn’t explaining them. Two participants could group the same items identically while reasoning about them in completely different ways. Understanding the pattern wasn’t enough — I needed to understand the logic underneath it. I pivoted to deeper qualitative sessions to probe the reasoning behind the groupings, which is what produced the insight that changed the project’s direction.
The deeper sessions revealed four distinct behavioral archetypes with genuinely incompatible mental models about what the platform was:
- Responders experienced PagerDuty as an alerting and triage tool — they needed fast access to incidents, nothing else.
- Managers used it as an oversight layer — they wanted visibility into team performance and on-call schedules.
- Administrators lived in configuration — they were the ones setting up the platform for everyone else.
- Stakeholders were mostly passive — they checked in, looked at reports, and moved on.
These weren’t just different user types. They had different entry points, different task flows, and different implicit answers to the question “what is this product for?” A single navigation structure couldn’t serve all four without forcing compromises that would fail at least some of them.
This reframed the design problem. The question was no longer which navigation model was right. It was: how do we build a navigation that can serve genuinely different mental models without asking users to adapt to the product’s internal org chart?
The finalist concepts — a JTBD-organized model built around four task categories (Activate, Prepare, Respond, Improve) versus a role-based view that surfaced contextually relevant surfaces — went through concept testing, then tree testing with 37 participants, then a 15,000-user A/B test.
What We Found
The winning structure was role-based, simplifying to three configurable views that filtered the navigation to match what each user type actually needed. The four archetypes didn’t collapse into three — the stakeholder archetype was absorbed into the manager view — but the underlying behavioral framework remained intact and became the more durable output of the project.
The JTBD framework (Activate, Prepare, Respond, Improve) persisted as a shared vocabulary for how Product, Design, and Engineering talked about user goals — not just in navigation decisions but in roadmap planning, feature prioritization, and customer-facing messaging.
Impact
The research influenced $2.3M in additional ARR. Measured outcomes against the previous navigation included:
- 22% improvement in trial-to-paid conversion
- 24% improvement in task completion
- 133% increase in feature discovery
- 47% reduction in time-to-first-value
The archetype framework was nominated for the company’s Leadership Award and became the reference model for new product onboarding, permissions strategy, and multi-quarter planning cycles.
Reflection
The methodological pivot at 25 card sorts is the thing I’d point to if someone asked what the project actually taught me. Quantitative data can tell you what’s happening; it can’t tell you why. I was sitting on a dataset that looked like it could support a structural decision, and the honest answer was that it couldn’t — not without understanding the reasoning underneath it. Stopping, pivoting, and spending more time in qualitative work added time to the project and produced a better outcome.
The other thing: treating internal stakeholders as research participants, not just gatekeepers, was how we discovered that the organization’s competing theories about the product were the root cause of the navigation problem. The users were confused because the teams who built the platform were, at some level, working from different assumptions about who the product was for. That’s not a usability problem. That’s an alignment problem, and it required a different kind of research response.