Does this pilot justify a wider rollout?
Apply the agreed criteria and keep the claim within the tested workflow.
The question
Northline tests SignalWorks on 20 labelled cases from one regional workflow. The agreed criteria are at least 95% correct routing and a p95 response time no higher than two seconds. P95 uses the nearest rank: ceil(0.95 × number of observations). All results are invented.
| Cases | |
|---|---|
| Tested | 20 |
| Correctly routed | 19 |
Judge correctness and response time separately.
| First | Second | |
|---|---|---|
| Cases 1–2 | 0.9 | 1 |
| Cases 3–4 | 1.1 | 1.1 |
| Cases 5–6 | 1.2 | 1.2 |
| Cases 7–8 | 1.3 | 1.3 |
| Cases 9–10 | 1.4 | 1.4 |
| Cases 11–12 | 1.4 | 1.5 |
| Cases 13–14 | 1.5 | 1.6 |
| Cases 15–16 | 1.6 | 1.7 |
| Cases 17–18 | 1.7 | 1.8 |
| Cases 19–20 | 1.8 | 4 |
The rows continue one sorted list of 20 observations. Read left to right, then down.
“Does this pilot justify a wider rollout?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Which criteria decide the result?
What would you do here? Open the move you’d make.
The agreed routing and p95 criteria.
The strong move. These are the success measures Northline accepted before the test.
Only the best response time.
Tempting, but it misses something. That ignores correctness and substitutes a different latency measure.
Whether the demonstration looked polished.
Tempting, but it misses something. Presentation quality cannot replace test evidence.
See how a strong candidate says it
“Use the agreed correctness threshold and response-time threshold, for this tested workflow.”
Why it works: Do not replace the acceptance criteria with audience enthusiasm or a better-looking metric.
Step 2: Does the initial test pass?
What would you do here? Open the move you’d make.
No, because the maximum latency is four seconds.
Tempting, but it misses something. The customer specified p95; changing it now changes the criterion.
Yes: 95% correct and p95 of 1.8 seconds.
The strong move. Both calculations match the agreed definitions.
Yes: all 20 cases were correct.
Tempting, but it misses something. One case was routed incorrectly.
See how a strong candidate says it
“19/20 = 95% correct. The p95 rank is ceil(0.95 × 20) = 19, whose value is 1.8 seconds. Both criteria pass.”
Why it works: The maximum is four seconds, but the stated criterion is p95, not maximum latency.
Step 3: How far can the claim extend?
What would you do here? Open the move you’d make.
Every Northline workflow is ready for production.
Tempting, but it misses something. The pilot tested only one regional workflow.
This test passed; define the next representative evaluation.
The strong move. This preserves the result while stating its scope.
Ignore the result because no small pilot can be useful.
Tempting, but it misses something. A bounded result can still inform the next decision.
See how a strong candidate says it
“Report a pass for these 20 cases and this workflow. Agree further tests before extending the claim to other regions.”
Why it works: A narrow sample does not establish universal fit or a stable performance guarantee.
Step 4: What does the new region show?
Northline now supplies 20 labelled cases from a second regional workflow. SignalWorks routes 14 correctly and records a p95 of 1.9 seconds. The acceptance thresholds are unchanged.
What would you do here? Open the move you’d make.
Approve the wider rollout because p95 is still under two seconds.
Tempting, but it misses something. The routing requirement also has to pass.
Report the regional failure and investigate before expanding.
The strong move. This states the failed criterion and limits the rollout claim.
Show only the initial region because it passed.
Tempting, but it misses something. That omits relevant new evidence about the proposed rollout.
See how a strong candidate says it
“14/20 = 70% correct, below the 95% requirement. Its p95 passes, but routing does not. Hold the wider rollout claim and investigate the regional gap.”
Why it works: Do not hide a failed workflow inside a pooled headline or let one passing criterion cancel another.
The answer, in short
- Decision
- Do not approve a wider rollout from these results.
- Because
- The second regional workflow fails the agreed routing requirement even though its p95 passes.
- What would change it
- A representative retest meeting both criteria after the gap is understood and addressed.
Common traps
- Changing the success measure after the result.
- Extending one workflow’s pass to every region.
Check your answer
- What was agreed before testing?
- Did I use the defined percentile?
- Which workflow does this result cover?
- Did every required criterion pass?
Last reviewed Nov 10, 2026