What a Pilot Can Tell You, and What It Cannot

The most common mistake with a policy pilot is to treat a small, workable trial as proof that a policy will succeed everywhere. That can lead decision-makers to expand a service before they understand who it reaches, what it costs to deliver in different settings, or whether the observed changes came from the policy at all. A pilot is most useful when its claims are deliberately narrower: it can show how a proposed arrangement operates in a defined setting and what questions need a broader answer.

To use a pilot well, start by deciding what uncertainty it is meant to reduce. Then set the boundaries, observe delivery, interpret the evidence within those boundaries, and decide what to test next. This sequence helps a pilot inform public decisions without asking it to carry more weight than it can bear.

Define the uncertainty

Before choosing a site or recruiting participants, write down the decision that the pilot is supposed to inform. A useful prompt is: “If we learn this, what decision could change?” The answer might concern whether a proposed appointment process is manageable, whether eligible residents can understand an application, or whether two agencies can exchange the information needed to provide a service.

Keep the question specific enough that someone could recognise a meaningful answer. “Does this policy work?” bundles together many issues: access, delivery, outcomes, costs and fairness. “Can residents complete the referral process with the help available at the library?” points towards observable evidence, such as where people ask for assistance, which steps they abandon and whether staff can explain what happens next.

Separate a process question from an outcome question. A process question asks how the policy is delivered: who does what, at which point, and where delays or misunderstandings arise. An outcome question asks whether people’s circumstances change, and whether that change is attributable to the policy rather than other influences. A pilot can often inform the first kind of question; it usually has a weaker basis for answering the second at population scale.

Write down the intended claim in one sentence before the trial begins. For example: “In the participating neighbourhood offices, staff can offer the new referral route during ordinary appointments.” That is more defensible than “The referral route will improve access across the region.”

Set the boundaries

A pilot is defined not only by its duration or size, but by the conditions in which it runs. Record the participating locations, the people eligible to use the service, the staff and partner organisations involved, and any extra support that will be available. These details show what the trial actually represents.

Choose a setting that can answer the stated question, rather than one that merely makes the trial convenient. If the uncertainty concerns whether a digital form is understandable to people with limited access to technology, a site where everyone can use a staffed computer room may conceal the difficulty. If the question is whether a local team can manage a new referral duty, a site with unusually high staffing may make delivery look easier than it would elsewhere.

Decide what remains outside the test. A pilot of a new appointment system may not assess the eligibility rules attached to the service, the quality of the service after referral, or the effect on people who never learn that the option exists. Naming these exclusions prevents later readers from assuming that an unexamined part of the policy was covered.

Also record what is deliberately held constant and what is allowed to vary. A scripted welcome message may be the same at each site, while staff arrange appointments in locally familiar ways. Those choices matter: consistent elements help describe the policy being tried, while permitted adaptations may reveal what local teams need to make delivery workable.

Choose useful signals

Match each signal to the uncertainty. For a question about feasibility, useful observations might include whether staff can complete the intended tasks during ordinary working hours, whether participants can understand the next action, and whether the service can be delivered without repeated informal workarounds. These are clues about operation, not proof of wider public benefit.

Use more than one kind of evidence where practical. Administrative records can show whether a referral was logged or an appointment attended. Short interviews with staff and participants can clarify why a step was difficult or what assistance made it understandable. Observing the handover between organisations can expose a gap that neither organisation’s records describe clearly.

Agree in advance what counts as a signal to adapt, continue, pause or stop. For instance, a team might decide that repeated uncertainty about consent requires a change to its explanation before further recruitment. The decision rule need not be numerical: it should be clear enough that people can distinguish an anticipated issue from a concern serious enough to alter the trial.

Record missing information as carefully as completed activity. If a form was not returned, the record alone cannot explain whether the person chose not to respond, could not access the form, or never received it. Keep the difference visible instead of treating every blank as the same kind of evidence.

Protect the interpretation

During delivery, note changes to the plan as they happen: staff turnover, a revised form, a new partner organisation, or a temporary support arrangement. A pilot can still provide useful learning after an adjustment, but readers need to know what changed and when. Otherwise, an apparent result may be attributed to a version of the policy that was not used throughout.

Keep an account of who took part and who did not, using information relevant to the policy question and handled appropriately. People who engage with a pilot may differ from those who are difficult to reach, have less time, need language support, or distrust the institution offering the service. The experience of participants cannot automatically stand in for the experience of everyone who might be eligible.

Separate observation from explanation. “Several participants asked staff to repeat the next step” describes an observation. “The written instructions were unclear” is one possible explanation; others may include a rushed appointment or unfamiliar terminology. Interviews or further observation can help distinguish these possibilities, but a small trial may leave more than one explanation open.

Say plainly what the evidence cannot resolve. A pilot may indicate that a new process can be delivered in one setting while leaving uncertain whether it improves outcomes, reaches people who are not already connected to services, or can be maintained with ordinary resources. Naming these limits makes the result more useful, not less: it tells decision-makers where further evidence is needed.

What can a pilot answer?

Test transferability

Transferability means judging whether learning from one setting is likely to be useful in another. It is not a simple claim that a policy “works elsewhere”. Compare the features that may matter: the mix of participants, staff capacity, local partner roles, available technology, access to transport, and the way people usually reach the service.

Imagine a council pilot offers residents a library-based appointment to complete a housing support referral. Staff in the participating library help people find the right form, explain the referral, and pass it to a council team. The trial may show that the handover is understandable when a librarian is available and the council team replies through its usual channel. If another area has no staffed library service and uses a different referral system, the observed process does not establish that the same arrangement will transfer. The next useful test could examine an alternative access point and how that team receives referrals.

Distinguish core elements from local adaptations. A core element might be that participants receive a clear explanation of what information is shared; an adaptation might be whether that explanation is given at a library, a community centre or by phone. If the core purpose is preserved while the delivery setting changes, the policy may remain recognisable, but the new version still needs to be checked in its own context.

Where transferability matters, list the conditions that seem necessary for delivery and identify which ones differ in the proposed next setting. Ask local staff and community organisations to challenge assumptions, especially where the pilot’s participants or service partners are not representative of the wider area. A difference is not automatically a barrier, but it should become a question to investigate rather than disappear into a general claim of success.

Make the next decision

At the end of the trial, organise findings by decision rather than by a simple verdict. A practical record can distinguish:

• Continue: the process operated as intended for the question being tested.
• Adapt: a specific feature caused confusion, delay or avoidable work.
• Investigate: evidence was missing or competing explanations remain plausible.
• Stop: a serious problem makes further use inappropriate in the current form.

These categories are not a scorecard. A service might be feasible but inaccessible to some intended participants; another might be well received but depend on extra staff support that ordinary delivery cannot provide. State both sides, and name the issue that should shape the next decision.

If the pilot supports another trial, make that trial answer a new or unresolved question. That could mean including settings with different delivery conditions, examining people who did not take up the offer, or comparing outcomes with a suitable alternative. If decision-makers are considering a broader expansion, they may need a fuller evaluation that can assess reach, effects, costs and unequal impacts; the right approach depends on the policy and available expertise.

For public policy evaluation, the most useful pilot report separates what happened from what it might mean. It describes the setting, the version of the service, the evidence gathered, the people missing from that evidence, and the questions still open. That gives decision-makers a sound basis for choosing what to change or test next, without mistaking a promising local process for a settled case for expansion.