Capability Checks
Part of Checking agency experience
Evaluating case studies without accepting every result claim
Examine an agency case study’s role, measurement and result claims without treating a past outcome as a forecast.
Use an agency case study to find out what the agency did, how it made decisions and how the result was measured. Treat the reported outcome as a claim to examine, not a forecast for your business. The example is most useful when its work and constraints resemble yours.
Establish what the agency owned
Ask what the client wanted, what the agency was hired to do and what the client or another supplier delivered. A website case study may combine strategy, design, development, content and a client-led campaign. Credit for the finished project should not silently become credit for every part of it.
Ask when the work happened and who made the difficult decisions. For a campaign, was the agency responsible for planning, media buying, creative, measurement or only some of these? A clear account of a narrower role is more useful than a broad claim the team cannot explain.
Compare the example's systems, approval process, client resources, audience and project stage with your own. Similar industries do not make different assignments equivalent.
Examine each result claim
For a headline figure, ask:
- What was countedvisits, enquiries, accepted leads, orders or revenue?
- What were the start and end dates, and why was that comparison period used?
- Is the figure a total, a rate or a percentage change? What is the starting value or denominator?
- Which accounts, locations or products were included?
- Did pricing, stock, the offer, the website or other marketing change at the same time?
- Can the agency explain the measurement method or show a suitably redacted report?
A platform label may need translation into a business measure. Ask the agency to define what it counted and how it was recorded.
If the event is a form submission, the count alone does not establish how many enquiries were qualified. Attribution, even when reported, does not by itself establish the effect of the whole agency project.
The agency may be unable to share a customer's underlying data. You can still ask for the definition, method and material limits of the claim without requesting confidential records.
Separate observation from explanation
Suppose an agency reports more enquiries after new pages went live. That can be an accurate observation. To assess its explanation, ask what else changed and what connects the pages to the enquiries.
To assess relevance, ask whether your business has a similar starting problem and resources. The example does not become a forecast merely because the reported change is real.
A precise-looking figure without a date, definition or starting point needs clarification. So does a broad headline supported only by a narrower result in the body.
Australian guidance on commercial claims addresses false or misleading claims, and the ACCC can require businesses to back up claims they make about their products or services. That guidance does not establish that a particular case study is misleading.
Australian regulatory context for claims in case studies
- Regulator
- Australian Competition and Consumer Commission (ACCC)
- Legal basis
- Consumer Data Right and Australian Consumer Law
- Requirement
- Businesses must be able to substantiate claims about products or services
Decide what the study contributes
Record the work demonstrated, the reported outcome with its limits and what remains unverified.
A case study may show sound handling of a difficult constraint even when the agency's contribution to the result cannot be isolated.
If the example is too unlike your assignment, ask for another one. Judge the decisions and delivery you can examine, rather than accumulating headline figures.
