Nine questions to ask about any case study
Run every case study, including ours, through this list:
- Is the client named, or is it "a leading home services brand"? Anonymity is sometimes legitimate, but it removes your ability to verify anything
- What was the baseline? A jump from 12 to 60 visits is a 400% increase and commercially meaningless
- Are the numbers traffic or revenue? Impressions and rankings are inputs; leads, booked jobs, and revenue are outcomes
- What was the time window, and does it span the same season as the baseline?
- What else changed? A new site, a new sales hire, or a paid campaign running alongside makes attribution to SEO a guess
- How large was the sample? One client is an anecdote, not a method
- What did not work? Any honest engagement has a failed experiment in it
- Is the traffic branded? Growth driven by brand searches usually reflects other marketing, not the SEO work
- Can you talk to the client? The strongest proof a vendor can offer is a reference call
Survivorship bias is the default
Every agency shows its three best outcomes and none of the accounts that churned at month four. That is not necessarily dishonest, but it means published case studies tell you the ceiling of a vendor's work, never the median.
The useful question is not "what is your best result?" It is "out of your last ten clients in my industry, how many hit their goal, and what happened to the ones that did not?" Vendors who can answer that are working from data. Vendors who deflect are working from hope.
Percentages hide more than they reveal
Percentage gains are the most common way a small absolute number is made to look transformational. Ask for the raw before and after: sessions, qualified leads, closed deals, and cost per acquisition. If a vendor will show percentages but not absolutes, assume the absolutes are unflattering.
The same applies to keyword counts. "Now ranking for 4,000 keywords" says nothing if those keywords have no volume and no commercial intent.
What a good case study looks like
It reads like a post-mortem, not an advertisement. It states the business situation and the constraint, names the hypothesis, lists the specific work shipped with dates, shows a chart with the intervention marked, reports the business metric, and notes the confounders honestly.
As an example of the standard: on an East Bay HVAC account we brought cost per lead from $87 to $34 over eight months. That figure is a paid-and-organic blended number on a single account, measured against the client's own CRM, in a market with seasonal summer demand that partially flatters the second half of the window. That caveat is what makes the number usable.
Apply the same test to your own reporting
If you are already paying an agency, run their monthly report through these questions. If it leads with impressions, keyword counts, and "work completed" rather than leads and cost per acquisition, you are being reported to rather than being shown results.