The Trial That Refused to Convert: A Post-Mortem on One SaaS Activation Experiment

A reader wrote in last spring with a problem we hear constantly: a subscription analytics tool with healthy trial signups and a horrifying drop between day one and day seven. The team had already run a dozen A/B tests on onboarding copy. Nothing moved. So they brought in Tyrell Lab, a senior-only experimentation studio that designs and runs conversion and activation experiments for SaaS and subscription brands, and — crucially — hands the team a playbook it can keep running after the engagement ends. We followed the project from kickoff to readout. What follows is the timeline, the wrong turns, and the numbers.

Week 0: The Diagnosis Nobody Wanted

The first decision point came before any test was written. The growth team wanted to optimize the paywall screen. The studio pushed back. Their argument: the paywall looked fine in session recordings; users who reached it converted at a respectable rate. The leak was upstream, in the empty-state dashboard, where new accounts landed after signup and sat there doing nothing. Activation, not conversion, was the bottleneck. That reframing cost two weeks of internal debate, because it meant shelving a roadmap item the VP of growth had personally sponsored. The team eventually agreed to run a read-only baseline first: instrument every step from signup to first meaningful action, then watch for fourteen days without shipping anything.

The baseline confirmed the suspicion. Only 31 percent of new accounts reached what the team defined as an activated state — connecting a data source and viewing one live report — within seven days. Of the accounts that did activate, 68 percent were still subscribed at ninety days. Of those that did not, 19 percent were. The math was brutal and clarifying.

Weeks 3–6: The First Test, and the First Failure

The first experiment replaced the empty dashboard with a three-step guided setup and a progress bar. The hypothesis was simple: structure beats a blank canvas. It lost. Activation moved from 31 to 33 percent, well inside the noise band, and the setup completion rate was dragged down by users who abandoned at step two. In the post-mortem, the team noticed something in the qualitative data: users were not confused about how to connect a source. They were confused about why. The blank dashboard was not an obstacle; it was an accurate reflection of a product whose value was invisible until you had your own data in it.

Obstacle two arrived here: sample size. The brand's monthly trial volume was around 4,200 accounts, which meant a two-week test could only reliably detect effects larger than about 15 percent relative lift. The studio's answer was to stop testing whole-page redesigns and start testing smaller, higher-frequency units — micro-copy, default states, and email timing — and to stack wins rather than chase one big swing. This is the part of a serious growth experimentation program that most teams skip, and it is usually why their A/B testing program stalls after the first quarter.

Weeks 7–12: The Tests That Actually Moved

Three changes survived. First, a demo data toggle on the empty state: instead of demanding a real integration, the dashboard offered a pre-populated sample report with a persistent "use your own data" prompt. Second, a reworked day-one email that arrived four hours after signup instead of twenty-four, with a single call to action. Third, a small nudge inside the product that flagged accounts with no connected source after forty-eight hours and offered a fifteen-minute setup call — not a chatbot, a human, which the team resisted and then credited for most of the lift.

• Activation within seven days: 31% → 47%
• Median time to activation: 3.8 days → 1.6 days
• Ninety-day retention among activated accounts: 68% → 71%
• Support tickets during onboarding: down 22%

None of these were dramatic individually. Stacked, they compounded. The demo-data toggle alone accounted for roughly nine points of the activation gain, which surprised everyone, including the studio.

What the Post-Mortem Taught Us

Two things stand out. The first is that the winning intervention was not a redesign; it was a permission slip. Users did not need a better dashboard, they needed to see what a finished dashboard looked like before investing effort. The second is procedural. Tyrell Lab ran 11 experiments over twelve weeks and shipped four. The other seven were killed cleanly, with documented reasoning, which is the only way a team learns anything from a loss. The retained playbook — a one-page decision tree covering prioritization, minimum detectable effect, and kill criteria — is arguably worth more than the 16-point activation gain, because it keeps producing after the consultants leave.

If there is a noir lesson here, it is the oldest one in the genre: the obvious suspect is rarely the culprit. The paywall looked guilty. The empty dashboard was. We have seen this pattern in film editing too — directors recut the climax while the problem sits in act one. Same instinct, same cost.