Jenny Park Back to work

AI Growth Operations B2B SaaS

AI Growth Operations Agent for Paid Acquisition and Creative Testing

Built a recommend-only growth operations system for a simulated B2B SaaS company, connecting AI-assisted creative generation, experiment governance, paid acquisition analytics, retention economics, and auditable budget recommendations.

Focus Creative Ops, Acquisition Quality, LTV:CAC
Role Growth Analyst & AI Workflow Builder
Methods CPAO, Cohort Retention, Z-Test, Fatigue Detection, Policy Engine
Tools Python, DuckDB, Streamlit, Plotly, pytest, LLM Mock Provider
$143 TikTok cost per activated owner, outperforming META's $666 CPAO in synthetic data
17.8% Creatives flagged for fatigue after a greater than 25% CTR drop from peak
2.03x TikTok LTV:CAC versus META's 0.48x, triggering a scale candidate recommendation

The Problem

Cheap Trials Were Masking Expensive, Low-Quality Acquisition.

Wrong Optimization Metric: Trial CAC rewarded cheap signups, even if those owners never launched a campaign. The team needed CPAO: cost per owner who activated by launching a marketing campaign within 14 days.

Creative Fatigue Without a System: Performance was decaying after early creative wins, but there was no consistent rule for separating a weak channel from a tired asset.

AI Without Governance Risked Becoming Noise: A useful agent needed to generate briefs and explanations, but budget recommendations had to remain deterministic, auditable, and human-approved.

Tablr is a fictional B2B SaaS product for independent restaurant owners. The project simulates 12 weeks of paid media, product activation, subscription, and revenue data to test whether a growth team can use AI to move faster without giving up measurement discipline.

Measurement Strategy

CPAO Reframed the Channel Decision.

Trial CAC was treated as context, not the decision metric. The primary metric became Cost Per Activated Owner, defined as spend divided by owners who launched their first Tablr campaign within 14 days of trial signup. That behavioral milestone is much closer to future subscription value than a top-of-funnel signup.

The META Reversal Story
Trial CAC hides downstream quality
Trial CAC, lower is better
TikTok
$12
Best
Search
$21
Solid
META
$52
Costly
CPAO, lower is better
TikTok
$143
Scale
Search
$248
Observe
META
$666
Hold
Key Insight
META was not just more expensive at the trial stage. Its CPAO was 4.7x TikTok's, which means it acquired restaurant owners who were harder to activate. Optimizing on trial CAC alone would have kept spend flowing toward the wrong channel.

Acquisition Quality

The Decision Engine Ranked Channels by Activated Value, Not Click Volume.

Channel Trial CAC CPAO LTV:CAC Recommendation
TikTok $12 $143 2.03x Scale Candidate
Google Search $21 $248 0.96x Observe
META $52 $666 0.48x Hold
LinkedIn No data No data No data Insufficient Data

At a $150K monthly budget scenario, the allocator recommended increasing TikTok to 55.8% of spend while holding META to 12.0%. The system did not make the budget change. It produced a recommendation package with evidence, risk, and next-test logic for human review.

Creative Intelligence

AI Generated Testable Briefs, Then Analytics Decided What Deserved Budget.

Sixteen creative briefs were generated across restaurant owner personas, hook types, channels, and funnel stages. Each creative preserved the strategy behind it: persona, pain point, value proposition, opening hook, proof type, variable tested, control ID, and compliance notes.

Hook Type Trial CAC Strategic Role
CONTRAST $216 Before-and-after framing for skeptical independent owners
OUTCOME $234 Revenue result focus for owners already seeking growth
SOCIAL_PROOF $241 Peer restaurant success for evidence-driven operators
DEMONSTRATION $258 Dashboard setup walkthrough for hands-on evaluators
FOMO $271 Competitive urgency for growth-minded owners
PROBLEM_AGITATE $289 Pain-point amplification for cold audiences
Creative Fatigue Detection - 90 Simulated Creatives
17.8%
82.2%
16 creatives flagged after greater than 25% CTR decay
74 creatives monitored without refresh trigger
Operational Rule
Fatigue flags trigger creative refresh recommendations, not automatic budget shifts. The system separates "this asset is tiring" from "this channel no longer works."

Experiment Governance

Every Test Had a Primary Metric, Guardrail, and Decision Rule.

The experiment registry prevented the agent from treating every performance lift as a win. A result had to clear both statistical confidence and practical significance before being promoted, and guardrail violations automatically downgraded scale recommendations to hold.

exp_001 - Running
META Hook Type Test
CONTRAST vs OUTCOME for scrappy independent owners. Tests whether the META problem is creative-solvable or channel-structural.
Primary: activation rate
exp_002 - Running
Messaging Territory
SOCIAL_PROOF vs FOMO for growth-minded owners, with CPAO as a guardrail so a click lift cannot hide poor activation.
Guardrail: CPAO
exp_003 - Running
TikTok Format Test
UGC video vs static creative for new owners. CTR is primary, but activation rate prevents optimization for empty clicks.
Guardrail: activation rate
exp_004 - Evaluating
LinkedIn Channel Test
Job-title targeting vs META lookalike for delivery-heavy owners, designed around CPAO and activation quality.
Primary: CPAO
Decision Gate
A test needed 95% confidence and practical significance against the minimum detectable effect before action. The agent could explain the result, but it could not overrule the policy engine.

Architecture

The LLM Could Explain Decisions, Not Make Them.

1
Analytics Engine
Python + DuckDB + Parquet
Computes channel scorecards, creative diagnostics, cohort retention, LTV:CAC, and budget scenarios from reproducible synthetic data.
2
Policy Engine
Deterministic Rules
Sets recommendation states using explicit thresholds: LTV:CAC, M6 maturity, retention guardrails, trial floors, and missing-data checks.
3
LLM Analyst Layer
Narrative Only
Receives the policy state plus evidence dictionary, then writes rationale, tradeoffs, risks, and next experiment hypotheses.
4
Governance Layer
Recommend Only
Writes auditable recommendations and disabled review actions. No ad platform mutation, no budget change, no campaign pause.
Safety Constraint
RECOMMEND_ONLY=True is enforced throughout the system. The output is a decision packet for a human operator, not an autonomous media-buying agent.

Impact

What the System Made Visible

7.6%
Attribution gap surfaced
152 accounts were marked UNKNOWN, keeping data quality visible instead of burying it in blended metrics.
55.8%
Recommended TikTok spend share
Budget moved toward the channel with stronger activated-owner economics in the $150K scenario.
4
Governed experiments
Each test defined a variable, primary metric, guardrail, evidence threshold, and decision rule.

The most important output was not an AI-written ad. It was a repeatable operating model: generate strategically structured creative, measure quality past the click, detect fatigue, apply deterministic policy rules, and ask a human to approve the next move.

Limitations

The Simulation Is Honest About What It Cannot Prove.

  • Synthetic data: All restaurant owners, campaigns, spend, revenue, and outcomes are simulated for portfolio demonstration.
  • Short observation window: The 12-week period is not long enough for mature M6 retention on newer cohorts.
  • No live platform mutations: The project does not call Meta Ads, TikTok, Google Ads, or LinkedIn campaign APIs to spend money or alter campaigns.
  • Mock analyst layer: The LLM-style narrative layer is a controlled provider pattern, not a production deployment with full prompt evaluation.
  • Attribution remains imperfect: UNKNOWN source records are preserved as a visible data quality issue, not forcibly reassigned.

Tech Stack

Tools & Methods

Python DuckDB Parquet pandas Streamlit Plotly pytest Synthetic Data Generation Creative Brief Schema Experiment Registry Two-Proportion Z-Test Creative Fatigue Detection Cohort Retention LTV:CAC Modeling Policy-Based Decisioning Recommend-Only Governance

Source Code

Code & Documentation

Full project documentation, synthetic data generation, Streamlit dashboard code, decisioning logic, and tests are available on GitHub.

View on GitHub

Interested in working together?

Open to Data Analyst, Product Analyst, Growth Analyst, and Revenue Analytics roles in the Bay Area or remote.

Get in touch See more work