Good day detective, Aelia here.
Now that you know how to run a marketing experiment (ahem, last issue if you missed it!), it’s time to put your detective hat on and check in on your campaigns. 🔍
Trouble is, you might run into the the sweet, sweet trap of vanity metrics.
this is you when you think your demo request email experiment is doing well because you have have a great open rate but haven’t checked your demo booked rate yet. whoopsiee!
Vanity metrics are the KPIs that give you the illusion that you’re doing well and make you feel good, when the reality may be otherwise.
So what are vanity metrics and how do you steer clear of them so you don’t measure the wrong thing in the first place?
Here is the tricky part: vanity metrics are not the same for every single campaign.
A vanity metric for one experiment might actually be a valuable metric for another!
If you’re going Why, Aelia, must thou hurt me so? in your head, fear not!
For I come bearing the gift of a solution!
Step 1: Define your north star metric(s)
The best way to avoid the trap of vanity metrics is to define a metric that supports the goal of your marketing experiment (it can be more than one if your experiment requires it).
These will be your north star metrics.
How to do this? Simple.
Go back to the goal you set for your marketing experiment. what did you intend to achieve from it?
The KPI that ties directly to your goal will be your north star metric.
Let’s take an example:
you’re running an email experiment to ask for testimonials from recent customers.
your goal is the action you want your audience to take.
in this case, it’s to get responses with testimonials.
and so your north star metric becomes the number of testimonial responses you get.
everything else—open rates, CTR—becomes background noise, i.e., vanity metrics for this marketing experiment.
Here’s a quick visual for some examples of vanity and north star metrics (and why) for some foundation marketing experiments:
See what i’m talkin’ bout?
Step 2: Identify warning signs to create a guardrail
Guardrail: identify what can break during your experiment/what can constitute a loss
When setting up your metrics, you might get so absorbed running after your north star metrics that you forget to identify warning signs that your experiment is failing.
And you don’t notice it until you look at measurable impact and go:
but…i thought the experiment was going well…
For example.
You’re closing deals in a bottom of funnel (read: right before they go ka-ching!) marketing experiment, BUT you’re also leaking leads with high drop-off rates.
Does that make your marketing experiment successful?
Maybe not if those dropoffs are quality leads that left because of a problem at your end. 😬
or, Example 2,
You get good conversion rate from a paid campaign. but the CPA is too high.
Is it worth it? Probably not.
Especially not if the lead quality isn’t great.
Here’s a visual of some guardrail examples for marketing experiments:
Now, this is not to say you need to panic every time you notice something going wrong.
Just last week, we had a sudden drop in subscribers which left Sophia in a panic of “Waiiiiit, what went wrong?”
But a quick check turned out most of those unsubscribers were personal contacts, not leads. (it’s completely okay if your friends sign up for your newsletters but they end up unsubbing sometime after because it’s not their niche!).
Since you might also run into such issues, I’ve created a handy case file so you can panic only when you need to.
Marketing Experiment Case File
This is what your Marketing Experiment Case File looks like (currently, it’s prefilled with a sample experiment, but you can edit it out when you copy the template):
Now here’s how to use it
Step One: Basic Data
Drop in your basic details:
Experiment name
Start date
End date
North star metric
Target (your goal for your north star)
Guardrails 1,2,3 (drop your target threshold you need to meet here)
Step Two: North Star Check
When your experiment is finished, or when it’s running and you want to do a mid-run check to monitor, check the results of your north star metric. if they are in line with your target, you can move on to the next check for your guardrails, but if the results aren’t great, you might want to stop and investigate here.
Step Three: Guardrails Check
Now, to check your guardrails, see the results for each of them and match them with the threshold you set in the Guardrail columns. Choose “good” or “bad” from the dropdowns to list how your metrics are doing here.
When you’re done with the dropdowns, the “Decision” column will automatically update the status for you, and what you need to do.
Here’s what’s in that decision column:
Continue: You may continue running the experiment (if you’re doing the mid-way check) or you can scale it if you’re done.
Pause: You might wanna investigate if something is wrong. For example, if you’re getting too many spam complaints and a high unsubscribe rate, double-check the audience you’re targeting. Is it a cold list or were they opted in? If they were opted in, what did they sign up for and does your content match their interest?
Stop: Yes you can panic now. Your guardrails are warning you that something is definitely wrong. Stop, investigate, and pivot.
Now, take your case file and experiment awayyy!
Hello, Sophia here! 👋🏽
Want to have someone else run your next marketing experiment and do the detective work? ⚗️🕵♀️
I’m taking 2 client slots for July/August to design, build, and run your 4–6 week experiment from scratch.
Let’s chat! Here’s my Cal link: https://cal.com/sophia-o-neal/lets-talk
We’re headed out of office - checks watch right now! - for our our 10-day Agency-wide summer break. We’ll see you and your inbox in two weeks (July 17th)!
Cheers!
Powered by the joy of crocheting a toast-and-egg and blooming sunflowers from Sophia’s kitchen window.
Aelia ⚡🧕 and Sophia 💜👩🏽💻
