Bernard: In this episode...
Jacob: when does, ML based
signal engineering make sense?
Shumel Lais: A general rule that we've
seen is like, you wanna get at least
10 conversions per campaign per day.
just be careful about hiding behind
blended metrics, blended cac, roas.
Sometimes it can be quite
uncomfortable as well.
But again.
In, in today's day and age, I think
we can really focus on direct cac.
Jacob: Hey, Humel, I'm super excited
to have you here today on the podcast.
Shumel Lais: Thanks for having me.
Jacob: We're gonna dive
into mobile measurement.
We're gonna talk about a lot of things
on single engineering, but first, can
you tell me a little bit about how you
got started in the mobile growth world?
Shumel Lais: Yeah, I've been, um,
I've been in the space for quite
a while now, um, since about 2010.
Uh, started my career, uh, agency
side, at an agency called Fetch.
Uh, I think back then there was
only a couple of mobile agencies.
This was.
Pre MMP, free Facebook,
having ads on mobile.
so yeah, started, started, started there.
Jacob: Cool.
Yeah.
Very early days, uh, before, before
kind of the Facebook side, which,
you know, dominates everything today.
So, wanna talk about mobile
measurement maturity.
And how you evolve to, to
more advanced strategies.
Um, and a super popular topic,
which you are very well suited to
speak about is Signal engineering.
Uh, and I know day 30 is built around this
idea and, and I was reading something you
wrote, I really liked your description
of, um, choosing conversion events
that better represent user value than
what ad platforms can see on their own.
Um, can you tell me a little bit more
about that, how you landed on that framing
of, of signal engineering and for day 30.
Shumel Lais: Yeah.
So I think, um, the term sig signal
engineering, I, I don't think I can take
credit for coming up with that term.
I think that was, um, Thomas Petit
and, and Eric, they, they've been
kind of posting on the topic.
Um, but the concept of.
Coming up with a predictive signal.
Um, that's kind of what we started
looking into, uh, before that
term kind of became popular.
But in, in essence, it's,
it's the same thing.
so what we, uh, are aware, we can
see these ad platforms like meta,
they're becoming very algorithmic.
Um, so it's very.
Uh, they've kind of taken away a lot
of the controls that marketeers have,
whereas historically, you may be managing
your bids and budgets and targeting
and, and they're really pushing you
into this world of just like, give us
your goal and we'll try and find the
best users to, to reach that goal.
So the topic of signal
engineering is really like, how
do we feed these ad platforms?
The best goal, um, especially in this iOS
world where, uh, there's a limitation on
how far visibility the ad platforms have.
So if, if I take meta as an example.
They're typically, uh, limited to
seven days of, of visibility, terms
of that goal event that you can send.
Um, but what marketers typically care
about, or, or, or apps, subscription
apps in particular is, um, usually
something a lot further down funnel.
So typically, uh, revenue or,
or, or return on ad spend.
And a lot of that doesn't materialize
or outside of this window.
that's really where this topic of signal
ð ð ð Hey, Humel, I'm super excited
to have you here today on the podcast.
Thanks for having me.
ð ð ð Hey, Humel, I'm super excited
to have you here today on the podcast.
Thanks for having me.
ð ð ð ð Hey, Humel,
I'm super excited to have you
here today on the podcast.
Thanks for having me.
. Shumel Lais: Engineering comes in
of how can we, um, find something
that's better, correlates better.
To that longer term, uh, value
that you're after, that the ad
platforms otherwise couldn't see.
So that's really kind of how
we landed, uh, on that topic.
And, and some of it came from, um,
historically from, from, from some past
campaigns going back a few years ago, um,
where I teamed up with a data scientist.
And we ended up, uh, solving
for this problem where there was
just a delay in a user behavior.
Um, and that's got when I got
introduced into the world of
machine learning and data science
and how that could overcome that.
Um, so that was kind of
where the idea stemmed from.
But I think more recently, um, it is been
kind of coin signal engineering, which I
think is, is a pretty good name for it.
Jacob: Yeah.
Got it.
That makes sense.
And so maybe you can, you can
dumb it down for me a bit.
Like, what's, like, can you like a, a
simple example of like what's actually
happening here and how, like, you know,
some app, what an app is actually doing.
Shumel Lais: So I think, um, if, if we
park kind of machine learning and, and
data science for a second and just think
at the most of basic level, um, signal
engineering is really just the, the.
Process of selecting, uh, an event
or creating, uh, a conversion event.
so you always start with what's the goal.
So let's say our goal is a
paying subscriber of an app.
Um, let's assume there's a 30 day file.
That's what we're trying
to bridge the gap to.
So, uh, the status quo, what a
lot of apps would go after is
optimizing to a trial start event.
because for anyone to become this
paying subscriber, they would
go through this trial first.
So that's kind of the starting point.
then signal engineering is,
can we find something better?
So at the most basic level, you may
find other behaviors that correlate.
With someone actually
going on to subscribe.
So if you, uh, let's say a language
learning app, it could be the number
of lessons that you completed.
So could we do, you know,
start trial plus, um, someone
who's completed five lessons?
That would be like a very basic example
of how you could, uh, build, you know, one
output or, or a signal to the ad platform.
So instead of just going for trials,
go for people who do a trial and a
behavior as wellâ ð ð ð Hey,
Humel, I'm super excited to have
you here today on the podcast.
Thanks for having me.
ð ð ð ð Hey, Humel,
I'm super excited to have you
here today on the podcast.
Thanks for having me.
. Jacob: I've also heard, um, you
know, people talking about qualified
trials where trials where the
user didn't cancel immediately
in the first hour or something.
Shumel Lais: Yeah.
And that, and that's another
really good example as well.
Um, and that works when
it's, you're almost like just
purely trial focused as well.
Um, and then there's the question
of how long do you wanna wait?
give for those trials to cancel.
Um, and certain apps you'll see, like
usually you find it with slightly younger
demographic, um, focused, targeted apps
where they're just very familiar with
this ability to cancel immediately.
So you see a big portion of
those cancellations happening.
Straight away.
So that's where it works quite well.
And then you've got some apps that are
maybe have like a slightly longer user
journey, um, and slightly, uh, more mature
users who actually want to test the app.
So you may not get as many of those
cancellations early, but that's
also a very good example of signal
engineering at the most basic level.
Jacob: Do you think like this concept
of signal engineering is become.
Because of the trial, because trial's
essentially a low quality signal that,
because if you are, if people are buying
direct and just purchasing, that's,
that's the highest quality signal.
Right.
And so is it all kind of like this,
trying to get a proxy around this and,
and you know, is it, is this mostly
driven from that subscription model,
but then also like platform limitations?
Is that, how is that, do you think
that's kind of the, the genesis.
Shumel Lais: I, I think it's, um,
definitely driven by the, this kind of
structural issue with attribution and,
and, and the platform's visibility.
Um, and I don't think it's just
limited to subscription apps.
If anything.
Actually, I think this has been around
a lot longer in the gaming space.
Um, and in gaming there's a lot
more variabilityâ ð ð ð ð
Hey, Humel, I'm super excited to
have you here today on the podcast.
Thanks for having me.
. So it's not just has someone subscribed
just, or no, it's a case of how much value
do I think they're gonna, uh, be worth?
Or, you know, how many in-app tokens
they're gonna, they're gonna spend.
So there's a lot more to predict.
So for subscriptions, trial
is definitely an aspect.
Um, and I definitely agree, like we also
see people who have a mix of trial, a
non trial or initial purchases as well.
but typically those initial
purchases are shorter.
Uh, so they're typically like
weekly or monthly subscriptions.
And even in that instance, um.
You know, they, they're offering
those because they know over, you
know, three months or four months,
those users are gonna renew.
And so again, there's still an opportunity
for signal engineering there because
there's still this gap of visibility.
So yes, meta could see that initial
purchase and that will probably
be stronger than a start trial,
uh, for users who, who do that.
but there's still a difference between
a user who renews only, you know,
once or doesn't renew a tool versus
one that's still retained in, you
know, six weeks or 12 weeks time.
Jacob: Yeah.
So regardless if it's just trials,
you know, because you're right.
Yeah.
A lot of times if you have a, we.
Weekly plan.
You know, you may not offer a trial,
but you're still trying to, you know,
understand and predict what is the
true LTV of this user based on their
attributes, based on what we see, are
they gonna retain what's their real value?
And then it's also like, um, the, uh,
you know, combination of different
purchase types, upsells, uh, and then
sending the right signal for all of
those where it's like, if you're just
sending on a trial star, do you combine
that with your other purchase signal?
How do you differentiate all that?
And so it gets you
pretty complicated quick.
Shumel Lais: So I think visibility
is like one aspect of it.
Um, so you could argue, you know,
if you've got an app with a three
day trial, um, only, and, and then
they convert and it's only an annual
plan, do you need signal engineering?
You, you know, there's an argument to
say, does it fit within the seven days?
So that's where you have these other
aspects of signal engineering, which is.
It kind of plays to the specific
requirements of the ad platform.
Jacob: Hmm.
Shumel Lais: Uh, so if I take meta
as an example, there's another
variable which is signal volume.
So not just signal accuracy of
predicting feature value, but
also is it getting enough volume.
So sometimes you might find actually
you have visibility into purchases,
um, or there's a really short trial.
But the people who actually
convert are already 20% of
the people who start a trial.
So all of a sudden you lose a lot
of volume and platforms like meta
still need that signal volume.
So you're kind of balancing these
two aspects, which I'd say the,
the key aspects is the volume and
then the, the precision side of it.
Jacob: Got it.
Got it.
Um, that's a, that's a a good segue
'cause in, in your kind of signal, uh,
framework, you talk about these three
dimensions of precision volume and recall.
And, and I, I know you.
In one of your talks, you used, I think a,
a photo storage app example, and I think
you're talking about optimizing towards
heavy uploaders, improved precision a
bit, but hurt recall or killed recall and
excluded a third of a future converters.
So, um, I think that's exactly
what you're talking about.
'cause you're saying then the ml, ML
based signal improved precision while
keeping that, that recall, um, tell
me a little bit more about that and
like how that got builtâ ð ð ð
ð Hey, Humel, I'm super excited to
have you here today on the podcast.
Thanks for having me.
. Shumel Lais: Yeah, so I, I'll
start with what, uh, recall is.
Um, and it's a, a fairly recent concept
to me as well that I've, I've come
across over the past couple of years.
so recall, the best way to think of
it is, um, of everyone who actually
converted or did your goal event, how
many of those did your signal capture?
So if you were purely a, um, you
know, optimizing to the trial event
and everyone has to do a trial.
To become a purchaser, then your
recall for that event would be a
hundred percent because, um, any
start, anyone who converts would've
had to do a start trial first.
So in from that perspective, that's
one of the benefits of a start trial.
Um, when we talk earlier about,
you know, some initial early
levels of signal engineering.
So if, again, if I think of this.
A photo storage app where they
may look at, you know, how
many photos were uploaded.
and if you were to draw a line
and say, okay, the people that
converted on average, they did
more than X number of photos.
And you might draw a line to say, you
know, it's 10,000 or 20,000 photos.
challenge with that is that can
be quite a blunt tool sometimes.
And that's where recall comes in because
there may be, um, users who only download,
you know, uploaded 4,000 photos instead
of 10,000, but they still converted.
And because you have this hard
cutoff, you miss a lot of those users.
Um, and so measuring recall as
a, as a metric, and it's quite a
standard metric in, in data science,
um, it just allows you to quantify
how many people you're missing.
And if you're missing.
Too many.
Then that's when you can say, okay,
maybe this isn't the best signal.
Can I find something that
doesn't exclude too many?
And again, it's a trade off
between, uh, uplifting accuracy.
You may say, actually I don't mind
losing a bit of recall because I
know the people I am sending are
really highly likely to convert.
Jacob: Got it.
So part of it is about maybe
you have some basic rule-based.
Signal engineering.
And then you can use kind of maybe
some analysis or an ML layer to
understand, is this a good signal?
What is the loss here that we're seeing?
And then that can give you an
insight is, um, can we, can
we build a model around this?
Is there, is there a use case or is
it, no, actually this is pretty good.
It's working.
We don't need to do anything else.
Uh, um, but, but you know, that, that,
that's a good kind of, uh, early step
to understand if there's opportunity.
Yeah.
Shumel Lais: For sure, and I would, I
would say like with machine learning, if
you've got enough data, there's always
an opportunity to be a a, a basic rule.
Um, and in essence, the, the way it
works is, um, or in, in this example,
if, if I can talk through the steps to
get there, you start with having as much
kind of behavioral data as possible.
So it may be users who are doing
certain behaviors inside the app, what
buttons they're pressing, how many
times they're doing certain actions.
So all of these become inputs, um.
A lot of the times that that data
doesn't exist in an MMP, it's
typically in your product analytics,
like a mix panel or an amplitude.
so it's really first
ingesting all of that data.
And the way these ML tools work, in
essence, is it breaks these users into.
a or B of converters
and non-con converters.
And then it's looking for
patterns, um, uh, within that.
So a really simplistic way to think of
it in, in one of the most simplest models
is, uh, what we call a decision tree.
Um, and it will just work
its way down for each level.
So it may be, you know, the
first decision is trial.
Uh, people who canceled in, in
one hour and people who didn't.
And then it would go underneath
that and it might find people who
did a certain event and didn't.
Have different, um, conversion
rates and it'll go down.
It could differ by geo and it'll keep
going down to find these like pockets
or clusters of users, um, who will
share like similar characteristics
and then look at what are the
conversion rates for those users.
Yeah.
Jacob: Is the, is the end state that
you're sending back a, like you said,
a, um, converted or not converted?
Or are you also sending in,
um, like revenue values of
like a predicted LTV value?
Shumel Lais: Yeah.
Yeah.
So you, you ultimately get a
probability, um, at a user level.
Um, and you can also apply
that to revenue as well.
So you have both.
And um, that's also depending on, uh, you
know, what's the goal for the app or also
how much variability they have in revenue.
So do we typically find in subscription
out to probably slightly less variability.
so a lot of the times a did
convert or did not convert.
By the, the plan type that
they started works really well.
Whereas for something like gaming,
the revenue value will work better
'cause there's a lot more variability.
Um, but in essence, you, you get this
output at a user level and then you
can decide, um, if you wanna fire
those signals, yes or no, if you
also wanna up weight or down weight.
Uh, the revenue.
'cause you might see, you know,
some of these that are in, in
gaming are considered like whales.
Um, so, you know, do you
wanna normalize that slightly?
Otherwise, you know, meta may spend
tons of money just to go after those
whales and they may not convert.
so it becomes a bit risky.
So there's a bit of tuning
that comes with that.
ð ð ð ð Hey, Humel,
I'm super excited to have you
here today on the podcast.
Thanks for having me.
Jacob: Yeah, that makes sense.
Where the, the distribution of revenue,
you know, for a subscription app is
like, you know, we're getting, um, you
know, 5% of people to pay, but they're
all paying like similar ranges, right.
At the end of the day, because
we, we just need a certain level
of return if we're, if we're not,
we don't have a good business.
Uh, and so the, the variance
in revenue, like, okay, if it's
$30 or it's $40 or $50, like.
It doesn't really matter to meta,
like that's not a, not a big of
enough variance that it's really
gonna change the optimization.
But when you have games with a super
long tail of, you know, some people
spending a lot, people purchasing
a lot more, then, then that kind of
difference can, can be meaningful.
Do you have, like, and it sounds like
it's a bit more of an art, is there, um.
Um, do you have simple, like
heuristics you use to go, um, when
it makes sense to send revenue?
When it, or, or when it makes
sense just to send, um, like
a, you know, conversion or not.
Shumel Lais: Yeah, I think, um, you
almost wanna try both, um, because
depending on the app, it could
vary, but also like the budget.
the, uh, app has as well.
Typically the more budget you have,
the more types of tests you can do.
Um, and same with the signal.
'cause you know, we talk about
these probabilities for each user.
Um, and we talked about different
levels of volume as well.
So you sometimes you may wanna
test multiple signals and see
which ones yielding better results.
Jacob: Got it.
And so there's, I think there
was another example you shared,
uh, about a restaurant booking.
I think, you know, the team
was optimizing on, you know.
Behaviors that correlated with
bookings, but, um, the performance
didn't improve because I, like the
relationship wasn't actually causal.
Um, how do you, I mean, I think.
Any, any, uh, semi sophisticated marketers
should kind of understand the difference
between correlation and causation or
any, any student that, you know, went
through basic stats class in, in school.
Um, but, but how do you, you know,
for people you're working with, how
do you teach teams to kind of think
about correlation versus causation?
Because a lot of times we have to start
with correlation and, and so it's like
thinking about proving that causation.
How do you think about thatâ ð ð
ð ð Hey, Humel, I'm super excited
to have you here today on the podcast.
Thanks for having me.
? Shumel Lais: Yeah, so, so that
restaurant booking at that was
a really interesting example.
That's the one that actually
got us into this like.
Concept and idea.
Um, and I was the marketeer that was
getting it wrong and, um, optimizing to
the, to the wrong event, um, before I
met my co-founder and he, he introduced
me to, to some of these frameworks.
Um, so I think the first thing is
having, um, like clear measurement to
begin with, just to be able to see.
If your campaigns are actually working
or not and, and driving a difference.
And, and that's maybe
something we'll get into.
Um, then the other is like this
framework of back testing on data.
So previously we would just look at,
um, you know, event, uh, a like how many
times does someone look on a restaurant,
uh, page, uh, like a specific restaurant
page versus how many times did they book?
but what I missed was.
Uh, put putting users into specific
cohorts and defining observation
windows to say, okay, this is the
observation window of behavior.
Let's say it's 24 hours and then
this is the window we're gonna ask
them to, to see if they did book.
And then it's like a binary yes or no.
So I think starting with that framework
is a lot easier to back test and see.
Is this actually does, does
this event have a good precision
and recall, uh, which is that
metrics that we would look at?
Um, so yeah, I think that that's kind
of how we learn about, or how I learn
about this framework as a marketeer.
Um, and that's what we're trying to
help bring to, to the industry as well.
Jacob: Yeah.
And, and, um.
I think that's, you know, common,
common ML concept where you, you have
the historical data, let's, uh, build
a model on half of it and then, and
then try to verify on kinda the other
half to see if kind of this is, but
even so, like, you know, we have.
Um, we, we build a model
at the end of the day.
Like there's only one real true way to
prove causation is run it and see if
it works and, and run an experiment.
And I think that's, you know, that's true.
Whether you're running a simple AB
test, you're, you're doing a correlation
analysis to figure out an activation
metric of like, at the end of the day
it comes down to like, experiments.
Did this work?
Uh, or did it not?
And then I think, um, uh, we can.
We can get pretty close and, and pretty
high confidence with, with our models.
Uh, but you, you, you always
gotta launch it and see.
And then also once you're, you never
know what, you know Facebook's gonna do
once you start sending signals to it, uh,
uh, the, the, the black box of Facebook.
Yeah.
Um, cool.
Well, I think, uh, maybe,
maybe we do this backwards.
We dive into kind of the
complex concepts and dug in.
Really, let's take a step
back from the beginning.
Uh, I think that, um, you have, um.
Amazing insight kind of in your, in
your mobile growth, uh, a career and
experience with, you know, simple
apps and large apps helping lots of
different kind types of companies.
I wanna kind of work through a
progression of, um, let's call
it, you know, measurement maturity
of, uh, where you get started for,
for small apps and, and then we'll
slightly layer on more sophistication.
So, um, you know, some people, uh, some
people are old pro, some people are just
getting started in the app world, and so.
May, maybe at a high level, what
do you think about these, these
stages and, and let's focus on
subscription apps, uh, for now from,
you know, where, where do you start?
And then we can, we can dive in.
Actually, let's just, let's
just start, let's start with
a, a small subscription app.
Um, just launched.
They've got a, a small team, limited
budget, you know, what's the, what's
the minimum measurement set up,
uh, they need before they kind of
start spending money on, on paid ua?
Shumel Lais: Yeah.
So, um, whenever we do signal engineering,
the first step is we always try not
test if people's measurements ready.
So, um, we, we do dozens of these
audits, so it is given us a good
understanding of on different scale,
um, kind of what, what we see, what,
what people are, uh, starting with.
So yeah, you definitely do see
a pattern and, um, in terms of
maybe I'll, I'll start with.
How I see this measurement maturity
evolving, and then we can look at, um, you
know, if you're running paid ads, where,
where, where should you really start?
so when I speak to a lot of early
stage folks, um, what they're
doing, you know, a lot of them
don't even have an MMP in place yet.
so again, an m and p is there
to, to when you have multiple
sources to decide who gets credit.
For specific user or
behavior of that user.
Um, so a lot of the people just starting
out, they don't have an m and p at all.
Um, they're ultimately taking
the revenue from App Store
Connect, or they may have, um.
A tool like Revenue Cat or, or
adapt and, and so on in place.
they're just comparing that
revenue versus the spend.
So I'd say that's kind of the
most foundational, and you'd be
surprised, we still see a lot
of folks operating in that way.
And that's how, um, you know, at board
level or, or for, for some investors,
they may kind of look at it zoomed out.
So I think that's, that's
the first, uh, step.
Uh, I think then it evolves into
what the second step, which I'd
say is platform-based attribution.
Jacob: Can I, can I interrupt you?
Um,
Shumel Lais: for it.
Jacob: is.
Do you need an MMP to get started?
Um, where, where do you have simple rules
that, like, MMP value has changed a ton.
Right?
Um, and so maybe a quick, uh, quick,
uh, uh, direction of when you actually
need MMP or when you maybe can
start with just the Facebook, SDK
or other simple measurement setups.
Shumel Lais: Yeah, really good question.
And it definitely comes up a lot.
Um, so I'd say, you know, if you're
serious about paid user acquisition,
even if you are just starting out
spending only, you know, a few hundred
dollars a day or, or less, um, I
would still advocate getting an MMP,
and I'll kind of run through why.
And a lot of these MMPs do have these,
like free plans or, or free tiers as
well, which normally include enough
credits for people to get started.
So I, I think what you said is definitely
kind of right of, you know, the, the
second stage of, um, measurement maturity,
I would say is just having the Facebook
SDK and that can work quite well.
Or if people are directing people
through websites, you might just
have a website pixel or, uh,
through, uh, the Facebook, uh.
Facebook, uh, conversion pixel and so on.
Um, so I'd say that's a second
aspect and that might work well if
you're just running on Facebook.
where it falls down and where I think
you really need an m and p, which is
then I think takes you to the third
level of measurement, maturity, um, the.
Biggest thing it misses is
having cohorted, uh, visibility
or visibility on what is the
user's value in 30 days or so on.
So meta for example, uh, their metrics
are all based on the day of the event
happen, not the day of the install.
they're a little bit sneaky like
this where, um, you know, they
can end up just serving your
ads to people who already have.
App, um, or save your ads to people who
are already subscribed and the renewal
event is tagged into the purchase event.
Um, and all of a sudden they're
claiming all these purchases when
they're just users who are already
subscribed and already renewing.
So that's where an MPI think can really
help to differentiate between, you know,
what's actually driven from an install,
which is why I'd, I'd always advocate it.
And then you've got some aspects of if
you expand beyond meta, um, if you're
running Apple search ads and meta,
that's again, another reason to have.
A third party attribution tool.
Um, again, it's not perfect,
it's still based on last touch.
So Apple search will still, you know,
um, cannibalize some of your meta
conversions, which is a topic in itself.
Um, so yes, I'd say that's,
that's why I would really
advocate having an MMP for almost
Jacob: Yeah.
Shumel Lais: paid ads.
Jacob: That's helpful.
I think, um.
I've asked, I asked a number of
people this, and you're always
curious to hear the responses.
'cause kind of the, the value has changed,
but that, that makes a lot of sense.
Okay.
Alright.
So once that baseline is live,
you've got an mm p installed,
you know what, what is, yeah.
What's next?
Shumel Lais: I'd say the next piece,
um, on the m and p, the other thing
just to, uh, or the next level
that I see a lot of apps aren't
quite there yet, is understanding
the difference between cohorted.
Metrics and deliverables.
Um, so depending on the platforms,
um, some of them by default
will just show deliverables.
So again, it's what revenue was
generated today versus revenue from
apps who installed today as an example.
So I think that's like a really.
Key fundamental to get in place.
So internally aligning, like when we talk
about roas, like what is a cohorted day?
Is it like day seven, day 30, day 60?
Uh, same with CAC or subscribers,
because only then are you like
comparing apples to apples.
So I'd say that's kind
of the, the next step.
And then the last step on attribution
is, um, or, or kind of level five is
where you would look at, um, media
mix modeling or incrementality.
and that's where you're ultimately
adjusting for some of that
cannibalization to really test
like what's truly incremental.
But I'd say.
You know, quite a few subscription
apps are at that level of spend.
Like that works when you've got two years
of historical data, um, you're spending
in, you know, millions, uh, a month.
That's where it can get there.
So I still think last search attribution
can work really well if you understand.
The limitations of it's really
apple search ads that becomes
the main, the main issue.
so if you're aware that, you know,
targeting your brand terms is gonna eat
up some of your Facebook ad conversions,
'cause there's a lot of users who will see
an ad and then go and search for it later.
Then you're covered.
So I think that's how I think
of measurement maturity,
like those five steps.
Um, and this is all like, again,
unrelated to signal engineering,
but it's almost like a prerequisite.
We wanna kind of get to that
level three or four, which is
having an m and p, before we
even think of signal engineering.
'cause only then it allows
us to test it accurately.
Jacob: Is there a, um.
In, in the app space, you know,
MMP or uh, media mixed modeling.
Uh, and, um, um, uh, you know,
incrementality testing are, are
definitely more advanced strategies.
Let's say you need, you know,
millions of dollars a month.
What is the.
What is the threshold for
single engineering when
it makes sense to explore?
And of course, single engineering
is a broad term, right?
Like maybe, maybe we'll, we'll qualify.
Says when does, you know, ML based
signal engineering make sense?
ð ð ð ð Hey, Humel,
I'm super excited to have you
here today on the podcast.
Thanks for having me.
Shumel Lais: Yeah.
So I'd say, um, one of the requirements
of signal engineering, or it's like if
I, if I double down a bit on meta, is
where majority of people's spend is going.
there's, this isn't like a, a definitive
rule, but a general rule that we've
seen is like, you wanna get at least
10 conversions per campaign per day.
that is what I would set
as a, as a starting point.
If your current trial starts aren't
getting that, then you definitely don't
wanna be doing signal engineering because
you're not hitting that threshold.
And although 10 sounds like not a lot,
um, sometimes depending on the, the
markets that you're targeting, that
cost per trial can be quite high.
It can be from, you know, $20 or the up to
a hundred dollars, for example, depending
if, especially if you're targeting the us.
So if you multiply that by 10, all
of a sudden, you know, the, the
spend starts increasing quite a lot.
that's where I would say spend
is quite a big aspect of signal
engineering to, and that answers
should we go down funnel or not.
So I think that's just the
gen, the general principle.
whether we should get into ML based, um.
Uh, signal engineering is one,
is answering the first part.
The next is the, the
kind of data you have.
Um, and do you have enough as well?
So what we've typically found is it's
quite rare that, um, people have enough
behavioral data tagged in their MMP.
Um, they usually have to go into
something like a mix panel and.
The ML models are only
as good as your data is.
So, you know, again, if you are
like a language learning app, do
you have all the lessons tagged up?
Do you have different
tracks, um, tagged up?
Like all of this can become inputs.
so yeah, so I'd say once that's
how I kind of look at data.
It's less to do with like number of users.
It's more, um, the number of conversions
you're starting with to the ad platform.
Jacob: Why do you need it an MMP?
Um, uh, are you still
using the kind of MM.
E post backs to the
network to to optimize.
Shumel Lais: The MMP is more
from a measurement perspective,
Jacob: Okay.
Shumel Lais: so the MMP is really
more, um, one is yes, the MP data
can feed in, uh, into the ML model.
So some people have enough
behavioral data in the MP.
The benefit of the MMP for signal
engineering is purely measurement.
So, so what we would do whenever we do
signal engineering is we don't wanna
just blindly run it, we wanna AB test it.
So we would have a campaign optimizing
to start trial, and then we'd have
a campaign optimizing to, let's say,
this predictive subscriber event.
by having an m and p, that's how you
can truly see like your day 30 roas,
um, for the start trial campaign
versus a predictive signal campaign.
without an MP, you wouldn't
be able to answer that.
Jacob: Got it.
Got it.
So, so it's not necessarily for the signal
engineering side, but for the measure
measurement, understanding of kind of
a cohorted, you know, roas for example.
Shumel Lais: And, and where the MMP
can also help is it can also act as
like the gateway for the signal itself.
there's again, a couple of options.
Um, for anyone doing signal engineering,
you can either send these signals
or server side events directly to
meta using the conversion API, you
could send it to your MMP and then
they would forward it to meta.
The benefit of it going through the MP
is you could, that one point can also
forward it to TikTok, it could forward
it to app loving liftoff and all the,
all these other ad platforms as well.
Um, so, but both work in
principle, but that's just another
benefit of going through an MMP.
Jacob: And I, I think that most people
are recommending, like with any decent
level of spend or any sophistication, you
should be using kind of conversion APIs.
Is that right?
Shumel Lais: Uh, not, not necessarily.
Um, I think there are like some more,
again, structural challenges inside,
um, like how the m and p data goes.
To meta.
Um, and this is just from experience
having consulted and worked inside an MP
on the product team as well, like seeing
like, um, how fragile some of this can be.
So that's where I think a lot of people
have, uh, leaned on the conversion.
API as a bit of a
fallback, as a workaround.
Um, but a lot of the times it is just
configuration, like making sure the right
parameters are passed into the m and p.
The MP configuration, there's
certain requirements that meta
has for it to receive the data.
Um, so if you're, uh, hitting those
requirements, then I'd say, you
know, the, the m and p can be just as
strong as a conversion API as well.
sometimes like people suggest a
conversion, API, again, coming back to.
You know, if you think of
like scan and, um, where there
was a lot of like data loss.
Um, whereas I think with a EM,
which is, uh, Facebook's, um, kind
of probabilistic approach to it,
um, it negates quite a lot of that.
Um, so yeah, I've not seen
significantly better improvement
using conversion API versus uh,
running it through an m and PI think.
I think they both have merit.
Jacob: Maybe, maybe you can clarify
something I don't fully understand that
other people I, I've seen talk about
of in terms of, um, sometimes you're,
you're sending a, um, conversion event
immediately via SDK, but then you're
sending, uh, additional data via API.
Can you tell me about that?
Shumel Lais: I think, um, sometimes,
uh, you don't have an option but
to send it through, uh, the API.
And that's like, again, if you
think of, uh, trial converters,
they don't, it doesn't happen,
uh, while they're on the app.
It happens, you know, from,
from Apple or Google Services.
Uh, so if you think of revenue
card as an example, all of those
events are service side events and
I'd say they power a majority of.
People's conversion events in the MMP.
so that's where there's a need
to for it to, um, be sent later.
Uh, and sometimes we have seen, depending
on the MMP, those server side events
struggle a bit more to get matched.
It's a little bit similar with scan as
well, scan with, like really struggled
with server side events just to do with
the nature of when it sends it and also
the parameters that get attached to it.
So yeah, again, the MMPs have
special relationships with.
platform.
So there's probably slightly different
parameters that are getting passed
through the M-M-P-S-D-K versus if
you just send it, uh, server side.
Um, so that's typically
where the challenges lie.
Um, and conversion API is
just like another way to
strengthen that server side API.
'cause you can send meta like the
email address, the U user's phone
number if you have that information.
Again, not every app business has that
information, and not every business is
comfortable forwarding that data as well.
Jacob: And so you can.
Some additional information that
you may not have had at the initial
conversion, be the API at a later date.
Uh, also supplement it with more
PII to kind of have better matching.
Uh, and then like, speed
is pretty important too.
Like just, I think there's, you know, the,
the probabilistic matching, um, you know,
AKA fingerprinting, uh, uh, it is, um.
Also important if, if on time.
'cause if you is, is the right
understanding that essentially, um, if
you send an event very fast to Facebook,
um, there are fewer possible conversion
events that it could correlate to.
So it's better matching where if
you wait up a couple days or hours
even that, um, there's the, the
matching is harder because, um,
there's more possible events it could
Shumel Lais: Yeah.
Jacob: to.
Shumel Lais: Yeah, uh, definitely.
And again, it depends on like what
method you're sending it through.
So if you're sending it server side, um,
and it's relying on this probabilistic
matching, there's typically fewer
parameters for it to, to match on.
Um, so the longer that window is,
the, the less likely it is to match.
Um, whereas the client side,
it just sends a lot more, uh.
Matching properties as well.
So even if it comes a bit
later, it's just got a lot more
strength of, of the capability.
So I think that speed
aspect can help in two ways.
One is from a matching perspective,
but the other is just from an
algorithmic feedback loop as well.
So, um, I.
It just allows meta to, to
learn and optimize quicker.
'cause if it's seeing that these purchases
are always happening on day seven, it's
almost waiting for seven days after a
click to know if this user was good.
Whereas if you can send a quality signal,
you know, within a day, for example, then
meta can quickly decide actually this
user was a good user or not a good user,
like immediately, uh, or within that day.
So it just allows the algorithm to
find users a bit quicker as well.
Jacob: I think that's very helpful
people under, for people to understand.
Um, alright, so, uh, you've talked
about, um, I think it's some of, some
of your, your content about bigger apps.
Being willing to accept longer payback
periods, maybe like a six to 12 month
payback in order to scale faster.
Um, how do we think about that
maturity ladder and, you know, when
a can team actually trust their
numbers enough to make that bet?
Of course, like money in the bank is
always a big limiting factor as well.
But, but how do you think about kind
of that measurement maturity side?
Shumel Lais: I think, um, yeah, money
in the bank is definitely, um, a
key aspect or, you know, what kind
of financing they have, if they've
raised funding or got some, you know,
user acquisition funding as well.
so we typically find the indie, uh,
developers, um, who are bootstrapped,
they're really pushing these annual.
Uh, subscriptions just because they
don't have that later visibility,
um, or the, the time to kind of
recoup that investment is like they,
they need that to be a lot shorter.
Um, this then gets into just
more general LTV, which is again.
Related to signal engineering,
but more broader, looking
at kind of cohorts as well.
And you ultimately need
historical data to, to go off of.
Um, and that's where it can become
a bit shaky because the, you
know, the product's gonna change.
You might test different plansâ ð ð
ð ð Hey, Humel, I'm super excited
to have you here today on the podcast.
Thanks for having me.
. Um, so historical, uh, performance
isn't always a future, you know, the
best indicator for future performance.
So we have to be a bit careful.
But typically, um, you
know, most of the, the, the.
Curves of these predict, you know, these
LTV curves, a lot of it's determined early
on in the first kind of one or two months.
So as long as, um, you've
got good visibility, uh, in
that immediate short term.
it's just having enough buffer
to know, okay, I'm gonna monitor
this for the first 45 to 60 days.
Um, if it's really off track, then
I may wanna scale back my spend.
but you can computate that out
to see, you know, what do I
think this is gonna be worth?
but I'd, I'd say again, when people
are waiting for it to recoup their
investment, again, it depends on the
type of subscription app as well.
Uh, we definitely see like more
of a push to annual, um, or people
just looking to get their payback in
like the first 60 days or 90 days.
But even that is going beyond the
bootstrap companies who wanna recoup their
investment in seven days, for example.
Jacob: And, and sometimes those,
um, faster payback periods can, um.
Give you more rigor and kind of in your
ua, you know, overall process that if
you're, you're really forced to, um.
Have all your, your ads high
performing, you know, immediately
you have faster feedback loops.
Uh, you may still have, you know, making
more money over time and, and your,
your ads may actually a better roas.
So, so I've seen that sometimes
even, you know, big companies
really still have these, you know,
very high bars, uh, for kind of
payback, which, which can be helpful.
Um, but then if you're, I guess the
flip side is if you're, you know,
trying to grow a market and compete,
you know, sometimes, um, it, it.
Overall kind reach and.
Shumel Lais: Yeah, definitely.
And, and that's when you do sometimes
get back into this world of like
blended versus direct CAC as well.
sometimes people, you know, push
towards blended just to kind of justify.
Spend, sometimes it's required because
if you just go on direct, there can be
attribution issues, um, that may force
you to be too conservative and you miss
out on the, this halo effect or, you
know, apple cannibalizing some of your
Facebook, um, ad performance as well.
So yeah, it is a, it is
a balancing act for sure.
Jacob: And we're talking about
that kind of, uh, you know, predict
predicted, uh, revenue for the future.
And, and you can use, um, you know, for
the first 30 days or 15 days, you can
get some strong signals about what's
gonna happen because, um, you know.
Most of this, um, you know,
activity or engagement or, or
cancellations or renewals, you
know, comes in pretty quickly.
What are, like, are those the signals
you're using, uh, to kind of build
out that like, you know, retention
curve or renewal curve or revenue
curve, you know, for, for people
that may want to do this themselves?
Like what, what are kind of
the, the easy inputs where you
can build a pretty good model?
Shumel Lais: Yeah, I'd say, um, for signal
engineering, it's a lot more short term.
Um, and it's at a user level, so
it can differ quite a lot from
traditional LTV modeling, which
looks more at cohorts, it may look at
cohorts of users by week or by month.
And then extrapolating that out.
And that's a lot easier to be accurate.
Um, with signal engineering,
sometimes it can be a pitfall as
well if you try and be too accurate.
'cause at an individual user level
that can be like, really tricky to do.
so yeah.
So again, like I think you,
you almost wanna start by just.
Grouping your users into those who do or
do not convert, or those who, you know,
into revenue buckets or those who are
worth X, Y, and Z in in this time period.
And just looking at behaviors they do.
And there'll be some obvious ones
like, you know, auto renewals
being turned off early on.
But then beyond that, um, you know,
if it's things like, you know, what
behaviors they're doing, so there can
be some low hanging fruit, But again, as
we mentioned earlier, it can sometimes
be a bit of a blunt tool, but I would
almost say most of the times it would
be better than just a start trial event.
Jacob: Yeah.
Yeah.
So look out some revenue,
see that, um, you know, maybe
look at your historical data.
Go.
Okay.
We generally see.
30% of people that, um, are not
going to renew, uh, turn off
autorenew in the first 30 days.
So by third day 30 we can have a
simple multiplier and then maybe
if get a little more advanced,
we layer on some engagement.
Okay.
Well probably if they're never
using our app, that decreases their
probability to, to renew as well.
And then so you can kind of layer on
kind of sophistication as you go to
kind of get those predicted LTVâ ð ð
ð ð Hey, Humel, I'm super excited
to have you here today on the podcast.
Thanks for having me.
. Shumel Lais: Definitely.
And, and another aspect can also be
like onboarding questionnaires as well.
Um, sometimes like, uh, a lot of apps
are starting to capture age as well.
Um, and that can help 'cause
you might find different,
uh, LTV based on age as well.
So again, it can be like a, a
simplistic way to start clustering
LTV by by user groups as well.
Jacob: Cool.
Cool.
That's, that's helpful.
Um, alright, so, uh, if a founder is
listening right now, they're running
some meta campaigns optimized to
cost for trial with standard mm p set
up, um, what's the very first thing
they should, should, uh, change or
try to do to improve performance?
Shumel Lais: So I think the
first thing is just check the
foundations, um, are covered.
So the first thing is like, uh, your
match rate between meta and your m and
p, like how different are those numbers?
Uh, 'cause that might highlight
like a more structural issue.
And then the second is, uh,
just aligning on what your, you
know, cohorted, roas or cac.
is gonna be, is it gonna be seven
days, 10 days, 30 days, et cetera.
So that'd be the first thing.
Just, just get that covered.
Um, then I would say, yeah, you
can test with, um, some kind
of higher quality, uh, trials.
So if you're doing a trial today, you
know, think about a qualified trial.
Could it be able to renewal?
Could it be plus a specific event?
Um, but the key thing
to note is just that.
Uh, volume aspect as well.
Like, are you getting enough?
Like how far can you push it?
Um, and you could always apply
this analysis on historical data.
Just look at the last 30 days and
say, you know, if I was to apply this
logic of a qualified trial, how many
users would that reduce things by?
and, you know, would it still
meet the threshold for my budget?
Uh, so that's the.
Jacob: What is, how should we
weigh off the trade, off between
quality and quantity of signal?
Where like is, um, okay, so
let's say we, you know, we have
20, 20 conversions per day.
Um, is maybe removing, you
know, 30% of them because
that's now our qualified trial.
Uh, is that better or, or.
Is there, uh, at that quantity,
like, don't even bother.
Like you, you, you need to get way more
quantity of conversions to Facebook
and that's always gonna be the highest
leverage improvement for a while.
Or how do we think about that
quantity versus quality of signal?
Shumel Lais: I, I'd say, um, quantity
kind of wanes off after you, you
know, anywhere between 10 to 20,
10 to 30 is a good, sweet spot.
so.
Jacob: I.
Shumel Lais: You are, even if you're at
20, there's still an opportunity that 20,
30%, you know, improvement of, uh, volume.
Like, you know, you, you're sending 20,
30% less but getting higher quality, I
would definitely take that trade off.
Um, and then the flip side, you know,
once you're getting past that 30, 40
signals per day per campaign, that's
when it's almost like you really should
be optimizing your signal 'cause you're
leaving too much money on the table.
at that point you wanna bring
it down and then again, you can
go through these iterations.
'cause hopefully this
is driving performance.
So you can start scaling your spend.
You'll start seeing this,
you know, qualified trial
that you, you optimize for.
Maybe now that qualified
trial is getting to.
30, 40 signals a day so we
can go even deeper as well.
So that's how I kind of see this
evolution or iteration of, you know,
when should you go further down funnel?
And that's ultimately what
signal engineering is.
It's just going down funnel, but designing
in a way where you go down funnel before
the user's actually gone down funnel.
Jacob: Yeah, because we need that.
We don't have time to wait.
We need that convergence signal fast.
And so yeah, that, that, that makes sense.
That's, that's a really great
framework to think about everything.
Um, cool.
Uh, well this was all the,
all the big questions I had.
I've got a few last, uh,
quick hits left for you.
Um, and so, uh, I would, uh, I would like,
if you have any hot takes that would annoy
someone at, at a big mm p or ad network.
Shumel Lais: Um.
I, I dunno if it would annoy
people at an ad network.
Maybe more for marketeers.
I would say maybe two things is
think in 2026, um, you, you, we
shouldn't be hiding behind attribution
or measurement problems anymore.
Uh, I think those.
You know, days have passed us.
So, you know, there's definitely
solutions and workarounds.
So I think everyone needs to have like,
strong measurement foundations now.
I don't think there's really an
excuse not to, um, that'd be one.
And then related, again, is, is maybe
more for the marketer than the ad
networks again, is, um, you know, again,
you know, just be careful about hiding
behind blended metrics, blended cac roas.
Sometimes it can be quite
uncomfortable as well.
But again.
In, in today's day and age, I think
we can really focus on direct cac.
You know, we know it may not be
perfect, but it's, you know, if
you've got your measurement in place,
it's working strong enough, uh,
for you to be able to rely on it.
Jacob: Yeah, the easiest person.
Shumel Lais: will always be more
expensive than blended, right?
So,
Jacob: Yeah.
Shumel Lais: yeah.
Jacob: I was gonna say the easiest
person to lie to is yourself.
Yeah.
Um, cool.
Those, those are great takes.
And so, uh, since this is a Price
power podcast, uh, uh, you know, by
Botsy, where we're focused on, on AI
pricing to optimize your revenue, have
to ask, what's the most interesting
win, uh, you've seen, uh, with prices,
pricing, and packagingâ ð ð ð
ð Hey, Humel, I'm super excited to
have you here today on the podcast.
Thanks for having me.
. Shumel Lais: Yeah, I, I think, um,
monetization and pricing is, is
still like a really underrated lever.
Um, not enough teams I
think are testing enough.
Uh, a couple of the trends that I've seen.
Recently that have been really
interesting is like paid trials as
well, which has been really interesting.
Like, how do you do like
a, a lower paid trial?
And then the second is like knowing
when should you offer that discount.
So it is very common.
We see these like abandon mate, uh,
abandonment discounts where as soon
as someone presses the X on the pay,
straight away get like a 50% discount.
Um, and again, you, you know, you're
probably best placed to know, you know.
When's the best time to to, to surface
this, but um, yeah, we've definitely
seen like that taking a big share of,
uh, subscriptions where, why I mentioned,
you know, why start trials may not
always be, uh, you know, as strong
because you still get a big volume of
people that see those discounts and
say, you know what, I'll take a punt on
50% discount, um, and just sign up now.
Jacob: If, if, um, generally
yes, showing a discount to every
single person is suboptimal.
If you can figure out the right
people to discount, to right.
People to show different offers
to, and yeah, I, I've seen that.
Um, I think some people are calling
like, design your trial paywall
where you could pick a, a longer
trial, uh, for a higher price or.
Or, or a longer trial, but you pay
for it like 99 cents or a few dollars.
So yeah, I've seen that become, uh,
kind of trendy recently and yeah,
some people have told me it works.
So yeah, those, those are great takes.
Um, cool.
Well, well this was awesome.
Um, a amazing insight and kind of
the nitty gritty of how to evolve
your measurement layer and how to
think about signal engineering.
So yeah, really appreciate you joining.
Shumel Lais: Brilliant.
Thanks for having me, Jacob.
Jacob: And, uh, anything else
you would like to promote?
Uh, we can have everybody go to
day 30, uh, and, and reach out.
Uh, we can link anything in the
show notes you want, but yeah.
Anything else you'd like to,
uh, promote to, to the audience?
Shumel Lais: Yeah.
Again, like, um, we're, we're still
quite early stages, a company and,
and our number one goal, we're
not like chasing revenue for us.
It's really like just validating
this as much as possible.
And working with as many apps as possible.
Um, one of the ways we make this easy for
everyone is we do these free signal audits
as well, um, which we don't charge for.
Um, and that actually allows us to just
check if those foundations are working.
Uh, and also some of those frameworks
I mentioned of precision and recall
or like calculate that for people.
Um, so that's like our kind of easy
way to just start getting into the
world of signal engineering without
thinking too much about investment.
Jacob: That's awesome.
I think that's an amazing
thing to offer people.
And I know I just sent someone to you
and they said it didn't make sense at
the time, but you gave amazing feedback
and advice and uh, and they really
appreciated the guidance they told me.
So even if, uh, uh, it doesn't, doesn't
make sense to work with day 30 today,
uh, Humel will be, uh, amazingly
helpful and happy to share as knowledge.
Yeah.
Cool.
Alright.
Thank you so much.
Have a good one.
Bye.
Shumel Lais: Thanks.
Speaker: Thanks for listening.
Hope you enjoyed.
Please go to price power podcast.com
to see all the episodes.
Go to Spotify and YouTube and
give us a subscribe and follow
so you don't miss any episodes.
Alright, talk to you next time.
All.