How Backtesting Can Protect You From Unrealistic LTV Expectations
You have a revenue stream. Check. You’re spending on paid UA. Check. You modeled out a future projected LTV - and you’re reasonably confident that you’re profitable on your UA.
What could go wrong?
While oftentimes a healthy projected future LTV is a good sign that you have a sustainable, profitable UA strategy, what can sometimes get lost in the optimism of strong projections is that LTV is a made up number - and the map can sometimes differ from the territory.
Your actual revenues can sometimes diverge from your projected revenues - and this can result in huge planning challenges that can impact your cash flow. There are 3 key reasons why this can happen:
1. Downstream monetization or retention features may be weaker (or stronger) than upstream features.
An app may have very strong retention or monetization for its early users, who may be hooked - and this may result in its LTV curve looking strong based on early upstream metrics.
However, sometimes the downstream monetization or retention features may not be nearly as strong. If, for instance, there isn’t a strong core loop - or if there aren’t strong metagames to keep users hooked, downstream monetization or retention may taper off in ways that aren’t predicted by the upstream monetization or retention.
2. Many early users may be ‘casual browsers’ - and thus not truly engaged.
This is a behavior that is especially prevalent in apps that are highly competitive - or in commodity categories. Users who, for instance, play match 3 games or play casino games often tend to have little loyalty to any one game - simply because there often isn’t a lot of differentiation between games.
What this often means is that users can often show strong early retention or monetization because they are interested in checking out a new app - but not necessarily because they expect to be loyal long term users. Once the hit of a new game mechanic wears off, they go to a new app - and so the early retention and monetization signals typically do not translate to long term retention & monetization, and thus long term LTV.
3. Downstream user behavior can differ by source, OS & geo.
Oftentimes we see that apps can have wildly varying retention and monetization profiles for different countries, OS-es, user acquisition sources - or for that matter creative formats. Looking at an aggregate projected LTV across multiple geos, sources or OS-es can often conceal the fact that some geos can have significantly stronger(or weaker) downstream retention and monetization profiles compared to others.
How do you address this? Backtesting.
The best way to address this potential issue of actual LTV being different from projected numbers is by backtesting.
Backtesting involves periodically checking our projected metrics vs. actuals. If you modeled out a d30 LTV of $2 based on a d7 LTV of $0.50 last month, you calculate what your actual d30 was. You do this exercise for d60, d90 and beyond as well.
Was your actual d30 LTV close to $2 - or was it wildly off? If it was wildly off, you should try and understand what behaviors caused it - so you can tweak your models accordingly - and get accurate data.
Getting your projections right can often make the difference between going bust and staying financially sustainable. In fact, as Adam Lovallo mentions in the first episode of the How Things Grow podcast, the first sign that he and his team at LivingSocial saw that the daily deal space was cooling down was when they backtested actual LTVs versus their projections - and saw that the early monetization metrics were mostly driven by the press and hype around the daily deal space; and these did not translate to the kind of downstream LTVs that they had projected for themselves.
It doesn’t have to be hard to stay financially sustainable - and to accurately plan for your cash flow needs if you have a systematic, regular process of backtesting in place.
Shamanth Rao is CEO of the mobile user acquisition firm RocketshipHQ, which helps mobile apps grow in a capital-efficient manner. For more tips, pointers and strategies about user acquisition, visit RocketshipHQ’s blog here.
Pollen VC provides flexible credit lines to drive mobile growth. Our financing model was created for mobile apps and game publishers. We help businesses unlock their unpaid revenues and eliminate payout delays of up to 60+ days by connecting to their app store and ad network platforms.
We offer credit lines that are secured by your app store revenues, so you can access your cash when you need it most . As your business grows your credit line grows with it. Check out how it works!