I'll be presenting a talk at the Facebook Developer Garage SF Wednesday evening. You can learn more about the event here. It's hosted by Kontagent and sponsored by Intel. Most of the content for my presentation is drawn from my original article on engagement loops, with new diagrams courtsey of my friends at Kontagent and a few new examples. Without further ado, here are the slides (feedback welcome!):
Eric Ries Engagement Loops The Levers And Metrics Of Engagement
View more presentations from Eric Ries.
When I work with startups on improving engagement, I really try to emphasize the importance of their most powerful lever: positioning. In most applications, most of the time, customers return to use the application not in response to a notification or event, but under their own volition. These decisions are critical to the success of any high-engagement product, and they take place entirely inside the minds of customers. Companies have an opportunity to influence these decisions, but only if they act well in advance of the result they are trying to achieve. Unfortunately, it's easy to lose track of positioning effects when optimizing for a single metric.
This is a common problem that results from viral-loop optimization. By copying the exact same registration flow as every other successful viral app, many viral apps completely lose their positioning. Customers can't even remember what apps they've signed up for, and become entirely dependent on notifications to bring them back. This starts a downward spiral: as more and more apps become indistinguishable, they send out more and more notifications, which leads to increasing fatigue on the part of customers. As notification channels get stuffed full of these messages, customers tune them out (or platforms have to put in place dramatic limits on access).
The solution for app developers caught in this vicious cycle is to develop competency in positioning. Luckily, a great example of the power of positioning fell into my lap recently. Some friends of mine at EA tried to break my World of Warcraft addiction by walking a copy of Warhammer Online over to my house. I dutifully installed it and played with them for a little while. But pretty soon the lure of WoW dragged me back, even though some of my friends stayed behind on Warhammer.
A few days ago, I received a great email from Warhammer Online. It's an example of an excellent synthetic notification. Take a look at this screenshot:
This synthetic notification gets everything right: it has a compelling offer and a clear call to action, it addresses me by my character's name, it's from a well-known NPC inside the game, it even includes the names of several friends who are still playing, and calls me to act on behalf of my guild (yes, it's called GUID). It then proceeds to list a whole host of cool new features the Warhammer team has added since I last logged in. Impressive.
Unfortunately, it didn't work, at least not for me. That's because I have much too strong an attachment to World of Warcraft. It's the ultimate high-engagement product. And yet, WoW never sends me emails. It doesn't notify me of anything, not even when my friends are logged in and about to start a raid. If I want to know what's going on, I have to log in and find out. It's up to me to decide when I want to do that. And how do I make that decision? Somewhere buried in my brain is a list, called something like "Things to do when you want to zone out and still have a feeling of accomplishment and power." At the top of that list is WoW. I'd only ever consider going to #2 on the list if #1 failed me completely. That's how most of us are - we only ever consider the #1 provider of any given service if it is available. Getting customers to see your service as #1 for a given category is what positioning is all about. (And manufacturing a new cateogry that you can be number one in is what resegmentation is all about)
In WoW's case, its positioning is established by the gameplay itself. WoW is a fun and addictive experience, and once it's sucked you in it's pretty hard to stop. But that is not the only source of positioning: brand advertisers have been using packaging and TV ads to do this for years. And most web applications do their positioning right in the first few screens of the app. This is why the registration is so important. At IMVU, we would routinely find retention effects that would stem from registration changes and have impact days or weeks later. One example I like to use is this: we added a YouTube video about IMVU to some landing pages. It was not prominently featured, but it did auto-play. We split-test that change, and watched the effects on engagement. Customers who saw the video were materially more likely to be active customers of IMVU ten days later.
The impact on behavior was pronounced, even though the immediate effect of the change was subtle. If I had to guess, I would say that if we had interviewed customers in the experimental group, they would not have been able to consciously recall the video they had seen during registration. But, unconsciously, it had affected the positioning of IMVU in their minds. Mission accomplished.
Anyway, for those of you planning on attending the Garage event, please come say hi. And for everyone else, please consider leaving your feedback - positive or negative - about the form or content of the presentation as a comment to this post. Your help is always greatly appreciated.
The team I work with has been using and measuring engagement loops (we call them "news loops") for about a year, now, and I'm one of the skeptical ones.
ReplyDeleteHere's why I don't think engagement loops are that helpful: unlike viral loops, it's not bad if it's not self-sustaining. In engagement loops users only get "credit" if their activity directly spawns more activity.
But in talking face-to-face with teenagers about how they use social sites, they say it's very rare for them to actually click on a notification. Rather, a notification "primes" them for the experience and once they get enough notifications (or enough high-value notifications) they just go to the site and do whatever they'd normally do.
Put another way, imagine the best case scenario. Your product has become part of a customer's daily ritual, like Facebook has for a lot of people. In that case they might never click on a notification. Instead, they just do whatever needs doing when they return to the site a few times per day.
In that situation the loop coefficient is zero, but that's a good thing!
In short, I think this metric is "too composite" and hides, rather than illuminates, useful information. I'd be reluctant to do an A/B test which optimizes the engagement loop coefficient, for example.
Thoughts?
I'd just add that since I come from a pure math background this idea was very appealing to me when we first came across it. Why, we just unified acquisition and engagement!
ReplyDeleteBut from a pragmatic perspective I think it's a dangerous thing to rely on when making concrete product decisions.
@Jesse - my attempt in this article was to simultaneously explain the engagement loop and also to give insight into when it should be used. I do think the concept allows us to unify acquisition and engagement, and is important for that reason. However, just like the viral loop, optimizing the engagement loop is a tactic. It only makes sense in certain situations. Your experience is not universal - in some contexts, notifications are very powerful. It all depends on:
ReplyDelete1. the fatigue rate of customers in that notification channel
2. the fatigue rate of customers with your notifications
3. your positioning
I think you'll find, for example, that the original growth of sites like Facebook and LinkedIn was dominated by notifications. They only became positioned as a must-do-every-day product later.
Sure, I understand that. But fatigue is only one possible explanation.
ReplyDeleteWhat I'm saying is that from the perspective of the engagement loop, failure and success look very similar. How do you distinguish users no longer clicking because of fatigue versus users no longer clicking because of the behavior I described above?
The engagement loop alone can't tell you, can it? This is unlike the viral loop where the coefficient really is the coefficient -- the higher it is the more viral you are and the faster your growth will compound, period.
And if we're going to be bringing in supplemental metrics to help us understand the engagement loop, why not just use the constituent metrics in the first place?
I just think there are less complex metrics, or set of metrics, that serve the same tactical purpose and don't suffer from this problem.
@Jesse - I don't think you're responding to what I am saying, or to the original article or slides. I am not arguing that we should use engagement loop metrics to guide product or business decisions, and I'm certainly not advocating that we use them primarily.
ReplyDeleteI see. So this is mostly academic, then? What good are these metrics if they don't help guide product or business decisions?
ReplyDeleteI'm not trying to be snarky, honest! I'm just not sure I follow you.
This comment has been removed by the author.
ReplyDeleteThanks for being patient with me, BTW. I feel a little stupid for pushing the issue this hard -- I'm sure I'm missing something.
ReplyDeleteThey are good only in certain circumstances. If you understand your product's positioning, then you can decide which of the levers of engagement makes the most sense to use to drive up engagement.
ReplyDeleteFor example, consider a site like Facebook in its early days. Most people are going to hear about the site for the very first time via an organic notification, when a friend signs up for the site (or tags them in a photo, etc). No positioning exists yet in the mind of the prospect. Most of the time, they will ignore the notification, but they will remember that their friend sent it. Thus, each organic notification serves to build up the positioning of the product as "something these friends use."
In that situation, focusing on the engagement loop might make sense. Again, I would not advocate for focusing on the engagement loop exclusively, nor for making its metrics the company's top-line goal. Instead, I always recommend making macro metrics the main goal, like the actual engagement rate of a cohort of customers (see the slide on "Measure the Macro").
Still, even in this case, engagement loop metrics serve two important purposes:
1. helping you diagnose why you are not getting as many organic notifications as you would like (because, for example, the engagement factor is < the fatigue rate)
2. helping you balance engagement vs acquisition. if you are over-optimizing your viral loop, you may be trading away important engagement benefits (see the original engagement loop article for more on that topic)
In any event, even in the case you cited above, where customers don't ever directly respond to notifications, I think they are still important. Customers remember that they received the notification and this impacts their behavior as well as the positioning of the product. I would bet (although I don't have the data) that for normal (non-hyper-addict) Facebook users, the time between logins is strongly correlated with the number of high-quality notifications they receive.
So, in summary:
1. always think carefully about which lever of engagement to use in your situation
2. try to understand how the engagement loop affects your efforts
3. even if you split-test small changes, always judge success by their effect on macro metrics
Eric,
ReplyDeleteOh, yes, ok. Insofar as thinking about the "engagement loop" helps you understand the constituent variables I agree it's very important. Each of those levers -- conversion rate, action/user, msgs/action, etc. -- are important when thinking about how and why users engage with your product.
It's the roll-up into one all-encompassing number that guides your product decisions that I'm skeptical about, but I see that that's not what you're interested in at all.
Thanks for taking the time to respond!
Best,
Jesse
Thanks for the good back-n-forth, guys. Very helpful extension of the original post.
ReplyDeleteEric, Great presentation tonight. I enjoyed the content and the message of placing more emphasis on engagement and positioning to drive value.
ReplyDelete