Suhail Doshi - How to Measure Your Product

We are very grateful to have Suhail from Mixpanel, who co-founded Mixpanel almost 10 years ago now and is going to talk about how to measure your product, which, as you heard from Gustav, is really the other side of the coin of growth and everything that helps you build something that people truly want and will use. So with that, Suhail!

"All right, hello everyone! Happy to be here! Awesome! I thought I would start off by just kind of like—I sometimes try to change my presentation a little bit on the fly depending on the audience—and so I thought I would just ask like a really quick question, just a quick show of hands. I'm going to define a user as just someone that uses your product, whether it be a B2B company, paying, free, doesn't really matter; like, they did something in your product. How many people have like zero users right now? Okay, cool! Awesome! And then how many people have a hundred or less? Well, okay, you have greater than zero, between 0, 1, and 100? Okay, awesome! Great! And then how many people have like more than 10,000? Okay, awesome!

All right, so we've got a majority of people with zero users right now. Cool! Awesome! Great! So I'm going to start off with just a quick about me. For those who haven't heard of Mixpanel, I started this company called Mixpanel. I think the first line of code was written in like October 2008. Technically, I was like 20 years old. I was in my parents' house, in my bedroom, and, you know, 10 years later, the company has about 300 employees. We've raised a bunch of money and we have about 7,000 paying customers, and we're starting to close in on around 100 million dollars in annual recurring revenues, just to give you a sense of, yeah, 2008 to now. Cool! Awesome!

If you don't know what Mixpanel is, we make pretty graphs like this. We help you measure what people are doing inside the product. So this is like, you know, someone inviting a colleague to their app. We might help you measure a funnel and see the conversion rate of people opening your app to like viewing an article. We'll help you measure retention. I know Gustav went over that a little bit today, and I'm just showing you this just because everything that I talked about today is basically something that you could probably do in Mixpanel or even any other tool. There are a bunch of tools that are great to measure these things, but I want you to know that if you walked out the door and you wanted to apply any of it, you totally could. There's like a freemium plan, and we cater very, very much to startups. Those are all of our early customers, so feel free to do that. Awesome!

Great! So today what I wanted to dive into was—I thought I would take a top-down approach to thinking about analytics and metrics. Instead of coming at you with charts and visualizations and numbers and things like that, I thought it would be better to kind of think about the problem that you're trying to actually solve and sort of ignore for a moment that you have to like do complex analysis and calculations. You will figure that out; it's not hard! Or you'll just use tools that will make that a lot easier for you. So that's one thing that I thought I would do.

I picked these three things. I tried to think pretty hard about like what the heck did I care about when I started a company? Like, what mattered to me? What was on my mind? And I thought, well, I think the first thing is just like do people even understand what I made? And Mixpanel, I had a really hard time with that in its first 18 months. We had to change a lot of things, and most people do. The second thing is, okay great, they kind of understand what it is, but do they find it relatively easy to use the product on day one? And that—that's kind of this never-ending mission that you'll end up taking when you start your company and when you ship your product.

And then the last thing is, you know, I kind of tried to pare it down a little bit, but I thought the most important thing would be then next, are people coming back and using my product? And I think that you'll find that—I know Gustav talked about this, and I'm going to be extremely repetitive because it's one of the most important things that early founders make a mistake with.

Yeah, so from there I thought that I would try to also— I thought that it'd also be helpful for you to see it kind of broken down in a formula. And so this is the formula, and I think I like to think about things from first principles, right? Like, what are the things that would cause you to grow? And they're pretty simple, right? There's just like people that visit some page or the app, right? Basic—you experience this every day. And then there's like people that sign up, and then, of those people, like, who did something? How many of those people did something valuable, like watching a video, or making a recipe, or, you know, taking a scooter ride? And then of the people—like how many of those people end up coming back some period of time later? And then how many people will like spread that product—like tell a friend, share it, maybe you even have a sales team, who knows? Right?

And these five things basically equal growth, and these are your levers. The thing that I wanted to kind of point out, and I want to stress, is that today I'm going to cover the basics, but I want to kind of implore on you that the basics are really important. Even thousand-person companies get the basics wrong. In fact, you'll find that if you don't get the—even these thousand-person companies that get it wrong, they tend to overcomplicate the number of things that they should actually measure and track. And it's really easy to make twenty-five metrics, conceive of them, align them, you know, divide them, ballpark by, you know, give each team five, and then really overcomplicate something that is actually quite simple to do.

When you overcomplicate it, what happens is, is that companies—even large companies—have immediate paralysis. Even midsize, large companies have this like decision-making problem. I used to have this funny phrase at Mixpanel where I used to say that all data is like contestable. You know, you could like see this number go off into the right and go, oh well, I don't know, it was like sunny outside and that's why people did it. People will like have these arguments no matter what because causation is really hard to discover.

What I find is that just picking like three to five things on a North Star and really simplifying it will do more good for you. And I know because not only have I watched thousands of companies make this mistake at all sizes, I have made this mistake. I had a piece of paper that had like twenty-five sub metrics, and it was nearly impossible to actually keep track of. What people end up doing is— in reality, what people care about is there are metrics that guide the team, and there are metrics that help you monitor and assess if anything is going wrong. And today we're going to talk about things that guide the team.

Okay! Cool! Great! So with this formula, you should be able to assess anything going wrong in the company, right? If you have a really confusing landing page, you should be able to figure that out. If your signup process is too difficult, you should be able to figure that out. If you have a product that people don't really value, you will figure that out with just that formula. It's just that simple! You can get more complicated after that, right?

You could say, okay, landing page views are down all of a sudden, why? Who is it by? Let me look at that segmented by country. Maybe something happened! Maybe I changed something in a different country. So you should be able to assess any reason why you're not growing just starting out with that. And that's why I say starting simple will last a very long time.

So this is what it looks like, right? Linking those two ideas together, you have this funnel of people going from visits to, you know, eventually retaining and spreading the product. And like on each one of these things, you're trying to basically solve one of these three problems: is it easy to understand, is it easy to get started, or are people coming back because they find it valuable? And then it loops.

And the funny thing about this is, even if you're like Airbnb and you're thousands and thousands of people, this process of like optimizing each one of these steps is never-ending. No matter what scale you're at, there will always be changing something, and you probably have competition, so you'll have no choice.

So awesome! And then once you are able to measure those five things and assess what's wrong, you have your levers and you should be able to take action and fix any of those things. Put a team on it. Maybe you're fixing it—it was really that simple!

Awesome! So we're going to start with the first thing, which is: is my product easy to understand? In order to illustrate this idea, I decided that it would be somewhat sympathetic, because I could show you like really awesome landing pages of like great companies, or I could just show you like what it's like to start out. And I also wanted to mention that having a really bad landing page—there's this really awesome quote that PG used to have in our batch that I thought I would impart on you, which is that like people basically just have their mouths like hovering on the back button, and they're just waiting to say, I don't know, what this is confusing, back! And that's immediately how most companies are just like losing most of their users from moving on.

So just remember that your enemy is the back button! So this is what Mixpanel's first front page looked like in 2009. That's not what our logo looks like anymore. This was a design by me—original design by me—and in all of its glory! If you like actually go and read the words, there are like tons of grammatical mistakes because I was horrible at writing. And for some reason in my mind I thought, I know what the tagline for the company should be! “Metrics that'll make you drool.” I don't know, I guess I think the reason why was because I thought these graphs were like so pretty looking compared to what was out there, that I thought, like, that would be like the value proposition of the company.

And of course, it's like totally wrong—like you're all laughing! I wish I could have like made this landing page, presented it to you, then you'd all laugh at me. Okay, I should change that thing, but I didn't have that! I had zero users at the time, you know? And I tried to target, you know, small startup, big company; I didn't know who I was targeting—application developer. I was trying to target everybody. Like, this is like every mistake that one could make on a landing page is like right here, right?

Like the signup button, at least I kind of got that right. Thank you, Endora, for teaching me how to optimize buttons! At least I made it like yellow, but that's about maybe all that I got right, right? Just maybe the signup button! And then, you know, after a lot of work, a lot of hard work, a lot of iteration, a lot of talking to customers, a lot of measurement, you know, eventually a couple of years later it kind of turned into this, right?

I like managed to get like the logo right, the signup button's like ginormous and yellow! And what we figured out though, after a lot of time and a lot of energy, was that we transformed “Metrics that'll make you drool” to “Actions speak louder than pages.”

And you'll notice it's like extremely prominent. The reason why was because the number one question that people that we would get as a company was, “That's cool, but like how are you different than Google Analytics? I don't get it!” Right? And so it turned out like this tagline for us at the front page lasted like the next five years of the company because it like very much differentiated us as something where we would measure engagement. We did something different; Google was about pageviews, we were about like what people did in your app, and that resonated.

From there, again we tried to explain like what various—we tried to urge people to go down the website and like look at different features and see if those things were valuable. We tried to be benefits-oriented and the copy tried to show customers and case studies who maybe found it valuable. We've probably made like four changes after this, but I thought I'd present this one just because it was the one that maybe lasted, like, the copy and the general essence of what we did instead—we did what we said we did kind of stayed the same largely but lots of design changes.

So, the real question is, you know, great! So like how do I measure how did you measure whether people really understood what they understood what you were doing? And so I think there’s like one tried and true number, and it’s like very simple; it’s just did people even bother to sign up? Bothering to sign up doesn’t mean that they’re going to use your products—that’s a whole different matter. Bothering to sign up is like, I don’t know, I’m kind of interested. I’ll kick the tires; this seems kind of interesting and cool to me. And that’ll tell you a lot about whether what you’re doing is even something valuable for people.

The second thing that you can try—these are just ideas. Trying to find the ratio of people that are going from just like hitting your page, right, and then doing any kind of thing after that. So instead of, aka, instead of hitting the back button, right? Like, did they click on segmentation or funnels or retention? In that case, right? Did they bother to do that?

I know there are a lot of B2B companies. These days—actually, one quick question—how many people are making like a B2B company? Yeah, great! So that's about half the room. So I thought I'd give a B2B slant. One thing that I discover was that people—a lot of your users will click on a pricing page link at the top. It's like second or third like most clicked-on link if you're a B2B company. I think, generally speaking—I, Jenna, I think you know this is totally contestable—but I find that like people clicking on your pricing page is actually a pretty decent indicator of like, okay cool! You have something kind of valuable. Okay, how much?

And so even something like that I think can tell you a lot. And then if you're in consumer, the benefit of being in consumer is that you generally usually will have like hundreds or thousands of users such that you could actually go do an A/B test, whereas in B2B it’s a lot harder to achieve that. And so I really—I think Gustav is absolutely right that you should try to conduct as many experimentation experiments as you can and really A/B test your copywriting and find out like what resonates with people. But these are all things that you can do to basically figure out, hmm, is this easy?

Cool! So the second, the second one is, well, is it easy to get started with my product? And the thing about this is like a lot of people—the thing about these first two things is that if you don’t make the first step easy, if it's easy to understand, a lot of people put so much energy when we’re trying to build a company. We try to think about like all the cool features and things that we could make that’ll be really valuable for customers.

The problem with that is like if you don’t even get step one right, you don’t even have a chance at any of the subsequent steps. So when we think about like is it easy to get started with my product, you know, forgot about password is kind of a useless feature if you have like a very small percentage of people that are basically like not even bothering to get started. In fact, I remember there was a YC company called Zen Tur, which turned into like Google Slides, and I remember Robby Walker during one of our Tuesday dinners said that they had—they didn’t even implement the save button for like their online PowerPoint because they knew that none of their users would ever ask, like, “Hey can I save this?”

They just didn’t even bother to do it because they wanted to see what anyone would ever use the PowerPoint presentation product. It was like that was in that one day, and part of that on me, I was kind of like, wow, that’s crazy! And for like, I don’t know, maybe like 18 months of Mixpanel, we didn’t even have forgot about password because we were just so ruthlessly trying to prioritize like things that really would really matter to customers, and it turned out we didn’t get very many requests anyway.

So this is really important, and here's an example of like easy to use something you use every single day: that is literally one step! You just go to Google, you type it in, and you immediately get to use the product. There’s no signup, there are no CAPTCHAs, there’s none of that, right? A more complex example, life is like Airbnb, who might have like, you know, ten steps or something like that.

We worked with Airbnb for a while helping them optimize initial user experience, and you know, even though Airbnb probably has like 11 pages of—you know, like you have to go to Airbnb and then you have to fill out this form and then you get taken to the results and then you click on those results. And then you click on many different results because you’re trying to price compare and you’re going through this very complex checkout process.

So there are actually many steps to booking something online, but Airbnb had to kind of figure out like how do we A, describe what we’re doing, what we offer, what’s useful for the world, and B, how do we help people get started as quickly and easily as possible? And like this is Airbnb.com right now. And it’s—you know, they have this very simple, like book unique homes and experiences. You fill out these very basic things, and you hit search, and that's it! That’s their getting started experience! It’s not their entire flow, but it’s the first way to get started!

And they even try to like kind of impute something like what you might find something imaginative on Airbnb. But the thing is, is that this seems simple, right? This seems like something that you could just like easily make. It doesn’t take—how hard is that, right? We can do that with Twitter Bootstrap in no time! The problem—the thing that’s really hard is like figuring out that that’s the right thing! That’s really hard—that takes like—they probably changed that you know 50 times before they figured it out—like this is the right current optimal thing at this stage for the company.

And so it takes a lot of work in, and you have to grind very hard to get to this at this point. So the question is: like, okay great, so how do you measure something like that? Well, I think the really simple answer is a funnel. How many people go from a landing page to signing up to doing that valuable thing—like watching a video, doing an Airbnb space? Who had been just doing a search—just one search—at least once! You get past that—measuring your funnel for the entire initial user experience.

There’s like weird things where like there’s actually situations in games—games can have like 20 steps, and I’ve noted—I’ve seen hundreds of gaming companies’ funnels, and even though they might have like 20 or 30 steps, sometimes they can actually convert very highly. Like they might get like 80-90% conversion rates through all those 20 steps. Just the first two are like actually the ones that have the greatest drop-off.

And then the last thing is just like actually speed—this is one that I think a lot of people don’t actually do. They’re just like, how fast can someone just get started, and that will tell you a lot about how complicated your experience is? Sometimes slow is okay; it depends on the business, but speed is really important. Does it take them like five minutes to get something figured out, or does it take a minute?

In Mixpanel’s case, we cared a lot about—we had a complicated flow because you not only had to sign up, but you had to write code—it was really complicated. And that was the one downside. We weren’t really sure; where people, with Google Analytics, you just copy and paste the JavaScript with Mixpanel, you had to like actually go and track a line of code!

And so we had this question in our mind: was that easy enough? And so speed became really important to us. How fast can someone do that? And I wanted to give you a couple tips and tricks, things that I have seen that companies do that are generally not great.

Number one, you're a startup— you don’t have lots of fraud problems. You probably don’t have a lot of like fake actors in the system trying to sign up. So if you don’t have those problems, don’t optimize for them until you do! And I just want you to know that E-mail and text confirmations have an enormous drop-off—like I'm talking like you get of the hundred percent of the people that come, like only thirty percent will like basically go on and like click on those email confirmations, or forty percent.

You spend a lot of time trying to optimize that, and then, to make matters even worse, like you have to hope to God that it didn’t go to spam! Like it’s just this really horrendous experience, so just be really careful! You’ll lose a lot of users that way. It’s really important to iterate on your initial user experience; that will be a never-ending process. You will do this forever!

Even as the press, especially as the product, it becomes really critical to like have someone on it. So at least one responsible person just thinking about that all the time! And then the last thing is, not every company—not every product—is capable of doing this, but to the extent that you can, just let users into the product. It’s always better to do that! In Mixpanel’s case, it wasn’t possible, because like we needed to get your data, so you had to sign up.

But we experimented with like, what if someone just copied and pasted the JavaScript but they didn’t sign up, and we just collected their data, and then after that we like asked them to sign up? We did experiments like that! So to the extent that you can achieve this, that’s awesome! And good examples of this are actually like Airbnb and Google, where you just try it.

But there are always exceptions to these rules. For example, like Pinterest makes you sign up, and they didn’t used to, but they do now—there are good reasons for that. And then, and then the last one, definitely not least, are people coming back to me. This is probably one of the metrics that is greatly ignored by most startups and it’s often the reason why I’ve seen companies even with millions of users actually die.

I’ve seen probably 15 different companies that grew virally and then just died. The company died, and I want to demonstrate this in a graph. I call this the shark fin effect, and the reason why I call it the shark fin effect is because, well, one, was because in Mixpanel, the default line color for your first like data point is like blue. And so, it always, to me, it always looked like a fin, like in the water, and it looked a lot like this.

And what happens is you fought—you like you’re slaving away and you find a way to go viral, right? Maybe you're going super viral on Facebook or on Instagram or something, and you're like, wow, I have struck gold! I'm acquiring tons of users; my life is awesome! I'm going to be a billionaire! This is great, right?

And the problem with this is that if it happens too early, but you didn't quite think about retention, it becomes really problematic, and this hurts companies. Basically, what happens is the app goes viral, and then eventually, the rate at which you're losing users becomes high enough that you can't acquire the new ones.

It turns out that like even though we like to try to be extremely rational people, it's really hard because when the app’s going viral, you’re just like, oh my god! I need to like just take advantage of this moment because what if I don’t get in? What if my competitor finds it? So I'm just going to optimize, optimize, optimize! But it turns out that it ends up not being very valuable because if you're just losing all the users, it’s really, really hard to reactivate.

If you talk to people that have ever done like a reactivation campaign—your activation campaigns generally don’t go very well. You usually just like lose those users until they decide that maybe they want to give the product a second chance. So it’s not like—like your response is like when this happens a lot of people's response is like, oh we lost the users because our app wasn’t valuable. Let’s just email them again and see if they’ll come back, and like that’s usually not a great idea.

So it can be really—it’s really important to really think through retention, and you really don’t want to be like this guy. The shark fin effect is really, really bad! In fact, LivingSocial is a great example; a lot of people don’t know that like LivingSocial was basically this app that was on Facebook that had nothing to do with like indeed like the Groupon deal-cutting thing, and they went super viral on—I can't exactly can't remember exactly what the product was.

LivingSocial literally had to pivot the entire company to basically compete with Groupon because the app wasn’t very retentive in those days. So there's some pretty basic ways to measure that people are coming back. The most obvious one that I hope most people know is that there's just new users. So making sure that you're tracking whether a brand new user—someone who just signed up—what percentage of those users will come back like a week later or thirty days later is really important to track over a longer period of time because a week later is just not a harsh enough metric.

And it’s important to see how fast you’ll end up losing users because you’ll have to figure out a way to re-engage to find new ones, and that becomes really, really hard at some point if the number of users you have becomes starts to become quite high. And then the second one is just daily active users. There might be people that disagree with this— I don’t think there are too many—but I now think that monthly active users has become like the new BS metric that’s like very similar to like just like a number of registered users.

Like the number of registered users—like a very silly metric. I remember seven or eight years ago—I think like I want to say it was like LinkedIn came out and said, we have 200 million registered users, and there’s like, who cares how many people are using LinkedIn? And I now start starting to think that like MAU is becoming quite close to that now. Daily active users are really hard to maintain.

I mean, how many products do you use every single day? If you just look at your phone and you go through all of the possible apps, like how many of those do you actually use every single day? So I think it’s a really harsh metric, and we actually found at Mixpanel—so even if you’re a B2B company and you might think, well, I don’t know, I’m selling to businesses, so like kind of really maintain like a daily active users, especially on Saturday and Sunday.

And we actually found that there’s like a really, really strong correlation with daily use and rate of churn—like rate of like people like stop no longer paying for the product. And it didn’t take very many users. It was like one point for daily actives would like reduce churn to like a dollar churn by like to like I don’t know, some 10 percent—it’s incredible!

And so I really stress measuring a DAU! And then the last thing is, a B2B slant again is something that I think a lot of B2B companies make an early mistake with. It’s actually not just a B2B company; it’s actually companies that do subscription. So I don’t know if there are any companies that would ever think that they may offer a subscription of any sort.

You know, like if you're a meditation app or something like that, it’s just dollar churn—its monthly dollar turns. If you’re charging your customers monthly, many businesses charge annually. But usually most startups start with monthly because an annual commitment sometimes can be hard for early adopters. It’s just measuring revenue churn.

I want to give you, I'm gonna give you a slight story. If you have revenue churn and your revenue churn is say 7% per month, right? You might think awesome! I'm keeping 93% of these dollars every single month; this is awesome! I'm doing really, really, really well. The problem is, it’s seven percent monthly churn, and if you extrapolate that out over 12 months, you’re actually losing fifty-eight percent of your revenue in a year!

It’s really harsh! A lot of people—like for some reason this thing is not mathematically intuitive to most people. I don't know why! It wasn’t even mathematically intuitive to me early on! If you think about it, like you're spending all of this energy with marketing and improving the landing page and improving the getting started flow, and you're doing all of that, and if you don't have retention—if you're losing that seven percent every single month, you have to somehow make up for that fifty-eight percent that you lost going into the new year.

And if you start thinking about it, you're like, wow! We’re making a million dollars, and we're losing five hundred and eighty thousand dollars. We're starting the year like that! Now you have to do that and grow on top of that, and once your base of revenue gets really high, it becomes really hard to keep up! And you kind of have this like really long arc of like a shark fin effect basically.

So it’s really important to look at your monthly churn, and this will be the single cause of like death! But the problem is, it's like the worst kind of death! It’s like death that occurs—it’s really like it’s really flat growth, but it occurs like eventually in like four or five years if you're able to acquire users. And I mention this not only because, you know, I talked to some founders about this, but this is also something that really hurt us in the early stages of Mixpanel.

And we really had to figure out a way to fix this, and there are all kinds of ways you can use product—you can use pricing; there are all kinds of ways to experiment with this—but this thing probably gave me the greatest number of nightmares and lack of sleep, I think, ever! So I really want to stress that on all B2B companies here; really think through this number and think about like almost graph, like okay, what would our rate of growth be if churn stayed constant? Really important!

Cool! I wanted to go over two things. One, I recognize that there are a lot of people here that have zero users, so they're like, "What the hell do I measure?" So, what if I have less than 50 users? Well, the truth is I couldn't really think of a better idea than just basically—you just have to talk to your users! And I thought I'd give you a mini story about at least that makes the panel what we did when we had like no users, and I had that page that just said “Metrics you'll drool over.”

One trick that I used is that I had like maybe 10 or 11 or 12 customers, and honestly, I just put all of them on—I am like on gchat or something. I don’t know what the kids use these days—maybe like WhatsApp—but honestly, just like would just badger them on like WhatsApp or, you know, gchat or whatever and then just ask them!

I remember one time we were trying to redesign our funnel UI, and we used to have a vertical funnel, but we had a competitor that has a horizontal funnel. And it seemed like that was a better idea; it seemed like that was more intuitive for customers. And so, I didn’t really know how to like get the data, because like how was I really gonna find out whether they liked one of those two UIs? And I didn’t have it—it was just me and a co-founder—so like how the heck were we gonna like do an A/B test on that feature? There’s no real way to realistically do that!

And so all I did was I just made like a really crappy version of the UI that I thought would be better—the horizontal version of a funnel. I was kind of greyscale, and I made like a really beautiful, colorful version of like the vertical one, which was kind of our control. And then I just asked like eleven people which one’s better! And I want to impress this thing because Gustav mentioned that it’s the Delta; it’s the difference that matters in an A/B test—not that this is like a true perfect A/B test, but like I asked eleven people, and I got feedback in a day!

Ten out of the eleven said horizontal funnels, and then we just made it, and we never looked back! It was totally the right decision in retrospect. So, I don't think there's any way that you can skirt by talking to customers! But here's the benefit of talking to customers—even though it feels maybe it’s more tedious than looking at a graph, you are going to get way more info and depth from talking to customers than you ever will looking at a data point in a graph, no matter how much you slice and dice and segment the data, no matter how mathematical you are about it.

You will never get as much information into in building your own intuition than just talking to customers. And it’s really critical that you do it! And the one thing that I wish I had done back then was I just wrote—I wish I wrote down tons of the feedback because I think that it would have helped/made my co-founder, and it would have helped employees in the company down the road. It would really like go on the journey, make them feel like they went on the journey with me rather than having our roles kind of feeling like they were so split up.

And then the last thing that I thought I would bring up is to impart what I kind of said at the beginning, which is that I think— I think that one of the biggest mistakes that people make with analytics is being really overcomplicated; thinking that they need to be super sophisticated; thinking that they need to track all kinds of crazy cohorts, and they need to have a dashboard with like 30, you know, panes on it that all—like, loads—you have like Mission Control.

I don’t think that is necessary. I had that; it didn’t work. It was really hard to run the company that way. And what also happens is that when you build your team, it will be really hard. You will be confusing for your team of those 30 panes—like which one do I need to care about? And I think that we need things that are—humans just need to simplify things.

So I'd pick one North Star metric, and I think the North Star metric I would choose in this case is like, what is a number that you're willing to bet the company on, right? Like if that number goes south, you deserve to die! If that number goes up, you will like—you will have like made a huge dent in the universe! Like what is that metric?

And I'm not saying you need to choose that metric forever—but choose it for six months! Choose it; commit some time to it! If you find out that it's actually like the wrong metric—which it will be probably the first time you choose—it's usually your metrics are wrong—and then you can change it. But commit again for another six months!

But choose that one number, and then like if what you have to do at the beginning is basically just like print it out and put it all around the office, do it! Because people will start to be maniacally focused on it. They'll show up to meetings and go, yeah, but that number is like down! What are we gonna do?

And where you can get really complex is discovering why that number is down or why it's going up! That's when you can get really sophisticated. That's when you can slice and dice it and figure out what the retention is and things like that and measure it over a funnel, but keep things really simple!

And then the second thing is like, don't boil the ocean. Like pick three to five other things. I think less is more and just stay there. Telling—even large companies—most large companies don’t do this and it usually is very discombobulating for them, for the workforce. It’s totally fine to have numbers that you wish to monitor, but don’t focus on them!

Awesome! I think that’s it! And then for the people that were like slightly bored because I went over the basics, I thought I would help you out by just kind of having some advanced topics for later that you could read about. Really, these are things that I found really useful.

My favorite one is this one: the next feature fallacy by Andrew Chen. And it’s just this idea that we always think that like the next thing that we make will be the thing that will change the trajectory of the company—i.e., like the next features right around the corner, and we’re gonna be huge. And I would really recommend that you read that article; it’s really awesome—one of my favorites.

So for advanced stuff, these are just people worth following, reading about, and can give you more ideas than what I've given you today. Awesome! Thank you, Jeff!

Kuhn, cool! All right, yeah! I want to be fair. I don’t want to be like if every speaker—like choose them in the front! Yeah, yeah! So that’s the problem with that is that um—it’s—so—oh yes! I think someone asked what are like good benchmarks for like those given those five steps: visiting, signing up, you know, using the product—what would be good conversion rates for any one of those steps?

And the chart part with benchmarks like that is that it’s so dependent on the business. You know, in B2B like going from like a visit to a sign-up, like I think I got like a four to five percent conversion rate is pretty good. I think I like swap notes with Stewart at Slack because we kind of had similar landing pages. I think I asked him, you know, what was your conversion rate at Slack? And I think we both ended up being somewhere around like four to five percent from visit to sign-up.

But that said, like I don’t know what that would be for, you know, something like Airbnb, where it’s like getting to a landing page and then, you know, searching for your first place that you would want to book! So I think it’s really dependent. I know that Mixpanel published it on its blog, publishes lots of benchmarks, so I know that—the marketing team there that does that—and they like publish all kinds of benchmarks for like—what should your retention be if you’re a gaming company, if you’re an e-commerce company, if you’re a B2B company—things like that—social company, a video company, things like that.

So I think that you could probably go there and find various benchmark reports, but it’s really depending on the business!

[Music]

So, when you’re going through the signup process, how do you see numbers vary if you're large like creating an account and password, doing all that kind of setup, versus a single click-like button login with like, cool—how does that like transfer? Do more people sign up if they have that like super quick option? Do you see where they will drop off because they have to use one of those?

So, yeah, are you—I think the question is, um, what converts better? Like those like social buttons where you can just like quickly authenticate—like Facebook Connect versus just like standard signup, is that right?

Yeah, like all of these questions they require like a lot of specificity because it’s actually like really unclear! And actually, and in that case actually, I don't know—I truly don’t know! And so I think the way to answer this question is to find friends in tangential industries that like don't compete with you, but basically do that; and then just be like, hey, what have you—what is your conversion rate? Have you tried something?

I don’t actually think a lot of those numbers are like secrets in reality, right? You know, they could either give you the information and save you you know, six months—or they’ll just like say—or you try it out! But it’s just, it’s sort of strange to keep it a secret, so I would rather give you ways that you can discover that—I would just ask friends that have like have a startup or even if they work at a big company; sometimes they’re willing to share it!

Yeah, so the question is Airbnb’s landing page was super simple, and how do you balance like— I mean, what if a lot of people just know about Airbnb and then just know what to use Airbnb for, and that’s why they can keep it kind of simple, versus like having something more complex?

A good example of this is like Craigslist—it’s crazy! If you go to like the front page of Craigslist, it’s like absolutely nuts! It’s in like even Amazon! If you go to amazon.com, right, like even though Amazon has like huge brand recognition, you go to amazon.com and it’s like, whoa! There’s like a million things going on at once, right? So how have these companies found the balance? How’ve they found how to deal with this complexity?

And the truth is, is that I find that it’s important to have like a guiding principle—like what is it that you're trying to actually optimize for, right? If, in the case of YouTube, let’s take YouTube as a good example, right? YouTube could just like be like Google; they just like, “What do you want to look?” “What do you want to watch?” Right? They could do that, but they don’t!

And so I think that the question really is is like what is YouTube? What is the team at YouTube trying to optimize for? Are they just trying to get you to like watch a video? Are they—is Airbnb just trying to get you to search for any kind of hotel, or a hotel?

But Airbnb—and so I think that like the team has to first develop a hypothesis of like what matter—what do we think matters? And then from there, it’s important to experiment and try to figure that out! It turns out like it’s not—again, it’s about the Delta of like what people will actually gravitate more towards.

Like if you make a really simple experience but like people are like really confused because they don’t even bother to do it, or you ask them, then you kind of have like new information. And then I think the key is to find the minimum number of things you need to add to get someone started.

And you know, I don't think there's anything—I’m sure Amazon has tested that front page like crazy and they found that it’s like quite optimal!

[Music]

The can—it depends on the product! I'll give you an example in Mixpanel’s case. Oh, sorry, the case. Yeah, the question was, are video—are like one to two-minute video tutorials effective?

Yeah, so I think that it’s—it’s complex! We did a video tutorial video in our flow, for example, and you know, like I would say that like a lot of people didn’t watch it, truthfully! And this is what I mean by experimentation—like a lot of people didn’t watch that video. Vittorio and I actually, I think, eliminated it, you know, the first three steps!

But what we did find was this interesting thing that occurred as a result of looking and measuring things, which was that—well, we have a feature in Mixpanel where you can like see every action that a user takes. And so I would literally just like go to every user and then just like watch like a creepy person just like sort of watching like everything that like someone would do in the product!

And eventually when you do that—like when you do like hundreds of those and you’ve just watched hundreds of those, you start to develop some sense of like, okay, you—what you develop is an idea, a hypothesis. And all of these things—like all of these things require art and science! I think that companies that take it too far, they like take it too far to science end up getting it wrong!

They end up optimizing for like weird things—like you could always make a button more yellow and bigger, but that doesn’t mean that doesn’t make users happy eventually! And so anyway, to answer your question, what I found was that lots of people were just clicking randomly on like all the myth side menu options!

And my hypothesis was, well, they hadn’t integrated Mixpanel yet, so there was nothing to look at! So what the heck were you doing just like clicking on like funnels and retention? Like why would you do that? And my hypothesis was they were just curious! Like, what the product did, and so we did this A/B test where we like had a control with like nothing, you know, just the regular, you know, clicking around, seeing like some image—and then we had another one that had like videos. And then we tried to see like what caused people to like do their integration, their first integration, what was the conversion rate.

And it turned out like the video—like super one! Actually, we didn’t even—a more simple test than that, I think we just emailed people a video and said, “Watch this video.” Super basic! We didn't even bother to implement it on the website and had amazing conversion rate, and then we decided to add videos to all those tabs where there wouldn’t have been anything anyway.

And so that’s why I want to say, like, I want to stress this—it really depends! And what matters is that you run the experiment to develop a hypothesis, i.e., the art—not the science—and then measure it, which is the science, to find out whether the thing that you thought of was in fact right.

The percent loss in revenue over a year?

Yeah, so what—these are slightly two different things. One is one is like user retention, and then one is like dollar churn. And so in the case of dollar churn—like the reason I bring up dollar churn is that even like you could totally have a company where nobody uses the product, and the company continues to keep paying for it, right? But I find that like most companies are like pretty incentivized to cut their spending—and employees are incentivized because they lose it in their budget!

And so with respect to dollar churn, you can have a lot in a B2B company, for example. You can have a lot of use—you can have—you can have even one user using the product, but the company could continue to—you continue to spend money on it, and like you might see like a forty percent retention rate—like that might be okay, right? As long as the company is like continuing to spend.

But losing forty percent of your dollars in a year is really hard and not good retention! So make sense? So those things are not exactly linked—they’re not directly linked!

Yeah, yeah! Like to give you to give you kind of a benchmark, like a chat app that’s like often used a lot, will have like sixty, seventy percent retention—like those are some of the most retentive things like something like Slack would probably have like eighty percent retention. It’s kind of like unbelievable!

Whereas, um, you know, like a product like Mixpanel, when we first started out like the first year, I think we had like thirty, forty percent you know retention and that’s not that great, but who’s their first few years!

Yeah, yeah! So question is, are these metrics can they be used for a hardware startup?

Yeah, I don’t—I don’t see any reason why! We worked with Jawbone for a long time, and we worked with Fitbit, and most hardware—does your cart hardware company have a software component to it?

Okay, so it’s a hardware company with subscription model?

Yeah, yeah! Absolutely! It may be more challenging as a hardware company to track, but it’s like—sometimes—but it’s totally doable! Most hardware is like connected to the internet somehow, so I don’t see any reason why! A lot of the scooter companies, for example—they're like all connected to hardware and they have a software component, so there’s really no difference!

Yeah, so question is, the first page was like trying to target like various kinds of sizes of companies, and then like the second landing page of Mixpanel like in 2012 basically didn’t kind of dispose of that, and it had something very different. It was marketing like features and stuff, and how do you figure out like what to do basically from that point to the next point?

Yeah, so I think that the basic gist is that so you can clearly see like I had no idea who our customers were. And then after like a few years, you just—you talk to users! You ask them like what size of company they are if they sign up. If you want to, we didn’t do that, but we just spent a lot of time—we just—it was so obvious that like everyone that was signing up for Mixpanel at that time was a startup.

So for small—in our—in our case would be—we call that a small business. I think it took like maybe no longer than six months to figure out that we were not gonna sign up a large company.

Yeah, we just didn’t have the product; it just wasn’t there! We hadn’t worked on it long enough, so the only people that were willing to basically try it were companies that could take the risk to try it. And most of those companies didn’t have anything anyway, so it was even easier! Is it easier to target companies that had nothing and that were willing to bear risk? And then most of the time, I would always employ this trick where I would just—I would charge the money for it, and then if they said no, I would just give it to them for free because the feedback was more important!

But I always wanted to see if they would pay for it, and so the only people that are willing to do that are also small businesses. Turns out, so for us it was just like—it kind of like hit us; it slapped us in the face! It wasn’t like this hard thing to figure out!

Yeah, yeah! So we did it—to be clear, we weren’t necessarily building things to bring new users on. Oh, sorry! The question is you guys built forgot about—you didn’t—you just—you decided to not build forgot about password.

How do you know when to build quality of life features versus features that just acquire new users?

Yeah, so I think that we didn’t—we didn’t— it wasn’t so much that we only built features to acquire new users. We built features to acquire new users and to keep existing users, so we were very focused on like parts of the product that would actually make a difference, right? That would be clearly differentiating compared to all of the other possible options out there.

And like one of the things that’s like not different is whether you have forgot about password, right? That was like in the bucket of things that like maybe don’t matter. And so I think the question is like really like when does it become a problem enough for you that it’s worth building?

Right? Like if you’re finding that like it turns out that the number of requests that you’re getting for forgot about password is like increasing and then manually doing this process is like annoying, then start—then you should just build that feature because you know that if you don’t build a feature they can’t login, then you can’t retain them, and then you lose them.

So it becomes this like flywheel effect where like the only way to retain them is to do that! And in fact, even though Mixpanel didn’t build forgot about password for like maybe you know, twelve to eighteen months, we went eventually—we found—and found this clever trick that Facebook used which was that when you failed to log in to Mixpanel like three times on your third time, we just sent you an email that says, you know, we just sent you an email, just go to your email, click the button, you'll log in right now. Don’t worry about forgetting it, don’t worry about changing your password, just like login!

And it turned out that like that had this like great retentive effect because we were really focused on retention at that time!

Yeah, okay, a couple more questions!

Yeah, how do you gauge product-market fit? How do you measure product-market fit?

So I think there are a lot of people that probably have already will talk about or have already talked about like, you know, people will say very—and I think they’re there all right, so I’ll give you a kind of—I’ll give you a quantitative slant versus like a qualitative slant.

Some qualitative slant or things like when people are willing to spread you to their friends, that kind of thing. The quantitative version of this is really just measuring your attention—like how many people—it’s really measuring like DAU. It’s really measuring the percentage of people that come back a week later or thirty days later.

It’s the number of people that like become almost like—they like need your product or they like feel like life would be a lot worse. And I think that you can—you can totally measure that by how they use it!

And so I think the quantitative measurement is basically finding like some benchmark—some sense that for your industry maybe that's like thirty percent. Just remember like that benchmark is probably like average, and your desire is to basically be above average. And so if you can like significantly beat that benchmark, then I think that you’ve like really—you’ve done better than probably product-market fit! I think that’s one thing.

I think another thing is like you can try to measure like overall like repetitive use, like frequent use in a day. So you can make pretty harsh metrics—you can be like for example with DAU, you could say I don’t want it to just be daily active user—like they did one thing, I want it to be that user has to have watched five videos, and then—and then then only then, or do they count as a daily active user?

You can be particularly harsh about that metric, right? Someone coming in like just see—that’s why I say MAU is kind of a BS metric now because you could come and watch one video and then never come back again, and you that count, and that's not like a very healthy business! So I think that find ways to be harsh about your North Star, and I think that you'll find that you’ll reach the product-market fit.

Yeah, yeah! I think with Mixpanel it was like thirty or forty percent retention was really really horrendous, and we totally didn’t have product-market fit yet! It’s like a Peter Teal question: what’s one piece of conventional wisdom that you think is generally wrong about startups in general?

Gosh, that’s a hard last question! It’s like everything else is like fine! Gosh, I think—I don’t know! I—I need to think about it, but I think that one thing that has like become pretty clear to me is that I think—I think after having done Mixpanel for about a decade, I kind of figured out that I tried to impart it on you in the talk today, which is just that it’s it’s much better to just like—I think that we tend to complicate things as humans, and we don’t—we tend and that comes that comes through in like everything. For example, like we tend to have—we tend to like make a version one of a product, and then we get to like version two, and we’re like, okay, we’re done!

We like—we did it! We solved the problem! And then like often a founder will like kind of get distracted and then go and like work on the next product on the next thing, and they haven’t yet figured out like their retention is probably not great or they’re operating at negative margins or like all these things. That same thing even happens with numbers—like we pick our three metrics, then we continue to get more complex as a business, and then we add the next twenty things, and all we do is all of these things to like in hopes of like feeling like we control the situation better.

And I just think that those are all like distractions and mistakes, and I think it’s really hard for the average founder to actually just be focused on something boring! We tend to have like a TD or something, and like we just, I just decide to get very easily distracted.

And I think that it’s hard to do something—do the same thing for four or five years because we don’t like feel challenged anymore, and I think that that’s a huge mistake that founders often make.

And I don’t know that anyone would say, oh that’s—that's like totally wrong Suhail! But for whatever reason, like every founder repeats it, so there must be something—I don't know—something there must be some kind of wisdom that we all think; this is like the right thing, and I made those mistakes!

I made new products that I shouldn’t have! I met too many things, over-optimized things that didn’t need to be optimized! So yeah, that’s really it!

Cool! Thanks! [Applause]