Ilya Volodarsky - Analytics for Startups

Hi everyone! My name is Ilya. I'm one of the co-founders at Segment, and I'm here to talk to you about how to set up analytics and the analytics foundation to build your MVP and to measure these primary and secondary metrics.

So this is going to be a little bit more of a tactical guide around what tools are there in the analytics space and the marketing space, which one should we actually be using, and how do I set them up?

Cool! Just before I get started, a lot of you may not know what Segment is. It's an analytics API where you can send analytics data and then toggle on different tools. We send the cool market position where we can actually see what tools startups are using and we can bring that information to you. Secondly, we've been around for about six years, and so we have information about a lot of different startups as they grew and what tools they used. So we're gonna be sharing a lot of those today.

Cool! So why talk about why even focus on the analytics? Obviously, primary and secondary metrics drive the MVP and product market fit process, and you're using that to actually test product market fit. You're also using it once you get out of product market fit strict search to actually focus the team.

So maybe there's gonna be an acquisition issue in the company that's preventing your growth, or maybe the users that you're getting aren't as engaged, or maybe you're having some monetization issues. The funnel actually is a forcing function to understand your business and where founders should be actually spending their time.

Then finally, all the way from, you know, a three-person team to Google with 1 million employees, you're actually using metrics to operate and drive teams. Eventually, you have an engineering team, you have a marketing team, and so what goal do you set in front of the marketing team? Use analytics for that!

Okay, so today we're gonna cover a few different things. First, you always start with the funnel when you're thinking about analytics. That's the sequential series of steps your users go through to actually get value and then pay you as well.

Then we're gonna talk about collecting data for your analytics tools. We're going to talk about the top three metrics, which will include primary and secondary metrics that work for most products, and then a product market fit methodology that you can apply on top of that. Then finally, we'll make recommendations about what tools are the best in the market right now that help the product market fit journey.

Okay, so to start where we started, the funnel! We'll make an example funnel for Netflix, which is a company super familiar with any B2B product or B2C product. It actually has this type of funnel where you acquire a user, you engage a user over a period of time; that loop is called retention, and then finally you monetize the user.

Metrics, both primary and secondary, are performance indicators on top of each stage in the funnel. So, on top of acquisition, you can ask yourself, "How many new users did I get this week versus last week?" and "What's my growth rate there?" For engagement, you take a cohort of users.

So, from, you know, Sunday to the following Monday, you have 16 people sign up, and then you can track that cohort of users week over week and see what percentage of them are still using the product four weeks later, which is a good example of how to track retention.

Then we talked about monetization, which is how much new revenue did I make this week versus last week. Okay, and then you apply your own custom business funnel to this. So if you're Netflix, we're all familiar. The users sign up for Netflix, then they play videos in a loop.

Netflix is obviously very sticky, watching it a lot. Finally, when the trial runs out, you do subscription upgraded, and you get access to more content.

Okay, so how do you collect data once you have this funnel? There are analytics APIs out there. I'm using Segment as an example. You basically want to say user 1, 2, 3—in this case—has done a user signup event, and they happen to be an organic user, which means they're not invited by someone else.

Then, if you're Netflix, you might say the user’s video played and eventually subscription upgraded. This is how you instrument your tracking in your mobile app or your web app. Then you think about event properties.

Imagine your Netflix, and you're holding one of these video played events in your hands, and you're wondering questions about it. So you know, "What video art is the user actually playing? How long is the video? How far did the person get inside of the video?"

Right, equivalent, if you're holding a subscription upgraded event, you're going to want to derive monetization as a North Star metric. So if you're a subscription business, you want to send your monthly recurring revenue. If you're a transactional business like e-commerce or retail, you want to send the actual value of the transaction.

Okay, so then you push this out into your web app or your mobile app, then you start seeing the data come in. You look at the debugger; you see, "Okay, user signup is here." Every good, you add your first analytics tool.

We'll use Amplitude as a good example here. Amplitude and Mixpanel are pretty awesome analytics tools out in the market right now. Then you start seeing data flow inside one of these analytics tools. So this is Amplitude. You can start seeing user signups growing as soon as you launched the real mobile or web app.

Okay, so now that you have analytics set up, it's time to focus on three different metrics. The first one is the acquisition metric: signups per week. It's really nice if you're a B2B business to cut this by the invite type.

So you have organic users, which are just signing up from coming to your website direct signup, and then some users are inviting other users, so those are invite types, right? When you're thinking about growth, it's really important to think about the organic user in that case.

Another example: why if I'm properties are important? The way you create this is you go to Amplitude, you say event segmentation report user signup. Next, here's your graph, right? So it's as easy as that; just website some data gets to Amplitude, and then you can see the number of organic users every week.

If you're working on the acquisition step as a secondary metric, you can basically say today 218 users organically signed up in the last week, but by the end of the month, we want that to be at 300.

We're gonna execute projects A, B, and C this month, and then we're gonna watch this graph every single day on a TV dashboard in our office or apartment, wherever we work. Then we're gonna see if our efforts are actually driving this, right?

So that's an idea of data-driven operation of a team. You set a metric and a goal, and then you drive towards that every day.

Okay, the second one is retention to cohorts. Someone recently asked about retention. What we'll talk about that right now. So with retention, you want to think about cohorts of users.

So you want to say Monday to Sunday, let's say December 10th through the 17th. Sixteen users signed up and then you look at those 16 users as they use your product on week 0, which is their signup week, week 1, week 2, week 3, and week 4.

The general idea here is like you can convince your mom or your grandma to use your product once, but even your mom or your grandma won't continue to come back and use your product over time every single week, right? So if you see users that are addicted that are coming back week over week, that's a really good sign of product market fit.

So this business can see that the December tenth cohort only 6.25% of those users are still around on week four, and that's a pretty low amount, right? So you probably want that to be somewhere between 20 or 30, at least.

You can set a goal saying, "I'm going to talk to a bunch of these target users and try to figure out why they're not getting value out of the product and then make some changes as well."

Okay, so what metric do you actually pick? This is taken from one of Gustaf's slides; pretty awesome. You think about what value your company is giving to your users.

So Airbnb gives you value by letting you stay at different rental properties around the world, right? They want you to do that at least one time a year, otherwise they consider you a churned user.

Equivalently, Facebook gives you value by letting you look at the newsfeed and connect with your friends, and they want you to do that at least daily or monthly once.

When you think about product market fit, you basically have these two different curves that happen. We have that cohort of 16 users that signed up in one week, and we track them over time.

What ends up happening is, for products that don't have product market fit, they tend to go to zero because people just don't care about the product, right? That's the definition of product market fit for those tools.

The products that do have product market fit will see some kind of natural plateau. Don't mind this axis; it should be somewhere between 20 and 30 percent at least.

Okay, so how do you create this graph? It seems kind of complicated, right? Luckily, both Mixpanel and Amplitude have really awesome reports for this.

In Amplitude, it's a retention analysis report. You say users enter the cohort with user signup, and then they return with the video player to the subscription upgraded event—that's the value event. Then you press next, and out comes this graph.

You could look at four-week retention for cohorts, improve the product, and watch as new cohorts that strike the four-week mark do better or worse, right? That shows you whether your changes week over week are actually improving.

Okay, finally, revenue. This is the primary metric you want to be thinking about. You'll use this for a subscription business, the subscription upgraded event.

You'll do a property sum over new plan monthly recurring revenue, you press next, and out come your weekly revenue graph. Then you could set monthly goals on top of this to make sure you're growing at the rates that you want to be.

Okay, finally, if you have a founding team, it's really good to basically put all this stuff on a dashboard and then put this dashboard on a TV in your office. It's incredibly, incredibly important.

Basically, this is kind of the the difference between being a data-driven team and not a data-driven team. A lot of founders actually set up their analytics, but then don't look at them ever again because it can be painful, right?

While a data-driven team will put it on a TV and talk about projects, talk about what those projects are actually changing, the metrics that they're trying to drive, and then just completely understand the business every single day, right?

This kind of company and this kind of founder will actually scale to build better, high-performance companies because the next team of employees they hire will also be looking at that same TV dashboard and be driven off those same metrics.

So really important: get a TV. Next, what you want to do is have some kind of social accountability around your metrics.

So if you have your friends, your parents, your advisors, your investors, package up how your business is doing into an email. This helps you synthesize what is actually happening and then send it out to those advisers and tell them where the business is struggling and what your plan is to fix it.

This allows the advisors to quickly understand the business and then respond back with much more appropriate advice.

Cool! Now we'll go into the startup stack. These are tools that we recommend that help this kind of tactical process of setting up these metrics.

So I'm gonna talk a little bit about that MVP business workflow that Michael talked about earlier. Initially, you're building an MVP. Segment built about seven different MVPs before we actually found Segment, and all of those failed.

Eventually, we found one that worked, and the process of actually building that MVP is incredibly important. Once you have that little experiment built, you want to enter private beta, which basically just means getting like 10, 20, or 30 customers to actually try this product and then having very direct lines of communication open with them.

What Segment does nowadays: every new product we ship, we open Slack channels with each one of our customers, and we have the product managers sit in those Slack channels and talk with the customers.

For the products that don't get product market fit, the customers just stop responding. We're asking, asking, asking; they're not responding, right?

For the products that do have product market fit, the customers are immediately like, "Oh, why don't you have this feature? This is broken. Might I try inviting my team?" This is not working.

Instead of you kind of pulling at the customer, the customer starts pulling at you. That's a good feeling of product market fit.

Okay, so at some point, the private beta is going well. You feel like people really care about this; you understand your target customer. Then you want to get a larger market segment to use it. That's the launch that we talked about earlier.

Try to get there as quickly as possible. A launch is just more users that you get to test product market fit on, and so if you feel product market fit there, then you can start scaling the company, right?

You hire salespeople, and you start doing paid marketing and things like that. So different tools will guide you throughout this process.

As you're building an MVP and you're about to give it to the first group of customers, install Google Analytics, install Amplitude. Google Analytics will tell you who's coming from the Internet to your website, and Amplitude will tell you which features are they using, how engaged are they with that feature set.

Unless you're able to stand over the shoulders of all of your users 100% of the time, analytics is the next best alternative for that.

We also install live chat on the page. So either Slack with your customers, or if you can't do that, then maybe have a live chat available.

In the beginning, Segment customers would ping us, you know, day and night, and that's where we got the most valuable feedback from them. So just as many open channels of communication as possible.

Next, data warehouse; this is something that we recommend. It used to be expensive; it is no longer expensive today. Basically, if you have a non-technical co-founder on your team but want to ask questions around the data, they'll always ask the technical co-founder who will have to provide the answers.

So data warehouse kind of democratizes the data, not only for the non-technical co-founders but for everyone else in the company that you hire after. Company dashboards, obviously.

We should probably move that to the left: email and push tools. So as soon as a user signs up, you want to send them an email.

I'll talk about that in a second, and then a help desk. At some point, you'll have so many support tickets if you start feeling product market fit, and if they're all going to your Gmail, one founder will just get overwhelmed and not be able to answer them.

So you want to have a shared inbox where multiple founders can respond. Okay, now I'm gonna go through a few different recipes of these different tools that we found really helpful in product market fit.

So the first one is improving product usability. Almost every product that's launched is unusable or highly unusable for the first three months while you have the kinks.

We see this with every single product, no matter how much effort we put into it ahead of time. As soon as customers hit it, they start using it in ways that you just don't expect.

So there's this tool called FullStory, which helps you look at sessions of customers as they use your website. So I'll tell a quick story on this.

This is a new feature in Personas, which is one of Segment's products, and we launched them. The metrics looked horrible. Customers are coming in, but they weren't actually completing it; they weren't using the product.

We thought, "Oh God, likely this doesn't have product market fit; we have to go back to the drawing board." Then one of the designers and her team had this amazing idea: let's look at the FullStory.

So we see this user going in about to start this creation workflow. They find this button; they clearly don't understand what the button does, they get so frustrated, they just exit the page.

We saw this with multiple different customers coming in, and we were like, "Okay, we just have to fix that button." We fixed that button immediately—all the metrics got better, right?

And so that's why it's important to have this type of viewing, either to stand over your customer's shoulders, which is great, what the Stripe co-founders did, or get a FullStory, which is a more scalable way to do that.

Okay, I call this the 43-minute founder email. So when we launched Segment, we would wait about 43 minutes and we would email the customer and say, "Hey, I'm Ilya! Thanks so much for signing up for Segment. Your next step here is to add a source to Segment, and if you have any questions at all, please email me or call me anytime. I'm available for you!"

Since we launched that email in 2013, we've had hundreds of thousands of responses to it, so it's the connection between you and the customer over email that if they get confused, they'll respond to it.

What you can use is a tool called Customer.io; it's a behavioral email tool which will say every time a user signs up, wait 30 minutes, 40 minutes, 50 minutes, whatever, and then automatically send them this content.

You could template the first name, the company name, and so forth based off of your analytics data. So huge recipe, recommend!

Then finally, for democratizing data access, I say this is more advanced. This is after you're in your MVP stage; you're feeling good about product market fit. You might want to install a data warehouse like Google BigQuery, and then Mode Analytics is a BI tool that works on top of it.

This lets you ask questions on top of the raw data that you might not be able to do in Amplitude and Mixpanel; just any kind of question you can ask in SQL.

Then even the non-technical co-founders will eventually pick up SQL and then start asking these questions themselves.

Okay, one common failure mode that we see with customers is trying to pick the perfect tool: Mixpanel or Amplitude, you know, BigQuery or Redshift, and spending way, way, way too long thinking about that.

The truth of the matter is you shouldn't optimize for picking the right tool right now. Both Amplitude and Mixpanel will give you exactly the same result at your stage.

Instead, just get through that decision as quickly as possible, but set yourself up for change in the future. So this is an example diagram that shows a customer of Segment that used different tools over a period of about three years.

You can see that they used about eight different tools between 2015 and 2017, and they either hired someone or they decided that their tools were no longer doing the job and they switched from one set of tools to another.

So best-in-class tools change every two years. Just be prepared for a change and don't spend too much time trying to perfect your choice right now.

Okay, this is my recommendation of what tools are the best for the MVP process. I'll walk you through them right now.

So Google Analytics is for understanding what users are coming to your website. Amplitude is for feature analytics. Google BigQuery is to democratize data access with a data warehouse, which is just a database of data.

Mode is the tool you use on top of that to ask questions on top of BigQuery. Intercom is like a list of all of your customers; they're a really good CRM for early-stage folks. FullStory is for improving product usability, and Customer.io is for emailing your customers.

Once you get big enough, you can start using Google Ads and Facebook Ads to actually do the data acquisition.

Okay, so I'll just jump straight to the slide. When we were really young, when we were in 2011-2012, we didn't have a lot of money, and so we didn't want to use all these different tools.

We used Google Spreadsheet for a CRM. We didn't use Asana or Trello because they were too expensive, so we used emails. The only things we paid for were basically GitHub and AWS to host our products.

Right, so today, happy to say Segment is now free for early-stage startups. So for all you folks, there's a Bitly link down here that I will send afterwards and it's in the deals.

Also, Segment went out, and we did a bunch of deals with these customers so that if you're early-stage, you can now get all of these tools for free.

So enjoy that and start using them to basically accelerate your product market fit process. Thank you!

[Music]

You're asking about growth rate. So you're asking if you're starting with a very small base of users, what growth rate you need to be on to make growth meaningful?

Yeah, it completely depends on the product, right? So if you're an enterprise B2B company, then you're closing deals for, you know, 10, 15, 20 thousand dollars.

Only a few of those customers can give you enough capital to hire more people to do more marketing, to hire a larger sales team. If you are selling, you know, shirts and pants, then you probably need a higher volume.

So totally depends on it! It depends on the kind of company you're building.

Yep, so the question is what kind of retention period do you want to look at if you're renting properties like an Airbnb?

Yeah, that's a really good question. It really—those kind of periods depend on the kind of company you're building.

So I'm giving one presentation, but there are hundreds of different companies that need to apply the information slightly differently.

If you're Airbnb, you probably would expect people to at least come back to your app, you know, like, how often do people travel? Right? They travel, you know, once a month, once a quarter, right?

And so that's the period that you want to make sure that they come back at. Yeah, so it's not like week is correct for everyone, but it's correct for the majority of companies.

Exactly! Keep a pulse.

Yep! Awesome. Alright, well, thank you so much!

[Applause]