Gustaf Alstromer - How to Get Users and Grow

On company updates, please be honest. They're for you and not for us. And if you make them clearly crazy, like you know we're never ever launching, we're launching in four million years, we'll get the hint. So don't do that.

There have been a lot of questions about the graduation requirement, which was that you had to submit nine out of ten weekly updates. This course was originally a ten-week course, and things changed around a little bit. It's actually more of a nine-week course now. It was a little bit difficult for some groups to get going because of our little snafu in the beginning of the course.

So here's the deal: eight out of nine updates is just fine, but we're also going to extend the class actually two weeks beyond the last lecture so that there will be plenty of time to do ten updates if you so choose. You can choose to do nine out of ten, but eight out of nine will be the absolute requirement for consideration for the four, the $10,000 grant. So we will be flexible, but make them good updates. Make them real updates.

Another reminder: those weekly updates are due at 11:59 p.m. every Sunday Pacific Standard Time. There's been some confusion about that, as we've written it a number of times, but please recall when that is. If you have questions again about anything — about your moderators, about your groups, about your updates — please send to startup school at Y Combinator dot com.

My very last note is a reminder to launch if you haven't already. The only way to find out if you've made something that people really want is to have people really use it. So get your products out there if you haven't.

Now, I'm going to introduce our first speaker, who is my esteemed partner at Y Combinator, Gustov Ahlstrom, who in 2012 was part of the group that built the growth team at Airbnb. If you know anything about Airbnb, you know from 2012 for the next five years, they grew an enormous amount. This is something that Gustav knows better, or at least as well as anyone in the entire world. So I'm thrilled to have him talk today about growth.

Thank you very much. Before I begin, thank you so much to Steven, Jeff, Adora, and Bill for putting all this together so we can be here and do this today.

I'm going to talk about growth. My background, before joining YC last summer, was being one of the first persons on the growth team at Airbnb and then growing that team from, I think we were three people in the beginning. By 2015, we then grew to two hundred and twenty people or something like that. Almost everything I'm talking about today, I owe to this group. That’s sort of where I learned all this stuff. Some of the best ways to learn to work on user growth is to actually work on a growth team, and I was fortunate to do that for almost five years at Airbnb.

Before I start, one of the questions that might be obvious — the answer might be obvious for some people but not for everyone — is sort of like, why should we do this? Why is growth important? I'm surprised how often I actually get this question, but it might be worth thinking about it for a second.

When you make a product, some people think that if you just put that product into the market, people will come and people will start using the product. In my experience, that's not really how it works in the world. If you start a startup or a company that aims to be a startup, growth is important because that’s actually what a startup is. If you make something that has the potential to be really big, then making that really big is sort of what startup is all about.

There's actually a post that Paul Graham wrote called "Startup Equals Growth," where he talks about specifically why growth is so important for startups and why it’s not important — why not every company is a startup and why growth isn't important for every company. But for startups, it's really, really important.

I'll talk in a second sort of about what it means to be intentional by growth. But the first question you kind of have to answer yes to is sort of like, "I have a startup; I want to grow." Now, how do we go about doing that?

So who is this talk for? In my experience, those companies that work on user growth started with consumer companies. Those were the companies that started embracing a lot of the tactics and strategies and technologies that growth teams were doing. Now, I would argue that almost every company that sells anything online or gets users online should be or could be using a lot of the skills and knowledge that I’m going to talk about today.

So, you don’t need to be just a consumer company. You don’t need to be just a social network to apply these things; these are applicable to a wide range of companies. If you're a B2B company and are excellent at growth, you have a massive advantage over other companies in your space because they are probably not going to spend as much time doing growth as you are.

Now, there’s one type of company for which this is relevant, but it might apply to most of you right now. And this is where this talk is dangerous because working on growth at the wrong time in your company’s history can be detrimental for you. It’s sort of like when you are kind of off to the races a little bit too ahead of time.

So the most important thing, which is what a lot of lectures here at Startup School is about, is how you make something people want and how you actually find product market fit. If you apply a lot of the things I'm going to talk about here to a company that hasn't built something people want or hasn't actually found product market fit, really bad things happen.

You will very often — and this is a very common graph — grow like crazy in the beginning, but then when you realize that sort of like that fuel they had isn't fueling anything, it just kind of evaporates, you come straight down. So that’s why it’s dangerous.

Now, I can go into more details about that in a second regarding applying some of these things before you actually have product market fit.

So the first thing I'm talking about today is going to be around measurement around product market fit because that's actually the most essential part for continuing growth. Now do growth teams have impact? Or are growth teams sort of like, how do you know that growth teams actually matter? How do you know that these things matter? I'm going to talk about that as well, like my experimentation.

But before I get there, I want to tell you a story from Facebook. This is not a story I came up with myself; I found it in one of the talks from Facebook, and I think it's so essential for describing the level of impact you can have if you are applying these skill sets to a company.

So this is the timeline of Facebook from 2004 to 2015, sort of their growth story. Some of this is public; some of this wasn't.

One thing Facebook had was an excellent data science team from the early days. They were really good at measuring sort of like forecasting how big Facebook was going to be. So back in 2006-2007, when Facebook had just started the growth team, they had a bunch of data scientists doing this sort of forecasting. How big will Facebook eventually be? They looked at all the types of metrics at facebook.com, and they came to the conclusion that Facebook will be roughly 400 million by 2015. That was their forecast by using all the data they had.

Now we know that wasn’t really true. They actually started growing really fast in 2008-2009, and this actually was a result of something that the growth team came up with. No one here had an idea what made Facebook grow so fast in 2008.

Roughly speaking, yes, they did, but that wasn’t as big of a growth moment as the news feed was in 2005. The significant change, though, was that Facebook was hitting a ceiling on the number of people who spoke English. They assumed that just because Facebook was in English and the content was in local languages, it would continue to scale. That wasn’t the case.

So they implemented this new platform that automatically translated Facebook into hundreds of languages, and the growth team accelerated again. The same thing started to happen again. I think they were doing very important things in 2010-2011, something substantial was happening at Facebook.

How did they make some really large changes, to some extent also driven by sort of like using data to figure out these things? Any idea what happened in 2010? It was a really big acceleration. Mobile!

So actually, what happened was that the data scientists at Facebook were forecasting Facebook to be about 700 million users or something like that, and because most of those people were using Facebook on computers, they hadn’t intended this massive shift that was happening in 2010-2011, which is people getting smartphones and Facebook switching the entire team. They even had large training classes for engineers just to learn about mobile.

So this was the big change that happened then. Then actually one more thing happened sort of in 2013-2014, where they made a forecast. They kind of hit the ceiling of growth again, but then something again happened.

Any idea what happened? So 2013-2014, Instagram and WhatsApp could be good answers! That’s not what I'm looking for here; Messenger is a good answer; it’s not what I’m looking for either.

So Facebook actually ran into the ceiling of the Internet, the number of people actually being online. This started this thing called Internet.org, which was intended to get more people online. They went to carriers and worked with them to get people online to get free Facebook.

This was very important for the continuous growth of Facebook. Now what can we learn from this? Well, we can learn that the initial forecast of Facebook’s growth was about 400 million people, but Facebook today is a 2.1 to 2.2 billion user platform. It’s a very large platform, and those forecasts were wrong. Now, the forecasts weren’t wrong; they just didn’t take into account everything great that the company was going to do.

So if there's anything we can learn from this, it's that if you’re intentional about growth and you're really trying to sort of break through these forecasts and ceilings, you can grow really, really fast. So this applies to every company, and I showed this graph to everyone at Airbnb, like Aaron and Leek joined the growth team and tried to get them to think in the same way.

So natural adoption, which is sort of like that initial forecast of how fast your product is growing without any of that work is always going to eventually slow down. But if you do the things that you do as a growth team, you can sort of continue to push the growth of your company.

Now I mentioned this before: having growth teams before you find product market fit isn’t really useful. Let’s talk about product market fit is it? It’s an important term, but it’s hard to define what it is exactly.

One way that I think about measuring product market fit is these two things. First, find the metric. You can do this for yourself after this talk. Find the metric that represents the value that your customers get from your product, and then you measure the repeat usage of that metric.

It sounds pretty simple, and we’ll see if it works. You can make this kind of a table if you want, where you have the company, the metric that represents the value you get from your product, and then you have the ideal frequency.

Let’s take Airbnb. You get value from Airbnb when you are booking and staying. When you’re actually traveling to Airbnb, that’s when you find that Airbnb is really valuable; it’s an amazing experience. Unfortunately, people don’t do that more than annually.

So if you want to figure out the product market fit of Airbnb, you can always ask people after the first stay, but you wouldn’t really know until they booked again. When they come back and book again, so because the booking cycles for travel is so slow, annually would be a good metric here.

Let’s take Facebook. If you come back to Facebook, being involuntarily, as an active user, you’re doing that because you find value. There’s something that makes you come back to Facebook or Instagram, or anything like that. The question then is how often should I come back? Well, probably daily or hourly or something like that.

They started this. They started looking monthly, and then they went down to daily. I think they might even be going more than that. Let’s talk about Gusto. What’s the product value as a customer of Gusto? What do you get out of using Gusto? Well, when you run your payroll, which is sort of what Gusto does for your employees, that’s one of the best ways to measure the value you get out of Gusto.

I know often you do that. Well, you run payroll every other week or every month. I’m not going to go into details of all of this, but for Lyft, it might be rides; for Checker, it might be background checks; for Stripe, it might be transactions. Now, for your company, you should figure out what is the one metric that I can measure easily every day or every time it happens, and what’s the ideal frequency through which that should happen.

If you can answer those two questions, you can make a cohort analysis of your own company right now, and you can start trying to figure out if you have product market fit. So then you want to measure these things.

So you take a graph; on one axis you have time, and the other one you have the metric that we just decided. From that, you can try to figure out if you have product market fit. So this company is measuring this on a weekly basis. The ideal use case of this company is using this product on a weekly basis. So every week here, there’s a dot which is the percent of people that use this product.

On week zero, the first time they used it, they used it again. So let’s take the Startup School example. I joined Startup School, and in week one, only 60 percent would come back to school, and then in week four, only thirty percent would come back. Well, that would suggest the Startup School wouldn’t be a very good product, but that would be a good way to measure if Startup School is actually a good product; people come back for all the content they were creating.

I’ll take another product. This is a great product; this is kind of like a normal product. A normal product would look like because almost always you have a little bit too many people in the beginning of the product then it will go down over time, but it will sort of flatten out where you keep measuring these events of your metrics.

This might sound technical, but it’s actually just a representation of how many people are using your product for that metric that you decided on that time window. Most good products flatten out. Let’s look at some examples here.

So these are examples that are based on payment retention. So this is a company that you all heard of that retains 10% after one month and then 12% after 12 months. It’s sort of like unusual. Who can that be? Stripe? Is this too hard? This is Shopify!

So Shopify has this product where a lot of people sign up; not that many people continue in month two, but then they kind of stick on forever. So what do we know from this? Well, if this curve would go down to zero or sort of keep going down, then Shopify would be a bad product. It wouldn’t be something that actually found product market fit and should have been working in growth. In this case, they have good product market fit, and they should be working on growth.

They should do all they can to continue working on growth. They should do something about sort of like the initial onboarding here where they lose a lot of people in the beginning, but they should be working on growth.

Here’s another company: 50% retention after one month and then 10% after 24 months. So similar to Stripe but like or similar to Shopify but they continue to lose users all the way down to even 24 months. Maybe there’s some flattening out at the very end here, but even 12 months after people start paying for this product, every month fewer and fewer people pay for the product.

Is this product market fit? Hard to say, but it’s not an obvious case that they have found product market fit. So this is Blue Apron. Here’s one that’s pretty good: 70% retention after 12 months and then 30% after 7 years. This company definitely had product market fit.

Any ideas of which company this might be? You’re all using it? Amazon? That’s not right. Apple? No. This is Netflix! So 70% of people that start paying for Netflix pay for Netflix 7 years down the line. That’s definitely a company with market fit.

Let’s just spend every time everything they can work on growth.

So raise your hand if you’re measuring your retention right now? Not that many people. Well, there are other ways to figure out retention. If you’re really small, you should go out and talk to users. You should ask them questions such as, “How would you feel if you could no longer use my product?” You should sort of like stay as close as you can to your users.

This is by measuring retention like this; it’s very hard when you have 10 users. When you have a thousand, it’s easier. But when you have 10 users, this is not the way to do it. You can actually go and just talk to any person. So there are other ways to measure it, but you sort of have to know that you have a product that’s retaining; otherwise, you shouldn’t be working on growth because you’re going to end up burning cycles and things that don’t matter.

Okay, so a lot of people might be wondering, how does growth and marketing relate to each other? Isn’t that the same thing, or how should I think about it historically? The way you had a company 20 years ago is you had a product team that made the product, and then you had a marketing team or a product marketing team that marketed the product. That’s how things used to work.

A lot of the sort of like hierarchies in companies or sort of organizations are still based on this idea that you have a product team separate from the marketing team, and these are different teams and different skill sets. Engineers work over in the product team, and then the marketing people work over in the marketing team. Actually, that’s not how things work anymore.

So the way you should think about this — and I’m going to come to this in a second — is that there are three different types of people and organizations that can drive growth. There is what I call product growth, growth engineering. This has different names. This is effectively product managers, engineers, data scientists, and sometimes marketers that work on growth, but they work on growth using technology.

So they’re actively changing the product to drive more growth, and much of this work here is about people that have already arrived at your product but haven’t really found the value of your product yet, and they’re changing the product to help you grow faster.

Commercial optimizations fall into this group; some of the growth channels actually fall into this group too. Now there’s this big other group here: performance marketing, which is effectively Google and Facebook marketing, which is also super technical and super data-driven. I would argue that these things are very similar.

So five or ten years ago, you would go to a company, and you’d see these being very different groups, maybe different floors in the office. Now, in my opinion, they should sit together. Engineers should sit together with performance marketers and vice versa because they are actually doing very similar things.

Now there’s this fifth button, which should be number four — sorry, it should be number before three here, which is brand marketing. Brand marketing is sort of like the hardest to measure, the hardest to measure type of marketing where you’re not really having a direct response; you’re not having someone directly giving you feedback on how that marketing is perceived. You can’t measure that very easily.

So this is not something that startups should be doing in the early days. In my opinion, startups should not be doing brand marketing until way down the line, when they sort of hit the limits on these two things. So startups should be doing these two things, and there are some qualifications to which you can do this, but almost everyone can do this if you’re a startup.

You have engineers, and you can do product growth or growth engineering or things like that.

All right, let’s talk about that first part: product growth.

So your product is a funnel. What does that mean? Your product has many, many steps between the first user and the person, sort of like completing what they’re trying to do on your product.

Let’s say I’m on an e-commerce store and I’m trying to buy something. There are many steps in that funnel. What the product growth team is working on is funnel optimization or conversion rate optimization.

So if you look at what we did at Airbnb on the growth team, many of those growth teams were conversion rate optimization teams. They were working in a specific part of the funnel. So the funnel could start with SEO; it could start with performance marketing; it could start with referrals or virality, but it would jump through a number of different steps.

Say sign up would be a very common step, or if you’re an e-commerce site, payment conversion or buying conversion. All of these steps are things that as a conversion rate optimizing part of the growth team should be working on, and these are some of the easiest things to get working out.

I’m going to be a little bit high-level here because to go into depth about every area here is going to — we can go on for hours.

So some good ideas of areas to start working on for conversion optimizations: one of them is translation. If you have an international product, it's not translated. You should be thinking about that! That immediately drives and drives more people to start using a product.

The second thing is authentication. Most of your products probably have some idea of user accounts, so you can sign up to your product, and you can come back and log in to your product.

You'd be surprised how much how hard this is. From Airbnb I know we spent many years working on just authentication, signing up, and logging in. It sounds so simple, but it turns out that’s a very fragile moment users flow through your product that you can always continue to make optimizations.

If you go to Airbnb and you go to Pinterest today, assume that whatever is there in terms of authentication is the most optimized version. These companies spend enormous amounts of time optimizing sign up conversion.

Another big area for conversion optimization is onboarding. So when I come to your product, what’s the first thing I experience, and sort of like how do I work? What can the product do to bring me toward the value of the product as quickly as possible? Those are things that you’re working on when you work on converting.

Then another big area here is purchase conversion — sort of like when I’m about to buy something, a product, and actually can take the final decision around what you do. There are so many things you can do here.

All right, so then there is something called growth channels. What is it growth channels? Your channels are sort of how people discover your product. Now, when you’re a small company — and I don’t know how big you guys are, but let’s say you have less than 50 users — you shouldn’t really be thinking about growth channels. Even if you’re less than 500 users, you probably shouldn’t be thinking about this because it’s too early.

But the things that you do when you're small that don’t scale have the word "don’t scale" in it for a reason. They don’t actually scale. There are very few companies who kind of grow big without growing on one of these scalable growth channels. They’re not that many platforms, and the channels/platform can be used in the same way.

There are not that many things that are really, really big in the world on which you can build a large company. So let’s talk about what those channels are.

The first thing here is basically you think through the behavior of your product. Let’s talk about the first one here, which is if the way I discover your product is a rare behavior where people use Google to find a solution. This is actually how many products in the world are being discovered.

If you have something through which to answer the question around what you do once a year, you probably if you’re building a company trying to answer that question should be on Easy B for Google.

So a good example here might be buying a house. You aren’t buying a house more than once or twice in your entire life, which means when you go and buy a house, you’re probably going to go to Google. It’s not surprising there’s something like Zillow or Redfin or other sites that allow you to buy homes online are entirely optimized for SEO and sometimes paid search.

If you are using something every day, you're not going to go back to Google every day. You’re going to go straight to that product; you’re going to open the app on your phone; you can go to the straight to the website, whatever it might be. You’re not going to go to Google anymore just to kind of figure out which of the different things I’m going to use.

Now, Google is not the only search engine, and there are other search engines you can optimize for too, but Google is the one that still matters.

Next behavior: do people of my product already share the product using word-of-mouth? If that’s the case, then there are lots of things you can do around virality and referrals. So you can grow your product by accelerating that behavior from your existing users.

You can either incentivize them with referrals — where you could be getting paid — or you can do it for free and with using virality techniques.

That’s having more users on my product actually improve my experience. What I mean by that? Well, if I am building the next LinkedIn, it makes sense that the product is not as valuable when it’s just all of us on LinkedIn. But when there’s a bunch of people and companies on LinkedIn, it makes sense; the value kind of increases. So in that sense, you should absolutely continue to do virality because every single new user is sort of the opportunity to bring in more users.

So there’s another way to think about virality, and this is that common question I ask companies in sort of the early days of YC: can you make a list, literally a spreadsheet, of all the people in the world that would use my product?

Let’s say I sell to buyers who sell to doctors’ offices; they sell to doctors’ offices. Well, I can probably make a list of all the doctors’ offices in the United States or in California. It wouldn’t be that hard; it’s totally possible. So I would find a way to make that list and then go out and do sales. This is surprisingly often something that you should start with. If you can make the list, if you know those people are, you should go on new sales.

The last one here is: do my users have high lifetime value? Do I charge enough for my product for it to be valuable? Well, then I should definitely go and do paid acquisition, for example Google and Facebook. I shouldn't do acquisitions or paid ads unless I actually am charging for my product.

All right, I’m going to go through each one of these. It’s not going to be possible to go into super deep detail for all of them, but I’m going to go into a little bit deeper into some of them and not the others.

All right, let’s talk about referrals. So I worked on a referral program at Airbnb for a very long time. The way to think about referrals is sort of an engineered word-of-mouth. So if people are already talking about your product, referrals is a way through which you can engineer more people talking about your product.

One way could be just making it easier; another way could be by using financial incentives. So in the Airbnb referral program, we had a financial incentive where, as a referee, I would make $20 for every user that would sign up in travel credit, and the referee would get $40 as a new user of Airbnb in travel credit. We would start with that principle and then try to get as many people as we call through our first phones as possible.

So, you notice I use the word funnel here again. So every product is a funnel, and even our full product, which is sort of a product of itself, inside the Airbnb had its own funnel.

I’m not going to go through all of these details, but the way you think about something that’s kind of engineering and product-led is you break it down into different steps and then you measure every single one of these steps.

Then you kind of measure the conversion rate to the very end. The first step here is weekly active users that saw the referral program. So how many users saw their referral program? If I wasn’t measuring this step, I wouldn’t know how many people did that.

When we started measuring this step — and I’ve seen this with many other companies that have a referral program — is that a small percentage of your active users see their referral program. Well, how could you be expected to use it if you can’t see it?

So we started measuring this in the early days of Airbnb. It turns out there’s a lot of opportunities just telling people about you having your referral program. But then, there are all these opportunities to optimize through that funnel.

So there are sort of people sending invites, how many they sent; the conversion rate to new users; new guests; and finally booking their first-time nights.

Here’s another new display — and this slide is going to be available online afterward so you don’t have to take notes or photos. We kind of separate the funnel into even more detail, and we would continue to optimize this for years.

All this matters a lot, and we would continue to optimize for years. I’m going to go through one example of sort of like one of those conversion rate optimizations we did for the referral program.

So here’s the referral invite email. So if someone else — if I invited one of you guys — you get one of these emails. The email would say, “Gustav Ahlstrom from Airbnb sent you $40 on your first trip on Airbnb! You can book rooms by signing up by May 25, 2018,” and then there’s a button to accept the invitation, and then there’s a photo of me and my name — it looks pretty good.

Now, this email is a result of dozens of experiments. Nothing here is random; everything here’s for a reason.

Let’s talk about that. The first one is the subject line. The subject line has my name. If it’s sent to any of my friends, that makes them more likely to open it. The sort of like the headline of the email has clear value. It’s very, very clear what this email is about.

You’d be surprised how many emails don’t have a clear headline and the clear value is what the email is most about. This email is about that I sent you $40 on your first trip on this website called Airbnb.

This — which is sign up by May 25, 2018 — is urgency. You should sign up by May 25, 2018. That’s the urgency that makes you actually go. That increases the chance of people who see this email actually going and doing it.

The name here, “Accept Invitation,” is the sense of exclusivity. It doesn’t say, “Sign up for Airbnb” because anyone can sign up, but by accepting an invitation, it sends an idea of exclusivity.

And finally, it has my name, where I live, and how long I’ve been a user of Airbnb with my photo. That’s really strong social proof that I endorse this website. I endorse this product.

So the way you think about this is sort of how you think about all conversion rate optimization. It’s a set of optimizations.

Now, paid growth — this is one of the areas I’m not going to go too deep into the options here. The things that matter for paid growth are that you shouldn’t do it unless you have revenue. Too many companies are buying ads when they don’t have any revenue or know how they’re going to make any revenue. That’s the big mistake; you shouldn’t be doing that.

If you have revenue, there are a couple of concepts that really matter. The first one is how much money am I paying for each new user that I’m acquiring? It’s called customer acquisition cost, or CAC. What’s the lifetime value or payback time of the users that I’ve acquired through paid growth?

So what that means is what’s the longest that I can, with some level of accuracy, forecast how much these users will be worth? If a user is paying $10 per month, we can sort of have this cohort analysis that we had at the beginning with the attention; we can figure out exactly how much our users are going to be worth.

And if my customer acquisition cost is lower than my lifetime value, then that’s good. That means you have a great payback time. If you only pay $20 for a user, that means if you make $10 per month, you probably have around a two-month payback time.

That’s more complicated than that, but it’s sort of like a simplified version. These are the most important concepts to keep in mind for paid growth.

Attribution — so this matters. If things get complicated, if you’re using both Google and Facebook, if you’re getting to some level of scale, you need to understand what it means to activate a new paid user relative to a dollar that you spent.

I’m not going to go too much into detail on what this means, but this is something that you should learn if you’re paying time on paid growth.

And finally, in my opinion, there are only four channels that sort of matter at scale: Facebook, Instagram, Google, and YouTube. These are the very, very large paid growth channels through which many companies are actually built on these days.

Sort of like one of the unspoken truths, in my opinion, is that a lot of the free channels for growth are going down, and the paid channels are going up. A good example of that is that it’s hard to grow on the Facebook newsfeed unless you’re paying Facebook money. The same is true for Instagram.

It’s sort of like changing constantly. When there’s a new platform, they often kind of promote free usage in the beginning and then start to monetize that. That’s sort of what was happening for both Facebook, Google, and Instagram today.

All right, search engine optimization — SEO. Some people might say, “Well, this is something of the past.” In my experience, it is not. It still matters because as long as we go back to Google to make our decisions about what we want to do in the future, this matters. Google is probably one of the largest websites in the world; it’s a big deal.

The one thing you should know about SEO is that there’s a difference between what you see on a website and what Google sees. Google can’t view images; Google can’t view JavaScript very easily.

So if you go to Airbnb and you see all of this, just remember that Google can’t see this. If you’re trying to communicate to someone who’s searching for “Stockholm” on Google or “apartments in Stockholm,” this is not what is going to be delivered back to Google. Google is going to see a bunch of lines of text through which you’ve marked up in your code. They should have hopefully indexed if you’ve done the right thing, and if you haven’t, then it’s kind of like your fault.

So there are a bunch of basics that you can do with SEO you just have to get right. The easier thing you can do is to run your website through an SEO browser and try to figure out if you only saw this. Would you understand what your product is about?

So using clear language, not in JavaScript, describes what your product does; it’s the most basic thing you can do for SEO.

There are two different areas of optimization for SEO — and this is as high-level as it gets. So at Airbnb, there’s a team of 20 people working on this, of which 12 or 13 are engineers. This is a really, really big deal; but this is sort of the high level.

There are two types of optimizations for SEO: things that you can change on your website, and other people in the world linking to your websites. This is the two main levers that you have in SEO.

For the on-page optimization, the right way to start is not to start with like a list to make some changes. The right way is to start with the strategy. What am I trying to rank for? What keywords exist on Google today that I might want to be the number one result for?

Now to do that, you have to do what’s called a keyword analysis or keyword research. You have to try to figure out what are all the things that people are searching for that might relate to my product, and then hopefully, they have some amount of large volume that they’re searching for.

Then I can try to say, “Well, these keywords that have sort of medium to high volume, and they don’t have much competition around these keywords, those are the ones I want to rank for.” When you decide that, the other areas of impatient optimizations are easier because now you know you’re trying to rank at scale.

I’ve seen this at a small scale too. SEO at this point is about SEO experimentation, which means you can do a set of best practices in SEO. It might take you to be a decent size company, but if you want to become a massive company, a really, really large company that’s growing through SEO, you have to use experimentation to make those decisions.

The other thing that matters here is off-page optimization. Who’s linking to you? There are lots of tools you can use to figure out who are all the websites online linking to you and what’s their domain authority? Are they actually authoritative on the web?

There’s a significant difference if I will link to your website and if you have x link to your website. It matters a lot in sort of how in Google’s perception of your site.

One surprising way there may be is that we had a lot of press — a lot of people in the world from media would come and write about Airbnb, and that was surprisingly important for off-page optimization. A few people writing might either impress or in other areas. That’s actually a great thing.

If you don’t have anyone writing about you, you’re not going to have many links because the web has changed. There aren’t that many links anymore; it’s not like everyone has a website these days that links to each other.

So things have changed. You have to be strategic about who you’re getting links from. The easiest way you can do it is whenever you get written up by anyone in the press or whatever, just ask them to link to you. It matters to you.

The last thing I’m talking about here is on growth teams. So a growth team today is typically engineers, data scientists, designers, product managers, and user researchers. These are not all the people you might have in your company today, but these are the companies that you will have when you start making decisions to run growth.

The way you organize the growth team is there are sort of two options. You either have the growth team as its own team, and the rest of the practice is the product team. That’s actually sort of the challenge here; when you’re a small company and you’re somewhat ignorant about growth, what you often say is, “I’ve hired a growth engineer; I hired a growth product manager. That person is going to be doing all the growth in the company.” That’s sort of not really a recipe for success.

So there’s a fine balance here between saying, “Everyone is responsible for growth,” which doesn’t work, and having a small team that is responsible for all the growth. You kind of have to find a balance here, and the right way to find that balance is to set very clear goals and very clear dividing lines in your product.

A good example here might be everyone that works on the core product, which is sort of the value of your product. Let’s say I’m Gusto, the running payroll for employees, which is the core product. Now everyone trying to get to that experience that’s sort of like the growth area of Gusto.

That’ll be one easy way to kind of make a distinction between growth and product.

So how do you decide what to work on? You make a simple calculation of what’s the effort, what’s the minimal viable thing I want to try here, and sort of how big of an outcome can that be?

I always try to forecast the outcomes before you start doing the work because if you’re forecasting something in a best-case scenario and it’s small, you shouldn’t be doing it even though it’s low effort.

All right. I have two small, short sections left of this talk. The first one is called making decisions. If you ask any product manager at Airbnb today what is the most important tool or learning that you ever learned at Airbnb that you’ll apply to your next thing, they would say experiment or experiment framework or some way of making A/B tests inside the company.

I’m so happy that someone is talking later because I stole a quote from his investor deck online. This is the quote: “Most of their world will make decisions by either guessing or using gut. They will either be lucky or wrong. If you keep making decisions without using data and experimentation, you will be lucky or wrong, and this is a huge problem.”

So if you kind of get to a scale of A/B or something even smaller than that, then every decision you’re going to make that you don’t use A/B testing for, you won’t actually know what the full true outcome of that decision might be.

So whatever Airbnb — we use experimentation and A/B testing for every single major decision in the entire company.

So this is how that tool works. A/B, but my talks are sort of like more about how that looks like. Let’s say in your company you decide to ship a feature, and this is how many measurements of my daily metric per day, and the Wednesday here is when I ship that new feature; then I look at it two weeks later, and it looks like the metric of that feature went up. So that was a good thing, right?

Well, it’s not that easy because there are so many more different factors. Who knows what happened between here? If you're a soccer app and the World Cup started, well, then you wouldn’t know if the feature has made a difference if this was just peak season or whatever you’re doing.

This is a school app, an education app, and this is September; then you wouldn’t know either. So the only way you would know is what's called a counterfactual, basically an A/B test. By running two different versions of the same feature at the same time, you would be able to know the true difference between making these decisions and not making them.

This might sound a bit technical, but it’s very, very important to internalize because you will get to a point where you are successful up until that point, and you think that you’re so good at making these decisions and then they get harder to make. You have to have a framework to make these decisions.

So what we did — because this was so important to Airbnb, we built something we called experiment review. Experiment review is where the whole growth team would meet in a room like this and go through all the features that we had recently built.

Before we told you which actually should have the different features in the experiment, we would actually ask the audience. So I’m going to do that with you guys here.

So here’s a photo from an experiment review. We would do this every two or three weeks; it’s really fun, but it drove home this one thing, which is that making practices are really hard.

All right, let’s get started. So the goal of this project was to increase the number of shares from the mobile Airbnb app, specifically the number of shares of listings. Back at that time, we had two options.

We had the native share sheet that you’ll know about, and then we had this experiment share sheet, which takes up the entire screen, has more colors, and the same type of buttons. You can see the difference here.

Now, before I show the answer, how many here think that the native share sheet led to more shares? Raise your hand. It’s half the room. How many here thought the new share sheet that one of our engineers built drove more shares? Raise your hand.

So the most many hands would not raise. I’m assuming you guys think there was no difference because this is so hard. You have to make a decision; you have to have an opinion. If you don’t have an opinion here, you’re basically saying I can’t decide.

So this share sheet led to 40% more shares, so very importantly, in this case, we’d used experimentation because if we just gone by gut, we would have been wrong. More than half of the room would have been wrong!

Let’s do one more. So we send out this email to existing users of Airbnb — sort of when I would book, make a booking — inviting, let’s say this email to the people that I listed as my co-travelers. The email looked in the control like this, where it would have the itinerary, the address to that listing, and the button on that email would say, “Join Gustav’s trip.”

Now, we tried a new version where the email looked a little bit different — less content. And then we had a different button here called “Accept Gustav’s trip invitation.”

How many here think that joining Gustav’s trip in the control led to more signups? Raise your hand. Some people. How many here think that accepting Gustav’s trip invitation raised your hand? That’s good. How many here think there was no difference?

So this is a 14% increase of just changing basically the name of the button! Let’s try one more for simplicity. So in this case now, they’re sharing experiments. I’m kind of giving you the very easy-to-understand ones.

So the control here was sort of like a bunch of sharing options on the Airbnb listing, which had some Twitter icons and Facebook icons, email icons. We got another version of that, and the icons were round, and then we had one that was called sort of square buttons, and it was just displaying email and Facebook, and then you can click on more.

How many here thought, for sharing, that the control with the icons won? Raise your hand. Some people. How many thought the round buttons were better? Raise your hand. Some people. I leave every thought the square buttons?

Wow, you guys are really good. How many thought there was no difference?

Well, both of these were winners! This was by far the biggest winner, and these are the kind of things that we debated. Because we ran experiments, you don’t have to debate anymore.

So product decisions are really hard at scale. You’ll want to use experimentation and A/B testing to make these decisions because otherwise it’ll be the loudest voice in the room that makes the decision, and you don’t want that.

So to summarize sort of my talk: if you’re running a startup, you should be thinking about growth before you start working on product-market fit. You should be measuring your retention and knowing if people are repeatedly using your product.

You should pick a metric and then pick a goal for that metric and drive that metric toward that goal. Very simple. Then eventually, but not right away, you should start running A/B tests for the decisions that are hard to make. In the early days, decisions are hard to make as they might be obvious, but the moment they’re not easy anymore, you should be building and doing this with experimentation.

Thank you! I think we have time for a couple of questions.

Yes, do you have any frameworks or ways you think about trying to experiment?

So the question is around experimentation in SEO. How do you go about doing that? So what you’re trying to test when you run an on-page experiment is sort of the amount of organic traffic you get from Google and how your ranking shifts.

Google actually is changing the rankings all the time, so if you make a big change on your website — say half a, say let’s call you have an A/B case listing pages, sorry, search result pages — so Stockholm, London, San Francisco, we buy at one group, and then you have New York, Paris, and Barcelona be another group.

You would change the content on the first group of pages but not the others, and not too long of a time, you would see more or less traffic going to either of these groups from Google. Sometimes there’s no difference, but if there is a difference, you’ll see more traffic.

To give you an idea of a very small change you can make, let’s change the title tag from “AirBNB Listings” to “The 20 Best Listings in Barcelona.” One of them is going to rank better because you can be more inviting to click on, and then you run that experiment towards the search engine and then, by measuring, you’ll kind of tell you which one is better based on the amount of traffic and search you’re getting.

Yes, [Music].

So the question was around A/B testing. How do I determine if it’s right for me when I’m really early? So A/B testing is a function of change. So you can have a medium to small-size audience. If the change is large enough, it can actually be significant.

Now what you can do is you can go to Google, type in an A/B testing calculator, and there’s kind of a form that will pop up. You can type in sort of the metric that you have and then the change, and you can see how big the change has to be for you to be able to see a difference.

Now, if you’re really, really small, you probably shouldn’t be doing A/B testing at all. So I would do A/B testing when you have large enough traffic that you can see a medium to small size change within, say, two or three weeks.

So let’s say a small change would be a couple of percent if you can see that in a couple of weeks. That sort of is my general recommendation. I’ve certainly seen a lot of companies and they are early embracing A/B testing relatively early and then have that guide their decision-making, and I don’t think that’s a bad idea because I think it’s much better to start a little bit too early than to start way too late if those are the only two options.

Of course, you can start at the right time.

Yep. How do you apply growth to high barrier to entry markets like health insurance?

So I think you should separate the acceleration of your growth in that market and the market itself. So if it’s hard to reach people, that means you have to probably try to reach more people before you actually grow.

So if you high — tell me what you mean by high barrier?

Yeah.

So if you have risk and regulation involved, I don’t think that the principle of growth changes. But if you’re purely growing through sales, for example, you can apply a lot of automation and technology to how to do sales.

So if I’m a company and I’m selling to insurance buyers at startups, I might be starting by emailing 10 people and, in fact, I can email 100 people or 1,000 people with the same type level of the email itself as kind of a field as personal and as direct to that receiver as if I send one email.

So, there’s certainly kind of things you can apply to your growth that allow you to reach a lot more people. Now, growth doesn’t solve any of the risk challenges at all. It doesn’t solve any kind of major market issues. And we certainly had a lot of issues with legal challenges in different markets, and sort of growth was disconnected from that; that wasn’t sort of our goal.

If you’re in a startup, you have those problems. I don’t think you should solve them. Solving them and solving growth are two different things, and they are separate from each other. Growth is a way to accelerate you getting to more health insurance buyers; it doesn’t actually say anything about that’s a good thing or a bad thing.

[Music]

Do I have any wisdom in using non-sustainable tactics in a new market like Uber?

So if you're doing things that don’t scale, you sort of have two options. One is they don’t scale, so you should stop doing them eventually, or you build sort of a playbook where you take those skills and you try them in a different city.

Now, they might not eventually scale. So let’s say I am manually recruiting Uber drivers. Now, that might be an unsustainable growth tactic; it might be working for the first 20 drivers, but eventually, it won’t have high ROI in comparison to say running ads on Facebook to recruit drivers.

So I think you kind of have to determine this manual thing that I'm doing in the very beginning, which is doing things that don’t scale — which is sort of recruiting every single user to become a user of your product. Eventually, it won’t work anymore, and you kind of have to find another channel or it won’t be high ROI in comparison to the other channels.

Most companies will go through this transition period where you go from, say, writing content manually to engineering, changing your website so that we get more search traffic, for example.

What about doing subsidies or incentives for rides?

Well, I actually think the incentives are super scalable. Like at Airbnb, we still have the referral program that’s sending up on hundreds of thousands of users every day almost, which a large portion of them are signing up through referrals.

So that is an example where subsidies are actually scalable. Now, handing you a coupon in your hand is not scalable, but doing that through an email system or through WhatsApp or through Messenger or some other channel is actually scalable as long as you’re not losing the money.

So you have to have a very good ROI calculation on the money that you’re handing out and knowing that that’s incremental money that if you didn’t get, those users wouldn’t start using your product. As long as you have a good handle on that, that is totally fine.

[Music]

So the question is: sometimes experimentation can be really hard to execute. Is there an ideal sort of frequency at which which experiment should I be actually running? For something like a marketplace, experimentation is really hard; that is correct.

So in that case, it’s easier — it’s harder than just setting up a simple tool online or using Mixpanel to set up your simple A/B test. It’s more difficult than that. If you’re having a simple product funnel, it’s generally not that hard to set up experimentation if you have an engineer, and if you have a decent amount of traffic, it’s not hard to set it up.

Now there are different kinds of products that make experimentation harder. I think that if you’re at this stage where you have a lot of traffic, you, in my opinion, don’t really have an option. You sort of have to invest in some level of infrastructure to use data to make decisions. If you don’t, the alternative cost is you end up making a lot of bad decisions, or you’re either lucky or you’re lucky because you’re right, or you’re wrong.

So you don’t have any other option but to run experiments. It is true that there are some types of products through which experimentation is easy. If you’re running a mobile app, experimentation is not that difficult; there are lots of tools you can use to run experiments.

If you’re running out of ideas for experiments, well, then you should go back to user research and you should look at other products that look like something like Pinterest or Airbnb or Facebook. They’re probably highly optimized, so a lot of the things in their funnels are there for very specific reasons because of A/B testing.

So I don’t have a better answer than if you don’t do it, you’ll have a lot of other problems. In terms of the frequency, there’s a cost to setting up an experiment, and that cost should be minimized as much as possible. But it’s sort of like you should just try to get better at it and pick the easiest, smallest thing that you can test and just test that.

If you have a big new feature you want to test, well, just test the first part of that feature. Don’t test all of it and see how people react to the kind of first part of the feature.

I can talk for hours about this, but thank you very much!