Startup School Q&A Week 1

Okay, any questions? The question was, have I experienced any different differences between running user surveys in person or online? Honestly, you'll probably figure out what the best solution is for yourself. Online honestly gets a lot of it right. I conduct a lot of interviews over Zoom. I think it's, again, it depends kind of on the context of the problem. Sometimes I really find it powerful to stand behind a user and ask them to use their existing tools. That can only really be done kind of well if you're in person. But I have seen some people do that pretty effectively over Zoom as well.

There's something that I keep in mind is um a UX researcher named Jared Spool, and he talks about some research where he says, like, he noticed that in product teams, products tend to get worse over time if you find out that the product and design teams are not interacting directly with their users a minimum every six weeks. Some amount of time, some set of hours. And so, it's one of those things where it's like, there's just so much to be gained from like just talking and having a small conversation where it's like you just miss a ton of stuff because it's like, I don't know how many surveys you fill out, but the people who do tend to fill surveys, they fill a very weird bucket.

Oh yeah, sorry, I mistook the question. It's just you online or offline? Yeah, no, you definitely need this open-ended conversation with a user rather than like a survey. Yeah, absolutely, so you always want to supplement.

Do you have any examples of when your user is different from your buyer? So, for example, my startup involves potentially selling in two countries. So, I will be trying to get entire countries to sign agreements to buy from me, but they will be the users of the products are going to be the people in the country. So, who am I?

Yeah, yeah, we should repeat the question. Yeah, the question was: what happens if the users of your product are different from the buyers? And in the context of the asker, she was asking whether in countries maybe the people using it, but the buyer may be an individual.

Yeah, this is a tough problem. This usually calls for more user interviews. So, the best answer here is just to have conversations with different people in the buying chain. Remember that a great user interview is not actually a sales pitch. It's actually learning more about the context in which that person encounters the problem.

So, in your case, I might consider having several different layers of interviews with potential buyers, potential users of the product, potential influencers. Oftentimes in an enterprise sale, there are some people in the buying chain that aren't even involved in using the product whatsoever, but they have a say in maybe the budget or the time frame in which they'll purchase or use new solutions.

And so, it helps to actually have conversations with them as well so you can get context around maybe past attempts that they've tried to solve this problem, you know, millions of dollars that they've already spent that went down the drain. And you can learn how they may be helpful or how they may hinder your sales process in the future.

Just a quick add, so that I would say for the economic buyer in the company or whatever, government or whatever it is, talking to them and trying to understand how they're going to make the decision and what they actually value is probably most important. The only thing I would add to that is that when you're talking to your user, there's usually like in a complex B2B enterprise transaction, you have someone that's acting as your champion, someone who's like super excited by using the product, usually they're a user, and/or there's someone who are like, "I don't have to do this anymore if you implement this by my company."

And so, you need to use them as your like Virgil. Like you're gonna basically say, "Help me map out who's all involved, who needs to be in the room, and what are their motivations?" And so, you're basically looking to get Intel for all that process.

And then the whole question is like you need to have all the arguments for things that they sort of care about. One of the hacks that I like to use that helps if you're still in university is you can actually kind of talk to big companies as if you were, say, the reporter for a school newspaper or if you were working on a school project. You can use this as like a hack to actually get a pretty high-level contact with someone who might become a champion simply by kind of saying, "I'm just really excited about budget management software. I would love to learn more about this. I know that you work on this a lot."

And so, you can actually like interact and talk to people at a pretty high level before you even announce that you're working on a solution for that. It's like, "How much money do you control for your budget?"

Oh, so for some of us, we've already lost. We're already talking to a bunch of users a week, but as we scale, that might get beyond what any one person can talk to. How do you slice down the subset of users they want to interview?

The question is, at scale, how do you segment just the users? What's the reason for segmenting these, you know, housewives? They might have different feedback than some of those housewives.

So there's a couple good tools for helping if you'd like. The question is like basically: how do you say let's say you have a really broad audience that you're trying to figure out what their use cases are? How do you best handle that segmentation so that way you get useful feedback about the specific direction of the product?

So there are a couple useful tools that I have used in the past, and I've recommended to other people. So Google Consumer Surveys and SurveyMonkey Audience. So both of those, you can ask really simple questions and choose the demographic. SurveyMonkey is actually a bit more expensive, but they actually have proper survey methodology that will balance it out based on like U.S. population.

So, for example, most people will fill out surveys, let's say for money or for nonprofits. It's mostly skewed towards women, and they tend to be older. So, like if you were to ask Google Consumer Surveys, a lot of that stuff will tend to be skewed that way. And for them, you can actually specifically handle it, and they'll actually use statistics to sort of balance it out.

So, that's number one. It's like directly asking. I tend to not for broad product do you like general surveys. Usually, it's almost always to my direct users to figure out, like, okay, who seems to be just already a fan of my product? Like, who have I accidentally acquired who's like a superfan? I'm trying to figure out how to get more of you, and then I'm doing everything I can to get them to talk to me.

So, I'd rather, like, I will spend like 50 bucks on a gift card to get that user to have a conversation with me so I understand who he is and see if there's ways that I could multiply him. And one of the things to remember is like, never forget that, you know, as the founder of a company, you can actually reach out directly personally to any one of your users and ask if you could just jump on the phone and chat with them.

This is like something that is kind of crazy because, you know, no big complex office is going to do this. Facebook's not going to do this. This is a competitive advantage that you as kind of a small early-stage company have. You, as the product creator, the founder, can actually have a direct connection with any user in your user base.

Some people productize this and do like tons and tons of emails, but others just choose to have maybe 10 conversations or 20 conversations with individual users every week.

Yeah, one of the things I always do if you have enough data, in your case like lots of customers, is to start tabulating for each customer figure out the characteristics of each one—like female, where they live, income level, so on and so forth. And if you have all the emails, there are actually services out there that you can qualify all these people.

And then after that, bucket all the customers into like: here are my best customers, here are the ones that paid me 100 bucks or more, and then try to find patterns and then go segment that way and talk to those people. And like Kevin said, how do I get more people who naturally pay me more? What are the patterns? What do they do every day? And offer them gift cards or whatever to do these interview sessions.

I would aim like if you have enough data and you have enough those types of people. I worked really am to do a session once a week with a handful of people. I find that's the most informative thing, the best thing you can do for your business.

And it's just like an in-person session. It can be or can be over the phone. It doesn't be in person depending on the product. So let's say you have an MVP built, you're starting to do this music interview. You're listening and not talking.

When do you do the sales pitch? To playoff Fedoras point, once you begin to understand within your user base who are the best users, who are the people that are the most disappointed if they couldn't use your product anymore, who are the ones that get the most benefit, you can actually go sideways and you can try to dig deep and find more people that reflect the same kind of demographic.

Some of the tips or tricks that we use are like where do those people live on the internet? Which subreddits do they use? Which Facebook pages do they follow? Which kind of other services do they already use? And then you can like use the fingerprint of the users you already know to find more and find out where they live online.

So my strategy would be if I was in that case and I had a wonderful user and I interviewed them and it turned into looking like, "Oh god, what I've built is perfect for this person." Instead of being like, I feel like it's the wrong moment to be like, "Ah, gotcha, here's my thing," I'd rather be like I'd go home, maybe wait a day or two, and be like, "Oh my God, thank you so much for the conversation that we had. Based on what you talked about, I built the first version of this. I would love for you to check it out and let me know what you think."

It's totally based on your feedback, and it's more likely that they're gonna be honored, their ego stroked, and then we're likely to use it. So to me, I feel like that gives you maximum benefit versus it feels like, "Did you even care about me at all? Was this really a sales pitch hidden?"

That's it exactly, and that's what you don't want to have after like being such a conscientious, empathetic listener.

And know that quick, a programmer that they programmed it while they were talking.

Yes, right here. Yes, I was at the AMA when I saw you about the comment regarding beta releases and I was curious to hear more of your thoughts about why you should release a beta. We want to limit people that are on the platform just to have the people or talk to.

So in the AMA, I kind of whoop beta versions or labeling your project as beta to artificially constrain growth on your product. And so the main idea for me is that it creates like an affordance. It basically is a sign that says, "My product's not ready for you. Don't use me because it's not ready for production; expect bugs," etc. And the reason a lot of founders or engineers like to put beta on theirs is because it's like, "Hey, you know what? This I know it's not that good. I built it myself. I know how many bugs there are and I'm so scared of people being disappointed. So if I put this beta on, it's just going to remove all liability."

And to me, I feel like there's no product I've ever seen in production that doesn't have bugs. And almost always, what good founders do is they listen to the people, they respond to the bugs, they fix it as quick as possible. And even in your beta, you will probably do the same.

So to me, I see no reason why you need to put that beta on. So I don't like anything that artificially can change growth because like all that stuff you're gonna do for the beta to be a good founder: listening, fixing, etc. You’re going to do that even in production, so that's number one.

The second thing about beta is that you're gonna spend your whole company's life trying to get as many users as possible. And so if you're thinking like, "I've built something so good that if I don't put beta on there, there's just going to be overwhelmed with demand," I'm gonna tell you that the hypothesis is probably wrong at all.

And so again, if something does go wrong or some certain issue that you're actually super scared of, like don't have that feature reveal. If something does go wrong, just like, "Hey, thank you so much. I've only been working on it a week, or I've only been working on it a month. I'm gonna fix it really quickly and then get it back to you. Please let me know if you find anything else."

And then you're more likely to get a more positive response and people actually using it the way that you're hoping. You might find people use it as a result. I like real world honest tests about do people want this thing and then is my thing ready?

Yeah, the real fear that founders need to get over is, "Okay, I've built something crap. People are gonna come if it's like not good enough. They're never going to come back." Like hopefully, you're building a business that will, but millions of people can use anyway.

And so just the four or five people that come and don't have a good experience, get that feedback and then just iterate towards something better. You don't have to worry about them not coming back and I think that's actually one of the main reasons why most people don't launch early or they put beta tags on everything.

So how do I stop interviewing and then just stop feeling, I mean like economy interviews, but like maybe people say you know 100, but the ones that we also women that I can start, I wouldn't do it wait for 100. To me, they're going to happen in concert with one another.

Oh yeah, so the question is like, "Okay, how many interviews do I have to do before I start building?" etc. And so if you don't have anything built, you don't even really know like what the solution should needs to be. You're just trying to figure out what is the problem set out there.

You basically do user interviews until you have an insight. Be like, "I think I can build something that will solve the problem." And then you immediately start building. But for the most part, you want to do them in parallel. And just to jump in, Kevin offered a framework to think about: is the problem that I'm facing, or is the solution to the problem big enough to be a company?

You can actually extract that from those early user interviews more effectively than after you have a product because you're not biased by thinking, "I already like this. I have a great idea already." You can actually be in that kind of pure mode where it's like, "Is this a real problem? Is this really a burning problem that people are facing?"

When do you charge? I love it early and often, as early as possible. And so, like if you can get away with taking a pre-order, wonderful. If you're building like, we often give advice to companies who are considering making a complex B2B enterprise app. We're like, "Go try to sell that first before you go commit six to twelve months making it."

It's putting, it's asking people to put their money where their mouth is. Remember how I said before that, oftentimes, users don't really know what they want. They won't be able to describe the future product, but they will be able to know it when they see it.

And at that point, you can extract data from them by saying, "Well, this person has the problem so acutely that they want to solve it that they're willing to pay some random startup that doesn't even have a product." If they're willing to pay them, that's like a very strong signal that you're onto something.

Eric's model for talking to users is very similar to what the best salespeople actually do when they're trying to sell their product. They're truly trying to understand the user's potential customer's problems and then they're immediately trying to craft it like, "Oh yeah, we've got that, we're able to solve that."

They're able to reposition their product back to them. And so this is why you're able to actually, for even complex enterprise software, if you truly understand the user or talk to them, you're like, "Oh, absolutely. We could totally build that. We can have that this time frame and I think we can do it at this cost."

Would that be something of value to you? Would you like to sign a contract and have us committed? We can start with a patron or whatever. The whole thing is like through the idea, you can sell it to the right customer if they believe that you're capable and that you truly are going to build something of value to them.

And so I did something where like for my first product, even before Pebble, we set up a landing page with Shopify or we set about we set up a sales cart where people could actually go through the checkout process to buy the product. We didn't feel comfortable about charging people's credit cards, but we made the checkout process look exactly like they would have to pay.

So we figured we would get a better signal from those users instead of just like an email signup form like, "Hey, are you interested?" They actually went through the flow thinking that they were gonna have to pay something, and at the end we said like, "Oh, we're actually accepting payment in a couple weeks."

But at that point, they were already so invested in me, we think that it was a more effective way of kind of gauging interest. So he's asking, is like the open source model where we release a product out for people to start using for free as a way of like building up an audience that you eventually will sell to? Like is that pretty much how that model works?

And so I think for most of the successful open source like base companies, like say for like GitLab, yes, that's pretty much how it works. They're thinking of building up an audience. I think for us like that advice for us came directly from Jason Fried of 37signals.

So he had talked about this story at South by Southwest doing big things with small teams, and they had been blogging for years on their blog 37signals before they finally launched Basecamp. And they realized like that was super valuable to them because people didn't start from zero when evaluating their product.

They felt like they already had a relationship with these people and there were all the exact kind of people, the audience that they knew that they were building for.

The Segment story is pretty good in this way as well.

Yeah, yeah. So the founding of it's kind of Segment, the story around how Segment got started, they actually created an open source library that analytics taught Jas that you can embed into your website, and it would kind of duplicate all of your analytics or would send analytics from your site to all of the different existing kind of aggregators.

And it was an open-source project that went crazy on Hacker News and they ended up using that as a way to then sell the rest of their offering from there. I think it was like a crazy small number of lines in the first version, and then the CEO at the time was like, "Not a chance, nobody wants your stupid little code, why are we announcing this?"

And it became the whole company and then it like just went by. Oh, it just went nuts. Like people in Hacker D's just couldn't get enough of it. People were actually, "How do I get access to this code?" etc. And then that became the counterpoint to the charge money early because that was a free open-source project that had no business model whatsoever at the beginning but again got a bunch of users.

Then they figured out like, "Okay, how do I charge? How do I turn this into..."

I love talking about this because this is a problem that comes up all the time. So especially for hardware companies, but I would have tracked the problem to the question: like broader, like if your MVP actually takes a lot of work to do, if there's a lot of building that needs to happen before you can offer kind of that first modicum of value to users, how do you do that?

Do you do a paper prototype, etc.? So I love this because I think that there are great ways that you can hack around this problem. It's very easy to say like, "Oh, I'm building the super complex piece of software or hardware. I need to spend at least two years before I can talk to customers or before I could find out whether they're interested in buying this." That is not true. You can start talking to customers on day one and here's how.

Talk about the existing ways that they're solving this problem. Figure out what are the products they currently use to solve it and figure out what the pain points, what the problems are from there. The way that you validate that they're interested in buying something is hack together off-the-shelf existing technology that kind of approximates the problem.

Obviously, it's not going to be as amazing as your beautifully designed perfect thing that you're working on, but you'll get an approximation of it. Show that approximation, show that hack to the person who just described that they have a problem and try to sell it. Dress up other people's technology as your own, like put it together—put your own 3D printed box on it, you know, duct tape it together, put your name on it, and try to sell it.

Because what I've learned is that if people aren't even interested in buying or using the hack-together solution, then the thing that you're gonna work on for two years may not actually be like that much of a solution to the problem that makes it exciting.

Like really, if you're working on a problem that's so burning, that's such an acute problem, the user should be willing to work with like a kind of crappy hack-together solution before you finish your real one. You can do so much with a landing page just to understand interest whether your message, the idea resonates. They'll be excited.

They see a light. If they have such a burning problem, they see a landing page for like this product that finally solves their, you know, Wayas hopes and dreams, like they'll get in touch with you.

I think the other thing is like Dropbox actually started that way. So Dropbox was launched as an After Effects movie that approximated what the experience was going to be like to share Dropbox. And then everyone went nuts. It’s like, "I want to know when that software is available!"

And it like almost no lines of code would have actually been written, and so it was basically a product demo that company.

[Music]

[Applause]

So the question is, you know, it's harder to get a consumer product out there, like the digital space on Instagram, Facebook, everywhere. It's getting much more expensive. And it's true, I see a ton of like new products that are launching on Instagram and I keep thinking like how much money did they pay to get one ad in front of me? It's crazy.

So what can you do at an early stage without that much money? Kind of going back to the theme of this talk—you can talk to users. Talking to users is actually pretty cheap because people on an individual basis are probably willing to pick up the phone and talk to someone.

Like I said before, oftentimes, like they've never talked to a tech CEO. They've never talked to someone who could actually build a solution to their problem. So if you position yourself as someone who may actually be able to help them, they may talk to you just for free because they see an opportunity.

Just as a side note, like oftentimes when I talked to a founder who's working on a product that may potentially help me, I know that they're potentially willing to build like the exact thing that I want if I just tell them that.

And so you may find people that are willing to kind of open up the kimono and share that for free. And kind of that's kind of a way to get a couple users to talk you. If you want to get more in aggregate, the most scalable ways that we've seen work for our founders are really like putting yourself in the demographic of interested users and figuring out where they live in the world.

Whether that means like in the physical world. Like we have companies that hang or tag on houses because that's like an effective way of getting people's attention. And kind of the digital equivalent of that is infiltrating communities where people who share, who have this problem are sharing their thoughts.

Like oftentimes, there's a Facebook group about a particular medical condition or like a particular travel destination, and it's actually pretty effective to just kind of like position yourself as an expert in the space and kind of organically enter and be part of that community.

We're gonna add in questions, so you guys got 15 minutes to talk to one another. Thank you so much for coming to the first lecture. We're gonna stick around for a little bit, and then we have to clear out of this room, but enjoy, and we will see some of you on Saturday in New York and then back here again on Wednesday. Bye, guys.

[Applause]