Breaking Down HackerRank's Survey of 40,000 Developers with Vivek Ravisankar

All right, the Veck, why don't we start with what you guys do, and then we'll rewind to before you even did YC?

Yes, sure! I'm S. V. Ivent, one of the founders and CEO of HackerRank. Our mission at HackerRank is to match every developer to the right job, with the underlying driver being skill. I mean, resumes are just a very poor correlation to skill, so that's how we got started.

The way what we do as a product is that as a developer, you could come on to HackerRank to practice and improve your skills across different dimensions, whether that's algorithms, artificial intelligence, or data-based skills. When you're applying to a company, instead of going through the traditional route of uploading a resume—it's mostly going to be a black hole—you take a company's coding challenge. If you want to apply to Airbnb, you take an Airbnb coding challenge; if you want to apply to VMware, you take a VMware coding challenge. If you have the required skills, basically we automate the whole code and give you a report. Then you get called in for an interview. So it's great for developers because it's a merit-based process. It's great for companies because now you're able to attract the right developers for their interview process.

And what might one of those challenges look like? For instance, Airbnb, are we being put into the code base in any way? What would it look like?

Yeah, I think that it varies. We built a platform where it's super custom for companies to go ahead and create. You could have a simple problem-solving challenge or something close to a real-world challenge where you could give access to our GitHub repo. Okay, and people can go ahead and modify the code, fix a bug on your own repo, and do it. So there's a wide range of problems that you could give on the platform.

Okay, gotcha. So you guys haven't always been doing this, correct? In its exact instantiation?

Yeah, so let's go backwards. Before YC, what were you working on?

Yeah, sure! So Socati is my co-founder. Right, we knew each other from college. We used to do a bunch of things together in college, and we started the company. Technically, we quit our jobs. I used to work at Amazon as a developer before this, where I did a lot of technical interviews, which is kind of the way I saw the problem. Yeah, you know, the resumes were not a good correlation to skills. So we quit our jobs. The way we started to solve this problem of resumes not correlating to skills was to help prepare students for their interviews. So we started Interview Street; that was our first version.

What it did was mock interviews. So let's say you had an interview call with Amazon or Google next week. You could come onto our site and attend a mock interview with either somebody from Amazon or somebody, an ex-Amazon person, who can give you a walkthrough of the things. We would do a mock interview, give you ideas that you need to improve, and tell you where you stand on all of those things. The students would be paying these people for their time.

That's right.

Okay, so it's gonna be hard for people to comprehend the value here in the US because we were doing it in India, but we were charging 350 rupees in Indian currency from the student and 200, and we used to pay 250 rupees to the interviewer. So 100 rupees goes to our pocket, right? So you could almost imagine that to be something on the lines of—I mean, technically it's six dollars and five dollars.

Yeah, but as you mean, you earn different amounts. You could say a student would pay $200, something like that, and an interviewer will get about $150.

Okay, so then if you then think about it in that way, it was pretty expensive for students. And we didn’t have any good payment gateways or anything of that in India. So we used to set up campus ambassadors in different colleges, and Hadi and I used to do a trip every month to visit all of these colleges to collect money from these campus ambassadors in cash.

Yeah, so our accounting was all pretty funny. And from an interviewer's perspective, 250 rupees, or like $150 or something on those lines, here is not very enticing. All right, if I have to spend an hour of my time doing a mock interview and giving you feedback, is it worth it?

So I still have the Excel, which projected us to make, I think, somewhere on the lines of a hundred million dollars, or hundred crores, whatever. It's like fifty million dollars and along those lines, yeah. In the beginning, I think we made about 4 thousand rupees, or maybe about—that's like equivalent to a hundred dollars here.

Okay.

After a year or a year and a half.

So clearly, you could just drag the cells in the Excel in whatever way you want and get the numbers, but it's very hard to execute. So that failed. I mean, we had a lot of lessons that we learned from that. Most importantly, I mean, the students didn't have cash. There was a lot of scheduling issues. And I think one of the things that YC stresses on the most, I wish we had actually done YC at that point. I think this was done by Paul Buchheit, “Make something that a hundred users really love versus making thousand people or ten thousand people sort of okay.” I think we should have focused on, you know what, we're just going to focus on these two companies, mock interviews, right? Versus any tech company anywhere in the world.

And we were scrambling for interviewers from Amazon, Google, Microsoft. Rather, we should just focus on: “You know what, these are the three companies we’re going to focus on. If you want to get a job here, you can come in and make it really happy and then figure out a way to scale.” Looking back.

Yeah, of course. But did people like the product?

Yeah, whenever people did the interviews, they really liked it. In fact, there were a lot of really interesting testimonials from people who actually attended the interviews, got their jobs; they knew what are some of the areas that they need to improve. They need to prepare and all of those things.

Yeah, that's great.

Yeah, I mean, I remember all the interviews I did during college. If I could have just gotten the email of someone at one of these companies, it would have been invaluable to me.

We used to do that sort of informally in college, right? When we were in the final year of college, we used to train people for mock interviews and others, so this was more like an extension.

Yeah, why can't you do this over the web across everyone?

Interesting. So you weren't necessarily off the mark around finding something that people really liked, but you found that you kind of hit a ceiling somewhere and it was not very focused either.

So there were a lot of different cohorts of people. Some of them were students; some of them were a couple of years into their work experience. Some of them wanted to do technical interviews; some of them wanted to do automobile interviews. And then we had to go ahead and scramble and figure out, can we get somebody from the automobile division to come and take the interview? We had no contact, so we had to figure out a way to scramble through LinkedIn and Facebook. I think it was sort of okay, but it really taught us to just stay strong.

Yeah, okay. And so, at what point do you decide, like, you know what, we have to change this; this isn't working?

Yeah, it's probably about 18 months. Well, we had applied to YC.

Or you did!

We did apply to Y Combinator with this idea. Of course, we got turned down. Then I think it was about 12 to 15 months, something on those lines, when, I mean, clearly this wasn't going anywhere. The Excel was completely off the trajectory of the Excel graph and where we were. And then we switched our business model or idea to something different.

Okay, so there's a big trend, or there is still present in India, which is once you graduate, you have to do your masters in a school here.

Oh, it's huge! I mean pretty much, you know, I don't know, 80% to 90% of people would attempt their GRE, which is your standard way of getting here, your TOEFL. They will prepare their statement of purpose, they will do all of those. And the application form is pretty expensive, right? You could only apply to maybe five or six universities because it was so expensive. You couldn’t just apply to all the schools in the US.

So what we did was we connected people who were already studying in these schools who could review your statement of purpose, your resume, and tell you if you had a chance to apply to the school.

Okay, so that was our pivot. It actually took off a bit, but then after three months or so, the traffic was down to zero. Turns out you can only apply twice a year. So we didn't know what to do for the rest of the year.

We don’t actually apply to YC even with this idea.

There you go.

So I think this whole thing about mock interviews and helping people apply for the Masters maybe a year and a half to two. Something on those lines, okay, that we worked on.

Okay. And then what sparked you to start working on where you are now?

Yeah, well, we were close to bankrupt. So there was a real need. And you know, of course, we had really good jobs, and Amazon was a great company.

Were you in India working on all of this?

Yes, in India.

Okay, and so then we realized, okay, so just the urgency really.

Okay, how are you making the connections with people here if you were based in India?

Sure, so what we did was we used to take the directory of people from our school who went ahead and did masters at all of these different universities and who are working in different companies. So we used to go through the directory, keep calling people, emailing them: “Hey, are you interested? We are from your university, that university!” Touch in to get them on board. So it was just friends’ networks, second degree. And I used to make, I don’t know, 50 calls every day.

Wow!

To get this going.

Okay, and just for context for me, people from India want to do a master’s here because it’s a gateway to getting a job here, or what?

I don't know how it started. Maybe it was just—I think it is sort of a checkbox. I'm not sure if this has changed now, but it was definitely, I mean, I graduated eight years back, or nine years back. It was definitely a pride, “You know what? I'm going to do my masters here.” And there’s of course some glamour associated with the great American Dream, coming here and working in Silicon Valley. Silicon Valley is sort of this mythical place for all technology developers to be here.

Okay, so it was more like a checkbox. I still think it is the case that you would have to apply if you had a good GPA and if you had good grades. The question is, why aren't you going? Why are you applying for a master’s?

That's funny, even among the very talented smart kids going to great schools in India.

Yeah, sure. I think you're applying for your master’s; why aren't you doing your PhD here in one of Carnegie, Stanford, or Purdue? And there are a lot of—actually, Hadi and I might actually be the odd men out, 'cause we didn't do it.

Yeah, we didn't. Anyway, yes! And I’m happy because I'd applied to six schools. I was so dejected when the school held me down, but in hindsight, it’s great, right? I don’t think I would have started the company had I come here.

No, probably not. This was a conservative move.

Yes! And once you graduate, you'd have to get a job because you have to pay back the loan, which is pretty big. So you do two years of master’s, then you do two to three years of working in different companies to pay back the loan. And there’s all of these visa things, and by the time you want to start, you’re already, I don’t know, close to five, six years invested in this.

That's really common.

And the friction or the inertia to go ahead and start another company when you’re happy working at Amazon or Google, it’s gonna be hard.

Yeah, definitely.

And so, okay, so you're basically out of money?

Yeah, we're still close to bankrupt.

Yeah, yeah. And yeah, so we had a clear choice, which was we could go back and get a job. I think we could have gotten that, but something, we just wanted to keep going. We just wanted to try this out once more, and I have no idea what kept us going at that point in time.

And there was some conversation— we were just walking on—which was, it sounds obvious: if you have to make money, you have to sell something to people who have money. There’s no way you’re gonna make any money. We were selling to students, who were already in whatever they had, called student debt, and they had very little money.

So that's when we said, “Hey, can we invert this?” Which is, can we build a product for companies to help them identify the right developers based on their skills? Of course, we didn't have the mission written down like, “Match every developer to the right job,” but looking back, I think that was the core of every decision that we made. And we felt, you know, we'll still stay true to our core, because mock interviews is one way of doing this. But if you actually give a platform that can enable developers— does it matter what school you went to or which company you went—but you have a chance to showcase your skills and get the dream job that you want, then that's great!

You know, we should build a platform for that. You know, that's what we got started, and that got the early attraction. So that was the third time we applied to YC.

And yeah, YC called at that time?

All right, and was it because you were showing significant growth at the time, or the idea was just working out?

I mean, I don't know! I definitely didn't want to ask why we were chosen. This is a mistake, sure.

So that was our first trip to the US, yeah. So, and we had some struggles there as well. Hadi, my co-founder, couldn't get his visa to come here, and I still think it’s the stance of YC, but YC cares about co-founders a lot. So I got my visa; Hadi couldn’t get it. And so I flew in here just for the 10-minute interview. This was, yeah, six years back.

And fortunately, we got in. I remember, you know, I think there was a panel of interviews; it was PG, Sam, Harj, Jessica, and Paul Buchheit. There were five people doing the interview.

Okay.

And you know, that's the foot I landed in the morning; in the evening, I was at YC because I couldn’t get my visa on time.

Oh, the last day of interviews!

Yes, on the day of interviews, only the last one, like, yeah, the last one as well.

Could it be?

And then you see all these five people. He’s the creator of Gmail. You're obviously a little bit stressed, and then you don’t have a co-founder. And PG asked, “Where is your co-founder's visa?”

But it was a good, intense ten minutes, and frankly, I didn’t think we’d get into YC. So I was—maybe it worked to our advantage because I was just completely open and honest about everything because I didn’t fake any of my answers or try to impress them. Look, this is the problem. I really want to come here and go ahead and do this, and it probably worked out in our advantage.

Interesting!

Yeah, I mean, so what you were working on before has this separate difficulty in the application because you have to educate someone about what it’s like in India versus here, where we have the exact same problem of like, “Oh, we're this company. We have all these jobs. We don't have enough engineers. Let’s sort this out.”

Yeah, yeah, I think there is an imminent way. Things— I mean, yeah, you're right, the environment is different. I mean, I remember I used PG’s—so I didn’t have—so I used to sit in the Pioneer office, white-dominate office for most of it. Actually, our first office was right opposite to the YC Pioneer office.

Yeah, it was a rethink TBS old office, okay, which we actually took. So it was right opposite. So I used to be in Y Combinator for the most part. I remember I took PG’s charger, Mac charger, by mistake. Oh, and this was the second day or the third day.

And I was— I was so kind of—I didn’t know what to do because, “Oh my God, what is PJ gonna think?” And then I started emailing, “I’m really, really sorry about the charger.” He said, “No problem, just keep it, just keep it. There tomorrow.” And I said, “Okay, are you gonna come in tomorrow?” And he said, “He's not gonna come in tomorrow.”

I, because the way, at least I was brought up and for most part in India is: if you keep a charger or if you keep the Amazon box right outside the door, somebody's gonna take it away, right? And even the tiny things. And then I had to learn about how the environment is different.

How people talk, how people react to it. In dinner time here it is 6:30. So I remember sending a calendar invite for dinner to a person that I want to recruit for 9:00 PM, and he said, “That’s kind of my bedtime so I can meet you in nightly.”

Yeah, so there are all of these things that you have to learn. And it's not just the macro part of the developer how it works, but even the tiny things about how you interact, how you work with people, and all of those things. You have to learn.

Yes.

Yeah, has that been incorporated into your product? Because I imagine you have all these international people applying to US companies.

Yeah, how do you help educate them on the cultural norms?

Yeah, no, it is a really good question because, right from how you sort of... how the types of sentences or words that you use varies quite a bit between what you would say to somebody in Asia, what you would say to somebody in the UK, and what you would say to somebody here. So all of those nuances we need to take care of because you want... there is a different culture aspect on how you sell, how you talk, how you get people going.

Here it's really difficult.

How long does it take you in particularly?

I think it's a forever learning. I don't think I've ever perfected it. I’ve got in my dinner time now at 7 PM.

Yeah, shifted a little.

But I think it's a constant learning. I don't think it’s ever gonna stop.

Yeah, my friend moved to India for a couple of years and he explained the exact opposite.

Yeah!

It was tough for him.

Yeah, I mean, it's just the way you grow up is different. I mean, there’s nothing—that's right or wrong, it’s just different, and you have to get used to it. And I remember PG and Sam correcting me from everything to even the tiny ones. In India, you would probably say “programmer.” You would add the “a” at the end, but here people will not be able to follow. It's “programmer.”

So if you have to say— and when I was doing my demo day pitches, all of these little things mattered because it was just a five-minute pitch, or no, I think it was a two-minute or a five-minute pitch for the demo day. And every little thing matters a lot because when you're talking to investors, so it's not just the accent but how you talk, how you schedule meetings, how you follow up, and everything is different.

Yeah, demo day stuff universally, because...

Yeah, I mean you're pitching a master group. I think you said your batch was like 60 or 70.

I think our batch was 63 or 65.

Yeah, and so that's enough companies or one to just fall into monotony.

But interesting! I really enjoyed it because I've never been exposed to—exposed to such a large group of investors. And I was—I couldn't even sleep the previous night. I was looking forward to it. I wanted to pitch and get invested—get a seed stage going, so I frankly enjoyed it. I was looking forward to it.

That's very cool!

So you raised money after YC?

Yeah, and you're committed. You're like, okay, we're gonna stay in California at this point.

Yeah, I mean, after I came here to Mountain View, and I don't know, maybe that's just partially the reason. The first city—well, I technically landed in San Francisco but immediately came to Mountain View. It was—I just—I think everything looked great for most part.

I mean, right from the kind of the people that you interact with, the advice that you get, the weather, the food, sort of the perfect mix for me coming from India. You had the weather similar to Bangalore and the food similar to Bangalore and Chennai. And then you had all the people that I really wanted to talk to and get advice from in terms of building a company. So it was the perfect mix, at least for me. So I was very much determined to stay here and build a company.

Man! What was that like, you know, flying here and then seeing all the people who you read their blog posts— PG is an easy example. What was that experience like?

Yeah, it was—it was kind of surreal. I didn’t know it was—as I said, I thought I was going to just stammer and flunk the interview. But like I mentioned, I didn’t think we'd get it. Actually, we were the first India-based company to get into Y Combinator.

Really?!

Yeah! And so that was another quote-unquote—I don’t know if it helped or didn’t help because we thought, okay, they've never chosen a computing-based company, so our odds are low.

But that really helped because I was just completely free and open. But it was great; it was great to meet all of the people that you admire far off from reading your blog post or Twitter or watching their videos.

And what were your greatest learnings from the experience from YC?

Yeah, I think the intensity was very, very high because there was a dinner every week. And I mean each—I noticed that each founder had their own set of people that they interact with. Of course, I interacted with a lot of them, but you talk to a few people and then you hear their updates every week.

And you want to push harder. You have a fixed deadline three months down the road, a three-month deadline for demo day, which you have to go and present to investors. So you have a fixed demo, fixed timeline, to make sure that you have a good prototype. You could possibly have paying customers to go ahead and do it. And I think the bar continues to increase, right, at every demo day.

So I think the intensity—I think the learning is—I still think Hadi and I have that intensity for most part, which is good.

Wow! How do you keep it up? 'Cause people talk about this all the time. They're like, it’s so cool having a cohort that you can kind of benchmark yourself against. But once you're out, you're kind of out.

Yeah, I don't know. Maybe we're just its DNA or we're just consistently hungry to win. But I think the challenge is can you extend that across the entire company? I don’t think we've done that. We've done an okay job at that, right?

So I think that's something that we still need to get better at. That's a challenge.

So sort of fast-forward massively, yeah, now the reason we started talking about doing the podcast in general was that you guys released this developer skills report.

Yeah, so just for folks who haven't seen it, can you explain it?

Yeah, sure! So we have a developer community that’s over—I think it’s 3.4 million, as of last week—that’s slightly over 10% of the world's developer population who come to HackerRank to solve challenges to practice their skills and get better.

One of the things that we wanted to do was learn more about our developers, so we sent a survey. Actually, it had 40 questions. There's a lot of debate: you know, will developers fill this survey? And then you’re sending a survey to developers with 40 questions, something on those lines; what are the odds that somebody is actually going to fill this?

So we sent this to a million of our developers.

Whoa!

It was amazing. We got close to 40,000 people to complete the survey. That was huge! I think Stack Overflow had 60,000, and they have 20 million developers or so. So if you compare the ratio, the engagement, and how many people completed the survey was giant, and hopefully we can beat Stack Overflow's number next year.

Yeah, what was your email subject line? How'd you get so many people to convert?

That's a good question! We did a B test, but maybe Mykola, our marketing team, Radhika—they might have a better idea on what was the subject line. But we did A/B test a few things, and we also tested on the timing, you know, when to send to which cohort, depending on the country, depending on the geography.

And all of you, because you're fairly distributed, right, in terms of user base?

Yeah, so maybe it's about 26 to 28 percent in North America, about 40 percent plus in Asia-Pacific, and then you're talking about 15 to 20 percent in the UK and the remaining in the rest...

Okay, so for each of the regions, we also optimized the subject line, when to send them, and how do we actually position this. So we did a bunch of things on those lines.

Was it dialed in for language or all in English?

I know it was all in English.

Okay, yeah, I don't think we changed any language. So the questions were waiting from when did you learn to code to what do you look for when you're switching for a job? Or if you're a hiring manager, what do you look for in a candidate? So we had a wide variety of questions and some really, really interesting insights.

So we had engaged a firm who could parse the data, who could do real analysis on this, and we're also likely going to put this up on our website for all of our developers to go ahead and do the analysis with our raw data. So something might happen maybe in the next couple of quarters.

Yeah, it was a really interesting insight, and we published this earlier this year. We got over half a million developers to read the survey. I think this was a huge milestone and exceeded kind of at least my expectations in both the quality of the content, the design, the reception that we got. And of course, there’s a podcast with a day, so it exceeded my expectations in all dimensions.

So, all right, I guess the most important question is what programming language is the most popular?

I think it varies across different industries. And you know, for—I think the one that people want to learn a lot is Python, and the one that's starting to get a lot more popular is Go. But it varies across different industries, from what financial services want for this fast-growing internet companies want versus retail.

Just all of these companies are transforming into software factories, so each of them has a different variation of it.

Okay, so for a developer who's curious and sees the survey, do you kind of funnel them through a flow on your site? Like, if you're interested in getting into banking, you ought to be learning...

Yes, yeah, that’s a great question! So we are trying to do what the rivers, which is what do you want to learn? What language and skills do you want to learn? And then we'll help you recommend, you know, where can you most fit? Where do you see—where does HackerRank see you, the best fit, or the best job for you based on all the data that we've collected?

Okay, gotcha. So let's go through some of the facts and the fun facts.

Oh, sure! I saw there were, in the beginning of the report, it’s a lot of stuff about age.

Yeah, so when did people start learning how to program? When does it all begin?

Yeah, I mean, I think the most—the fun insight was one in every four developers learned to code before they learned to drive. Yeah, that was a really interesting insight that we were able to get, and it’s not super surprising for me as a developer. But when we announced this, it was just—I think it was the most popular insight. We even started printing t-shirts in our company; that was—that was an interesting one.

And it just shows the fact that self-learning is starting to become big. People are learning on their own. You don’t necessarily need to go to a particular school to do your four-year degree and then understand how to code or how to build applications.

And it's just going to—I think over, if you just extrapolate the trend line, I think this will be two in four or three in four developers will learn to code before they drive in the next decade or so.

Yeah, and did you collect any data around boot camps, coding boot camps?

Yeah, we did. We did collect—I think we collected data on how much the hiring managers, or others, valued the boot camps, and learning—it was mixed, to be frank. I think there were certain hiring managers who valued it a lot and certain hiring managers who didn't.

And my guess is this probably depends on the quality of the boot camp, right? I mean there are a lot of them who—there are a set of boot camps that say, “You know what? Come here, it doesn't matter whether you know how to code or not. I’ll help you become great in 90 days.” I mean, that's not how it works, right?

And it almost seems like a negative signal for sometimes which is, “Oh, you didn't know how to code 90 days before and now you’re claiming that you’re a React expert.” You know, that puts me off. So I think it depends upon the quality of boot camps and then—then on the other extreme you have this school which I was personally really impressed by; it’s called 42.

Okay, it's a school. Actually, PG tweeted about it.

It's referenced?

That's right! They just opened another one in Fremont, where I had a chance to visit, where you, of course, have to take a minimum level of coding proficiency challenge or a test.

Yeah.

And then you are allowed to explore whatever types of computer science topics that you really like and then figure out which one you have a natural attraction or a passion for—whether it's security or machine learning or front-end development—and then continue to get better at that and then will help you connect to the right job.

So that seemed to be a different approach, but it's very different, right? Because if I understand it correctly, they're giving people room and board. That's good, and it's like a huge loan!

Yeah, it’s free?

Yes, and it's over the course of years.

Right, right.

Yeah, so it is very different!

Be interesting to see how that experiment works because I think it was started by a really big billionaire in France or something; he just wants to help the world. So I'm not sure how it might work from a business value proposition, but at least in terms of net improvement to the world, to create more developers and to help improve technology, I think that’s a great start.

Yeah, so I think that’s why the boot camps had mixed responses from the hiring managers at least.

Okay. And then what are what are the differences between, you know, startups hiring and big, you know, like Amazon-type companies?

Yeah, I think the one common thing was everybody stressed on the importance of problem-solving skills, irrespective of whether you're a smaller company or a bigger company.

And the one thing that the startups or the execs cared about the most is—because at least in a startup you maybe you might have a recruiter or not, but most likely you're doing the recruiting on your own—they care more about your contributions or your open-source contribution.

So your profiles or portfolio—much more than what a large company cared about. But at the heart of it, people wanted strong problem solvers!

And also, one of the things that was present in the survey was how you wanted to dive in deeper on the knowledge of that particular stack, at least for a startup, because of course you have very low or pretty much no bandwidth to train somebody on the new stack that you’re working on versus a large company where you have a little more bandwidth to go ahead and train.

So you're okay with the generalist who can come and learn a new language, versus in a startup, I like you to know the particular language or the stack before you come in. So those were some of the nuances of the differences that you can see between a startup and a large company.

So in terms of structuring a job offering to someone, or rather just like a job posting, what are the things that end up appealing to developers?

Yeah, I mean, this was very interesting because I assumed differently. And of course, always data over here. And one of the things that we learned was how work-life balance beats all of the different perks that people offer—whether that's free food, or “we have a ping-pong tournament every week,” or things along those lines.

And then we—I actually didn't believe this data, firstly, so I wanted to do another survey for people who did the work-life balance, who actually opted in, “What would you really mean by work?” And what we learned was a lot of developers have their own projects that they want to learn, that they want to build on the side. And the environment that you're working should give a good amount of balance.

And probably even support if you can do on the things that they're learning on the site or building on the side versus what you're doing for the company.

Yeah, so that was kind of an interesting insight about work-life balance, and one of the things that, you know, I'd like to do with HackerRank—we're still discussing on how do you position this in the right way—is if you have any of the side projects that you're working on, yeah, we'll fund the infrastructure cost for that.

So, it's—we're still figuring out how do you position this, how do you give this away. And I'm pretty sure after people hear this podcast, they’re gonna email you stuff, but that's one way of helping or encouraging people to do that as well.

Oh, interesting! And so what are the other ways that companies signal that they have that work-life balance figured out? Or is it just like folks come in for the interview, they get a sense for it, and then they say yes or no?

Yeah, that is one—but I also think if you're actively contributing to open-source, whether that’s— I mean, for example, we are improving a lot of the open-source code editors that are there right now. We've not done a great job in letting the world know about all the improvements that we've been doing, but that's one way.

And of course, interviews helped a lot in understanding more about the developer culture.

Okay. And do you guys have any rules around working on side projects? Because I know that like at larger tech companies, there are totally issues. And like, you know, I don’t think they really flex that muscle very often.

And ever?

Yeah, but you know, like companies like Apple are infamous for not letting people do side projects.

Yeah.

Yeah, I mean, that's probably one of the advantages when you're small, which is—I mean, I don't think we have any rules. I don’t think we should put any rules because it's just freedom or creativity to go ahead and do it.

And frankly, if it’s gonna be conflicting with your core idea, and you're scared about it, maybe you should take the developer's idea and implement it. It's better! Or otherwise, you could just—you don’t need to be scared.

And you can just have the developer work on the projects that they want. So I know— I don’t think we should put in use, but I mean a large company like Apple or Amazon or Google will need to put in constraints.

Yeah, yeah. All the Google hack this by putting a 20% rule where you could 20% of the time work on your projects. I think that's how Gmail was born where I think Paul Buchheit worked on his 20% free time to build Gmail. So that's how they hack it.

It's not really technically a side project that you do outside. You use all the Google infrastructure and libraries and eventually contribute to Google. So that's one way you could try and do it.

Interesting! And what else is attractive to a developer?

Yeah, and the other thing that we had in the survey was really on continuous learning. I think there are just so many new things that come up, new layers of abstraction, right? Which is, of course, very enticing for a developer. New frameworks—I mean, there's a framework in Java that's launched every month or every two months, which is kind of driving me crazy because you just have to keep migrating from one over to the other.

And the other thing was continuous learning: can I learn new things? Can I learn how to get better? So that is a forever learning attitude or culture.

What else did you learn about how developers are actually educating themselves? So obviously some percentage are going to boot camps; some are doing CS. What is everyone else?

Yeah, it was really interesting. Even though you had a CS degree, a lot of developers said they actually learned a lot from YouTube work sites, like HackerRank.

It's not a plug for HackerRank, even though I think the title might say this or whatever. But that was really interesting because that’s a new—that's gonna change—it’s a paradigm shift right now. Because if you look 10 years back, the paradigm shift in recruiting is: if you look 10 years back you’re going to use GPAs and universities and the companies that you worked at as proxies.

Yeah, but now the proxies are changing, right? It doesn’t matter if you went to a grade school or a great GPA. If you learned on your own and built a lot of things on your own, that is all that is needed; that is all that is enough.

So the way that you look for in a candidate: Who do you look for? How do you assess? How do you get people in the door? And how do you remove biases that have been there for not 10 years? That’s been there for a hundred years.

Or do you go to this school, or did he work in this great company? How do you remove these biases and truly focus on skill? I mean, it's a lot of work to go, but that’s a big paradigm shift, and it’s a very good and healthy trend if more developers can learn on their own for free.

You don’t have to pay a hundred thousand dollars for tuition. It’s good overall for the world to do that.

Interesting! And so how would someone stand out? Like, what are the kinds of projects that end up appealing to a company? Because I know if you do a lot of these boot camps, like, “Great, you made your own CMS.” Like, so did ten thousand other people!

Yeah, did you get any insight as to, you know, when someone does care about these projects, what are the kinds of projects that appeal to companies?

Yeah, I think it’s hard to—I think there’s one part of it, which is your sort of motivation or your drive, which can really come through—which is, I mean, if you look at your repo and made a couple of edits in one file, it really doesn’t—you can’t claim that you actually contributed open-source or anything.

But I think one indicates a drive. The second is also—and which is probably hard to just look at a HackerRank profile or a GitHub profile or any of the profiles, and you can't just look and figure it out—is the depth of thinking and what are all the things that this developer wants to do more.

And that’s something probably you can get from the interview. But the bigger question really is, would you be willing to bring this person on-site and go through the interview process based on their skills and not on their pedigree?

And it sounds obvious, but we're trying to educate our customers and everybody, hiring developers: this is the right way to go, and we're sort of leading and pushing the movement for this, and hopefully we can be successful at that.

Yeah, cool! And how does this affect a situation where you're hiring remote developers? Are there any other particular cues that you pay attention to?

Well, yeah, what it’s worth, yeah, I think the communication aspect is always a big deal. How do you get everybody in sync? I think some of the companies have done an amazing job. GitLab is a YC company. I talked to their CEO, and it’s fully remote.

It’s crazy how—then it just shows how much effort you need to put in to make sure that people are in sync with each other. I do think—and you're seeing this happen a lot. GitLab is just an example.

But I know a lot of other companies are starting to do is—I think the future would be an ability for you to build out engineering teams wherever you want and not just restrict yourself to the Bay Area.

And a good common connective tissue that makes sure that things don’t break because of bad communication or because of time zone differences and others, I think that's where the world is going to move.

And which means ability to know their skills and how they can contribute when you're not in the office, and all of the others is going to play a very, very important part. Like ability, do you understand both of this at a deeper level? It’s gonna play an important role.

Yeah, and have you guys conjectured or started working on anything that might be able to test someone's efficacy as a remote developer? Because it's one thing to have the technical chops; it’s another thing to be able to work in, like, a co-working space or in your bedroom or whatever when you’re in a different time zone.

Yeah, I might get done.

Yeah, I mean, I think I think there is this one part, which is the IQ which is how good are you in all of these different skills, where do you need to improve? And the other part is EQ, which is what your personality is.

We’ve been doing some experiments or partnerships with a couple of them, but we've not had a real breakthrough on that yet. But I do think that is going to be—that's gonna be helpful.

I mean, the reason I'm saying that is there are a lot of companies who use HackerRank to build out remote teams in India and other parts of the globe. There is a company which I believe is Booking.com, one of the most popular travel companies in Amsterdam.

They are able to hire developers all over the globe using HackerRank, right? And they're doing it in a very efficient way because they understand the skills, strengths, and weaknesses before you come on-site and before you fly to Amsterdam.

So I think we're starting to see this movement at least happening in a lot of companies on the IQ part of it. And now, if you can do a better job, or if you can work with a partner or if you can do something even internally at Hacker, you can do the EQ, it can be a very strong solution.

And so this is all done through programmatic exams; there aren't any in-person interviews to assess EQ.

There are some companies who are working on this programmatic way. It sounds very interesting; we've not partnered with them yet.

People really understand how well it'll fit in, but it’s certainly a very, very interesting opportunity.

Okay, so in addition to maybe figuring that out in the next—all, how else do you see paradigm shifts happening for developers in the future? You know, obviously, they'll be all over the world.

Yeah, what else is gonna be happening in, you know, five years?

I mean, there are a couple of things—one is, I think it was there on Hacker News, as much as there are APIs and different levels of abstraction that have made it easier to get things started, to set up your environment. And this is for somebody who’s a newbie, who is just learning to code; it’s very intimidating to set up something and get it running in the compiler!

I mean, you could have—I think that’s one part where things need to get better. Here’s an environment, here’s where you can go ahead and start coding because a lot of them give up. Oh! I did this thing online and then I tried it out on my local; it didn’t work because I don’t know if you did, you have this library and there was a dependency failure and you have to go and figure it out, and it just becomes a big mess and you give up.

So I think that is still—you know, we still have to do a better job if we have to increase the number of developers in the world. And you can’t have people giving up after the third lesson or the fifth lesson and saying, you know, it’s not taking anywhere and—

All, so there needs to be a purpose. I mean, the purpose can’t be you’re gonna get a hundred thousand dollar job, right? The purpose has to be much more whatever is in your mind, you can go ahead and build it on your own.

Right? I mean, you have to encourage the fact that you can build versus, I’m gonna get you a six-figure salary to do it! I think that’s one part that needs to get better if you really have to create more developers in the world.

Well, why does that have to get better? Because a hundred thousand dollars an hour—I mean, I agree with you—yeah, but to be devil's advocate...

Oh yes!

Like, that's a lot of money to a lot of people.

Yeah.

Is that not work?

I think it will work. It will work for a subset of people, obviously.

Sure!

But I think, you know, in a very longer term, the really great developers are people who did it not because they'll get a job, but because they really wanted to build something. They really wanted to get this—whatever their idea that they had in mind out.

And I worry that if you had a—and this is just my hypothesis—if you had a monetary value associated with it, you might just focus a lot on that, and you will reach a ceiling pretty soon.

This is just my hypothesis versus if you really fall in love with the idea of building or creating new things, the ceiling is sort of infinite because your—there are always new technologies. There are always new things that you can go and build.

So it's more about what is the type of behavior that you're trying to create? It’s like a kid; you’re trying to get into the developer world, trying to do it and what is the type of behavior that you’re trying to create and why you’re doing what you’re doing?

So that should be taken care of. And the second big bucket is really the paradigm shift that I was talking about, which is more companies should come forward and encourage and say, “You know what? We’re about skills. We don’t care about resumes. We don’t care about what school did he go to. As long as you have the skills, we're willing to do it.”

I still think the proxy of your GPAs and company exists in a lot of companies.

Yeah, absolutely!

I mean, I think that like people constantly disqualify themselves before they even enter a race.

Yeah!

And we see with YC all the time, like, you know, people are constantly asking questions like, you know, I raised money, can I do YC? Like, I have a raise money; can I do YC?

Yeah, yeah!

I don’t have a co-founder; can I do YC?

Yes! And you often encounter this thing where it’s like, you can just apply.

Yeah! You just go for it.

Yeah, and like moving things toward a more meritocratic system makes a huge difference, yes.

What else does a developer value in the hiring process? Not just like about a company.

Yeah, what do they want?

Yeah, I think the counterintuitive thing is that whenever I’ve talked to a lot of companies is, you should also understand that developers are interviewing you. It’s not just a one-way thing.

And in it, developers are interviewing you on a bunch of different things, right? From the process, how tight and efficient are you? Are you actually doing their peers? And also the challenges or the questions that you’re asking and how deep.

Sometimes I’ve seen this at a lot of companies: people are scared to ask hard questions because it might turn off—it went the other way, which is, “I'm wondering is this the level?” I mean, if you just ask the easy questions, “Oh, here’s our developers that I’ll be working on.”

And of course, there’s a different nuance to the hard question, right? You don’t want to ask a complex math problem that has a trick answer. If you didn’t know this particular formula, you can’t get it, you know, the hard in the sense of relevant to the business, relevant to the job. And how do you go about doing it?

Yeah. The other thing that I'm starting to see happen across a lot of different companies, and even HackerRank has incorporated this, is the business acumen. The level of business acumen that you have as a developer.

Usually, it is—it’s always developers or builders; you have this whatever requirements or spec; you can go ahead and do it. But if you look at a lot of the great companies—Google or Facebook or others—they’re all developers at the heart of it, right?

And the reason, of course, they had great products, but they also had really strong business acumen, and really great developers have that. Or what I think Steve Jobs incorporated is a 10x developer—have that level of very strong business acumen.

So that’s something that people want to know. What’s your strategy? People want to know how they can contribute, how what they’re building helps in the business strategy.

Oftentimes, we think it’s just about users using in amusement and others, but no! I’d like to know how we’re building the company. And that’s something that we’re trying to do a lot.

Something that you should do in the interview process as well is my recommendation.

Interesting! So in other words, like, ask questions about product, not just technical stuff.

Yeah, and about the business strategy! You should be able to challenge HackerRank or challenge the company on, you know, why are you doing this? Here’s a different idea. Here’s another thing that you can do better.

Oh man, now you're just stressing people out! Like, more stuff to prep for!

Yeah, but I think it's fun because I like it!

Yeah, just sit in whatever, like, the armchair critic, you know, you can say, “Oh, why did you enter this geography? I think that's a bad move,” or “Why your business plan addition is not as conflicting with your free user addition to me?” You can have a good conversation, and frankly, the—and here’s—I mean, all of these things are—the nuances are super important.

Which is, it doesn’t matter the questions that the candidate asks, whether that’s the right or the wrong thing for the business—you have to judge the quality of thinking, right?

It could be completely off from your business strategy that you would actually implement because you have way more data and you’ve been doing this for six years or something on those lines. But it’s the quality of thinking and how do you—and what’s the framework of asking questions that you should be judging people on?

Okay, and I found those questions and developers really be at a much higher productivity at our company.

You know, what about the brain teasers? Are they still in fashion? Developers like them at all?

No, I think it’s—it’s not—the brain teasers are used. I don’t know who created that. Some company created that—clearly I’d have to go back and look at the—it's probably like McKinsey or something.

Yeah, it could be, right?

I think—I think the underlying goal is the right one, which is basically they’re trying to think about the computational thinking, which is, I don’t know how many tennis balls will fit in a Boeing, right?

Four or seven!

I think they're trying to—again this is the question: are they trying to figure out the quality of your thinking? How do you think about it? How do you—what kind of data you will analyze, or how do you—how do you structure the problem, how do you break it down into subproblems and go ahead and do it?

But I think there are better ways that you can assess the computational thinking or the problem-solving skills.

We just—

Interesting! Because that’s what the hiring managers care about the most in a much more developer-friendly, coding-friendly way of helping people do that.

Cool!

Okay, so in closing, I'm curious about your experience as a founder hiring developers.

Like we’ve handled—we’ve covered a bunch of stuff for like if you’re a company, this is how you hire your, like a thousand and first developer.

But if it’s just you and your co-founder, and you raise a little bit of money, or you're making enough money that you could finally hire someone, what are the stuff—what are the things you're thinking about?

One of the things you're testing for early on?

Yeah, I think Peter Thiel talked about this in I think maybe one of the podcasts or one of the interviews, which is the way PayPal got built was, they went and hired people who the others thought weren’t qualified but actually they were really, really good because they didn’t have the traditional degree or the traditional background in doing it.

I think the first 20 people or so at PayPal were very much on those lines. That’s kind of how we at least built the first five to ten hires of developers, and it’s really helped us a lot!

If you look at it, Hadi and I didn’t go to the IITs, which is the equivalent of Stanford, and you know, the first five to ten developers that we hired were people who were very smart and very skilled.

And it’s almost this quote-unquote untapped pool, and it’s the secret sauce! And that really set everything right, from your motivation and drive to do great things.

I mean, the first engineers are still with our company and doing amazing things, actually. And they are doing some great things right now, leading a team; they’re managing a team.

And the motivation of the drive to do really great things, the ability to learn, the ability to assess, making sure that the peer or whoever is going to join the company is as good or better than them—all of that has helped a lot.

And I feel that’s an important thing that I come back to on this whole proxy thing.

Yeah.

The current ways of doing proxies on GPAs and degrees is bad, and you have to figure out your first ten people from the untapped pool. I think it helps a lot in your driving motivation for the long term.

It's just my hypothesis, but I just have one data point from HackerRank.

Yeah, no, I mean, I agree with you, but like the short answer is that you have to do the work to find the great people and it's like in that arbitrage of you don’t have the right credentials, but you have all the skills so you're worth it.

Yeah, also, I think there’s some level people should be really passionate about the mission and—and you—you, again, again, the nuance comes here, right?

Which is, you can’t expect somebody to be more passionate than you; that’s never going to happen. But as long as they have a real feel or connect, you know what, we need to solve this problem.

And here are some other ways that you can actually do it, you know? That’s another thing that you need to test for the first n developers also.

All right, man! Well, thanks for coming on!

Yeah, thank you for having me!