Startup Technology - Technical Founder Advice

I would like to introduce Jared Frieden, my partner, and his esteemed panel who he will introduce to talk about technology. Thank you. Thank you, Jeff.

Okay, well, I am super lucky to have a very esteemed group of guests with me here today. Everyone on this panel is a tactical founder of a really successful company. Everyone here has built a really cool company in a really amazing technology organization. We've changed the name of the event for today. It was originally called CTO advice and on the advice of Lillian, thank you so much, we have changed it to tactical founder advice, which I think is a better description.

I'd also like to thank everyone on the startup school forum who wrote in with questions for the panel. We posted a few weeks ago and solicited questions; we got over 150 responses. So thank you, everyone who wrote in with those questions. We're going to do our best to cover as many of them as we can and then at the end, we'll open it up to the in-person audience for some questions as well.

Okay, so let's get started. Could we start by having everyone introduce themselves and tell us about your company and about your technology? Like what’s your tech stack? What’s some of the interesting technology that you build? How does your technology organization look like today?

"Oh you monster! Hello everyone, my name is Ralph Judy. I'm CTO and co-founder of Plain Grid, Clean Grid. We're 350 people based out of Mission San Francisco. We write beautiful, easy-to-use software for the 17 trillion dollar construction industry. So what that looks like to you, the analogy often uses, we're like GitHub for construction. Construction has blueprints; blueprints change rapidly, version control is extremely important. If you have changes, that means there are issues that are happening. Issues need to be tracked and then we build collaboration tools on top of that too, as well as a lot of other tools for the construction industry that go into some deep jargon.

Our stack? We're based on AWS. We used to be based on a variety of other things, we had to move everything into AWS over time. On our back end, mostly Python; on the back end, we’ve got some Go and other things. One of our challenges is we actually write native for every platform. So we've got iOS with Objective-C, Android all Java and Kotlin, the web Reacts, and then Windows. We have a full Windows app as well, which has done that."

"Hi everyone, I'm Calvin French-Owen and CTO and co-founder of Segment. Segment is a single API to collect, organize, and adapt all of your customer data into hundreds of downstream tools you might be using. Whether that’s an analytics tool like Google Analytics or Mixpanel, maybe a customer success tool like Gainsight, maybe an email tool like Customer.io. Segment helps you collect data from where it lives and get it where it needs to be. In terms of the company overall, we're a little over 300 people right now. We have our headquarters in San Francisco, but we also have offices in Vancouver, New York, and Dublin.

The engineering product and design team, which is kind of how we build product here at Segment, is a little over 80 or 90 right now. In terms of our tech stack, we're also built entirely atop of AWS. In terms of how we run our infrastructure, we run on top of ECS, Amazon’s container service, and pretty much all of our different services are containerized today. We're running about 300 different microservices which are kind of piping together various Kafka topics and reading and writing, transforming this data, getting it to where it needs to go. Our back-end is primarily written in Go."

"Good morning everyone, so my name is Diana, who I was the founder and CTO for SEO Reality and we're building the back end for augmented reality. And I say I was because my company just got acquired by Niantic, the makers of Pokémon Go. So what we were building in Escher, and now we actually continue doing it in Niantic, is building this back-end technology for augmented reality to enable developers to build AR experiences as easy as possible. So we handle all the complexity with the computer vision algorithms, with all the interaction with the back end, all the hard parts of getting things to render so that developers could build easy AR apps in minutes. So that’s what we do.

We provide a lot of advanced AR features like multiplayer, persistence, and cross-platform so that it is easy for developers just developing and let’s say on Unity, and it just works. In terms of tech stack, it has been a bit of an adventure. Back at Escher, we had a service that was hosted in AWS, but that didn’t matter too much because we had everything in Docker containers. Now, Niantic, I think is a heavy user of Google Cloud, so we’ve moved everything to Google Cloud. But we have a lot of the bulk of the code is actually for us native, means more C++. Literally, because we find it’s a lot more efficient for us to write the code and a lot of the algorithms once and then cross-compile it across all the architectures for Android or iOS and even the back end. So that we run some of the CV algorithms; we could prototype it in the phone and then we could easily move them into the server because our service actually also written in C++."

"Hi everyone, I'm Lillian. I am CEO, oh not CTO of Second Measure. So Second Measure, we're about 50 people. We got started back in 2015 and we're based in San Mateo. What we do is we analyze credit card data. So basically, we take billions of credit card transactions and try to build a daily view into the public and private company performance. We analyze these billions of transactions automatically every day and just clean them, enrich them, normalize them, and then service that in a front-end application for our clients, which include VC firms, hedge funds, and big brands like Blue Apron or Spotify to check trends, do any sort of competitive intelligence or customer intelligence. So we actually don't have a CTO, so that's a fun fact about us, and that's why I'm here on the panel. Myself and my co-founder actually both technicals; that’s been really pretty fortunate, we don't have to deal with as many of the challenges around starting with non-technical founders.

So our— I mentioned that our team is about 50 people, primarily technical. So our technical organization is about 30 people and that’s actually split evenly between data scientists and engineers. And that’s probably something that makes us unique is that our core product is actually the data itself. So our data scientists and engineers have to collaborate super closely on a day-to-day basis. Probably one of the things that has technically been interesting is a lot of our users want to explore and dive deep into our data. Building a front-end that allows that flexibility of exploration while still creating a good user experience basically requires us to figure out how to rapidly query data and run really complex queries on our back end.

Yes, about our tech stack? We’re also primarily on AWS, so for our pipeline, we leverage services like Lambda and Spark, and then for our front-end we React based and leveraged columnar data stores on the back end to service queries. So I think Redshift."

"Awesome, guys! That was super cool. It’s so cool the diversity of different kinds of technology organizations and products that are here today. Okay, so most of the company in startup school are really early, so I want to take everyone here back to their v1, the very first version before you had actually shipped anything. Tell us the story of how you built your v1. Who built it? How did you build it? How long did it take you to build it? And what things went wrong in the process?"

"I guess I’ll start. This is a great story. I’ve actually got my CEO and co-founder over there watching me right now, and um so that's the company. Hi Tracy! Yeah, hey Tracy, how's it going? So that’s where we started. Tracy told me about her idea of putting blueprints on an iPad. This was in 2011, right at the launch of the iPad, so it was very early for the technology. I said yeah, no problem, I could put blueprints on my iPad; it’s like a PDF, that’s no problem at all. So I was attempting to impress her really quickly by putting blueprints on an iPad, and it turns out there were actually some hardcore challenges with putting these giant images on a really low computer availability on the iPad; there’s no graphics chip at the time.

So basically, these are 20,000 pixel images and they would just crash or overrun the video memory, and they’re in PDF, which is a kind of painful format to deal with. So the first prototype sucked, um, it’s really slow and that actually told me that there’s something really meaty here, there’s something that’s actually a challenge, and it took me about a month to write the second prototype based off some—my background actually, I used to work at Pixar so I had some graphics experience there, didn’t use any proprietary stuff but used some off-the-shelf computer graphics knowledge to write the first blueprint viewer that ever ran on mobile. So that was our first prototype back then, and I think the best thing I remember back on it is, um, the only thing we really focused on the first prototype was what was technically impossible; there was no way you could load blueprints onto an iPad at the time and that was our prototype, which meant that all the data got there by side loading, which meant there was no web interface.

The little crappy web interface we had had a delete button right next to a publish button that was not a pixel apart, and there was no confirmation on the delete button. So we actually, as a team, managed everything on the back end; we kind of like behind the curtains loaded all the documents for our first 30 or 40 customers, but all they saw was the prototype that was fast and did what they thought it would; they never saw the man behind the curtain, so to speak. So that’s all the story of our first prototype."

"Cool, um, I’ll share a little bit of a story of our v1 and then maybe our v1 next. I think for those of you who have seen the startup school talk where my co-founder Peter talks about finding product-market fit, he goes into this story in way more detail than I could today, but I can share a little bit of our journey.

So when we first started, we were in the YC 2011 class, which sounds like a dinosaur now in terms of YC years, yeah? But at the time, we were actually building something completely different from Segment today. We were building this classroom lecture tool which would help professors get feedback about what their students were thinking in the middle of class. We were students at the time; we figured hey, like this seems like a great idea. Professors will sometimes go on for five minutes and they'll lose the entire class, and the class gets confused and just waste everyone’s time; let’s solve that problem.

We built it out over the course of the summer and we deployed it back in Boston in the fall, and the whole thing just kind of crashed and burned. Like, students would go to Facebook, YouTube, Wikipedia; like in retrospect, all the things that you would assume college students would do if they have a laptop in front of them in a lecture. But we didn’t quite see it that way, so we started looking around for a new idea because we’d just raised a seed round, and we said okay, what are the problems that we have with this college lecture tool? And one of them was that we couldn’t answer questions about how people were using our tool. We couldn’t tell you how a college student at Harvard was using the tool differently than students at MIT, or we couldn’t tell you how anthropology classes used it differently than biology classes.

So we switched again to something which also we don’t do today, but effectively it was a competitor to Mixpanel or Amplitude. This tool would allow you to cluster your users by different rules and break them up into segments, which is where the name came from, and from there you could then reach out to them or figure out what your most engaged users were doing. So we built out that system for about 15 months. The whole time we were doing revs on the back-end infrastructure, we were building it out and we're trying to get users and no one wanted our tool.

Fifteen months later, we actually moved back to the Bay Area after living in Boston for a year. We go and talk to PG and we tell them about kind of our whole saga that’s been running for the past year and a half, and he listens to our story and he listens to everything that we’ve done and when we’re finished, he just sort of pauses. I think it was maybe 400 feet out there walking around the roundabout. He says wow, so you just burned half a million dollars and you’re still at square one? And we're just like, yes. He said, well, on the plus side you still have some runway left, so try again. He wanted something new.

At the time, we had this internal tool that we had been using as kind of a growth hack to get ourselves more users, and the whole idea was that you would use the same API to send us data as you would send to Mixpanel, Kissmetrics, Google Analytics, all these other tools, and people actually liked that single API, and they were contributing to this open-source library and GitHub and starring it and using it. And so my co-founder Ian said, hey, why don’t we turn this into a product? Why don’t we launch this and see what happens? My co-founder Peter said, oh, that's the worst idea I've ever heard. It will never work. He’s CEO now, but he’s seen the light.

So I, he said, okay, I’ve got to figure out a way that we can test this idea incredibly quickly. So we cleaned up the library and we launched it on Hacker News and to our surprise it actually got about a thousand stars that first day on GitHub.

So our real v1 was that first week taking that library, and we basically just had a landing page up which said hey, if you’re interested in a hosted version of this leave your email, and about 500 people left their email. We built out that v1 over about a week where just everyone was in mad hackathon mode and building out the product which we launched a week later. And so today, I think there’s actually no parts of that product which still live on today, but it was enough to get us kind of the seed of product-market fit and then start getting us a user base. So one week is the short—one week! One week! Everybody hear that? One week.

Okay, that’s kind of amazing."

"One week. Sir story is definitely more than one week. So me and my co-founder have been thinking of building different things and one of the technology trends that we really saw was he comes from a robotics background with a lot of experience with SLAM. SLAM is this algorithm called Simultaneous Localization and Mapping and it’s one of the core algorithms that run today in AR, which helps tell where your camera position is while you also build this map of the world.

So what happened is there’s this category of algorithms that have been used mostly for robotics. Now AR had been kind of tried many times but never really being picked up; there’s AR with markers and that never really took off. Then we thought about why don’t we bring these algorithms and run them on the phones?

We got to the point where we thought we could do it because the computation at that point with phones, for example for iPhone 7, has the same computation actually as a MacBook Pro from 2013. So you think about that, that’s kind of amazing in terms of all the trends with Moore’s Law and some of the ones with power efficiency and compute. So we decided to go do it. I was actually still working and I was leading a data science team back then and said okay, I’ll do it because I was thinking of doing some transition and I was working part-time with another engineer, and this one of the engineers that we kind of got excited to work with us.

So we struggled a lot to try to get a version working; the first was very duct tape and it wasn’t very encouraging. It was only working at five frames per second which is terrible because if you want a good AR experience the whole thing is that you need it to render nicely and be at least 30 frames per second, so that was very discouraging in a sense. Then, because it was all kind of put together, this and then we started kind of really removing all the research code out and then it started working and seeing a lot of promise and it was running at least at 30 frames per second.

Note that when we were working on this was at least about a year and a half before ARKit had launched, and then we were excited about this. So we were both very more from a technology side, so we didn’t know product-market fit, way to go with this, so we had to find the market. The exercise we did is actually we went in and interviewed a bunch of people that we thought would use it and we found a really good fit with gaming. So we decided to focus on that, and that was our YC application.

The thing that really took us— and this took off, and this is actually, I guess, a good story too—is the first day of YC Apple announced ARKit, which is basically what we had built. That was sort of a moment of truth for us; it’s sort of this really moment where you’re looking at yourself and it is really go flight or fight.

So we decided why not? Let’s just take on Apple. So we decided to just double down and we had this idea that the start wasn’t just on the device, which we had all our demos and technology, it was just working on the phone. We didn’t have anything on the back end yet. So back in YC, we got it working with a back-end and also with Android and basically accelerating the roadmap that we thought it was about a year, shrink it and got it done in three months.

That was the thing that once we put it out there and let people sign up if they were interested, we go got about a thousand developers signed up in about a week. So they find okay I think we found something, and then after that we got a lot of interest and among those Niantic was one, and the story is we got acquired later."

"So how long was it from the time that you started working on the original SLAM algorithms until you had a working product in users’ hands?"

"So we were not really working full-time; it was probably about a year of part-time and even by the time was part-time it wasn’t really working. So when full-time, I took another six months of just me and another engineer to do it, and then during the summer YC we hired two more engineers, and that’s when we were able to really accelerate and get it working."

"Well cool! So for us, our v1 probably took, assuming full-time, two or three months to build. I actually would say that most of that time was just figuring out what our product was going to be. We knew that what we were working on was going to be really interesting; initially we were focused on investors basically trying to get them an information edge on sort of any investment decisions that they’re making, whether that be hedge funds focused on public markets or VCs focused on private markets.

We ran through a few ideas; we were like hey do these investors want predictions? They just want to know how like this company, public company's quarterly sales are gonna hit? And the short answer is yes, but no, that doesn’t really seem like a sustainable business model. So then we shifted a little bit like okay maybe these investors really, really want to go deep and they just want to cut data any way they want. So we put something together really quickly, put that in front of some of our friends, your investors and found out very quickly through some white user testing that that was way too overwhelming; our users wanted some guidance.

So what we ultimately landed on is we needed to build our own self-service platform where we can apply our opinions around how our customers should be viewing this data and getting quick access to insights. So once we figure that part out, I was actually pretty quick. My co-founder focused on building the analytics product piece, so the front-end application, and I’d say, you know together we kind of focus together—we built sort of the data pipeline and sort of the transformations on the data.

But I think some important choices that we made early on is—and maybe some that you guys are facing right now—is what technology do you want to use? Like what do you want to build this in? Ultimately we decided to build our first application in Groovy using the Grails framework, which is very not shiny and not that many people are doing that, but that was actually the code that we had the most experience in.

So my co-founder Mike had built large production scale systems servicing hundreds of thousands of concurrent users, and pretty much within a week had the bones of the application, and kind of like some of these other stories here, like none of us, it sounds like, are running our v1 anymore. So it’s more important in the early days to be able to iterate quickly, and usually that comes from using the technology that you know the best, building things in a modular way such that once you actually find that product-market fit and know what your audience needs, then you can focus on that area and actually select technologies that are best fit for that problem."

"Awesome! Okay, so I’m gonna go next to the very top voted question from the startup school forum, which is about the trade-off between engineering best practices like good test coverage and security and scalability and speed versus writing code as quickly as possible and shipping something.

So can everyone talk about how you made that trade-off for your v1 and then how it’s evolved since then and sort of like the timetable of its evolution as your company grew and those things became presumably more important? And just to mix it up let’s try flipping the order and we’ll start with Lillian this time."

"Sure! So speed is paramount. Nobody’s gonna pay you for having excellent test coverage. So in the early days, it’s, you definitely want sort of the bit of like what you need, right? So it was—that’s the risk to your business around sort of security and not having things working.

Obviously, you want your whatever you build to run every day; you don’t want to be fixing it, breaking all the time and not actually building, but really it is a balance, and I do believe that initially, it’s more important to have that speed of development and incorporate constantly testing and incorporating the learnings into your product than it is to have like the most robust, most scalable product. Great! You're trying to find something that people will pay you money for and solve a real problem, and that takes time and a lot of iteration that has definitely evolved since we got started.

You know once you find product-market fit a lot of these problems change; you know your system gets more complex, your team grows, you have a lot more people contributing code, a lot more ways for your systems to break, a lot more users who are relying on your product. So for us the way that manifested itself is, yeah, we started writing unit tests to verify each engineer would be verifying that you know their code works. Then we introduced CI/CD to make sure that that could work with the whole system. Our director of engineering who leads our engineering organization started developing and formalizing our best practices around like code reviews, testing, etc. in defining those processes and then finally building some more controls directly into our system to make it harder to break things.

And specifically like what was the rough timeline for those things? Like are we talking about like two months after launch, two years after launch?"

"I’d say most of that stuff I was just talking about happened probably the last year to year and a half. So, and that’s how many years after?"

"About a year and a half after launch. And the main reason for that again, were the growth and the complexity of our systems. And you know our technical staff tripled in the last year, so just that alone introduces a lot more need for process."

"In our case, it was always about trying to get to MVP that was reasonably working well, and all that was definitely duct tape code. There is nothing that I’d be proud of; it’s just Frankie's I had put together, and barely working as much as possible. So we were there even when we kind of private launch in a sense, when we have the private beta and got customers who kind of signed up, and we actually had a beta, a private beta that we had some number of game developers working with us and that was also the duct tape code.

The only time we started actually having all the best practices is actually once we joined Niantic because now we have to integrate into the games which Pokémon Go has hundreds of millions of users. So we’re bearing ourselves to get a lot of the, we pretty much, we wrote the whole codebase; none of what we had before is there, it’s all rewritten through pair."

"I think for us at Segment there are kind of three distinct phases of our development. I’d say the first one was when we were in full hackathon mode, right, where we definitely didn’t have any tests, no CI. I’d say there was an 80% chance that we were just going to throw it out two weeks later because it wasn’t going to stick and then we were going to move on to the next thing. I think because we had been burned so much by building out all of this infrastructure and investing a lot of these ways to provision your infrastructure and write tests and spin up different new services over the past year and a half, it really I gotten us nowhere; we had no users.

It has created this like pretty aligning homing instinct for us where we just said no matter what, the only thing that matters is getting users at this point in the game. So that’s where we started and I think that lasted us probably for about the first nine or ten months where we were just focused on rapid iteration, getting more users, making sure that we didn’t run out of money and then the whole thing wouldn’t have mattered.

I think from there the first phase that then shifted was when we brought on our first engineer, TJ. Wright, for those of you who know or are involved with the node community. TJ is one of the best engineers I think I’ve ever worked with. I think 90% of node shops run on some part of his code. He basically came into Segment and he looked around at well this mess that we created, the product that we created, and he said, well there’s no way for me to work in this; like I have a horrible time onboarding, I think it took him a week to get his entire laptop set up.

So we kind of moved from this point where the entire development team shares the same tube of toothpaste to now starting to have more and more engineers involved with the project, and when that happened it really stepped up our game in terms of testing, CI, and just reproducibility when it came to running the builds and running the stack because suddenly we had to expand outside of just the four of us.

And then I think so maybe that period lasted another two or three years. Probably in the past two, three or four years from now or from this point in time we’ve gone through another shift where we’ve started thinking a lot more about end-to-end testing, security, kind of best practices around handling all of our customers’ data.

And the real reason for that is that now we actually have a pretty significant amount both like revenue and customers and reputation to lose. From day one, we had no users, so it didn’t matter if the whole thing went down for a day or hours or whatever, it was just the calculus didn’t make sense there. But today we have thousands of customers who are relying on Segment to publish their data, to not lose it, to treat it and handle it securely and so for us, the investment makes way more sense.

And that last period, like roughly what scale did you have to reach before you started thinking seriously about those things?"

"Yeah, it’s a good question. I think when we started having enterprise customers who were paying north of six figures per year, that’s when it started to shift and we started thinking, oh, we really have to take care of how we’re handling this data because these customers are really depending on this to power their business."

"So I think my story's a little different than the other panelists. This might be interesting. I would say all three of those different facets I think I heard security, I heard scalability, and I heard engineering best practices. We really have to treat them independently at Plain Grid. Klinger's used to build in all and most hospitals, most jails, most heavy civil roadwork; we’re used on all the tech campuses, all of the different government buildings that go up.

So obviously, security from day one, I mean that’s key to us, so we had no choice but to always be super conscious of security and scalability for that matter because our customers, the only way we’re useful is if we’re used to build the project; like we’re the replacement for blueprints, which means in construction, if we’re not working properly, like no one builds.

And not building for a dam, construction can seriously impact the bottom line of these businesses. So for us, the first two—security and scalability—were always key. Scalability is a challenging one because it’s—I don’t know, I’m not sure if there’s a way to do it without it being kind of reactionary. You either over-engineer everything and then you maybe never get a product to market, or you eventually have to deal with scaling issues.

For us, we tried to architect it in such a way that we would last through the foreseeable future; the foreseeable future would come and some customer would find the first kind of flaw in the scalability. And we’re used, I mean our—to give you an idea of the scale, we’ve probably about—we’ve customers with projects that are like 500,000 blueprints with a ton of changes and maybe over 5 million annotations on one project and we work offline as well. So this is all downloaded onto a cache on the device; we can actually, our physical size on an iPad can take up like a hundred gigabytes for certain projects.

So we have like all kinds of scaling issues we’ve had to approach, and we always try to balance it between engineering like a year ahead of time but not maybe three years ahead of time, which is a trade-off. So, you know, find that trade-off yourself. I remember when we were in YC, we had all these stories of companies that had built the most ironclad architecture and just never had a product to market, so that was on our minds while we were building that."

"For engineering best practices, our v1, some of that code still exists in the product today, so some of the code I wrote for v1 still there. I would say it’s probably our measure of technical debt as an engineering organization, so that’s humbling. But at the same time, like I’ve seen some developers and you know we’re fully, uh, you know we do everything cool now; see CI/CD, we run integration tests, we’ve got a UI testing team.

So this is feedback I’m giving from the past, but you know, if you're an experienced developer, it’s pretty hard to write spaghetti craft code, you know? It’s actually challenging! Most experienced developers and experienced some people are just really good from the get-go, it’s normally something through time and writing a lot of production products that you just generally learned the tricks of not writing spaghetti code pretty quickly, and you know some of that code runs without heavy testing."

"The other thing to mention is, to properly scale test our product, it doesn’t—I mean it would take days during the integration test to download and upload all the data. So we really have a lot of functional testing to help us double-check. But the thing I’ve noticed with UI and unit tests is we’re a big fan of unit tests; we’re a big fan of test-driven development.

I’ve definitely seen developers write the most spaghetti test frameworks and the most spaghetti tests where they’re just testing their mocks and they’re doing this in the middle of a production environment, you know, like a production need for release, and they spent so much time writing these unit tests that actually were not testing anything rather than getting the product out the door. So engineering best practices are great; there’s a trade-off between writing a lot of tests and writing good code from the get-go."

"So speaking of test-driven development, a lot of the questions were about the various engineering methodologies: agile development, lean startup, test-driven development, extreme programming. How do you guys think about those? Do you adhere to any particular methodology in your company?"

"I would describe us as agile-ish, so we do not fully adopt any of those methodologies. Basically, we find what works well for our team and aren’t super dogmatic about you know, is this agile or is this not? You know, we run sprints, we have daily stand-ups, so we have some elements incorporated in our development.

But I think kind of with anything, you know a lot of these problems you’re going to be solving, a lot of problems for organization and you have to kind of find what works best for you. And so I think it makes sense to take bits and pieces that work for you and adapt it to your organization. And then I guess it’s just the one other thing I’d mention on this is, as you grow, a big challenge is just constantly evolving how you work, right?

So it’s very different to work on a small team before where everyone kind of knows everything, and everyone everything is then everyone’s heads versus a team of 30 or 50 or for some of us you know hundreds. That is very different. So you kind of constantly need to be assessing is this style of working or this methodology, is this right for this stage of the company?"

"Yeah, I think the point of evolving the process is something we’ve been doing a lot. So back when we were at Share, we were only with four engineers and me building the product. So back then it was very messy; we’re just trying to move as fast as possible really. So zero documentation, everything was kind of whiteboards and everyone kind of could handle everything in their heads and build it out.

In terms of we didn’t really do sprints because I think some engineers even quite liked it as much and there were just too many big tasks and we just trusted different individuals to go and tackle kind of one area and then we would integrate it. But now things are very different as we hired; I think our team has tripled already in the past eight months for the AR platform product. When you have a bigger team, you definitely need a lot more process to be able to be efficient and communicate because you don’t want to duplicate and not everyone knows everything, and the system grows in a lot of complexity.

So we went from zero documentation to a lot more. When we used to have a little bit CI, we now have very much of CI that actually builds to all the architectural flavors to Linux, Mac OS, Android x86, Android ARM, iOS, etc., and actually runs all the tests in all of them and actually catches a lot of bugs."

"And we test—we have coverage in all those things. But we also have to train and get the team to be okay with having more process, so now we do not quite like a sprint but we do weekly planning for the week and then we reconvene, and I think the teams are kind of breaking, break it down into sub teams that work."

"People kind of float in and out of their fan focus on areas; we split into three main focuses. One is sort of the back-end folder work with the back-end; I lower the work on the client to make it cross-platform, the unity API is, and the other the big one is bringing a lot of the computer vision algorithms to production.

So the funny thing is that everyone in the team has ended up being and trained up to be a computer vision engineer because that’s sort of the core. We have CV algorithms on the server, we have CV algorithms in the client and the core algorithms that run, so there’s been this, how it kind of shaped up to be."

"But we don’t really have a per se process where we do right now at Niantic. That’s okay; now we have all cares as well per quarter. So now we’re starting to plan longer, longer and have ideas of before you still have like a rough idea where things would go, but now this and this and this."

"Yeah, for us, we don’t have any set process that teams have to follow at Segment. Kind of individual engineering teams sort of like what Spotify does are able to self-organize; they are able to run however they want. They want to do sprints, at school, they want to use JIRA, they can do that. If they want to use Asana, whatever it is, they are allowed to run however they’d like to run.

The one thing that we’ve introduced over the past two years is similarly the OKR model for those of you who are familiar. This was a model that was developed at Google. It’s the idea of O’s or objectives and then key results. So you have your one objective where you’re trying to go, and then key results that are supposed to be objective measures of how you get there such if you do every KR, then it adds up to the full objective.

And so those are something that we do on a company-wide basis every single topic. Team in the company gets together and puts together their OKRs they say one week before each quarter. Then for that three-month period, they’re just executing on that plan. Some teams grade those on a weekly basis and check in and say how am I doing against these goals; some teams grade them on a monthly basis. It kind of doesn’t matter but at the end of the quarter, every team is saying hey, here’s how well I did against my stated goals.

Separately for the teams that I work with in particular, we end up running kind of just a weekly meeting where at the beginning of the week we end up planning out what we want to get done and then we have kind of a daily stand-up. I honestly don’t care too much about the content for those meetings so long as we’re always discussing the most important problems.

We’re pretty big believers that the tools and processes should be there to serve us, not the other way around. My experience at Plain Grid echoes all the other panelists; it’s agile-ish, self-organizing; every team can choose the tools that they want. Maybe some things that’d be helpful to you from the beginning, some things we’ve always found that’s been very huge useful in every engineering team are again the daily stand-ups. You’ve got to communicate on a daily basis as engineers; sometimes it can be a little difficult to want to communicate every day and not just program when you wake up.

But that 15 minutes—and you can do it remotely as well over Slack or something—has always been really helpful in just kind of unblocking people. Timeboxing, I mean I think that a lot of these management practices all roll up into some abstract philosophies and sprints or timebox methods where you’re just not going to work on something infinitely. That’s very helpful to kind of keep people’s realities in check because often estimates can be off and you know sometimes what we thought was going to take us a week can take us a month, myself included.

So timeboxing is a good way to keep categories of that. I’d also say some other lightweight management techniques you can probably employ right now in your team is probably one-to-ones, weekly one-to-ones. Just make sure if you’re talking to people in a group and you’re having stand-ups, make sure you have some time to actually connect with people individually and get to learn more about their wants, their needs, their emotions and their careers."

"That’s great, guys! Okay, so now I want to change topics to the right way to work with non-technical co-founders. And I think the topic that came up most commonly in the startup school forum was the one that Ralph alluded to, which is how to deal with the deadlines and timeframes, particularly with non-technical co-founders and I think particularly in the early days. So does any have thoughts on that? Any orders—go!"

"I'll volunteer for this since my technical non-technical co-founder is staring at me over there. A few ideas here. If you've got someone that’s really non-technical—and I’m not saying my co-founders really non-technical, I’m sure she can use your computer—but you know, if it’s really non-technical, this is an amazing testing opportunity for you.

I mean I remember I would write this software and I’d be so happy about it and I’d hand it to her and as soon as she touched it, it would break! I don’t know what she shook the iPad; she rotated it three times but she had a way to break the stuff I wrote and that was amazing for testing in the beginning.

So that’s one way to engage your non-technical co-founder. You eventually have to learn how to start pouting your deadlines, that’s really hard to do; I never got very good at this. I’m always optimistic, even to this day, I’m like, oh yeah, that’s gonna take me a week! So I never really got good at this; I’m just aware that I’m optimistic about it and then I’m always kind of off by a week and she’s aware of that as well.

So eventually, I’ve talked to other engineering leaders; they keep two books! They keep like a separate set of books. I talked to their technical co-founders, non-technical co-founders about saying, okay, here’s the deadlines here and then you have another set of books, that’s actually when you think you’re gonna get the job done.

I think one thing that can be really helpful is having a process, some very lightweight process of like here’s where we’re like testing and then here’s release because often some people might not realize that iOS has a release cycle that’s not controlled by the developers; it’s controlled by Apple. So you got to pad that into your plan.

I’ve found that a little bit of project management goes a long way when you’re dealing with a non-technical co-founder and even better if they’re good at project management; that’s a great place to employ them as well."

"So all of my co-founders are technical, so we didn’t really run into this problem in the early days. Kind of everyone's like, oh yeah, I know what’s happening here; if it slips, we know exactly why; it wasn’t as big of an issue. I found more recently with deadlines, kind of the trick that I’ve started using is in one-on-ones actually asking each person what they think will happen over the next three or four weeks and when I find asking one to one is that one, people don’t anchor against each other’s answers so you get a very unbiased view, and it like forces each person to think about the near-term future; what's going to happen.

And second, each person is much more wary of the parts that they didn’t work on, and so you get kind of a sense of like where there might be trouble spots, and typically the end results is sort of like a wisdom of the crowds where it’s kind of the average of all three or four answers, and so I’ve started using that as a way to get a better sense of deadlines where each person has their prediction or mental model of what’s going to happen and then you kind of average them all out to where it should be."

"Well, I guess in my case, my co-founder didn’t work in engineering or build products or shape products, mostly just in research. So we’ve been doing mostly a PhD, so in that sense, it’s kind of a little bit, you know, technical in the sense that not having experience in building and shipping products.

It’s mostly just kind of the algorithm side. So there was a bit of that work to understand why certain things take long and why they need to be built a certain way, but at some point it was more he just letting me handle all of that, and he got engaged mostly on the external communications business trying to find partners and all that. So that’s what happened. Sometimes that could be the split where as the technical founder, you end up owning the product and a lot of development and having clear communications and at least try to set expectations that with deadlines that’s the hard part; it’s always, I think this is always a trade-off between engineering and product and business, right?

The trade-off on when to get things and be able to communicate that clearly. So my co-founder is technical, so I don’t have much to offer here—not much to add either on the deadlines and time estimates but do your best guess, do some padding."

"I'd say one thing kind of on the non-technical front though that I do think is worth mentioning is, you know today a lot of you are probably building your products and in writing code, especially if you’re one of the technical co-founders. At least in my experience, that changes.

So I pretty much stopped contributing to our code base about a year into the company. And it’s for us, kind of as leaders, well I can’t speak for all of us, but at least for me, there’s a pretty big shift that happens once you start gaining traction where it’s less about building a product for a market and finding that fit, and it shifts more into turn like building an organization and a company and a system; like this system of humans who are going to build something long-lasting and great without a lot of your involvement.

So I guess what I’m trying to say is, you know at some point you may end up doing a lot more non-technical things, so get ready for that!"

"Yeah that’s great! Okay, so a lot of the companies in startup school are at the phase where they are trying to figure out the right way to structure their early engineering team if they want to hire people locally, if they want to hire people remotely, if they want to hire only employees who work full-time, or if they want to hand off pieces of the product to contractors or to third-party development firms. Can you guys talk about how you would think about that if you were starting a company now? Advice that you give them?"

"I think this is going to be the last question and then we’re going to open it up to the audience. I can start. So we did not start with contractors; we started by building like hiring full-time employees. I think the main guidance I would give here is you are building a product in a company and you’re developing these core competencies.

I’d say for the most part, like if you end up contracting that out, you’re actually not just contracting out through that technology expertise, but you’re learning a lot in these early days of what’s going to work for your clients, what’s the—what the product actually is, and so moving that sort of having that external to your company I think is a really big loss in the early days just in terms of all that knowledge.

I could see an argument, if you’re just like getting off the ground and you’re doing contracting kind of as a way to evaluate potential new hires, I could see that definitely being a path, but again, that sort of with the intention of building your team. On the outsourcing front, I think it’s a similar thing of you really need to look at like what is your core competency? That’s what you’re investing in; what’s your IP? You don’t want to outsource that.

At some point if you do, you’re pretty much just a marketing company. And so I’d say if you are—different business models might have different requirements. So for some of you it may make sense to outsource a large part of what you’re building, but for us, we have experimented a little bit with outsourcing. I’d say our approach has really been to make sure those projects are very well-defined and not on the critical path, so that we can experiment. It is a different type of project to manage sort of outsource talent, and that way you can sort of learn, see if that works for your business and if it does great, if it doesn’t it’s fine."

"Was there also a question about remote local versus remote employees? I think that depends on what type of culture you want to build. So for us, it was really important for us to have our team together, so we hired a local team and then sort of in the last year or so we have actually started tapping into remote engineers.

There's definitely like, you know, a little bit of a culture shift with that. We’re still predominantly local, but there are a lot of advantages including being able to tap into other talent pools and it has actually turned out to be a really good forcing function for documentation, right? Explaining your code, explaining your decisions, which is good for a number of reasons including just like, as you grow, communication is one of the harder problems you have to solve.

So I think for us, because so much of the core that technology is what our company is, we didn’t quite outsource any of that. We did contract people as more of an interview process, so we would work with and contact someone for a month before we gave them an offer. So that's what we did but we did outsource more things that we really thought they were not essential to our company. For example, building 3D models, some of the design, some of the actual writing technical documentation, some of it website things that were—we could do it if we had the time I wanted to, we knew we could do it, stuff like I could build a website, but our time was better spent on that core text."

"So those sort of things we did outsource and in terms of remote versus local, because everything was so complex with what we build, we chose the culture to be more all on-site. And to this day, we still are building the team locally, but I think you could handle a lot of stories of companies that do remote, but you have to kind of start remote first."

"Cool! Yeah, I can share a little bit from our story. So in the very early days, we basically only hired full-time folks. We did an experiment with contractors; kind of the only case where we’d outsource is when eight of us could take some piece of infrastructure that we’re building or like some new product feature and we could build on top of that. I think that was still probably the right move, especially in the early days.

It feels like a startup is really fragile and you want a bunch of people who were just all pointed in the same direction and really kind of in it for the long haul. On the local versus remote piece, we actually started hiring pretty much only remote people. We’re really involved with a lot of different open-source communities on GitHub particularly; there were some Node in the very early days, some JavaScript ones, component, etc., and so these were people who we had been working with and collaborating with for years at this point who then when we were ready to start hiring, we just said, hey, why don’t you like to work more closely together?

I think it came with its own set of trade-offs. On the plus side, we had access to these people who were insanely talented not in the San Francisco Bay Area but who were used to working with us already and who we very clear like built an insane amount of value for segments. I think the trickier part for us to get right was around the communication piece and all kind of being pointed in the same direction from a product perspective.

So while we grew remote from about the first 10 or 15 hires, we then only grew locally through about the next 70 and then only recently started opening remote backup. I think now it’s because we actually have enough bandwidth to create a really good remote culture where we’re putting an emphasis on that documentation, that communication piece and we’re actually really able to hire folks who have been working remotely for years as well."

"So, I think the best advice you’ve gotten already is just do what’s right for you at the size of your company and know that there’s trade-offs with all these decisions but there’s no easy answer for any of these questions."

"Great! Okay, I think we’ve got time for one or two questions from the audience."

"Questions? Questions? One over there?"

"You okay? I’m just going to repeat the question so everyone can hear it. If you were going to start another company now, what would you do to accelerate the development process?"

"Any takers?"

"Yeah, sure! I mean, I think we spent too much time building technology on my side to try to really get that product market fit as soon as possible. And it’s a challenge in terms of the areas that you pick because if you’re in a kind of frontier tech space with, like AR/VR or this summer biotech, those lead time could be really long and think about what kind of company you want to build.

I’d say get in front of users all the time; find your user base! It’s really, it’s actually a lot harder to build a product in this vacuum where you’re just like in your head maybe ideating with your co-founder and you’re like what if we do this? What if we do this? These all sound great; no this idea sounds bad, but it’s like way easier to just put it in front of somebody and have them say this is great or this sucks; that feedback loop is invaluable."

"Yeah, echo the sentiment on users, 100%. Get out in front of them, start showing things to them, start make sure you’re digging deep to make sure that you’re really solving their problem. I think if I were to start again today, actually I’d build two or three versions which I just intended to throw away. Maybe this is biased on probably prior experience, but I think the fact that you can get reps in a very safe manner, where then when you’re really ready to start, you can just kind of go nuts building out the v1, I think is really powerful.

You know, I love mobile development; I love Swift; I love Kotlin, but I don’t think I would choose to have to write for different platforms if I was going to write again. I probably would have picked one of the cross-platform development libraries. I don’t know which is best; they all— you’d have to tell me if anyone has like, after this, come with your strong opinions at which one of this is best; they all seem to have trade-offs. Probably the best advice I could give though is I’d stopped coding a little bit earlier and I would start hiring a little bit earlier because I think there’s a lot of me that wanted to passionately write those features and if I instead been passionately hiring other developers, we would have, you know, we would have built faster."

"Interesting! Okay, one more question over there."

"And your driving features given by the users? What would be the communication style? Get them in front of the users, bringing them to the meetings, get them over the phone—give the questions, you know, kind of set it up a little bit where you’re trying to bait the users into giving the feedback that you’re looking for or you won’t get the feedback you’re looking for.

And maybe there’s some adjustment there too, so get them in front of users as much as you can; that’s the most powerful tool you have, in my opinion!"

"Okay, thank you all so much! This was great!" [Applause]