Reference checks

Since starting Lattice in 2015 we've hired about 130 people. I've come to believe that one of the most underrated secret weapons for hiring is reference checks. Reference checks done well can contain real signal that can help you avoid making costly mistakes (either false positive or false negative).

People don't do reference checks enough because 1) most people are somewhat lazy and 2) they don’t think there’s enough signal to make it worth it. But the truth is references don’t take that long (especially compared to the countless hours you’ll waste if you make the wrong hire) and, more importantly, reference checks can be an extremely valuable part of your candidate evaluation process because they can distill years of data that you simply can’t reproduce in even the longest of interview processes.

So do reference checks and consider them a core part of your evaluation process. Here are the steps I've used:

Set expectations with the candidate

Let the candidate know that the final part of the process is reference checks so it's not a surprise when you ask for them. This doesn't have to be weird; usually during the first interview or while scheduling a second you'll lay out the steps, so just include this there. E.g., "Next steps are a meeting with X and Y, and if that goes well, we'll schedule a half-day onsite. From there the final step is reference checks and role scoping with your would-be manager."

Get a mix of provided and backchannel references

When you're near the end of the interview process, ask the candidate for 3-5 references. Ask for the list to contain people they've worked with in different relationships (a manager, a direct report, a peer, etc.), and ideally in different contexts within a company or at different companies. The closer they've worked together the better. Ideally, you’re looking to speak with people who have a strong business relationship but a weak personal relationship, since those are the people with the most referenceable content and the least partiality. In practice this is hard to find, and that’s okay.

Let the candidate know that you'll also do a couple of backchannel references, but check to make sure there aren't any off-limits groups. Frequently, the candidate won't have told their employer they're looking, in which case doing a backchannel reference with a current colleague could do real damage. Don't do this. But backchannel references can be an important part of getting a full picture on what a person is like to work with, especially with more senior candidates with longer work histories.

Running the calls

Generally, I've found 10-15 minutes is enough time for these calls. You don't need much small talk or tons of questions, but rather just some clear context setting and a few direct, important questions. Here's the basic format I use:

  1. Thank them for taking the time. Quickly explain what your business does, and give a little background on your stage and the general state of affairs.

  2. Tell them about the role you're recruiting for. Describe what you think the job will be like, why you're hiring it, and what kind of person you think you're looking for.

  3. I start by asking them to tell me about how they know the candidate, in what capacity did they work together, and a bit of context on the nature of their working relationship.

  4. Then I ask specific questions based on 1-3 areas where I most want to learn more. This might be ability to thrive in a startup environment, cultural / stylistic questions, management ability, technical proficiency, or whatever else. I usually pick a small number of areas so I can ask about each topic in multiple ways, kind of like those surveys where you're asked nearly the same question over and over. It feels strange when you're asking these, but it really does help get you closer to the truth.

  5. Wrap up by asking general questions to try and gauge how strongly they feel about the candidate. Questions like, "Is this person in the top half, quarter, or decile of everyone you've worked with?" or "Do you think we would be crazy to pass on this candidate?" or "If you were starting a new company or team would this person be your first hire for the role?" are good types of questions to help you get a sense of whether the person thinks the candidate is solid, great, or truly exceptional.

What you're looking for

On a reference call, the normal rules of interaction are slightly altered. You don't all of a sudden assume people aren't being honest, but you do discount the strength of assertions by a couple notches from what you might in a normal conversation.

This isn't because you don't trust people, it's simply because people tend to like the people they've spent a lot of time with, the stakes are high since a potential job is on the line, and the cost of embellishing is zero since the referencer frequently doesn't know you or necessarily expect to interact with you again. Think about what you'd say on a call about a colleague you knew well and liked, but who was a solid, not exceptional, performer. You'd probably be generous.

You're looking for very strong signal. If you ask them if they're in the top half, quarter, or decile of people you've worked with, you're hoping for an answer along the lines of "top 1-5%". If you ask them if they think you'd be crazy to pass on a candidate, you're hoping for something like "completely crazy. And if they do join please call me so I can apply too."

When referencing our VP of Product, one of the referencers didn’t know the candidate was looking and clearly start trying to feel out whether or not she could try to hire him. And when referencing our head of customer success, her former CEO’s answer to the “what percentile” question was “no percentile, this is the number one person I’ve worked with.”

Obviously, you won't always get such a strong answer, and that doesn't mean you shouldn't hire the person. But directionally I wanted to illustrate the type of answers you're ideally looking for.

How to work with a COO at a startup

Founding a startup requires a relatively high performance in a large number of relatively unrelated areas. Besides being good enough at product and problem solving to make something people actually want, you also have to be good enough at hiring, managing, fundraising, sales, marketing, culture-building, and much more. Not to mention specific areas of expertise you might need to master depending on your domain such as security, supply chain management, manufacturing, or whatever else applies. And especially not mentioning the skill of managing your own psychology well enough to guide your team through the years of inevitable ups and downs.

This is one of the reasons I think having some work experience is so helpful before starting a company. You don't need to wait until you've acquired every possible skill before starting a company, but if you're already proficient in multiple of these areas before you get started you have that much less to learn on the job.

It's also one of the reasons I've come to believe that more startups should hire a COO than probably do. Of course they will not be right for every company -- maybe not even most companies -- but my current view is that every startup that reaches product market fit [1] should at least consider whether it's right for them.

COOs are a poorly understood role, both because they do different things at different companies, and because people simply don't talk about them that much (although there is definitely good content out there like this video with Keith Rabois and this interview we did with Claire Hughes Johnson). 

Our experience hiring a COO early at Lattice

I want to share my very positive experience with hiring a COO so others can consider whether a similar role would make sense in their situation.

We hired J Zac Stein, former VP of Operations at Zenefits, as our COO in November 2017 when we were just over 20 people. The idea was that he would be a general purpose business athlete who could run a function for a while, build up a strong foundation, leave it with great leadership, and then move on to the next thing. An executive "gap filler", as Elad Gil described in a recent blog post advocating for something similar.

In the 6 months since he's joined, here's what he's done:

  • He's overseen the sales team through a leadership transition, allowing me to run a multi-month process to hire a new head of sales. Not only did the sales team grow our revenue by over 100% in the past 6 months, but they've also added early versions of sales development and sales operations.
  • He's built up and led our customer experience team, as well as made the decision to bifurcate the team into customer success and customer support. This was a core area of expertise for him coming into the job and it's given me the freedom to work on other areas knowing it's in better hands than it was with me.
  • He's been a one-stop-shop for legal, finance, analytics, and HR (aided by contractors), allowing us to not only hold off on hiring those roles, but also to have a much more experienced set of eyes on them than we would have with more junior hires.
  • Now that our new sales leader is joining, he's overseeing marketing where he can apply operational rigor and managerial guidance. This allows us to continue betting on our less experienced but extremely talented head of marketing, which is very important to me.

So he's served as multiple VP functions, both at once and over time. Arguably, a dedicated VP in each area would do slightly better because they wouldn't be spread so thinly and would be a functional expert. There is some truth in that, but it comes at the cost of countless hours of hiring, integrating into the team, managing multiple personalities, and other related trade-offs. And potentially more importantly, the "executive gap-filler" role allows for much faster adjustment of organizational structure, strategy and tactics, and all sort of other organizational frictions you get with VPs who were hired on for a more specific purpose. When you're still early in your company's life, this flexibility is extremely valuable.

Why don't early stage startups hire COOs more often?

Part of the reason people don't hire COOs early is that it's simply not on the list of hires we're taught to consider. Everyone knows that early on you hire engineers and designers to build the product, sales reps and marketers to bring customers on board, and people to support your existing customers. As you start getting traction and grow your team, you hire managers, and as things become more complex and strategic you hire VPs.

But there is less canonical knowledge and public discussion about the COO role, and a result founders often just don't even think about whether it's right for their company or not.

The other main reason is that people are often anxious about hiring roles with so much influence. Whether it's because of the sense of loss of control and influence, the fear of getting the hire wrong, or the relatively high expense of hiring a COO, there are many of these kinds of anxieties that give founders pause.

Yes, nothing in life is totally free and you will have to give up something to have this kind of person on your team. But great people pay for themselves many times over, great COOs know how to make their CEOs comfortable with their broad scope, and if things truly don't work you should trust yourself to make the difficult but necessary decision to part ways.

Great leaders are supposed to hire great people and give them lots of responsibility [2]. That's your most important job. It's how you scale and it's what's required in order to build something great.

Keys for the CEO/COO relationship to work

If you decide to hire a COO, this will be the most important relationship you have at your company aside from your relationship with your cofounder. Like any important relationship in your life you need to think critically about what makes it healthy and put work into building that. Here are the most important things for this relationship:
  1. High degree of trust and mutual respect
    In a startup environment, trust is largely a function of understanding a person's motivations and believing they are aligned with the company's and your own. Trusting the other person to be a reliable steward for the company's interests is built over time spent together, repeated interactions through challenging business questions, and an open line of communication.

    Relatedly, mutual respect is critical because there's simply too much going on to be in each other's business and second guessing each other's decisions. The CEO and COO should be thought partners to spar with each other on challenging decisions, but ultimately need to have the respect for one another to be willing to "disagree and commit" when necessary.

  2. Low ego on both sides
    Ego isn't inherently bad. It gives people the confidence to take on projects they aren't quite ready for, do things like start a company, and generally maintain a positive sense of self. However, in the CEO/COO relationship, you have to be careful to not let ego get in the way if you want it to be successful.

    As CEO, you fundamentally need to have the humility to admit you hired this person for a reason and will be allowing them to run a large portion of the company. This requires elevating your COO in the eyes of your company to allow them the authority and influence to do their job, and it means handing off functions that you might enjoy leading because you know it's better for the company. As COO it means towing the line between the ego required to accept that level of responsibility and the humility required to still respect the CEO's role. If things like handing off functions when necessary are going to be stopped by either person's ego, the relationship just won't work.

  3. Clear delineation of roles
    Geoff Donaker, who was COO at Yelp through a decade of meteoric growth, once sat down with me and J Zac and white-boarded out the three possible relationship configurations for the CEO and COO. One is the "Ted Turner model", where the COO essentially runs the company internally and the CEO is an external figurehead. This works only in the rare cases when the CEO is influential enough for this to be a valuable use of time. The second is the "two parent model", where both executives are equally responsible for everything. This generally leads to organization confusion, friction, and one of the two executives leaving the company.

    The third configuration, which he described as the only stable one for startups and is certainly what we are going for, is one of clear delineation of responsibility. The COO runs several functions and the CEO runs several functions. Everyone in the company is clear on what those are, and while the CEO and COO advise each other on challenging questions there is no doubt about who is the decision-maker for a given issue.

  4. Enjoy spending time together
    Finally, I think it's critical for the CEO and COO to enjoy each other's company. It's simply too much time together, too many things you need to discuss, and too many issues that are intellectually and emotionally challenging that you will go through together. The relationship won't work if you don't actively enjoy each other on a personal level. You shouldn't start a company with a cofounder you didn't enjoy being around, and the same thing goes here.

One final thought for founders considering this role (or other very senior roles) is to remember the human truth that the more ownership and responsibility you give someone, the more they're going to care. Being a founder can be a lonely job, and bringing on an executive who grows to deeply care about your company makes the job a little less lonely.

[1] My favorite working definition of product market fit is the ability to hire a stranger who can sell your product to another stranger in a repeatable way. YMMV but this framework has been helpful for me.

[2] As an aside, I think people are often afraid of hiring people truly great people and giving them lots of control and responsibility. This is an especially common pitfall for first-time managers who are afraid that if they hire and enable someone who is too good at their job they will lose their job to them. In practice this never happens, and the exact opposite is the case; the hallmark of a great manager is someone who builds and empowers a great team, and anyone who knows what they're doing will evaluate a manager that way.

Platforms for job creation

The other day, my uberX driver told me that he'd been unemployed for almost a year. He started driving last month, and now he can pay his rent and work whenever he wants. A friend of mine makes his living as a full time renter on Airbnb, effectively acting like a hotel while he travels around world with the profits. At Teespring, some of our most successful sellers are programmers who left their high-paying jobs because the opportunity to build their own business on top of Teespring was more appealing.

There's an interesting link between many successful startups that gets surprisingly little attention. It was a foreign concept before the rise of the internet, and now I believe it is one of the driving forces behind the growth of Airbnb, Uber, Teespring, Postmates, Patreon, Verbling, and many more.

Companies can now empower people to create their own jobs.

This is a huge deal and has major implications. Job creation gives employees their livelihood, it gives companies champions who fight to make them succeed, and it gives economies legs to stand on.

For employees, it used to be the case that to get a job, you had to be hired; now you don't. Companies now have the ability to create jobs not only internally but externally, and they can increase the size of their workforce much more quickly.

At Teespring we have just over 100 employees, but hundreds of sellers are making their living with our product, and thousands more are getting close. Uber is creating 20,000 jobs per month, a rate of expansion that simply wouldn't be possible with normal hiring models.

Creating jobs across platforms

Perhaps even more interestingly, there is also a way to empower people to create their own jobs even when it’s not all about your own platform. While the companies mentioned above are built on the users who make it their full-time business to sell services through the platforms, it is also possible to create jobs for users on other platforms.

Consider companies like Instagram, YouTube, and Wordpress; the very top creators on these platforms certainly make good livings, but the majority of them don't. Figuring out how to give more of these users the ability to earn money is a massive opportunity, and unlocking this potential will establish a new degree of engagement on these platforms. The companies that figure out how to help users on these platforms turn their passion into profession will do very well.

Creating jobs is a powerful force, and the rise of ubiquitous and mobile connectivity has given companies an entirely new ability to create jobs remotely and scalably. The companies that see the power in this and figure out how to leverage it will do well.

Why we're not in a tech bubble

It seems like almost every day now I see someone who believes we are in the midst of tech bubble (a world in which companies are valued far above their true worth). The most common arguments people mention are 1) startup valuations are massive and dissociated from underlying value 2) public technology companies are overpriced, and 3) MBA-types are moving to Silicon Valley to get a piece of the gold rush. In this post I will address each of these points and argue that we are not in a bubble.

Startup valuations are too high

Argument: Nosebleed startup valuations are the most commonly cited indication that we are in a bubble. Airbnb and Dropbox are valued at $10 billion each. Snapchat and Pinterest are valued at $4 billion each and they don’t even have revenue. Facebook bought Oculus for $2 billion before consumers could even get their hands on the product. Whatsapp is just a joke.

Most people have become comfortable with the idea that even though young startups are likely to fail, seemingly high valuation can be reasonable if there’s a sufficiently likelihood of success. Early stage venture investing follows a power law, and the billion dollar outcomes can justify valuations for a whole set of startups. This is why two founders with an idea and a prototype can raise money at a $5 million valuation; early stage startups are treated as part of a basket.

For mature companies like Microsoft, valuation is best understood on an individual company basis, as a discounted stream of future cash flows. Huge companies trade at a reasonable multiple of their earnings because dramatic growth is typically out of the question.

I believe the mental error here is categorization. When people criticize multi-billion dollar valuations, they are bucketing them alongside mature public companies, when really they should still be considered as a basket that will follow a power law of outcomes. The right question is not “is Snapchat worth $4 billion?”, but rather, “is there a 10% chance Snapchat is worth $40 billion?” [1].

We live in a world with over 1.5 billion smartphones and more sophisticated technology than ever before, which means startup potential is greater than ever. When you combine this with an understanding of these companies as part of an overall basket, multi-billion dollar startup valuations seem reasonable.

Public markets are overpriced

Argument: Tech high-flyers like LinkedIn and Netflix are trading at hundreds of times their earnings. We are seeing the most new tech IPOs since the tech bubble in 1999, and many of them are for companies that are quickly burning cash. The NASDAQ is closing in on its all-time highs from 1999 bubble, and the small-cap tech stocks are at some of their highest P/E ratios ever.

There is some truth here; tech stocks are expensive right now by any measure. However, that doesn’t mean they’re overpriced. 

While it’s true that younger smaller tech companies are richly priced, the mature tech companies like Microsoft, Google, Apple, Cisco, Yahoo, etc. are trading at very reasonable levels. Also, younger, struggling companies like Groupon, Zynga, and King have stock prices that reflect their situations. 

Investors are evaluating each company on an individual basis and valuing businesses that at least make sense and have workable business models. This is very different than the last bubble when companies erroneous business models got great valuations as long as they were “tech”, and mature companies valuations were rising with the rest of the tide.

What we really have in the public tech markets is highly priced young companies and moderately priced mature companies. If you believe that some of the young companies are poised to disrupt the incumbents, or if you take a basket view to young tech companies, then public market valuations aren’t so scary.

Outsiders are migrating to startups

Argument: Five years ago, consultants were competing to get into business school, MBAs were funneling back out to Wall Street, and bankers were quitting to work at PE funds. Now, everyone is just joining a startup.

With this argument, the premise is true; more people are starting and joining startups than ever before, and they’re coming from a much more diverse set of backgrounds.

What’s not necessarily true is the conclusion, that this migration of talent is a bad thing that indicates a tech bubble.

The largest new source of tech workers is the veterans of banks, consulting firms, and business schools. This demographic is widely assumed to be “in it for the money”, and so skeptics believe they will be unable to build the revolutionary companies that drive innovation forward and create massive value.

I have a more optimistic view [2]; I think the majority of this crowd was unsure what they wanted to do, but knew they wanted to work hard and feel important. It seems plausible then that they could discover the startup world a little late and fall in love with it just like anyone else.

Instead of viewing this influx of workers as a sign of a bubble, it should be viewed as the logical extension of an increasing attractive industry in a country with fluid labor allocation.


A bubble is one of those things that can only be identified in hindsight, not because we don’t have enough information right now to know, but because it has not yet been determined. If companies collapse and lose their ability to grow and earn money, then we will have been in a bubble. If enough startups succeed in the ways we hope and expect, then we will not have been in a bubble.

I believe the latter is more likely to be true, that tech is poised to continue on a trajectory of massive growth. The market for startups today is different than it has ever been. Over a billion people have a computer in their pocket at all times, and billions more have access to the internet. Software can be deployed from the cloud and run on a massive scale instantly. Industries that never used technology before are welcoming software into their businesses with open arms. Large amounts of capital are available to startups at every stage, and there is a bigger pool of talent wanting to work in startups than ever before.

Yes, capital is plentiful and cheap, expectations are high, and new tech workers are flooding in from every corner of New York. This may scream, “bubble!” to you, but remember that technology is not zero-sum game. All of this could very well signal the influx of capital and labor that is necessary to create a virtuous cycle that can grow an industry at a massive scale. 

[1] In fact, the bar is even lower. The real question should be is the integral of graph of probability vs. outcome worth more than $4 billion.

[2] As a side note, although on some level I understand the reaction to be skeptical of MBAs entering startups, I think it’s an unsubstantiated bias and drives an unwelcoming culture that we should not encourage. 

Customer service isn't a department

So far during my time at Teespring, my most striking observation has been the importance of great customer service. Every conversation at work is laced with the subtext of improving customers’ experience, and every employee does customer service at one point or another. Good customer service helps you understand your customers, define your company, and grow your business, and should be nurtured carefully.

Customer service is the best way to understand your customers

Every single employee at Amazon - even Jeff Bezos - spends at least two days per year doing customer service. A lot of interesting things start to happen when people from every part of a company take the time to interface directly with customers. Maybe a product person realizes a clearer way to present users with the information they care about, or an operations person recognizes that a certain supplier is causing too many problems. In these kinds of ways, customer service helps you define your business.

When the entire company is relentlessly focused on serving customers, the company's value proposition tends to increase. Great customer service is not just a department, it’s a characteristic of a company that focuses on its customer’s needs and works backwards from there. It should appear in all parts of the company, the way that the DNA encoding your eye color can be found in every one of your cells. 

Good customer service drives growth

Good customer service can do as much to grow your company as any other avenue. Not only does it increase your user retention by keeping existing customers happy, but it also creates an army of advocates who will tell their friends by word of mouth how much they enjoyed their experience. 

As a driver of growth, word of mouth has a very special quality; as your service grows, you get more advocates. Ideally, you retain a constant ratio of advocates to users; e.g. 20% of your users generate a new user in any given month. This is why startups with very happy customers tend to grow exponentially.

Growth is the silver bullet that solves all problems in a startup, and customer service is one of the best ways to get growth.

Just because it’s easy to do customer service doesn’t mean it’s easy to do well

Customer service is an essential business function. In fact, customer service is the essential business function; serving customers is what makes you a business. Because of this, it deserves incredible attention.

Too often, people dismiss customer service as one of the softer sides of business. Decent customer service is easy, but great customer service is hard - companies like Amazon pour massive time and financial resources into getting it right.

There is an erroneous tendency to give less credit to activities that have a lower barrier to entry. Anyone can play chess, but becoming a grand master is extremely difficult. Even though customer service is easy to do moderately, companies should invest heavily in doing it well.