How Do You Know?

I often ask a simple, unreasonably useful question: “how do you know?”

I will ask that question of myself (“how do I know?”), a team (“how do we know?”), or another person (usually “how do we know?”, because “how do you know?” can sound aggressively challenging).

Credit where credit is due: I learned the depth and usefulness of this question up from my friend Ian when he was my boss at Home Instead. He’s incredible at asking this question at exactly the right time to make things better.

There are two applications of the question that I’m going to cover, in these categories:

Facts or perceptions
Functionality and correctness

I’ll deal with the first fairly quickly, because, strangely enough, the second is more interesting.

Facts or Perceptions

“My teacher hates me.” “Tony in Sales is trying to set us up to fail.” “…all men are created equally, and endowed by their Creator with certain inalienable rights…”

This version of the question is about epistemology, evidence, and truth.

Asking “how do you know?” can move a conversation away from rumination¹ on a topic, and toward something more productive.

Instead of “my teacher hates me” and fear for what will happen in school tomorrow, “how do you know?” can lead to a parent understanding what happened, and allow them to better emotionally and practically support their kid, and help their child explore alternative explanations for what happened.

Even when it comes to sometimes-fraught religious or political discussions, “how do you know?” can lead to examination of where people are coming from, and sometimes establish common ground. Another phrasing that I’ve found helpful is “how did you come to that conclusion?”²

As an aside, if your response is a variation on “it just is” or “everybody knows that”, there’s a chance that you have a belief that is not supported by evidence. I am wary of those beliefs when I discover them in others (especially if that person is my manager), and even more wary of those beliefs when I discover them in myself. After all, if I cling to beliefs that aren’t true, that’s neither admirable nor useful.³

I will caution you to use this question gently and respectfully to explore perceptions, not as a smug cudgel. Being respectful can go a long way.⁴

Functionality and Correctness

This is the application of the question that is most useful as an engineer or an manager of engineers.

“I fixed the bug.” “How do you know?”
“Adding support for the new billing system will take two months for three of us.” “How do you know?”
“Once your team rolls out our new AI chatbot, revenue will double.” “How do you know?”

I don’t always ask it in exactly the same words, but I always want to know the answer. What is it that makes you (or me) confident in the conclusion I’ve come to?

When I ask you the question, it’s not challenging your character or your intelligence. I want to learn from you. I’m approaching your conclusion with a “trust but verify” mindset. Reality has a way of finding its way through the best of reality distortion fields, and I’d rather know sooner if something isn’t going to work.

There’s an old joke that “weeks of coding can save you minutes of planning.” If I can spend five minutes now to find out I’m wrong about a five week project, I’d like to find out now. It seems like a much better use of time. (This is especially true if that five week project will also produce negative returns for the company, in addition to wasting the five weeks of effort.)

If an executive comes to me with that gleam in their eyes, telling me they’ve just solved our big problem, and all I have to do is implement X or support Y, I get suspicious. I’ll ask “how do you know?” How do they know X will support all our use cases?⁵ How do they know supporting Y will give the results we’re after?

This can be a hard question to ask. There might not be a good answer. The answer might be a lot of spin. Or it might simply be “I don’t know.”

So how can you have a better answer?

How Can You Know?

In the engineering world, there are plenty of ways to know—to become more empirically certain about the outcome to the proposed question. Not all apply to every situation, and certainly all do not give the same level of confidence. Several apply much better before building the thing, while others apply better after.

Let’s take a look at a few of the most common—if nothing⁶ else, if you’re feeling stuck⁷ figuring out how to know, these might shake a thought loose.

Read and Learn

It might be surprising to know there is a lot of information on the Internet. Blogs, magazines, books, podcasts, many written by experts in their field. Source code that lets you know exactly how your framework or library works.

You might not be able to find an answer to your exact question. Even if you can, well, it turns out that just anyone can write a blog post,⁸ so finding a blog post that says version 3.7 of VeryCoolFramework magically solves all your development issues might be too low a bar to say you “know”.

However, if you break it down, you probably can find answers to smaller questions that will lead to confidence in answering your bigger question.

And if you bring some healthy skepticism to your search (e.g., looking for more than confirmation; maybe even looking for the best information suggesting you’re wrong⁹), that will help build confidence over time.

You can also find information to build up a conceptual framework so you can intuitively understand the domain better and reason through your questions. This takes longer, but can pay off over longer timeframes.

Ask Someone

Whether inside your company or outside, you might be able to find someone who can help you increase your confidence.

Writing a pricing page that needs to integrate with your company’s pricing and checkout APIs? It’s worth spending some time with the team responsible for those APIs, running through your proposed solution. If you’re lucky, they’ll say “yes, we support all that” or “yes, except you’ll need to use this API instead of that API”. If you’re less lucky and your proposed solution won’t work, at least now you know. You can adapt, whether that’s re-working your design or asking for API updates.

Trying to build a YouTube clone in Ada for the TI-82 calculator, but uncertain how long you’ll need to implement the video compression algorithms? Look, I haven’t got the faintest idea where to point you, but I’d bet with some work you could find a video compression expert out there. Maybe even one who’s worked in Ada before. Exchanging a few emails could help you be sure it’ll work, and even help you estimate the project.¹⁰

Yes, this “someone” to ask could be ChatGPT, with all the usual caveats about healthy skepticism, information sources, and a recognition that ChatGPT doesn’t truly have “knowledge” in the same sense a human does. Still a potentially very useful tool.

Review the Work

After the work (or a part of it) is done, you can review it. For code, that’s code review. Documentation, proposals, presentations, and incident write-ups are all reviewable. Ideas, concepts, and designs can be reviewed for flaws and improvements.

Review gives less confidence than testing, for all kinds of reasons¹¹. But review can be valuable for things that testing can’t touch.¹² A test can confirm that a piece of code produces the correct outputs. A code review can confirm that the piece of code solves the correct problem.

If a review is part of answering “How do you know?”, make sure the reviewers are asked to focus on the right areas and to do a rigorous review. I remember reviewing a particular piece of code. It was elegant, well-designed code. It was only after I approved the pull request that I learned it was intended to solve a problem that it most definitely did not solve. Don’t do what I did. And don’t let your reviewers do what I did.

Automated Tests

For code, automated tests are an incredible safety net. They can confirm that your code does the right thing. If built well, they’ll run fast so they don’t get in the way.

You can look at type checking and static analysis (in languages that have those features) as a kind of automated test that can also be incredibly helpful. More than once, I’ve kicked off a refactor by changing the type signature of a method, then used the compiler type errors as a to-do list of methods to fix.

How do I know I changed the type everywhere? Because it compiles!

How do I know the code still works properly after the refactor? Because the unit tests still pass!

Even for small changes—not major refactors—knowing everything still runs as expected, and that nothing accidentally broke, accelerates an engineer’s ability to get things done. Not having to run a manual test for every change is a great feeling and frees up a lot of mental bandwidth. Speaking of manual tests…

Manual Testing

Even in programming, not everything needs automated tests. Sometimes, running a quick manual test (“does it look right now?”) can be the best way to build confidence. (Outside of programming—say, the realm of team practices or marketing approaches—a manual or real-world test might be the only option!)

Remember that your memory isn’t perfect. You might know, today, exactly how to confirm something is working. Will you remember in a year? (And if you just said “yes”, how do you know?)

Documenting your test procedures, even as very brief notes in a README or Testing.md file might help you remember how to know that everything still works. (Larger projects will probably have more structured test documentation.)

Try It!

I suppose trying things out is the ultimate form of manual testing.

The most straightforward “try it” is to build everything and try it out. If it doesn’t work, tweak it and try again. For a simple shell script, or an easily-reversed change in how a team handles incoming work, it’s a pretty good solution! (I would not recommend this for, say, spacecraft.)

Building everything can take a long time, though. If you’re already not sure the end product will produce the desired outcomes, you can consider trying something smaller than “everything”.

For example, you could run a quick spike or proof of concept to reduce risk and uncertainty in a short period of time. Instead of building the whole thing to your normal standards, a spike involves building a small bit of the system to work out the tricky part. For example, if you think your dog really needs a doghouse, but aren’t sure he’ll sleep outside, you could use a large cardboard box to “build” a “doghouse” and see. If your dog¹³ happily sleeps in his cardboard doghouse, you’ll feel much more confident he’ll sleep in a real doghouse. If he won’t sleep outside, you’ve just saved the work and expense of building a doghouse—or discovered you need to plan on investing time in training your dog to use a doghouse.

There are other flavors of trying things out to learn more. You could run a “concierge test” where humans openly do the work the system will eventually automate. Or a “Wizard of Oz” test, in which the system looks like it’s automated, but humans do the work behind the scenes.¹⁴ Or… well, there are plenty of other ways to try something out, depending on your constraints and how useful it is to know whether something will work.

Focus what you try so you maximize learning and risk reduction. You don’t need to build a login system and pretty web pages—you’re sure you can do that already. But maybe you need to prove that you can access all the data you think you need, stitch it together correctly, and turn it into something useful. If that whole data thing works out, you can build a login system and pretty web pages around it. If that data thing doesn’t work out, you won’t get value out of your pretty web pages anyway.

(It’s probably worth mentioning: vibe coding something can speed up building, which might help you try things faster. Don’t lose sight of the need to know that the vibe coded software works so you can know your trial is valid. “How do you know?” can be a recursive question!)

Take a step back

In a world that loves quick answers, hot takes, and instant certainty, take a step back. Check your facts. Ask “how do you know?”

…that it does what it’s supposed to
…that it’s the right solution
…that you’re solving the right problem
…that your facts are correct

Admitting early on that you don’t have all the answers can lead to greater certainty and much better results in the long (or even medium!) run.

Being a little less certain up front can make you much more certain and more successful later.

And wouldn’t it be nice to be more successful?

Repetitive thinking or rehearsal on a topic. In psychological use, typically repetitive thinking on negative thoughts. Even when used in a more general sense, rumination means no progress is being made. [↩]
If, in sometimes-fraught religious or political discussions, you find it unfair or uncomfortable to be asked “how did you come to that conclusion?”, and this source happens to carry any weight with you, allow me to point you to 1 Peter 3:15, which instructs readers to “Always be prepared to give an answer to everyone who asks you to give the reason for the hope that you have.” [↩]
This is true even in religious discussions. St. Paul, who wrote most of the Christian New Testament, acknowledged as much in his first letter to Christians in Corinth (1 Cor 15:19), when he said “…if only in this life we have hope in Christ, we are of all people most to be pitied.” What he meant was “if this stuff I’m writing you isn’t true, there’s no value in any of this stuff I’m writing to you (“…your faith is useless”, v17) and everyone should feel sorry for us being so deluded.” The Dalai Lama said something very similar in a New York Times column: “If science proves some belief of Buddhism wrong, then Buddhism will have to change.” Even if some things are hard to prove or disprove, it’s still worth examining the basis for one’s beliefs or conclusions—even if the beliefs being examined might be less consequential than one’s religious faith. [↩]
If your kid is crying about their teacher hating them, or their coworker is upset because of something that happened in a meeting, leading with “how do you know?” is not going to be helpful. Especially if you ask it in a “you’re wrong and I’m about to prove it” tone of voice. [↩]
At a previous job, we were working to solve a big hairy company-level problem to solve. Let’s call this The Foundation Problem. There was a somewhat-related problem that I’ll call The Presentation Problem. Our estimate was that The Presentation Problem would take about a year and need focus from several teams. We were using a vendor of expensive tooling on The Foundation Problem, and they suggested they had a solution to The Presentation Problem. Contracts were signed and declarations made of how efficient that solution was. Six months later, the vendor of expensive tooling was hyping the performance of their system that now took only ten seconds to display a web page that would be customer-facing (no, I am neither kidding nor exaggerating). There was no way to speed it up more, because our business was more complex than their platform could handle. That 10 second page load was after using all of their normal speed-up tricks. So we said goodbye to a very large pile of money and dusted off the plans for implementing our internal, vetted solution to The Presentation Problem. [↩]
Did you know that nothing starts with n and ends with g? [↩]
If I’m feeling stuck and need a buck I don’t rely on luck because the hook brings you baaaaaack! [↩]
Who knew? [↩]
King Ahab of ancient Israel is recorded as saying “There is still one prophet through whom we can inquire of the Lord, but I hate him because he never prophesies anything good about me, but always bad. He is Micaiah son of Imlah.” Ahab chose not to listen to Micaiah, which directly led to his dying in battle soon after. Read the whole story in 1 Kings 22. [↩]
When you’re done building this TI-82 YouTube clone, let me know. It sounds amazing. I’m curious whether you plan to serve regular videos from TI-82s, or downscale the videos to 86×64 and play them on a TI-82. Either way, it’s way cool. [↩]
“Beware of bugs in the above code; I have only proved it correct, not tried it.” – Donald Knuth [↩]
DO-178B (Guidelines that were used for building safety critical aircraft software) allows inspection to be used for certain aspects of software validation, especially aspects that are not testable. However, it’d be very unlikely to be able to use inspection for all verification of even a tiny, low-criticality software package. DO-178B has been replaced by DO-178C, but I assume the verification standards are fairly similar. [↩]
Who, I’m sure, is named Spike. [↩]
“Pay no attention the man behind the curtain!” [↩]