On-Site at the Customer — How Do You Find a Problem Worth Solving?

It is a capital mistake to theorize before one has data. — Arthur Conan Doyle, "A Scandal in Bohemia," The Adventures of Sherlock Holmes

At the first on-site meeting, the customer sits down and hands you a list of seven wishes: smart quoting, meeting-minute generation, photo-based quality inspection, vendor-email sorting... Someone in the room raises a hand for each one, and every single one sounds like something "AI could probably do."

It sounds like a wishing well. But the delivery team only has the headcount to build one of them.

How do you choose? Many projects don't fail because the execution was bad — they fail because the wrong problem got picked at the very start. Pick the wrong problem, and the harder the team works, the more completely that effort is wasted.

This chapter is about exactly that: where to dig for the real problem, how to run what you dig up through five filters, and what's left once you've cleared all five.

I. The Real Need Is Rarely Found in a Meeting

Start with how discovery actually works. Most projects understand "discovery" as "interviews": book a string of meetings, take a stack of notes, go back to the office and write it up. The biggest risk on that path isn't missing a question — it's that every sentence you hear has already been translated once. By the time it reaches you, it's no longer first-hand data.

A widely-circulated field method calls this step "shadowing": follow a real user through a real day of their work — not interviewing them, but sitting beside them and watching them work, watching which systems they open, which spreadsheets they copy-paste between, where they frown, which "official process" they quietly route around. What you're watching for is the workaround, not the workflow: the official flowchart tells you how the organization is supposed to run; the workaround tells you how it actually runs. Anthropology calls this method participant observation; the Toyota Production System calls it genchi genbutsu — "go and see for yourself." OpenAI's FDE team did exactly this on the John Deere project: they flew out to Iowa and followed agronomists and farmers into the fields, watching how spraying decisions actually got made, which information actually entered the decision, and how the hard deadline of the growing season governed everything. The old process they eventually replaced — agronomists calling farmers door to door, giving verbal advice on equipment use — appears in no document anywhere. You only see it by going out to the field¹ (see Figure 6-1).

Figure 6-1: The real need hides in real actions, exception handling, and unwritten rules.

Why does it have to be seen with your own eyes? Because users themselves can't tell you — and "can't tell you" happens on two levels.

The first level is that they won't say it outright. Ask directly, "do you think this plan is good?" and you'll mostly get politeness back. There's a small startup-world book devoted entirely to this problem: Rob Fitzpatrick's The Mom Test. The title comes from a pointedly cruel piece of advice — don't ask your mother "is my business idea any good?" Because she loves you, she'll say yes no matter what. A customer's compliments work the same way. The fix is to move the question from opinion to fact: don't ask "do you think you need this" or "would you use it in the future" — ask "the last time this came up, what did you actually do about it." Ask about the past, not the future; ask about one specific instance, not the general case. Politeness can't be checked against anything. Facts can be cross-examined.

The second level is that they genuinely don't know it's worth saying. A soldier's need for a roadside-bomb early-warning tool is one no interview will ever surface, because soldiers don't know "software could actually do that" — they assume the dread that comes with every patrol is just part of what patrolling is¹. A Chinese enterprise AI deployment consultant makes a strikingly similar point: pain points rarely get spoken out loud; "they're hiding in the manual work people repeat every day, in the mess at some post nobody wants to own. You can't dig them up with a questionnaire — you have to camp out on-site"².

When you can't make sense of it, there's a blunt fallback: ask "why" three layers deep — the first layer nails down what they want, the second asks why now, the third asks what happens if it doesn't get done. By the third layer, the real problem usually surfaces. A customer says "I want a knowledge-base Q&A system." Three layers down, the real problem might be that a veteran expert is about to retire, thirty years of experience can't be carried out the door, new hires take half a year to get up to speed and make constant mistakes in the meantime — the surface ask is a knowledge base, the deep problem is knowledge transfer.

There's a subtler layer of distortion still: the person in front of you is not the user at all. When the people making the request are a mid-layer digitalization team, "you think you're building a system for the actual users, but the whole conversation runs through a group of people carrying their own digitalization KPIs — by the time they relay front-line needs to you, it's already passed through a filter." The fix is straightforward: route around the middle layer and reach the front line directly — "even two or three interviews with real users produce a stronger signal than ten meetings with middle management." When there's genuinely no way to reach them, list the requirement assumptions one by one and have the middle layer confirm each against what the front line actually wants — not what the middle layer assumes they want².

A real example: a company deployed an agent to handle refund requests. It read every request, checked it line by line against the refund policy, and rejected anything that didn't qualify — every single rejection was "correct." A few weeks later, long-standing customers started to churn. No one could figure out why, because every rejection the agent made really was called for under the written policy. Eventually someone sat down with the people who used to process refunds by hand, and found an unwritten rule that had never made it into any document: orders paid with a corporate credit card get approved automatically — those customers order every month, and haggling over one refund could cost the whole account. The policy never mentioned this; it was tribal knowledge the human processors had worked out for themselves years earlier. The fix wasn't a different model — it was turning that check into a fixed rule in the workflow, applied before the model ever makes a judgment call³.

This case lays bare what discovery is really about: the model being correct is not the same as the business being correct — a layer of unwritten process knowledge sits between the two. That knowledge isn't in the requirements document, isn't in the system logs; it only lives in the hands of the people who handle the work every day. There's a similar case: a file-format migration had stalled for an entire year, and everyone assumed it was a technical problem, until someone sat down next to the engineer who kept blocking it and watched him work — he'd always double-clicked files to inspect the data, and the new format had nothing you could double-click. The team built him a tool that night that let him keep double-clicking. Two days later, he signed off on the migration³.

Which is why discovery needs people who understand the system to go in first. A digitalization lead at an investment firm, reviewing internal practice, put it this way: when you're packaging an AI skill, "the most important thing is knowing exactly what you're trying to package" — and getting clear on that takes architectural judgment that has to come from someone at the mid-management level or higher, guiding the process from the start⁴. Interviews collect the wish list; the architect judges the real need behind each wish. Both have to happen at once.

II. Filter One: Does This Problem Actually Matter?

Don't rush to assess technical feasibility on the list you've dug up. The first filter asks: does this problem actually matter inside the customer's organization?

Bob McGrew, a Palantir alum, put this bluntly: "the scope of a traditional implementation might be you start with something that's pretty close to what the product does, but you want to be solving one of the key problems that leadership has identified. If you're not solving one of the top five priorities for the CEO, it's probably not going to work. They probably won't have the energy to persist through the much more challenging route of getting effectively a new piece of the product built in a way that worked for them."⁵ His reasoning isn't technical at all: if the problem requires an on-premise deployment, you'll have to fight the customer's IT team; every layer of the organization has someone who has to say yes before you can move, and those people don't think like a startup does and aren't aligned with the end user's goals — you have to find a way past them. Which is exactly why the problem you solve has to be important enough that you can "bring in someone from the top to say, 'Yes, give them authority to operate. Give them, you know, the ability to use'" whatever they need⁵.

There's a second reason for the importance threshold: picking your battlefield. Which core production line is AI actually going to touch — R&D, contracting and delivery, manufacturing quality, sales operations, or some piece of customer service? It has to connect directly to delivery, revenue, gross margin, risk, or customer experience — not just "look like it saves time." Take the hard, high-value problem over the easy one; otherwise the payoff is negligible.

This lines up with a view held by some practitioners in China: what most existing AI solutions actually accomplish is "cost and efficiency gains within a limited scope" — bolt a little AI onto the old process, get a bit more efficient, spend a bit less; "your company's original processes, people, and business all stay exactly the same." And "if the people and the organization don't change, AI's capability simply can't come through"⁶.

Put the other way around: a problem worth handing to an FDE is usually one that, if you follow it far enough, leads straight into process and organizational change — that's exactly what earns it a place among the CEO's top five problems, and what makes it worth bringing in someone with the authority to say yes.

One of Palantir's early customers turned filter one into a textbook case. The FDE's first real customer was Airbus's factory in Toulouse; he moved there for a year and spent four days a week on the production floor. The entry point wasn't "here's what AI can demo" — it was the biggest problem the CEO told them about directly: ramping up A350 production. The team responded by doing exactly one thing: pulling work orders, missing parts, and quality issues ("non-conformities") — scattered across multiple systems — into a single interface where people could check off tasks, track parts, see the production schedule, and fuzzy-search past quality issues. Under the hood, none of it was more than basic software functionality, but actually shipping a "best-practice" interface into a real production environment ended up helping roughly quadruple the pace of manufacturing⁷.

The direction was worked backward from the biggest problem the CEO cared about — not forward from what the technology happened to be capable of.

Why does this filter come first? Because if the problem isn't important enough, every filter after it is wasted effort — a successful validation that nobody claims, a deployment that stalls because nobody's willing to sign off. Chapter 16 covers this in full: why AI adoption is an executive-sponsorship undertaking. For now, just hold onto the question: is this on the CEO's top-five list?

III. The Remaining Four Filters

Once a problem clears the first filter, it still has to pass four more: value density, verifiability, data reachability, and tolerance for change. Together with importance, that's five filters — not a flat checklist of equal-weight items, but a narrowing channel, where each filter eliminates one specific way the project could die (see Figure 6-2).

Figure 6-2: What gets stopped isn't a bad idea — it's an idea that doesn't yet meet the conditions for going live.

Filter two: value density. Three questions on-site are enough: who's doing this right now? How many times a day? What happens if it's done wrong? Answer those three, and you've basically diagnosed how healthy this problem actually is.

There are three plain signals for judging value:

High frequency — something done every day pays off from even a small improvement; something that happens three times a year won't save much even at ten times the efficiency;

Currently done by hand — only where the work runs entirely on manual labor, experience, and veteran know-how is there room for AI to step in;

Can get ten times better — a 10% improvement convinces nobody to switch; a 10x improvement has people fighting to use it.

So "if a scenario is only 10% faster, don't even touch it — save your energy for the high-frequency, hand-done, ten-times-better ground"². Don't forget the list of seven wishes from the start of this chapter: value density isn't just about the "how many headcount does this save" ledger — it's also about the "what market does this open up that we couldn't reach before" ledger⁸.

Filter three: verifiability. What's the current baseline for this problem — the baseline being the status quo: how long does the manual process take, what's the error rate, what's the cost? What will you measure improvement against? For a problem with no baseline, build the baseline first — the AI conversation comes second.

This filter stops the kind of problem where everyone applauds when it ships and, six months later, nobody can say whether it was actually worth it: the impact can't be quantified, and it's the first thing cut when budget season comes around. Experienced teams put "acceptance metrics" before "go live" — starting from a concrete scenario, they have business people and engineers score the output line by line, and the scores feed straight back into iteration. How to build acceptance metrics is a subject Chapter 12 covers in full.

Filter four: data reachability. What data does solving this problem require? How many systems is that data spread across? Who owns the permissions? How long does approval take? This has to be nailed down during discovery — it can't wait until production. In one real case, an internally launched customer-service agent project didn't try real customer data until its second iteration, at which point data-security concerns landed on the table immediately, and the project was forced into a budget-constrained, on-premise small-model deployment that badly capped the model's capability⁹. Data-security constraints are real, but they need to be understood — and designed around — during discovery, not discovered during production. One thing worth flagging when you draw the data map: in almost every organization, the official data source and the data source employees actually trust are not the same thing — some veteran employee's private spreadsheet, a report that only ever circulates by email, is often the one people doing the actual work use every day. Connect to the wrong source, and no matter how accurate the system is, nobody will trust it.

Filter five: tolerance for change. Once the solution goes live, whose job changes? Is this problem someone's actual power base? Whose position did the process you're about to automate used to support? This isn't office gossip — it's the project's survival. On demo day, the department that cooperated the whole way through applauds, and in the corner, a manager from another department says nothing — the process he's responsible for maintaining is exactly the one you just automated. The mature move is to design a "path forward" for the people who stand to lose, rather than a dead end — turning the gatekeeper into a coach, and turning a veteran's experience into training material for the system¹. The list of problems this filter surfaces gets reused later — Chapter 15 on adoption and Chapter 16 on executive sponsorship both come back to it.

The filters get run wrong sometimes too — four mistakes come up most often:

Starting from the highest-value problem — usually unverifiable, and you've bet big before you've even started;

Starting from the easiest problem — usually nothing anyone really feels the pain of; even a successful build is just self-congratulation;

Starting from the CEO's personal preference — a scenario the boss picked off the top of his head, usually neither high-frequency nor measurable, and nobody dares push back;

Taking on a "just build us a demo for the board" job — a demo optimizes for looking good, production optimizes for being usable, and the two are different builds from the very first line of code.

IV. Running One Case Through All Five

Here is an anonymized real case.

The customer was a mid-sized manufacturer that walked in with seven wishes: smart quoting, meeting-minute generation, photo-based quality inspection, vendor-email sorting, an equipment-repair knowledge base, a sales-pitch assistant, and contract review.

First, filter one: this customer's CEO cared about exactly two things this year — delivery delays and raw-material cost. Of the seven wishes, smart quoting and contract review hung directly off delivery and cost; the equipment knowledge base was adjacent (downtime hits delivery); the remaining four were things "each department happened to bring up." Four got cut here.

Filter two (value density): smart quoting happens dozens of times a week, each one taking a salesperson two hours flipping through price sheets and past deals, and a mistake means losing money directly — high-frequency, hand-done, ten-times-better: passes. Contract review runs about thirty times a year, two hours per review — genuinely painful, but not high-frequency; parked for this round. The repair knowledge base: every minute of downtime burns money, but a "knowledge base" by itself doesn't stop downtime — back up one "why" (is the real problem retiring experts taking their knowledge with them, or is it slow parts lookup?) — parked pending further digging.

Filter three (verifiability): smart quoting has clean historical data — three years of quotes and closed prices serve as the baseline, and improvement is measurable: passes.

Filter four (data reachability): the price sheet lives in ERP, past deals live in CRM, and permissions sit with two different departments — reachable, but approval takes two weeks, so the request goes in during discovery. Passes, with a condition.

Filter five (tolerance for change): the sales director who owns the quoting process is a longtime direct report of the CEO's and openly supports the project; the extra step for the sales team — "confirm the AI-suggested price" — can be designed to default to acceptance: passes.

Seven wishes, five filters, one survivor: smart quoting — a one-sentence problem statement ("cut sales quoting from two hours to ten minutes without raising the error rate"), a baseline (current average turnaround time and price-revision rate), and a data-permissions checklist (ERP price sheet plus CRM history, two-week approval). The process of cutting six carries more information than picking the one that survived. Every wish that got cut left a record behind: why it's not happening now, what condition it's missing, and when it's worth raising again. That record is itself next quarter's roadmap.

V. The Discovery Deliverables

Discovery is done. What should this stage actually deliver?

Not a slide deck — the discovery deliverables: three things that let next week start with actual work.

1. A problem card: a one-sentence problem, the current baseline, and a target metric — one page, no more. If it doesn't fit on one page, you haven't actually thought it through.
2. A data-reachability checklist: what data is needed, which system it lives in, who owns the permissions, how long approval takes — the raw record from filter four.
3. An interactive prototype: not a demo video, but something the customer's own people can click through with their own hands. A Japanese FDE, two months into the job, wrote down this field lesson: "over a perfect spec document, a prototype people can actually touch. Watch how the customer reacts, and correct course on the spot."¹⁰ A spec document locks a direction in place; a prototype exposes it. The first thing a customer says the moment they get their hands on something real often overturns whatever consensus came out of the previous three meetings (see Figure 6-3).

Figure 6-3: A problem card, a data-reachability checklist, and an interactive prototype let discovery hand off straight into next week's work.

With the three-piece set in hand, discovery is over. The one-sentence problem on that problem card now has to become a system that actually runs inside the customer's environment — how small and how real that first version needs to be is exactly what Chapter 7 takes up next.

---

Footnotes

1. Fan Bing, Forward Deployed Engineer (FDE) (XDash open-source book) ↩ ↩² ↩³

2. "100 Questions About FDE" (open-source ebook) ↩ ↩² ↩³

3. "FDE Engineer 101: Bridging the Gap Between AI and Business" (Chinese-language compilation video) ↩ ↩²

4. Yilu Tongxing (一路瞳行), Episode 134: "From AI Assistant to Digital Employee" ↩

5. Y Combinator, "The FDE Playbook for AI Startups" (Bob McGrew) ↩ ↩²

6. Turning Point (破局点), Episode 31: "FDE, Chinese Style" ↩

7. "Reflections on Palantir" (Nabeel Qureshi) ↩

8. Silicon Valley 101, Episode 240: "The Hottest New Job in Silicon Valley — FDE" ↩

9. "AI Job Sense: Stop the Agent Rat Race" ↩

10. "I Thought FDE Meant Fighting Alone at the Customer Site" — What Two Months on the Job Actually Looks Like ↩