Synthetic Data to Retrain Drifting Targeting Models, and a Missing Test Standard for AI and Autonomous Systems

▶ Watch on YouTube · YC Root Access · 10:11

The defense industrial base of the future will be built by small, fast-moving teams working on some of the hardest problems in national security.

Live from Washington, D.C., Y Combinator brings together founders, senior government officials, and military leaders for a day of conversations about rebuilding America’s defense industrial base, working with the government, and closing critical capability gaps across autonomy, munitions, space, secure communications, manufacturing, and more.

— Channel description, YC Root Access (event description; this page covers one chapter)

The last founder talk of YC Root Access’s day in Washington (see the event page). Kristopher Luo, co-founder of Null Labs, says he and his co-founder dropped out of Stanford and worked at Google DeepMind. Null Labs generates labeled synthetic training data for military computer-vision systems. In ten minutes Luo makes two pitches: synthetic data as the way to retrain a model that adversaries have learned to defeat, and the same generator as a way to test AI and autonomous systems before the government buys them.

It earns its own page for two reasons. It gives the corpus a case of model drift with a named cause: an opponent who changes what the model sees, on purpose and on a weekly timescale. And it states the testing problem from the buyer’s side. There is no common test standard, so buyers pick the vendor they already trust, and a live trial covers one time of day, one weather and one terrain. The talk is a pitch and gives no measurements; Debates below says what rests on what.

TL;DR

• Two products from one generator. “What we do right now is we provide companies with data to train more lethal AI systems.” The second, “more esoteric opportunity I see is basically applying these same mechanisms of the training data that we create … to how we evaluate and test autonomous systems and AI systems” (7:48:08).
• Drift with a cause. Computer vision for finding targets became standard with Project Maven in 2017, “essentially putting computer vision on every single drone to identify targets” (7:49:03). Adversaries now counter it. Luo’s example is Russian vans fitted with mirrors “almost as like a dazzling effect” so that Ukrainian drones’ AI no longer detects them (7:49:27). The result: “deployed models that Palantir are putting out or you know other contractors are essentially drifting and decreasing in performance over time as adversaries are adapting their own valuable assets to essentially not be able to be detected by our AI systems” (7:49:46).
• No real data to retrain on. “The enemy changes its look next week. We’ve got zero images for us to train our new AI system on” (7:50:07). Collecting means getting hold of the adversary’s target, booking range or flight time, waiting for the right time of day, weather and angles, then labeling, “which I’m sure some of you know is actually very expensive to do.” Null Labs generates “the exact target, your CONOPS, the domain of interest” and the sensor, already labeled. “One 10,000 image data set, we generate that in 45 minutes. To collect that, that’s 2 to four weeks or even longer” (7:51:03).
• The risk, named and weighed. “Synthetic data has a risk. You can overtrain on it. You can, you know, over generalize onto synthetic data, maybe learn synthetic artifacts, but I think the bigger risk is deploying AI systems that are failing, are not working, are not detecting our adversaries” (7:51:33).
• No common test standard, so the trusted vendor wins. “Right now there’s not really a common standard for testing AI and autonomous systems. … How are they supposed to pick the right one? So oftentimes they’re just going to go with the trusted vendor” (7:52:50). From his own experience, cold applications to SBIRs and challenges return “nada”, and “the real contracts that we’ve gotten are from knowing someone on the inside” (7:53:30).
• Why AI breaks the old procurement model. Procurement still works the way it did for “procuring like an aircraft carrier” (7:53:54). With a tank, edge cases fall to the operator’s judgment. “But when an AI system is in those edge cases, you don’t know how it’s going to perform and you don’t have that human input that is going to control like making the right decision” (7:54:27).
• A live down-select tests one slice. About 20 systems, shortlisted from white papers, try to shoot down 20 drones. “You’ve tested it at one time of day. It’s clear skies, whatever it is. … We’ve only covered one aspect of the entire spectrum of the operational environment” (7:55:06). A digital range “tests a lot what the range cannot”: swarms at scale, other locations such as the Middle East, “adversarial threats … of tomorrow’s world”, and all of it out of sight of satellite surveillance (7:56:14).

Visual canon

Luo pitches from a slide deck, and the slides carry what the narration leaves out: photo dates and credits, sensor bands, the size of the vendor field, a YC batch and an Army research agreement. A targeted Gemini search cut 19 stills from this chapter; 10 are published below, each transcribed from the still and checked against it, and Gemini’s reading was corrected on the collage (eight panels, not nine), on the training-and-testing grid (it missed the SWIR band and the two amphibious-vehicle panels) and on the Ukrainian drone frame (it skipped the detection labels and their confidence scores).

1 · Machines have found targets for decades

Still at 7:49:13 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Nothing new. Machines have found targets for decades.

When	System	Slide text
1991	Tomahawk	Missiles match what their camera sees to stored scenes to find the target.
1990s	Predator	Drones stream live video from over the battlefield.
2017	Project Maven	The Pentagon puts computer vision on drone video to flag targets automatically.

Photos: U.S. Navy, U.S. Air Force (public domain).

Still vs. transcript: the narration goes straight to Maven, which it says “standardized it, essentially putting computer vision on every single drone to identify targets.” The slide adds the two earlier steps and words Maven more narrowly: computer vision on drone video, flagging targets, with no claim about every drone.

2 · Adversaries are adapting to fool AI

Still at 7:50:04 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Why now. Adversaries are adapting to fool AI.

• Left, AI finds targets from the drone feed: Ukrainian drone footage, AI detection on armored vehicles. Three boxes on vehicles along a road, labeled id:26 armored_vehicle 0.38, id:21 armored_vehicle 0.84 and id:18 armored_vehicle 0.31.
• Right, So the other side dresses up to fool it: Russian van covered in mirrors, Aug 2026 · Photo: via JFeed.

Still vs. transcript: the narration describes the mirrored van but gives no date or source. The slide dates the photo to August 2026 and credits it via JFeed; the credit was not checked. The left panel shows the detector’s own scores on unaltered vehicles: 0.84 for one, 0.38 and 0.31 for the other two.

3 · One 10,000-image dataset

Still at 7:51:12 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Let’s make this obvious. One 10,000-image dataset.

• Generate with Null Labs: < 45 minutes (a sliver of a bar)
• Collect it: 2–4 weeks (a long bar)

Still vs. transcript: the narration says “we generate that in 45 minutes” and that collection takes “2 to four weeks or even longer.” The slide states generation as an upper bound, under 45 minutes, and drops “or even longer”. Neither says what resolution, sensor or compute the figure assumes.

4 · Generated scenes across domains

Still at 7:52:08 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

No text. Eight generated scenes, left to right and top to bottom: a drone over a snow-capped coastal cliff; a sailboat on open sea; a ship among islands in low sun; open water seen from above, with two small detection boxes; an arctic bay with icebergs and a small boat (a wide panel); a warship in a grayscale infrared-style view; a split view with colored bounding boxes over grassland, vehicles and a small building; and a forest with a tank by a stream. The box labels are too small to read.

Still vs. transcript: Luo calls this “the realism that we’re working with across all domains from surface to space.” The collage shows sea, coast, arctic, grassland and forest, one infrared view and two scenes with detection boxes. None of the eight panels shows a space scene.

5 · Generate, and you get there first

Still at 7:52:31 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Build the model · Illustrative. Generate, and you get there first.

A schematic with no values on either axis. The vertical axis is labeled Model ready to field, the horizontal Time →.

• Generate with Null Labs (solid line) climbs to ready early and holds.
• Collect real data (dashed line) stays flat for most of the period, then climbs to just below the solid line.
• The shaded area between them is labeled The gap: time without a working model.
• At a dashed vertical line, Adversary adapts: the solid line drops to zero and climbs again as fast as before; the dashed line restarts from zero and is still climbing when the chart ends.

Still vs. transcript: this is the slide Luo presents as model drift: “the performance of a model is going to dip when the adversary adapts their target … It’s definitely not with real data. We’re going to get that model accuracy back up using Null Labs, using synthetic data.” The slide is marked Illustrative and has no numbers. It plots readiness to field, not accuracy, and it draws real data reaching readiness too, only later: the gap it shows is time, where the narration rules real data out.

6 · Same engine, training data and testing data

Still at 7:53:27 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Same engine. Training data and testing data. Eleven generated images, each with a bounding box on its target:

• Two large panels, left: an amphibious armored vehicle in open water, seen from two angles. The lower box is labeled zbd-05 #1 (ZBD-05 is the designation of a Chinese amphibious infantry fighting vehicle); the upper label is too small to read.
• A three-by-three grid, right: tankers and cargo ships at sea and in port, each tagged with a sensor band. EO on five, LWIR on two, SWIR on two.

Still vs. transcript: this slide is on screen when Luo says “some of these images are from current customer engagements with SOCOM as well as PACOM”, so these are the images he means. The slide names no customer or command, so it does not settle PACOM against INDOPACOM. The narration speaks only of “the sensor of interest”; the tags show three bands: visible (EO), long-wave infrared (LWIR) and short-wave infrared (SWIR).

7 · The status quo: fly a few drones in Huntsville

Still at 7:55:37 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

The status quo. Fly a few drones in Huntsville. See what gets shot down.

100 vendors pitching → Paper: white paper downselect (flagged Who got left out?) → A few: brought to a live event → 1 gets picked

Still vs. transcript: the narration gives the middle of the funnel, a downselect “from the white papers to like 20 systems” that try to shoot down “20 drones”, placed “in like the Mojave Desert, I don’t know”. The slide adds the field at the start, a round 100 vendors (the slide before it is a ten-by-ten grid of vendor tiles under “100 vendors. One decision. How do you pick?”), and puts the trial in Huntsville, which matches his later “when we tested it in Alabama”. Its flag sits on the paper stage: the cut it questions happens before anything is tested.

8 · Test what the range can’t

Still at 7:56:48 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Digital first · live to confirm. Test what the range can’t.

Swarms at any scale	Train in Alabama	Tomorrow’s threat
No range can fly this safely	Validate for the Middle East	Test against adversaries that haven’t shown up yet

And it’s more secure: nothing flies, so there’s nothing for an adversary to watch.

Still vs. transcript: the three columns and the security line follow the narration. The kicker does not: Digital first · live to confirm keeps the live range, after the digital one, as its check. The narration never gives live testing that role, and neither says how a digital result would be confirmed live.

9 · One environment for the whole life of the model

Still at 7:57:09 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

Null Labs. One environment for the whole life of the model.

Generate the data → Train the model → Test before fielding → Deploy to the warfighter, with a dashed arrow from Deploy back to Generate: Refresh whenever the threat changes.

Hardware-agnostic: USV, UAV, ground, space · Y Combinator F25 · CRADA with DEVCOM Army Research Lab

Still vs. transcript: the narration lists the same four steps. The slide adds the loop back to generation when the threat changes, and its footer makes three claims the talk never speaks: the platforms it serves (uncrewed surface and aerial vehicles, ground, space), Null Labs’ Y Combinator batch, F25, and a cooperative research and development agreement (CRADA) with DEVCOM Army Research Laboratory. None was checked.

10 · The ask: help Congress know this exists

Still at 7:57:42 from YC Root Access, “The Startup Industrial Base: Building for the Next 250 — Washington, D.C.”.

The ask. Help Congress know this exists.

Save dollars → Save warfighter time → Save lives

Field the best systems. Stay ahead of the threat.

Still vs. transcript: the spoken ask is to “connect with folks” interested in “making big changes in how we’re procuring autonomous systems and how we’re training them”. Only the last line, “help field the best systems and stay ahead of the threat”, is spoken. The written ask is aimed at Congress and frames the reform as a chain of savings that ends in lives.

Why this matters to the wiki

1. Synthetic data where real data does not exist yet. The corpus’s one computer-vision case of synthetic training data is Carrier’s syringe-defect detector, recorded on the industrial-ai-agents page: real defects are too rare to train on, so they are generated. Luo’s case has the same shape with a harder kind of scarcity. The target’s new appearance has never been photographed, and the adversary decides when it changes. APIGen-MT is the corpus’s detailed account of generating training data for language agents, and most of that paper is about validating what was generated: a review committee, reflection on failure, rejection sampling against executable ground truth. Luo names the failure modes (overtraining, learning synthetic artifacts) and describes no validation step in this talk.

2. Drift caused on purpose. Elsewhere in the wiki, drift is something to monitor: Huang lists eval monitoring and drift-watching in the development stack a company takes on when it owns its models. Here a deployed model’s accuracy falls because an opponent changes the input to defeat it, and the response Luo proposes is retraining on newly generated data. He describes his slide as showing accuracy dipping when the adversary adapts and recovering with synthetic data (7:52:11). The slide itself (still 5 under Visual canon) is marked Illustrative, has no values, and plots time to a field-ready model rather than accuracy.

3. Testing in simulation, and what it costs. The second pitch is an argument the ai-benchmarks page holds for software agents. At the Hugging Face agentic-evals workshop, Andrews (Meta) describes GAIA 2 on the ARE simulated environment: simulation buys reproducibility, observability, safety and low cost, and gives up realism, the sim-to-real gap. Luo lists what a digital range buys for physical autonomous systems: coverage of weather, time of day, geography and future threats that one field trial does not reach. He does not discuss the gap for testing. The ai-benchmarks page records that gap measured for robots: 89.4% success on RLBench’s simulated manipulation tasks, 12% on real household tasks (AI Index 2026).

4. No agreed bar, and who gets picked. On the AI Index panel, Perrault says nobody knows how reliable AI has to be to be acceptable in legal, medical or financial use, and that he sees no fully autonomous systems reaching scale. Luo states the procurement version of the same gap: with no common test standard, buyers fall back on the trusted vendor. His edge-case argument, that a tank’s operator decides what to do at the edges and an autonomous system has no operator, is the distinction the automation-vs-augmentation page draws from Narayanan: automation needs a higher reliability threshold than augmentation, because no human catches the failure first. On the buy side, the enterprise-ai-adoption page holds Collison’s report that enterprises now buy from unvalidated startups because the status quo looks riskier. Luo describes US defense buying in which that shift has not happened.

Dynamic capabilities. contextual/external-triggers: adversaries changing their vehicles’ appearance to defeat deployed computer vision is the external change that degrades a fielded AI system and forces it to be retrained. contextual/internal-barriers: on the buyer’s side, procurement methods built for aircraft carriers, no common test standard for AI, a default to the trusted vendor, and contracts that come through insiders rather than SBIR applications.

Linked entities and concepts

• Entities: Y Combinator (YC Root Access runs the event), Google DeepMind (where Luo says he and his co-founder worked after leaving Stanford).
• Sources: the event page; with typed edges, Carrier 2026, Prabhakar et al. 2025, Hugging Face workshop 2026, Stanford HAI 2026; without a typed edge, Huang 2026 on drift-watching, Collison 2026 on the enterprise buy side, and AI Index 2026 for the RLBench figures.
• Concepts: industrial-ai-agents (synthetic data for rare-event training), ai-benchmarks (simulated environments, the sim-to-real gap, no agreed standard for evaluating agents), automation-vs-augmentation (the reliability threshold when no human is in the loop), enterprise-ai-adoption (vendor selection by buyers).
• Dangling (single-source mention, deferred): Kristopher Luo, Null Labs, Project Maven, Palantir, SOCOM, PACOM (US Indo-Pacific Command), SBIR, Jason Cornelius and Perseus Defense (an earlier speaker at the same event, named here as a company that did not reach a field down-select).

Debates and supersession

• Every number is the founder’s. 10,000 labeled images in 45 minutes, against 2–4 weeks to collect, is a claim made in a pitch. No image resolution, realism measure, compute budget or model result is given. The claim that synthetic data brings accuracy back after drift (“It’s definitely not with real data”) rests on a slide marked Illustrative with no values on either axis (still 5), and no numbers are spoken. That slide draws real data reaching a field-ready model too, only later.
• Synthetic-to-real transfer is asserted, not shown. Luo names overtraining and synthetic artifacts as risks and weighs them against fielding systems that fail. That is a judgment, not a measurement. The wiki’s one measured sim-to-real figure is for robot manipulation (RLBench, 89.4% simulated against 12% on real household tasks, on ai-benchmarks): a different task, with the risk running the same way.
• Training and testing from one generator. The pitch uses the same mechanism to make training data and test data. The talk does not say how a synthetic test would be checked against live results, or whether a system’s test data would come from the same generator as its training data. The deck’s testing slide is headed Digital first · live to confirm (still 8), which gives the live range the job of confirming digital results without saying how.
• Customers and incidents unverified. Luo says some images on his slide are “from current customer engagements with SOCOM as well as PACOM” (ASR: “paycom”; PACOM is the former name of US Indo-Pacific Command, and he may have said INDOPACOM). The slide on screen as he says it (still 6) shows generated maritime imagery, ships tagged EO, LWIR or SWIR and an amphibious vehicle labeled zbd-05, but names no customer, so the PACOM/INDOPACOM question stays open. The mirrored Russian van carries only a photo credit on its slide (“Aug 2026 · Photo: via JFeed”, still 2, not checked). The remark that failing AI systems are “currently happening in Iran right now” is stated without a source and is as transcribed.
• Procurement claims are one founder’s experience. Cold SBIR applications returning nothing, contracts through insiders, and contractors who “don’t deliver” over a 24-month period of performance come from his own company and from conversations “with the other Defense Tech founders here”. They sit beside Collison’s enterprise buy-side reversal. The two describe different buyers, US defense and enterprise, so neither contradicts the other directly.
• Vendor pitch. The talk ends with an ask to connect with people interested in reforming “how we’re procuring autonomous systems and how we’re training them”; the closing slide puts it as “Help Congress know this exists” (still 10). Null Labs sells into the reform it argues for.

What was actually ingested

Chapter 7:47:41–7:57:52 of the livestream: 225 caption segments. The talk ends at about 7:57:41, with the speaker’s thank-you cut off by the chapter change. The transcript is raw auto-generated ASR, not cleaned at acquire. Every quote was checked against it at the cited timestamp and corrected only for names, acronyms, fillers (“uh”, “um”) and repeated words: “no labs” → Null Labs, “Google Demon” → Google DeepMind, “Palanteer” → Palantir, “conops” → CONOPS, “sibers” → SBIRs, “paycom” → PACOM, “oftent times” → oftentimes, “You you can” → “You can”. The speaker’s name comes from the chapter heading and his title slide; he does not say it on air, nor name his co-founder. The team slide at 7:48:55 (not published here) names the co-founder as Niel Ok, CTO, and puts the Google DeepMind logo under Ok’s name only; under Luo’s it shows the Stanford Nanofabrication Facility. The talk is slide-led: a targeted Gemini still search cut 19 stills from this chapter, every one was viewed and checked against its pixels, and 10 are published under Visual canon. The 9 left out are the title card, the team slide, four cards the narration reads out nearly word for word (zero real images; procurement is broken; no common test standard; contracts through insiders), a collect-or-generate comparison the narration covers step by step, a vendor-grid slide the funnel repeats, and a fielded-versus-never-tested card that restates his point about contractors who do not deliver. Null Labs’ website, its customer claims, the Project Maven history and the Russian-van example were not checked.