Last week a due-diligence pipeline returned a 4,000-word report about a token that did not exist in its input. Nine analytical dimensions. Comparison tables. Confidence grades. A risk matrix. The extraction stage had failed — the source sat behind a paywall, the crawler returned whitespace — and the analysis stage never noticed. It formatted the void.
I have read a great deal of bad crypto research. This was the first piece I had seen that was internally rigorous about having no content: a field-by-field status table, every row marked unverified, followed by a professional-sounding conclusion that the input was unanalyzable. The model was correct. The pipeline was not. That gap — between a true statement about failure and a system that shipped it as a deliverable — is the part worth auditing.
Two-stage LLM analysis has become the default architecture for token due diligence. Stage one extracts structure: title, source, thesis, information points, project names, timestamps. Stage two consumes that structure and generates the nine-dimension report — technical, tokenomics, market, ecosystem, regulatory, team, risk, narrative, supply-chain transmission. Funds run this over inbound decks. DAOs run it over governance proposals. Small audit shops run it over whitepapers before quoting a scope.
The architecture is sound in principle. It keeps extraction and judgment separate, which is exactly what you want when you need to trace a claim back to a source span. The problem is what happens at the join.
In this case, stage one returned a well-formed object: every key present, every value an empty string, an empty array, or null. No exception thrown. No HTTP error. The crawler had hit a login wall and returned 200 with a body of navigation chrome. The extractor found no thesis because there was no thesis. It reported that honestly. Stage two received a syntactically valid payload and did what it was trained to do: it produced a structured document.
The failure mode has a name in systems engineering — silent null propagation into presentable output. Solidity auditors have been flagging the identical bug since 2016. A low-level .call() returns a tuple; the success flag is a boolean; ignoring that boolean lets execution continue after the call has failed. Every reentrancy post-mortem I have written includes a variant of it. It keeps reappearing not because it is a logic error but because it is an interface-contract error: both sides behave correctly in isolation, and the seam between them has no gate.
The fix is unglamorous. Validate on information density, not key presence.
def gate(extraction, min_points=5):
if not extraction.get("core_thesis"):
return {"status": "ABORT", "reason": "no_thesis"}
if not extraction.get("project_name"):
return {"status": "ABORT", "reason": "unresolved_target"}
if len(extraction.get("information_points", [])) < min_points:
return {"status": "ABORT", "reason": "insufficient_density"}
return {"status": "PROCEED", "payload": extraction}Twenty lines. Fail-closed. The report that reached me existed because no equivalent twenty lines existed upstream. Schema validation passed, because "N/A" is a string and [] is a list.
The second observation is less obvious and more useful. The report's most valuable content was its own admission. It flagged that "unrated" is not "low risk," and graded zero disclosure as an independent risk of medium-to-high severity, because an investor cannot perform diligence on a black box. That read is accurate. Absence of disclosure is not neutrality; it is an unquantified liability. In a market where LPs are withdrawing from protocols with three-line documentation and no audit trail, the distinction between "we found nothing wrong" and "we found nothing" is the whole ballgame.
Third: template tokens as a leading indicator. On-chain data does not lie, and neither does text. Count sourced assertions per section. If more than 60% of the output is framework furniture — headers, caveats, restated methodology — the analysis is decoration. I ran that ratio against the report. The technical section contained zero performance figures, zero repository links, zero audit reports. Nine dimensions, zero data points. The taxonomy was complete for the same reason a sealed box is complete: nothing had entered it.
The instinct will be to blame the model for hallucinating. It did not hallucinate. It refused to — and that refusal was then packaged into a document that reads like analysis. The dangerous output is not the confident wrong answer. It is the hedged empty one, because it looks like diligence and survives review. A reader skimming for red flags sees a risk matrix with no red in it. The formatting is the exploit. Vulnerabilities hide in plain sight, usually inside a table.
The report even recommended a workaround: skip extraction, submit raw text directly to stage two. Sensible for a single ad-hoc query. Catastrophic as a pattern. Drop the extraction layer and you lose the provenance chain — no assertion can be traced to a source span, no confidence grade can be justified, no reviewer can distinguish a quoted fact from a generated one. You gain latency and lose auditability. Metadata is fragile; code is permanent. Do not trade the first for the second.
Six years of auditing AI-adjacent trading infrastructure has burned one lesson into me: guardrails fail at the seams, not at the centers. Each stage in a chain can be individually sound. The join between them is where a null becomes a conclusion, where an unverified field becomes a table cell, where a failed crawl becomes a four-thousand-word deliverable.
The forward question is not whether models hallucinate. Assume they do. The question is whether your pipeline has a gate that fails closed on empty input — and whether anyone downstream would notice if it did not. Silence is the loudest exploit. Logic remains; sentiment fades.