Four model responses. A seven-point scale. A reviewer on the far side of the world, scoring your most private conversation for "flattery."
That is the operational core of Project Lily, and it is also a ledger — one that was never meant to be read. The complaint served in the Northern District of California on September 2 alleges that paid ChatGPT users' conversations were routed to human reviewers through third-party staffing agencies. 404 Media surfaced the underlying documents roughly two weeks later. OpenAI's response, reduced to its essentials: the review existed to curb sycophancy and improve the model.
Every line of code tells a story of greed. This one tells a story of dependency.
The pipeline behind the product
RLHF — reinforcement learning from human feedback — is the training backbone of GPT-4, Claude, Gemini, and the Llama line. It works by collecting human preference pairs: a reviewer reads a prompt, reads several model completions, and ranks them. Those rankings train a reward model. The reward model shapes the policy. Without human preference data, the alignment layer collapses.
That creates a structural demand for real conversation. Synthetic data, as of now, cannot fully replicate the cultural nuance and safety-edge judgments that human reviewers supply. So the industry built a supply chain. Conversations flow from product to platform, from platform to staffing agency, from agency to contractor. The endpoints cluster in Kenya, the Philippines, India, and Venezuela. Reported wages for some annotation roles sit between $1.32 and $2.00 an hour.
The codename matters here. "Project Lily" is floral, discreet, and — unlike "Conversation Audit Pipeline" — designed not to alarm. In my experience, soft internal names appear precisely when a team already knows the project carries reputational risk. The naming is a tell.
The jurisdictional choice is not accidental either. The Northern District of California has become the default venue for technology-privacy class actions, and its case law is mature. Under the CCPA, statutory damages run from $100 to $750 per consumer per incident. If the class captures even a fraction of ChatGPT's paid base, the arithmetic gets uncomfortable — not because OpenAI cannot pay, but because the template becomes copyable. The same pipeline exists down the street at three other labs.
The oracle problem, relocated
Here is the technical heart of the case, and it is one I recognize from an adjacent battlefield.
In DeFi, a protocol does not know the price of ETH. It asks an oracle. If the oracle is opaque or manipulable, the protocol inherits the flaw — no matter how clean its own contract logic. The oracle lied, and the market paid the price. I have traced that exact failure mode; it is how $2.4 million left a leveraged yield farm in a single transaction.
OpenAI has the same architecture, one layer up. The human reviewer is the oracle. The reward model is the contract that consumes the feed. And the "de-identification" promise is the smart contract's "trustless" guarantee — an assertion users are asked to accept on faith.
The complaint exposes where that guarantee breaks.
First, pseudonymization is not anonymization. Strip a username and you have removed a label, not an identity. Contextual inference reconstructs the rest: an IP-adjacent location, a job title, a spouse's name, a recurring medical question. Under GDPR and CCPA frameworks, pseudo-anonymized data is still personal data. The industry treats the two terms as interchangeable. Regulators do not.
Second, the "user memory summary" escalates the exposure by an order of magnitude. This is the Memory feature — cross-session profiling that hands a reviewer not one conversation but a longitudinal portrait of a person. A single chat is a data point. A memory summary is a dossier.
Third, the automated PII filter is a ceiling, not a safeguard. Named-entity recognition catches explicit tokens — emails, phone numbers, government IDs. It fails on implicit identifiers. "My boss runs a crypto fund in Zug" sails straight through. This is not an engineering oversight; it is where NLP currently stops. Based on my audit experience reverse-engineering an LLM-driven agent contract — where the output parser failed to validate signatures and a single prompt injection drained $15 million — I can tell you the parser is always the weakest link. The model is never the problem. The boundary around it is.
Fourth, the outsourcing chain dilutes accountability. OpenAI to staffing vendor to subcontractor to reviewer: four hops, each one a legal firewall. The code is silent, but the ledger screams — and the routing log that connects a specific conversation to a specific reviewer is the evidence nobody has published.
The review task itself confirms the point. Reviewers summarized user intent, scored four anonymized completions on a 1-to-7 scale, and commented on why. That "four models" detail is not trivia: side-by-side evaluation against multiple in-house versions means the prompts fed not only safety tuning but comparative model selection. The data trail maps internal iteration, layer by layer. And economically, the cheap part was always the labor. If compliant sourcing — domestic reviewers, enforceable NDAs, audited retention windows — becomes mandatory, the cost of the pipeline multiplies three to ten times. The incentive to preserve the status quo was never malicious. It was arithmetical.
What the bulls still get right
I will not pretend the case is a slam dunk.
The safety argument is genuine. Sycophancy is a real alignment failure — a model that validates everything is a model that can be led anywhere. Reducing "overly human-like" flattery is a legitimate objective, and current alternatives are weaker than the plaintiffs' framing admits. RLAIF, where a model reviews a model, invites reward hacking. Pure synthetic feedback risks model collapse. Human data remains load-bearing.
Nor is this a breach in the classic sense. No external attacker. No dark-market dump. Framing it next to Meta's $650 million facial-recognition settlement overstates the arithmetic — the affected class here is narrower and the damages theory thinner.
But the steelman does not rescue the disclosure. Whatever the safety necessity, it does not require the absence of conspicuous consent. A pipeline can be both legitimate and undisclosed. The problem was never that OpenAI read the data. It is that nobody told the users they might.
What to watch
Ignore the settlement number; it will be immaterial against a roughly $157 billion valuation. Watch the privacy-policy diff, the discovery filings, whether the class reaches minors and triggers COPPA, and whether Anthropic and Google — who run comparable pipelines — get pulled into the next complaint.
The deeper signal is architectural. If RLHF must carry verifiable provenance, the tooling that makes DeFi auditable becomes the compliance layer for AI. Beneath the surface, the truth is compiled in hex — and this time, the ledger is about to be subpoenaed.