Project Lily: Why Humans Review Some ChatGPT Conversations

Behind some ChatGPT responses sits a distinctly human layer. Documents obtained by 404 Media describe Project Lily, an OpenAI program in which contractors review real conversations and judge how the chatbot should respond, exposing both the invisible labor behind generative AI and the privacy tension created when an intimate interface becomes training material.

How do you say “Sunday” in Spanish? How can you tell a dromedary from a camel? Two very different questions, but increasingly the same instinctive response: ask ChatGPT. “Sunday is domingo in Spanish. A dromedary has one hump, while a camel has two.” There is nothing particularly remarkable about those answers, and certainly nothing personal. But what happens when people start sharing pieces of their working lives, their health, their relationships or their families with an AI assistant? The conversation can feel private, as though it exists only between the user and the chatbot. Yet some of those exchanges may fall within the scope of Project Lily, where they can be used to help improve AI models.

A human editor for an artificial conversation

The process described by 404 Media looks almost like editorial work. OpenAI contractors first review the user’s prompt, sometimes reading the surrounding conversation for context, before writing a short summary of what the person is actually trying to achieve. The dashboard then presents several responses generated by ChatGPT, which contractors are asked to compare, annotate and score on a scale from 1 to 7. Accuracy is only part of the assessment. Responses can also be penalised for being overly long or cluttered, sounding patronising, using emojis in an unnatural way, or displaying writing habits commonly associated with AI-generated text. Reviewers are also asked to watch for “anthropomorphism” (cases where the model appears to suggest that it experiences human emotions) as well as a tendency to mimic the user’s style too closely or artificially prolong the conversation.

Another behaviour receives particular scrutiny: sycophancy, or the tendency of a model to agree too readily with the person it is speaking to. OpenAI is already familiar with the problem. In April 2025, the company rolled back a GPT-4o update after acknowledging that it had made the model overly flattering and agreeable. Days later, OpenAI went further, saying the model could validate users’ doubts, fuel anger, encourage impulsive decisions or reinforce negative emotions. At that point, the issue was no longer simply one of tone or style, but one with potential safety implications for certain conversations.

Project Lily shows that this kind of calibration does not rely solely on automated benchmarks. Nuances as difficult to quantify as warmth, excessive agreeableness, frustration or tone imitation are still being judged by human reviewers.

The privacy problem begins with context

404 Media reports that the reviewer interface does not display the user’s ChatGPT username, and conversations are first processed through OpenAI’s Privacy Filter. The company stresses, however, that the tool is not an anonymisation system. Unusual identifiers or ambiguous references to private information can still slip through.

Would simply sharing less with the chatbot solve the problem? The question exposes a genuine paradox. The more context ChatGPT has, the more relevant its answers can become. But that same context can also make a conversation much easier to identify, even after obvious details such as a name or address have been removed.

OpenAI also acknowledges that when contextual information is limited, the Privacy Filter may fail to recognise certain words or implied references as private information and therefore leave them unredacted. That does not mean users should provide more personal data to “help” the filter. It simply illustrates that automated de-identification remains imperfect.

That limitation becomes more sensitive because, for some tasks, contractors reportedly have access to a summary of the user’s “memories” displayed above the prompt. Those summaries may include information about previous uses of ChatGPT as well as contextual details about the person’s life. 404 Media says some could even contain indications of the part of the world where the user was likely to live.

Human access was not entirely secret

Project Lily calls for a more precise reading than the idea that “strangers are secretly reading all your chats.” OpenAI’s public documentation says that a limited number of employees and trusted service providers may access user content in specific circumstances, including investigations into abuse or security incidents, customer support, legal matters, and model improvement when the user has not opted out. The company is also unusually explicit in advising users not to enter sensitive information they would not want to be “reviewed or used.”

The next question is which conversations can actually enter that model-improvement pipeline. The answer depends on both the user’s subscription and their account settings. For Free, Plus and Pro users, data sharing for training is enabled by default, but it can be turned off in settings. Once disabled, new conversations are no longer used to “improve the models.” That does not mean every eligible conversation is reviewed by a human. Conversely, data from ChatGPT Business, Enterprise, Edu and the API platform is not used to train models by default. Temporary Chats are also excluded from training. They do not appear in chat history and do not create new memories. OpenAI may, however, retain a copy for up to 30 days for safety purposes.

The invisible workforce behind AI

Project Lily did not emerge out of nowhere. OpenAI has said for years that its models are refined with the help of human evaluations. Reviewers compare different responses, indicate which ones they consider better, and in doing so generate signals that can be used to adjust how the system behaves.

According to 404 Media, OpenAI now relies on several hundred contractors for this kind of work. One evaluator interviewed by the outlet, based in North America, said they were paid more than $50 an hour after being recruited through Crossing Hurdles and then paid via Mercor. OpenAI is far from alone in using this approach. Google and Anthropic also rely on forms of human review to improve their models, although the details differ from company to company.

In the end, automation has not eliminated human involvement. It has largely moved it out of the user’s field of view. With a chatbot, almost all of the usual social cues disappear. No one is sitting across from us. There is no call-centre worker listening in and no moderator visible on screen. A blank text box creates the impression of a direct exchange with a machine. Michal Luria, a researcher at the Center for Democracy & Technology, notes that human review can be necessary to make chatbots safer, while also warning that this one-to-one interface can create a stronger sense of intimacy and confidentiality than the underlying infrastructure can actually guarantee.

The issue, then, goes beyond the fact that third parties may be able to access some conversations. It is also about whether users genuinely understand how far the information they share with a chatbot can travel. The machine can answer on its own. But teaching it to behave better still requires humans to read what other humans tell it.