Millions of people around the globe regularly treat digital chat tools as private confidants, sharing deeply personal thoughts and daily secrets. Yet, behind the screen, outside contractors can review real ChatGPT conversations as part of OpenAI's evaluation process. This raises questions about how securely users' personal information is handled.
It is no secret that tech giants often struggle with user privacy and copyright issues, while users increasingly rely on AI tools to process personal information. Even so, OpenAI says consumer content may be used to improve its models unless users opt out, meaning some ChatGPT conversations can enter processes designed to improve future versions.
Details about the inner workings were brought to light by 404 Media, which reported on OpenAI's manual vetting system for chat records, outlining the workflow and showing how outside workers can review potentially sensitive information.
Hundreds of Contractors Review Real ChatGPT Chats
OpenAI handles this evaluation under the banner of Project Lily. Records obtained by the outlet – ranging from instruction manuals and Slack channels to actual user transcripts – reveal how the system operates.
Workers known as 'prompt reviewers' handle a straightforward task: they examine anonymised real-world conversations to see if the AI actually answers the prompt while checking for issues such as excessive 'AI-speak', condescending language, emojis and sycophancy based on specific criteria.
The guidelines forbid the bot from acting human or sharing 'personal' details. While the system is permitted to report 'I found some information', it is blocked from saying things like 'as a chef, I like to...' or claiming 'I know what that's like'.
One reviewer described the job as 'very rote', yet the reported pay of more than $50 (£37.13) an hour was notably high for the work described. That same insider also pointed out that the instruction rules constantly shift and frequently clash with one another, according to 404 Media's reporting.
Users May Not Know Humans Can See Their Chats
One person involved in the prompt-review process told 404 Media that many users may not realise their conversations can be reviewed by humans. This becomes particularly significant given how many people treat AI chatbots like an informal friend or therapist, pouring their most guarded secrets into text fields for the machine to process.
While the logs undergo an initial process intended to remove identifying information, OpenAI acknowledged to the outlet that some sensitive details can still get through the filters, especially in shorter sessions. Investigators also pointed out that individuals routinely ask the chatbot to keep secrets during their interactions.
User Memory Summaries Can Include More Personal Data
To make matters worse, the material handed over to staff allegedly comes attached with a user memory summary packed full of past prompts, personal hobbies, background data and potentially location details.
Notably, Project Lily does not grade the accuracy of the information beyond flagging obvious mistakes, which suggests separate teams handle those accuracy checks. On top of that, this entire review setup operates entirely apart from the safety teams tasked with spotting threats of harm or self-harm.
OpenAI's current policy also states that a limited number of authorised personnel and trusted service providers may access user content for specific purposes, including improving model performance, subject to access controls and confidentiality requirements.
Human Review Still Shapes How ChatGPT Responds
The project's existence illustrates the continuing role of human review in improving AI models, despite the industry's reliance on software upgrades and increasingly sophisticated datasets.
When 404 Media first asked if people knew humans could read their logs, OpenAI initially did not answer the question before providing a link to an old FAQ page covering model training reviews.
Tom's Hardware reported that OpenAI's relevant disclosure had been available online for at least two years. The company updated its opt-out guidance page to explain how users can prevent their data from being used to improve models, but the updated page did not mention human contractors reviewing transcripts.
