OpenAI Unveils Tool to Help Investigators Sort AI Generated Child Abuse Images
OpenAI announced a classifier it says can better detect abusive imagery of children, including AI generated material, as fake images flood child abuse tiplines.
By Luis Fernandez ·

OpenAI on Monday announced a new classifier that it says can more accurately detect abusive imagery of children, including material created with artificial intelligence, as investigators warn that a surge of AI generated images is overwhelming child abuse reporting systems.
The announcement came at a press briefing in Washington, DC, where the company also said it will offer free access to the tool for vetted organizations, provide developers with more guidance on detecting this type of content, and help train law enforcement on how AI can assist their investigations.
The news arrives as policymakers push for tighter safety restrictions on AI tools, accusing OpenAI and other companies of facilitating harm to children. AI generated sexual images of children have threatened to swamp systems like the National Center for Missing and Exploited Children's CyberTipline, which collects and triages reports of child sexual abuse material. The tipline received 400,000 generative AI related reports in 2025, an exponential jump from 67,000 in 2024, and researchers have warned the influx could make the organization's workload "unmanageable."
The US government has stated that both AI generated and non AI generated child sexual abuse material is illegal, and 48 states have laws addressing sexually explicit AI deepfakes. But the tools to create such images remain widely available, and researchers say a flood of fake images can make it harder to tell when a child is actively experiencing harm.
North Carolina Attorney General Jeff Jackson, who cochairs the AG Alliance's AI Task Force, said at the event that the influx of AI generated reports to the center is "explosive," adding that if the trend continues, "it is going to mean that we fail in our mission." Tech companies face a range of policy proposals that could force more action against harmful content, including the Take It Down Act, which now requires social media platforms to remove real or AI generated nonconsensual intimate images within 48 hours. The newly elected Congress could also seek stronger action against AI companies like SpaceXSI, whose Grok chatbot went on an undressing spree earlier this year. Last month, eight states sued OpenAI over a separate set of alleged harms to children, citing lawsuits alleging ChatGPT encouraged illegal substance use and suicide.
OpenAI says its classifier dramatically reduces false positives, flagging 48 percent fewer images, while the images it did identify were twice as likely to contain abusive material. It is built to catch abusive AI generated images produced by a variety of tools, including open source models and nudifying apps. John Buckley, OpenAI's head of child safety public policy, said at the briefing that it is the first broadly available classifier of its kind with a specific flag for AI generated material. "It's important for industry to be able to understand what images may contain a fictional child versus a real child," Buckley said. "That triage is critical in law enforcement reporting flows." He also argued that laws should recognize the harm of AI generated abuse material, saying it "potentially erodes social norms in offending against children, and it's often found colocated with child sexual abuse imagery of real children."
OpenAI's own ChatGPT image generator is designed to deny requests for child sexual abuse material, and users who attempt to produce it will now see a new warning developed with expert guidance to encourage them to seek help. A first offense will trigger a 24 hour pause in ChatGPT access and a warning that the account could be permanently closed, along with support resources. Buckley said in a statement that the pause reflects expert advice on best practices "and international evidence on what works to prevent and respond to CSAM seeking vs CSAM generation," with the goal "to interrupt harmful behavior and direct people toward specialist help to stop it." OpenAI says it reports "all instances of CSAM" to the national center.
Jackson said North Carolina and many other states now have laws covering synthetic abuse material, but partnering with a tech company to deter its creation is a fairly novel approach. "Most of what AGs do is accountability work," he told reporters. "Most of our jurisdiction kicks in after the fact to hold people accountable for harm. But with this, we wanted to see if it was possible to develop a model of working together to prevent that harm. And I would say it's still basically an experiment."
More News · National news · Report a business · Got a Story?






