Sora is even fooling human deepfake detectors

It used to be that artificial intelligence would leave behind helpful clues that an image it produced was not, in fact, real. Previous generations of the technology might give a person an extra finger or even an additional limb. Teeth could look odd and out of place, and skin could render overly blushed, like something out of Pixar. Multiple dimensions could befuddle our models, which struggled to represent the physical world in a sensical way: Ask for an image of salmon swimming in a river, and AI might show you a medium-rare salmon steak floating along a rapturous current. 

Sure, we were in the uncanny valley. But at least we knew we were there.

That’s no longer the case. While there are still some analog ways to detect that the content we see was created with the help of AI, the implicit visual tip-offs are, increasingly, disappearing. The limited release of Sora 2, OpenAI’s latest video-generation model, has only hastened this development, experts at multiple AI detection companies tell Fast Company—meaning we may soon come to be entirely dependent on digital and other technical tools to wade through AI slop. That has ramifications not only for everyday internet users but also for any institution with an interest in protecting its likeness or identity from theft and misappropriation. 

“Even [for] analysts like me who saw the evolution of this industry, it’s really hard, especially on images,” Francesco Cavalli, cofounder of one of those firms, Sensity AI, tells Fast Company. “The shapes, the colors, and the humans are perfect. So without the help of a tool now, it’s almost impossible for the average internet user to understand whether an image or a video or a piece of audio is AI-generated or not.”

Visual clues are fading

The good news is that at least for now there are still some telltale visual signs that content was generated via artificial intelligence. Researchers are also hunting for more. While extra fingers appear less common, AI image generation models can still struggle to produce sensible text, explains Sofia Rubinson, a senior editor at Reality Check, a publication run by the information reliability company NewsGuard. 

Remember that surveillance video of bunnies jumping on a trampoline that turned out to be AI-produced? You might just have to consider whether rabbits actually do that, Rubinson says. “We really want to encourage people to think a little bit more critically about what they’re seeing online as these visuals are going away,” she adds.

Rubinson says it’s possible to search for whether a portion of a video has been blurred out, which might suggest that a Sora 2 watermark used to be there. We can also check who shared it. Toggling to an account’s page sometimes reveals a trove of similar videos—an almost-certain giveaway that you’re being served AI slop. 

On the flip side, usernames won’t necessarily help us discern who really produced content: As Fast Company previously reported, it’s somewhat easy, though not always possible, to grab a Sora 2 username associated with a famous person, despite OpenAI’s rules on using other peoples’ likenesses.

Ultimately, we may need to become fluent in a model’s individual style and tendencies, argues Siwei Lyu, a professor at the State University of New York at Buffalo who studies deepfakes. For instance, Sora 2-generated speech can appear a little too fast. (Some have dubbed this an “AI accent.”) Still, Lyu warns that these indications “are subtle and can often be missed when viewing casually.” 

And the technology will improve, which means it’s unlikely such hints will be around forever. Indeed, researchers say the visible residue that AI was involved in creating a piece of content already seems to be fading. 

“The tips that we used to give in terms of visual inconsistencies are disappearing, model after model,” says Emmanuelle Saliba, a former journalist who now leads investigations at GetReal Security, a cybersecurity firm working on detecting and studying AI-generated and manipulated content. 

While incoherent physical movement used to indicate AI’s use in the creation of an image, Sora 2 has improved significantly on mimicking the real world, she says.

At Reality Defender, also a deepfake detection firm, every one of the company’s researchers—half of whom have doctorates—have now been fooled by content produced by newer generations of AI. “Since the launch of Sora, every single one of them has mislabeled a deepfake as real or vice versa,” Ben Colman, cofounder and CEO of Reality Defender, tells Fast Company. “If people who’ve been working on this for 5 to 25 years cannot differentiate real from fake, how can average users or those using manual detection?”

Labels won’t save us, either. While companies have touted watermarking as a way to identify AI-generated content, simple workarounds appear to foil these tools. 

For instance, videos from OpenAI’s Sora come with a visual watermark—but online tools can remove them. OpenAI, like other companies, has committed to the C2PA standard created by the Coalition for Content Provenance and Authenticity. That specification is supposed to encode the provenance, or source, of a piece of content into its metadata. Yet the watermark can be removed by screenshotting an image created by OpenAI technology. Even dragging and dropping that image, in some cases, can remove the watermark, Fast Company’s tests with the tool show. 

OpenAI concedes this flaw, but a spokesperson said they weren’t able to reproduce the drag-and-drop issue. When Fast Company posed questions about this vulnerability to Adobe, which operates the C2PA verification tool, the company said the issue was on OpenAI’s end. 

Updating methodologies

Of course, the companies Fast Company spoke to are interested in selling various products designed to save us from the deepfake deluge. Some envision that AI content detection might go the way of virus scanning and become integrated into myriad online and workplace tools. Others suggest that their platforms will be necessary because the rise of tools like Sora 2 will make video call-based verification obsolete. Some executives believe their products will play a role in protecting brands from embarrassing AI-generated content. 

In response to the release of the Sora app, a few of these firms do say they’re seeing growing interest. Still, like humans, even these companies need to update their methodologies when new models are released. 

“Even if the human cannot spot anything from the tech point of view, there’s always something to investigate,” Sensity’s Cavalli says. This often requires a mixed-methods approach, one that takes into account a range of factors, including studying a file’s metadata and discrepancies in background noise. Sensity’s detection models are also retrained and refined when new models come online, Cavalli adds. 

But even this isn’t always perfect. Lyu from SUNY Buffalo says that while the detection systems his team has developed still work on videos produced with Sora 2, they have lower accuracy compared to their performance on generative AI models. And that’s after some fine-tuning.  

Hany Farid, a UC Berkeley professor who cofounded GetReal Security and serves as its chief science officer, says the company’s forensic and data techniques have seen “better” but “not perfect” generalization in the latest models. In the case of Sora 2, some of the company’s video techniques have remained effective, “while others have required fine-tuning,” he says, adding that the audio detection models still work robustly. 

That’s a change from earlier eras of generative AI, when forensic techniques had to be continuously updated to apply to the latest models. “For our digital-forensic techniques, this required understanding specific artifacts introduced by the AI models and then building techniques to detect these artifacts. For our more data-based techniques, this required generating content from the latest model and retraining our models.”

Whether these deepfake detection methods will continue to hold up is unclear. In the meantime, it seems that we’re increasingly heading toward a world flooded by AI but still building its seawalls. 

The name of GetReal Security has been updated.