Did OpenAI really not see the Sora 2 copyright controversy coming?

OpenAI's launch of its Sora 2 AI video generator and Sora social media app is in disarray after it backtracked following complaints about copyright theft. The company's claim to have been surprised by the controversy and users' rapid circumvention of new controls demonstrates an outstanding lack of thought before the launch of such powerful tech.

To recap, Sora 2 is a new AI model that can generate much more realistic and controllable video than its predeccessor. OpenAI, the company behind Chat GPT, launched the model along with an iOS app that it intended as a new form of social media where users can generate deepfakes of themselves and their friends.

Initially, copyright holders were told they had to opt out if they didn't want their intellectual property to appear in videos generated by the model. After a week of chaos in which people generated AI videos of things like a Nazi SpongeBob SquarePants and ads for 'Epstein Island' children’s toys, OpenAI backtracked and switched to an opt-in policy as media companies and bodies such as the Motion Picture Association complained of copyright infringement.

To try to sway people to opt in, it promised to give rightsholders more control over the generation of characters. Bill Peebles, OpenAI’s head of Sora, posted on X (above) that users can specify how their cameo is used through text instructions, such as “don’t put me in videos that involve political commentary” or “don’t let me say this word.”

Some users have reacted furiously, claiming that Sora 2 is now practically useless because they continuously receive warnings that their video generation requests breach the new guardrails. But others are already finding ways around the new controls, using unofficial images or changing character names to avoid the app detecting that third-party IP. Ultimately, the problem is in Sora's training and OpenAI's whole approach to AI ethics.

Incredibly, OpenAI's CEO Sam Altman has suggested that he wasn't expecting the app to be so controversial. Considering the well-documented concerns about copyright theft in generative AI training, including several ongoing lawsuits, that suggests he must have been living under a rock for the past few years.

The Verge cites the CEO as saying in a Q&A response: “I think the theory of what it was going to feel like to people, and then actually seeing the thing, people had different responses. It felt more different to images than people expected.”

Sam also seems to have been surprised to learn that users might not want their deepfakes to say offensive things. He said expected people would either want to make their deepfakes (dubbed 'cameos') public or not, but not that their decision might be more nuanced. “They don’t want their cameo to say offensive things or things that they find deeply problematic,” the billionaire CEO was surprised to learn.

Another fear around Sora is that it could plunge us into a crisis of misinformation and fake news. Sora adds a watermark to videos by default, but it's hardly difficult to remove. Sam said in the Q&A Monday that he knew that “people are already finding ways to remove it”, suggesting that the CEO of one of the world's biggest tech companies has just learned that AI watermark removers exist or that you can easily mask out a watermark in a video-editing program.

The whole debacle shows OpenAI's haphazard approach to launching its tech. It veers to and fro between trying to paint a picture of responsibility and promoting its models for their ability to rip off well-known art styles and copyright material. For the launch of Chat GPT 4, it actively encouraged the brief AI Studio Ghibli craze with Sam using a Ghiblified picture of himself as his profile image on X.

Is the company and its leadership really so out of touch that it didn't predict the reaction that Sora generated? Or is it playing a game of testing how far it can go, releasing things without controls to generate hype and then rowing back when it fears legal fallout or harm to its relationships with major media companies?