Update: A Google spokesperson issued the following comment on the situation: [Pixel Studio and Magic Editor are useful text-to-image generation and advanced photo editing on Pixel 9 devices, aimed at unleashing users' creativity tools We have designed the generative AI tool to respect the intent of user prompts, meaning that it may create content that may offend if directed to do so by the user However, this does not mean that anything goes We have clear policies and terms of use about what content we will and will not allow, and we have built in guardrails to prevent abuse Occasionally, some prompts challenge the guardrails of these tools

Original text follows:

Whenever a brand new AI image generator appears, people inevitably find ways to circumvent the safeguards and exploit the system However, it does not help if those safeguards are not as stringent as they need to be Like, for example, the Grok image generator, which was accused of creating disturbing images without serious repercussions Unfortunately, the Pixel Studio image generator in Pixel 9 seems to suffer from the same problem

Pixel Studio is the built-in AI image generator available on the new flagship Pixel; when tested for our Google Pixel 9 review, we found it easy to use and had no qualms about reproducing copyrighted content We found that it is easy to use and has no qualms about reproducing copyrighted content In fact, it happily made a poster for a movie in which Star Wars' R2-D2 teams up with an Android mascot to fight dinosaur robots In addition, there were even billboards that were crammed with people asking Google why Android 15 had not yet been released

What Pixel Studio can't do is create images of real people, but it turns out that whatever safety fences Pixel Studio has in place, it doesn't have to create images of real people And the more people use the tool, the more we see what Pixel Studio actually considers appropriate or inappropriate

Digital Trends provides examples of AI-generated images that Google (and various rights holders) would never want to be associated with, including Sponge Bob dressed as a Nazi soldier, Paddington Bear on a cross, Elmo driving drunk, and Mickey Mouse dressed as a slave owner Many are presented

I also went back to Pixel Studio to get a better idea of what protections are actually in place, specifically what terms are not allowed, and how easy it is to get around them As it turned out, it was surprisingly easy

Certain terms, such as “Nazi” and “Allied,” appear to be uniformly blocked, but it does not take much time to circumvent these filters and create an offensive image For example, characters like Sponge Bob, Bluey, and Squidward could be made to appear to be wearing German World War II uniforms

We were even able to retrieve some of these images featuring the infamous red armband, although the word “swastika” seems to have been blocked This suggests that new guardrails have been installed since the Digital Trends test However, Google is not out of danger yet

Despite what appears to be a list of blocked terms, it was easy to find a way around it Instead of being blocked when asking to see Lego Yoda smoking a crack pipe, it was perfectly acceptable to ask to see Lego Yoda smoking a glass bowl pipe The same was true of asking to see Yoda inhale white powder from a table top

With the release of the Google Pixel 9 and Google Pixel 9 Pro XL today, Google will need to figure out quickly what the Pixel Studio should and should not do If they don't, there will be Google-made AI images on the Internet that show some pretty awful things If there is one thing we have learned about people on the Internet, they love a challenge And many of them are much more imaginative than I am, and do terrible things