As a European, I of course received slightly delayed access to Googles new AI mode, so just now tried it for the first time.
Normally not one to judge a model by the first prompt, but I felt the need to share my first experience due to how poorly it went and as something of a counter to past reporting by others [0].
Asking for offers concerning the "Google AI Ultra" [1] plan, I received a hodgepodge of incorrect information [2], including the claim that such a plan did not exist for Austrians (despite linking to just the Google One page where it is advertised) and claims around a "Google AI Premium" offering that, as far as I can tell, has been renamed to "Google AI Pro" months ago.
I tried other models, of course, which did handle the task as expected [3].
Genuinely, I cannot for the life of me understand how this happens, especially since the model did find and link to the correct information, especially considering Geminis upcoming models seem very promising in my A/B limited testing. A cautious reminder though that even when it comes to information by the company making that model, using LLMs for (re)search requires users to actually step through the linked sources.
It also reinforces a notion I have had for a while, but never benchmarked due to lack of interest, that despite having the search advantage, Googles models somehow purely subjectively have been the least reliable in using online sources unless directly linked. Bard, Gemini, Attention is all you need, etc. have been a tale of missed opportunities, I feel this has been talked about to no end, but that they still struggle with search after years [4], when the frontier models they release have gotten competitive in most other fields is hard to understand.
As a European, I of course received slightly delayed access to Googles new AI mode, so just now tried it for the first time.
Normally not one to judge a model by the first prompt, but I felt the need to share my first experience due to how poorly it went and as something of a counter to past reporting by others [0].
Asking for offers concerning the "Google AI Ultra" [1] plan, I received a hodgepodge of incorrect information [2], including the claim that such a plan did not exist for Austrians (despite linking to just the Google One page where it is advertised) and claims around a "Google AI Premium" offering that, as far as I can tell, has been renamed to "Google AI Pro" months ago.
I tried other models, of course, which did handle the task as expected [3].
Genuinely, I cannot for the life of me understand how this happens, especially since the model did find and link to the correct information, especially considering Geminis upcoming models seem very promising in my A/B limited testing. A cautious reminder though that even when it comes to information by the company making that model, using LLMs for (re)search requires users to actually step through the linked sources.
It also reinforces a notion I have had for a while, but never benchmarked due to lack of interest, that despite having the search advantage, Googles models somehow purely subjectively have been the least reliable in using online sources unless directly linked. Bard, Gemini, Attention is all you need, etc. have been a tale of missed opportunities, I feel this has been talked about to no end, but that they still struggle with search after years [4], when the frontier models they release have gotten competitive in most other fields is hard to understand.
[0] https://simonwillison.net/2025/Sep/7/ai-mode/
[1] https://one.google.com/intl/de_at/about/google-ai-plans/
[2] https://share.google/aimode/Pey5o1s8OsrKsxN7J
[3] https://chatgpt.com/share/68ee0ec3-a188-8009-a5fc-aa103754c1... and https://claude.ai/share/0cfd06e9-b67b-442b-87dd-a6843f5b84a1
[4] https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....