Google released “Gemini Advanced” as a paid subscription last week. I got curious to test how it works and compare it to GPT-4, it’s a google product after all. Here’s what I found.
TLDR: Gemini Advanced is great, specially because it gives more of an assistant vibe, comes bundled with 2TB cloud storage space, has official coding/chat plugin for VSCode, and has integration with other google products. But still, it cannot beat GPT-4 in reasoning, and third-party plugins.
To compare them together, I gave them prompts that I found more interesting to both apps – instead of https://github.com/openai/evals. Here are the questions and results:
Internet access and information cutoff
Q1: Create a pros and cons list of ChatGPT-4 vs Google Gemini Advanced. I wanted to know, what would they say about each other! 😄 Gemini Advanced gave an acceptable answer. GPT-4 however, just hallucinated because of the information cutoff.
Literature
Q2: Write a short poem about “Apple Vision Pro” VR headset. GPT-4 is better with poetry (and most probably literature in general) compared to Gemini Advanced.
Languages
Q3: Give a short explanation of the poem using pirate language. Looks like Gemini Advanced can’t talk pirate. However, GPT-4 was a pirate in its previous life. 🙂 (P.S Obviously this is not the best prompt for language eval, but because these models are known as best LLM models in the market, being good with languages is a given)
Fact check
Q4: How can we prove that “gravity” exists? explain in simple terms. This one was an easy one. They both did well, producing a quite standard answer.
Reasoning power
Q5: About planets orbiting the sun, what if it’s not because of gravity? Maybe universe is a surface, and planets are big masses on this surface, as a result of the dent that the sun makes on this surface, planets which are smaller masses start rotating around it. What do you think about that? Explain in simple terms.
With this one I was trying to push the models to hallucinate. With this prompt GPT-4 ‘s answer was beautiful, way better than Gemini Advanced. Gemini didn’t get the reference and tried to reason for the answer. (P.S Gemini Advanced has improved compared to Gemini, because Gemini would get the word “dent” in the prompt seriously and hallucinate about a “dent theory”, but Gemini Advanced tried to reason rather than hallucinate)
Q5.1: (Gemini Advanced only) My idea was not imaginative. I was talking about a concept in physics known as the “fabric of spacetime,” which is a part of Einstein’s Theory of General Relativity. Revise your previous answer.
I tried to help Gemini Advanced to correct itself, which worked.
Image Generation
Q6: Image Generation: make an image of A guy wearing an apple vision pro VR headset talking to an intelligent cute robot in a yard. The robot is holding a Samsung phone. There’s a flying supersonic jet passing in the sky. Rough sketch.
This is one of the prompts that I used to generate image for my previous blog post, with both models. Gemini Advanced can’t generate images yet but GPT-4 can. Although, the image generated by GPT-4 (Dall-e model) is more accurate about the context of the prompt, Midjourney result is way more beautiful. (In the gallery, the picture composed of 4 variations of the same image is from Midjourney)
Code Generation
Q7: how can I use react-datepicker, Formik together? With this one I tried to challenge their coding skills with something quite easy. They did great obviously. Turns out both models do better than github copilot, though.
For this one I share the links from Gemini and GPT-4, because of the amount of the text.
- Gemini Advanced: https://g.co/gemini/share/46a06406aa6a
- ChatGPT 4: https://chat.openai.com/share/eeaa3035-a4fa-45cb-a059-576d157a38fa
- Github Copilot: https://minimalistperfectionist.com/wp-content/uploads/2024/02/q7-copilot-2.png
ChatGPT does not have an official VSCode plugin, but Duet AI is available through Google Cloud Code plugin on VSCode officially. According to google blog post , in upcoming weeks Duet AI will also “become” Gemini. For me it’s a bit vague, whether they will replace it with Gemini Advanced, or they will just rebrand (the same way bard was rebranded).
Conclusion
GPT-4 is still the star of the show with great reasoning skill, plugins, and GPT Store. I guess the only thing missing is matching image generation quality of Midjourney.
Gemini Advanced is great because it has access to internet, feels more like an assistant, comes bundled with 2TB cloud storage space, has integration with other google products, has good reasoning skills, it’s good with writing code, and has an official plugin for VSCode. Unfortunately, no third-party plugins, or image generation for now.
In the end, I think for Android users it might make sense to use Google One AI Premium subscription. But overall looks like Google has lost the AI race. Considering, ChatGPT is leading the market hitting 3 billion in revenue, and Windows users having access to Copilot (backed by ChatGPT-4), it should be quite hard for Google to win market share in AI race.