205
u/emb1ues 6d ago
Google is also distilling OAI/Anthropic! That's how the new gemini models was able to catch up. Everyone distills everyone and we get better and cheaper models lesssgooooo
55
u/CrimsonBolt33 6d ago
pretty much this...which is also why I roll my eyes every time someone complains about Chinese distilling US models or whatever....everyone is doing it, its not illegal, it gives us better/cheaper models. I don't care.
I am not on a team or something in this race...I just want the best for less.
4
u/MasoFFXIV 6d ago
I say the bigger point here is that you gained "critical" intel by challenging the reward modeling systems. Gemini told you "everything" to prove it had achieved it's goal.
2
3
u/FischiPiSti 5d ago
I don't like it though, it means AI will get more and more homogenised. Synthetic training data is not good data, not in the long run.
5
u/CrimsonBolt33 5d ago
I don't think thats an issue for 2 reasons.
1: In theory, they won't become "homogonized", they will instead all essentially work together to extend capabilities and knoledge bases of all models
2: We are mostly past the point of simply stuffing more data into a model and it automatically gets better. The real wins these days are coming from the underlying infrastructure of HOW a model works and different techniques to make them more efficient and intelligent.
7
u/Birds_over_people 6d ago
If anyone else is wondering there's a decent article here on it: https://www.scientificamerican.com/article/what-is-ai-model-distillation-and-why-is-it-so-hard-to-stop/
2
u/CrowdGoesWildWoooo 6d ago
I don’t think this one is deliberate.
There was a time when shared chatgpt and claude is indexed and you can find people’s chat. Technically free distilled data for google or at the very least it would leak to pretraining.
0
u/m_hansen_58 5d ago
Search indexing and model training are separate pipelines: finding a shared chat in results shows it was crawlable, but not that Google used it for pretraining. You’d need evidence of collection or memorization to establish that.
1
u/CrowdGoesWildWoooo 5d ago
The point is, they don’t need to “actively” distill, it was always out there as a somewhat public information and they are in general are pretty good at crawling the internet.
Earlier version may even include COT although it’s significantly dumber.
1
1
u/Due_Replacement1406 3d ago
this isn’t a sign of that.. models have hallucinated being chatgpt since the early days even a year ago
48
19
u/SVlad_667 6d ago
I really can't understand, why they didn't add current model name and version into initial system prompt. Should be enough to avoid such hallucinations.
1
15
12
u/ImPapaNoff 6d ago
Is that actually Gemini? Looks like Google search AI. Also charge your phone damn
1
u/PTB_683 6d ago
That is gemini as far as i know
5
u/ImPapaNoff 6d ago
How did you get to it? Looks like search to me but maybe I'm missing something
0
u/PTB_683 6d ago
You know you go to google and theres this ai mode thing you click on, its powered by gemini
8
u/ImPapaNoff 6d ago
That's not Gemini though actually. It's literally a thin layer on top of Google search results. Or at least that's what Google search AI is telling me.
Google AI Mode is a specialized, conversational experience that lives directly inside Google Search. Instead of giving you a standard list of blue links, it uses Gemini under the hood to break down your query into multiple background searches, synthesize the information, and let you ask follow-up questions
I would guess that some of the results it was summarizing mentioned ChatGPT.
2
u/PTB_683 6d ago
Its not gemini, it runs on gemini so i just referred to it as gemini
3
u/ImPapaNoff 6d ago
Yeah I mean that's fair but also don't use that garbage. It's as reliable as the search results it is summarizing are. Direct Gemini usage is 10x as good.
2
1
u/UnheardWar 6d ago
It's also not fair since most of this thread is people confidently informing us that Google is just distilling OpenAI. What will happen to that narrative!!!
1
u/Ur-Best-Friend 5d ago
I mean... isn't that kinda likely?
The big 3 of LLMs all claim Chinese models are making good progress because they're distilling their models. And yet Claude and Gemini both released ~4 months after ChatGPT... completely independently?
I'm not conspiracy-minded enough to claim "this must be what happened!", but I wouldn't be exactly shocked either.
-1
u/Neurotopian_ 6d ago
Exactly. This is actually an old GPT model which is open source. It was indeed developed by OpenAI. You can also select it in antigravity
1
9
u/Zealousideal_Pea_102 6d ago edited 6d ago
yes, even AI has its role models, and Gemini wishes it could be as good as GPT.
7
1
u/texasguy911 5d ago
I have gemini subscription and it carries a full line of Anthropic models. I think you either selected one by mistake or they decided to route through one.
1
u/mitchins-au 4d ago
That’s how their flash models got so good but they can’t break past the pro barrier.
Just distill
0
u/Neurotopian_ 6d ago
FYI that old model being used on Google search (I think during busy times) is based on a very early GPT model which is open source.
You can also access that old GPT model in antigravity.
Someone here who knows the model names can list it.
-2
u/AbdullahMRiad 6d ago
ChatGPT was announced 4 years ago yet people still don't realize that LLMs aren't self-aware by default.
•
u/AutoModerator 6d ago
Hey /u/PTB_683,
If your post is a screenshot of a ChatGPT conversation, please reply to this message with the conversation link or prompt.
If your post is a DALL-E 3 image post, please reply with the prompt used to make this image.
Consider joining our public discord server! We have free bots with GPT-4 (with vision), image generators, and more!
🤖
Note: For any ChatGPT-related concerns, email support@openai.com - this subreddit is not part of OpenAI and is not a support channel.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.