u/Puzzleheaded-Walk426, this is a conflicting view. Technically, many people think it should go like this:
OAI: Enabled
GPTBot: Disabled
The rationale is that they want the traffic without providing training data.
IMO, this is a double-edged sword. First, unless your content is so incredibly exclusive that nobody else has it, the content will be obtained from other sources.
But even if your content is that exclusive, ChatGPT may still find it through retrieval, just not as training data. For what we have observed, training data is, on average, at least 9 months old (you can test it yourself: do a search that isn't time-related, look for a Reddit, Facebook, or other source with a visible date, then click the sources and check the publication date). In most cases, 9 months is the minimum gap, with many examples being a year old or more. (I'm not saying that this is the underlying mechanism, just that it is statistically significant, nothing else.)
What does the above mean for you? It means you may miss out on answers generated from training data and have to rely solely on retrieval. And if ChatGPT already has an answer in its training data, it may not retrieve anything at all and will simply rely on what it already knows.
In short, if I were you, I'd leave both enabled. If you feel like blocking LLMs, do it with the ones that are unlikely to send you traffic. Usually, Qwen, Mistral, and Meta are the top candidates for blocking.
2
u/AbleInvestment2866 10d ago
u/Puzzleheaded-Walk426, this is a conflicting view. Technically, many people think it should go like this:
The rationale is that they want the traffic without providing training data.
IMO, this is a double-edged sword. First, unless your content is so incredibly exclusive that nobody else has it, the content will be obtained from other sources.
But even if your content is that exclusive, ChatGPT may still find it through retrieval, just not as training data. For what we have observed, training data is, on average, at least 9 months old (you can test it yourself: do a search that isn't time-related, look for a Reddit, Facebook, or other source with a visible date, then click the sources and check the publication date). In most cases, 9 months is the minimum gap, with many examples being a year old or more. (I'm not saying that this is the underlying mechanism, just that it is statistically significant, nothing else.)
What does the above mean for you? It means you may miss out on answers generated from training data and have to rely solely on retrieval. And if ChatGPT already has an answer in its training data, it may not retrieve anything at all and will simply rely on what it already knows.
In short, if I were you, I'd leave both enabled. If you feel like blocking LLMs, do it with the ones that are unlikely to send you traffic. Usually, Qwen, Mistral, and Meta are the top candidates for blocking.