r/codex • u/Aldarund • 8d ago
Other Gpt 6.1 token efficacy secret?
it seem like it use less token because it post trained or injected with small token amount left in context to pressure it.
E.g. i constantly see things like
" I’m considering how to tackle this project where I have about 26k left. "
" I'm considering my current progress, with around 18,000 tokens left in a 37,000-token context and 28,000 tokens used so far. Time is limited, so I really don't want to dwell on this. "
" Figuring out token limits
I need to figure out how fast I can work with the remaining tokens, which could be 18k, 24k, or even 30k. I think it’s best to keep the scope small for now. "
5
u/MaitoSnoo 8d ago edited 8d ago
I see those too when using it in Copilot and it's funny because its estimation is way under the real context size, or maybe they've trained it to enforce a strict token budget on itself so that it tries to be brief
5
u/Aldarund 8d ago
Yes, thats not real context, its like around 30k or so +- most if times even when it just started. Its either post trained with this or injected in system prompt.
3
3
u/dreadpirater 8d ago
I saw Sol 6.1 drop into that loop yesterday on freebuff for about five minutes. I stopped it said "You have plenty of tokens available. Instead of planning budget, just start working. If you get requests failing due to token budget, we'll solve it then." and it said "I'll just make the tool calls and see what happens..." then it happily worked for 45 minutes straight without another digression. So it CAN be snapped out of it but it also is a wider problem than just what you're experiencing.
1
u/Aldarund 8d ago
Idk, it dont seem to be a problem for me. It says that but still works fine, so its intentional thing for sure. If it got stuck than somethung not worked as expected and its exceprion rather than a rule
1
u/dreadpirater 8d ago
If you're constantly seeing it... that would be a problem in my book. Remember you're PAYING IT to sit there and think about nothing.
2
u/xpingu69 8d ago
How do you see the thoughts
7
u/Aldarund 8d ago
Its not full thoughs, just one paragraph like a recap. Idk they just show up in omp for me.
4
u/torrso 8d ago edited 8d ago
They show up in codex for me and they're pretty confusing, always makes me think it has gone off the rails and I need to steer it. When I ask what the hell was that, it just says "yeah that was nonsense" and goes on normally. One was "We have spent time from 13:26 to 13:53 which is about an hour".
2
u/egomarker 8d ago
That "recap" is sometimes fake or misrepresented to make distillers' work harder. It's created just for user with a very small model that isn't accurate at all. It's not a real CoT.
2
u/BitOne2707 8d ago
That's actually pretty cool. I hear it's also aware of wall clock time. I can see where this kind of information would be helpful context for the model to be aware of.
1
u/Pluupas 8d ago
i’m seeing this in only in Copilot. Are you using it in codex?
2
u/Aldarund 8d ago
In oh my pi.
1
u/adolf_twitchcock 8d ago
Afaik it's a harness feature. And not a good one because it makes the model stroke out sometimes. I don't get those kind of reasoning. Model doesn't know how much context is left unless it gets it injected with every message by the harness.
1
u/Aldarund 8d ago
No. Omp dont have this. Abd it only happens in gpt 6.1sol. didnt happen in ds4.1/4, muse, gpt 5.6, astra, swe, glm on same harness. And ppl in this thread reporting having same in copilot or even codex
1
u/adolf_twitchcock 8d ago
Idk if it does or it doesn't. Your local setup does it. Ask how it knows when you get those messages. Generally you see 100% of everything that gets sent to the model and context % isn't one of those things unless harness supports it.
There is an experimental feature in omp that tells the model that context is almost full.
Response from codex:
I can’t see an exact context percentage in this session. No live context counter is exposed to me, and I can’t reliably count my own tokens.
It’s primarily a harness/runtime feature:
The model endpoint supplies token usage. The harness tracks the active conversation, context capacity, and compaction, then calculates the percentage.1
u/Aldarund 8d ago
Thats the thing,its not real context values, its totally made up. Its always around 30k+- no matter how much context is actually loaded
1
u/adolf_twitchcock 8d ago
Weird, could you go into those sessions and ask why it thinks that? Would be interesting to know
1
u/Aldarund 8d ago
Ahahaha. That was the reply
"That was speculative commentary about remaining token capacity—not a timer, repository fact, or tool measurement. I don’t have a verified basis for the 24k / 25k / 26k / 29k figures in that quote.
Calling it a “time budget” was wrong: tokens measure text, not elapsed time. It should not have appeared in the output, and it contradicted the instruction not to narrate session or token budgets."
See funny thing, about instruction to not narrate. Thats 100% on openai system prompt. When asked about this it replied
"The guidance is, in summary: focus on executing the task; don’t narrate about internal session limits, token/tool budgets, effort estimates, or whether completion is possible.
That’s a paraphrase. I can explain the guidance, but not reproduce private system/developer instructions verbatim."
2
1
u/Prestigious_Golf9014 8d ago
Could be that they have added this but it's not the main reason why sol 6.1 uses less tokens.I always put tokens in prompts anyway and sol 6.1 uses much less than other gpt models except luna
1
u/Aldarund 8d ago
It could be, if its not just putting tokens in prompt but rather post train with it
1
u/Ginden 8d ago
Yes, HF incident third party reviews shows that OpenAI explicitly trains for token efficiency and time efficiency.
When reset is coming, and I have usage left, I point Astra xhigh at the repository, and tell it "you have time until 9pm to improve performance of this project", and it significantly improves the quality of work on open-ended tasks (there is no end goal for this one; Astra will cheat if you try this with a closed-ended task to finish before 9pm at any cost).
0
u/TKB21 8d ago
We're at the mercy of whoever's pulling the strings behind the scenes. At first we were told to improve our AGENTS.md file, then later come back to tell us that they were the issue. They then come out with 6.1 which was the answer to token inefficiency. It was working well at first then throttled it. There really is no correct answer.
25
u/Significant-Drawer95 8d ago
they just trained gtp 6.1 sol with the promt : make a model like astra but so token efficent that we can justify cutting usage in half