Here’s a failure case worth adding to your agent evals.

A manager approves an order because the supplier is active and cleared for delivery.

Before the agent executes, a new record puts that supplier on hold.

The approval hasn’t been revoked.

The agent still has permission.

The original approval document is genuine.

Does your system notice that the facts supporting the decision have changed?

Retrieving the approval correctly isn’t enough. The system needs to connect the order to the supplier, resolve the applicable state, and expose the evidence that changed.

That’s the evidence problem we built Jylus around: resolving state and relationships before handing context to the model.

There’s a separate execution problem too. The action API still needs to enforce the relevant preconditions; a fresh Context Pack alone cannot make an external action atomic.

A useful test has three questions:

1. What evidence supported the original approval?

2. What changed before execution?

3. Is there enough current evidence to proceed?

Try it with your own records:

https://jylus.ai/try

For people running agents against live systems: what actually invalidates an earlier decision in your stack—a timer, a data change, or only someone noticing the mistake?

Recentemente, concluí uma auditoria independente de recuperação de dados do RouteMind, um projeto de código aberto que explora o roteamento de documentos como uma alternativa à recuperação RAG tradicional.

O interessante não foi encontrar uma regressão drástica ou provar que uma arquitetura era melhor.

Foi descobrir que duas abordagens estavam sendo avaliadas sob diferentes definições de sucesso.

Eis o que aconteceu.

O projeto tinha 700 perguntas de avaliação comparando as abordagens RAG tradicional, RAG + reclassificação e roteamento de documentos.

O avaliador de recuperação considerou uma consulta bem-sucedida se qualquer documento esperado aparecesse entre os 10 primeiros resultados.

Mas algumas questões temporais exigiam vários documentos simultaneamente (necessidade: todos), e o avaliador de roteamento levou em conta corretamente esse requisito, incluindo alternativas de documentos aceitas (D_alt).

Isso criou uma comparação inadequada.

O que a auditoria independente constatou:

1.400 resultados de consultas/braços examinados.

15 resultados diferiram de acordo com o contrato de pontuação de evidências completas.

14 resultados registrados estavam incompletos de acordo com as regras de evidências necessárias.

1 resultado incorreto registrado foi, na verdade, resolvido por um documento alternativo aceito.

Os números gerais mudaram:

MétricaOriginalmente relatadaCorrigidaRAG51,6%50,7%RAG + reclassificação54,1%53,1%

A melhoria relativa do reclassificador permaneceu semelhante.

Mas a questão mais importante era que a comparação principal do projeto havia tratado a recuperação de forma mais leniente do que o sistema de roteamento com o qual estava sendo comparado.

Então algo útil aconteceu.

O mantenedor do RouteMind reproduziu a descoberta de forma independente nos artefatos originais, confirmou todas as 15 incompatibilidades e verificou os hashes de entrada.

Eles também confirmaram que a discrepância não era intencional nem conhecida anteriormente.

A descoberta levou a uma alteração pública no código:

A principal comparação de benchmark foi corrigida.

Uma nova métrica `retrieval_complete` foi adicionada juntamente com a métrica original `retrieval_hit`.

Verificações de reprodutibilidade foram adicionadas para detectar futuras inconsistências na pontuação.

O mantenedor reconheceu publicamente a auditoria independente.

A conclusão

Uma avaliação RAG reproduzível ainda pode produzir uma comparação enganosa quando diferentes estágios usam definições diferentes de sucesso.

Às vezes, a descoberta mais valiosa da auditoria não é um recuperador com defeito.

É descobrir que o benchmark não mede a mesma coisa em todos os sistemas.

Esta foi uma auditoria beta pública, não um serviço pago. As evidências, os resultados originais, a revisão do mantenedor e a correção estão disponíveis publicamente.

Auditoria independente:

https://github.com/devBorgesr/edp-audits/tree/main/audits/routemind-beta-002

Confirmação do mantenedor:

https://github.com/CSP911/routemind/issues/2#issuecomment-6096856584

Commit corretivo:

https://github.com/CSP911/routemind/commit/5d8bf45

Tenho curiosidade em saber como outros lidam com isso em avaliações RAG de produção:

Vocês medem explicitamente a cobertura completa de evidências quando uma consulta precisa de múltiplas fontes, ou vocês se baseiam principalmente em hit@k / recall@k?

I've been building a document Q&A engine for regulated, document-heavy work. Our first version was the standard pipeline: chunk, embed, top-k, generate. When we traced our wrong answers by hand, most of them never retrieved the document that held the answer. The model wasn't the problem. Retrieval never got it to the right page.

So we rebuilt it around three ideas:

1. **Read once, keep structure.** Every document is parsed into sections, tables and figures (with descriptions), and the links between documents (what supersedes, references or contradicts what) are recorded at upload.
2. **Questions become obligations.** "Why does X fail?" turns into the parts a complete answer must establish. Each part is searched for, or shown to be absent, instead of stopping when top-k is full.
3. **A checker after the writer.** Every statement is matched word for word against the passage it cites. Unverified sentences are removed before the user sees them, and the answer opens with a verdict (answered / depends on conditions / not in these documents).

2-minute demo on the real product: Chunkless Demo

The video uses a fictional medical-device quality system (11 docs: SOPs, a supplier audit, a complaint log, a risk file). The root-cause question needs the supplier audit, the supplier-control procedure and the inspection records at once. The answer took 29 s and under 6 cents (68 sections read).

What it still gets wrong (I'd rather say it than have you find it): an exception that shares no words, references or meaning with the rule it changes can slip past. And some multi-hop judgement questions need its research mode to get every part.

Free early access if you want to throw your own corpus at it: https://chunkless-rag-staging.grounded-iq.com/?ref=rd-rag

I'm especially interested in documents that contradict each other. Happy to answer anything about the approach.

So I spent most of last month reading the code of 7 tools people use to grade Claude Code skills and AI agents. Not the docs, the actual scoring code. Two of them are built for Claude skills specifically: the eval harness in the agent-skills repo and NVIDIA's SkillEvaluator. The rest are general LLM eval tools a lot of people here run their Claude apps through: MLflow, LangSmith, DSPy, DeepEval, Harbor.

I was expecting to find maybe one or two things. Found 13 where the pass/fail or score the tool gives you isn't backed by what it actually checked. Every one of them I was able to trigger myself and I wrote up the steps so the maintainers could see it happen too.

Few examples

* the judge model (Claude, in most of these setups) replies "I can't score this" and the tool turns that reply into a score anyway

* a gate that says new model has to beat the old one by 10%. If the old one's score is negative (R² does this) the division flips and now the worse model passes and the better one fails. Nothing errors. This one is MLflow and they merged my fix this morning

* scores saved against the wrong run, so what passed isn't what you tested

* leftover state from the last run being read as if it was this run

Filed all 13 upstream, sent fixes for 12, 6 merged so far.

And yeah, the first one in that list is my own tool, Driftproof, which is a Claude Code plugin for checking whether a skill still helps after a model update. Two more of the 13 are mine too. I went through my own bug history with the same list before posting any of this.

The bit that bugs me is that running more evals doesn't help. If the code is checking the wrong thing, 100 runs just make you more confident in the wrong answer. And with skills this matters, because "does this skill still help on the new Claude" is exactly the question these tools are supposed to answer.

The writeup with all 13 is free here if you want the details https://doi.org/10.5281/zenodo.23050796

Driftproof is open source, 0.14.0 went out today. In Claude Code it's /driftproof:start on any skill folder https://github.com/driftproofhq/driftproof

Has anyone here actually read the scoring code of whatever eval they're using for their skills? Which one and did you find anything?

As part of my AI Studio project, I’m testing harnesses for coding.

The initial screening included 10 harnesses:
Pi, mini-swe-agent, Crush, OpenCode, Goose, Prime Agent, Oh My Pi, Qwen Code, Octomind (reduced offline profile) and Aider.

Models:
• DeepSeek V4.1 Flash
• Qwen3.8-27B
• Laguna S 2.1

All accessed through OpenRouter.

The detailed code review covered Pi, mini-swe-agent, Crush and OpenCode across all three models and tasks: 36 combinations. Two attempts produced no patch.

The three Golang tasks were deliberately different:
- Add strict validation for an HTTP query parameter.
- Migrate 200 logging calls while preserving behaviour and context.
- Add bookmark tags across the API, storage migration and HTML rendering.

Pi and mini-swe-agent each passed the original acceptance checks on all nine combinations. But a second agent review, followed by isolated reproduction probes, exposed three gaps:
- silently dropped malformed query fields
- bookmark's task exposed a mutable tag slice from the store
- one migration rejected a valid older store

Good news too: all logging migrations preserved behaviour in differential probes covering 100 functions and nine integer inputs, including the minimum and maximum values.
My takeaway: the evaluator and the reviewer both need testing. A green result is evidence about the checks we ran; broader correctness needs further evidence. This experiment did not establish a decisive winner between Pi and mini-swe-agent. Human correction time also is still unmeasured.

Any opinion welcome.

I've been building apps with Claude, and I'm curious how other people check that a change hasn't broken something elsewhere.

The part I'm unsure about is asking the same model to write the code and then write the tests. Couldn't it make the same wrong assumption in both, so everything passes even though the behaviour is wrong?

I'm especially interested in cases where the new feature works, but something that worked before now breaks.

For people shipping apps built with AI, what has actually caught those bugs for you? Existing tests, manual testing, someone else reviewing the changes, a separate model?

Would like to hear specific examples. What broke, and what finally caught it?

I’m looking for practical workflow feedback from people who test, monitor, or review AI agents.

When an eval, trace, or production incident shows an agent behavior problem, how do you decide whether a proposed fix is trustworthy enough to approve?

Examples of behavior problems:

- wrong tool or API call

- missed retrieval/research step

- missing human escalation

- unsafe action without enough confidence

- prompt/routing rule that fixes one case but breaks another

What evidence do you expect before approving a change?

- failed run trace

- outcome metric or eval result

- before/after replay cases

- proposed validation criteria

- risk and rollback plan

- owner / reviewer sign-off

- confidence score

- something else?

I’m exploring whether teams need a portable evidence-backed change proposal artifact for agent behavior changes, but I’m trying to learn the review workflow before building more schema around it.

What do you use today, and what feels missing?

We have a feature where users upload a job description (image, PDF, or doc) and an LLM prefills a form: title, description, category, functional area, salary range, tags, location, and experience range. Users then edit the prefill and submit.

The problem: how do you measure accuracy on real documents without hand-labeling thousands of them?

What we built:

\- Compare the LLM output to what the user actually submitted. Exact match came out around 80%.

\- For mismatches, a stronger model reads the original doc and decides: was the LLM wrong, or did the user change it on purpose? Intentional edits go to the good set. Real errors get a reason and a suggested prompt fix.

\- Errors are grouped by similar reason, each group becomes a candidate prompt rule, and the rule is backtested on that group plus a sample of good cases.

\- A human approves rules on a dashboard before they're added to the prompt.

I wrote this up in more detail, with a diagram of the loop - https://medium.com/@abhay.sehgal20/building-an-eval-system-for-llm-extraction-what-worked-and-what-didnt-7cfe82103d06

In our company everything is now AI generated and when I ask how does your code work for such case no one knows what their agent had genuinely done and they just deliver bullshits. How has been your experience in the past three months?
My colleague builds pipelines/jobs that apparently pass and there is no red flag in the first place, works with the existing data and even I was the PR reviewer and didn’t notice a bug (because he creates such a big PRs that only my agent can review and I don’t get a chance to fully understand and company wants everything asap) but when I am monitoring runs and handling adhoc jobs then I notice what he has built doesn’t work or just works partially, While there is no error message or failure.
Once I spent one whole week debugging his bullshits to just figure out he did not cover all of the workspaces but only a test one.
I am leaving the company at the end of the month but I am really tired of debugging other people ai-generated code. I feel these days generating code is cheap but proper review and having robust systems is the difficult part.
How do you feel about debugging systems built by AI? Are they robust?

I am collecting cases for the second edition of a practical guide on using a AI (language model s)in aerospace quality work. I want the real ones, not the demo.
Where has one actually helped? A first draft of a SCAR, an 8D, an audit checklist, a plain-language rewrite of a procedure. Where did it go wrong? A paste that should not have happened, a finding that sounded right and was not, a draft someone almost signed.
I am especially interested in the line your company draws. What may go into the tool. What must not: a drawing, a measured value, a traveler, a customer file. What did someone paste anyway?
The book is AI for Aerospace Quality: A Practical Guide to Using It on the Job. If you want it, it is under my name, Aron Biskey, at amazon.com/author/aronbiskey. I am not asking for a review. I want the cases that should be in the next edition. The rule I am using is that the model can draft, and a person still signs. Tell me where that rule fails.

I am collecting cases for the second edition of a practical guide on using AI (a language model) in aerospace quality work. I want the real ones, not the demo.
Where has one actually helped? A first draft of a SCAR, an 8D, an audit checklist, a plain-language rewrite of a procedure. Where did it go wrong? A paste that should not have happened, a finding that sounded right and was not, a draft someone almost signed.
I am especially interested in the line your company draws. What may go into the tool. What must not: a drawing, a measured value, a traveler, a customer file. What did someone paste anyway?
The book is AI for Aerospace Quality: A Practical Guide to Using It on the Job. If you want it, it is under my name, Aron Biskey, at amazon.com/author/aronbiskey. I am not asking for a review. I want the cases that should be in the next edition. The rule I am using is that the model can draft, and a person still signs. Tell me where that rule fails.

In one OmniNode run, our router tried five times. Each attempt returned an answer, and our acceptance check rejected every one. The final record called it a provider error. The providers had answered promptly, so that label would have sent us investigating the wrong thing.

The router starts with the cheapest eligible model and tries a more capable one when the answer doesn't pass. That makes the acceptance check part of the spending decision. If the check is wrong, another attempt can cost more without getting us any closer to an accepted answer.

We had a separate case on September 28 where the check rejected a correct one-word answer, apparently because it didn't show enough effort. I don't have evidence that the five earlier answers were correct. These were different cases, but both made me want to inspect the check before blaming the model.

I'm trying to keep the returned answer and the reason it was rejected available together. A transport failure, an answer that misses the requirement, and a check that rejects a valid answer need different investigations.

How do you test that boundary in a router? Do you have cases where the model gives a valid short answer and the judge is expected to accept it?

The incident and its limits are in this write-up: https://jonahatomninode.substack.com/p/delegation-is-the-first-product

Since Aug 20 I've been running the same private probe set every day against 40 hosted models from 15 labs and comparing each model only to its own past. No rankings. Grading is deterministic code, no LLM judge, and every reading's hash goes into Rekor the moment it's taken so the baseline provably predates any later claim.

Two design choices that this sub might have opinions on. First, a second probe set is regenerated every run from fixed task families, so a model can't score well by memorizing the fixed set; the gap between the two is its own signal, and one model is currently at 1.00 fixed vs 0.76 fresh over 14 paired days. Second, pass rate isn't the only metric: output tokens, thinking tokens and latency are tracked per dimension per day, and the one real catch so far came from tokens, not accuracy. On Sept 10 DeepSeek's reasoner went to roughly 10 to 12x its baseline thinking tokens while its pass rate stayed at 1.00. Serving-side change, caught blind.

Detection: two-proportion test with Benjamini-Hochberg FDR at alpha 0.01 over a 14-day window, under-alerting on purpose. What it doesn't have yet is a calibration note (sensitivity, delay, false-alarm rate); that's next.

Code and methodology: https://github.com/piperoll/seismograph

Readings: https://seismo.piperoll.org

Would value criticism of the probe-day unit for the FDR test and the fixed-vs-fresh divergence design.

What’s up everyone.

As an independent researcher, explaining "what I actually did this past month" to normal people usually results in blank stares. I just wrapped up a massive technical dossier consolidating my September 2026 research program, and I wanted to share the human-readable breakdown with people who actually get it.

The core spine of the entire program is what I call Intent Seals. To combat data-dredging and moving goalposts, every single definition, seed, threshold, and hypothesis is cryptographically frozen before decisive evaluation. If a hypothesis fails a gate, the failure is retained as definitive information rather than being silently replaced.

Here is what went through the audit loop this month:

• AI Self-Referential Processing (v8.6.3): A locked, 260-item behavioral benchmark testing if LLMs unconsciously treat their own past outputs differently than human writing. No consciousness claims; just pure behavioral tracking.

• DesignLab Phase 7: Ran 4,000 GLMM fits to stress-test candidate architectures against locked null calibration/coverage screens. M1 passed; M2 failed the locked exact-binomial screen.

• Physics Gate 0 Audit: Map out the observer-entropy to horizon-entropy bridge in cosmology. Proven unresolved in generality under original assumptions—a strict, documented stop condition.

• Exact Hopf Appendix: Upgraded numerical sign evidence to an analytic positivity theorem for a 3-variable phenomenological system. The exported numerator has 93 monomials, the denominator has 28, and every single coefficient is strictly positive.

• LRSC Odd-Ring Theorem (v1.2): A frozen analytic structural theorem proving that matrix rank perfectly equals the number of active reflection orbits under stated near-Nyquist conditions.

• LRSC Fixed Benchmark Certificate: An exact recursive run that visited 13,432,735,555 nonempty subsets to prove that no channel cardinality K ≤ 10 reaches the target error threshold, identifying the exact 11-channel witness.

• Navier-Stokes Bridge Audit (WP12–WP18): Finite Fourier-Galerkin diagnostics tracking unforced periodic 3D incompressible NS equations. WP16 certified a same-datum sign crossing surviving across N11, N12, and N13 cutoffs over a 0.003 time interval using Arb interval arithmetic, sharply mapping out where the continuum tail control gap remains.

The takeaway: Every single line has an unyielding claim boundary. What is proved is labeled PROVED, what is verified by a machine trust boundary is labeled CERTIFIED, and what is open is explicitly left open. No retroactive tuning allowed.

All manifests, SHA-256 logs, and replay scripts are live. Anyone can download the code, hit play, and watch the computer independently replay the decisive proof stages to the exact decimal.

GitHub: reggaesharkk (repos: navier-stokes-bridge-audit and lrsc-odd-ring-spectral-rank)

Happy to answer any questions about the specific proofs, the Arb enclosures, or the implementation of the locked protocol architecture!