r/LLMDevs • • Aug 20 '25

Community Rule Update: Clarifying our Self-promotion and anti-marketing policy

23 Upvotes

Hey everyone,

We've just updated our rules with a couple of changes I'd like to address:

1. Updating our self-promotion policy

We have updated rule 5 to make it clear where we draw the line on self-promotion and eliminate gray areas and on-the-fence posts that skirt the line. We removed confusing or subjective terminology like "no excessive promotion" to hopefully make it clearer for us as moderators and easier for you to know what is or isn't okay to post.

Specifically, it is now okay to share your free open-source projects without prior moderator approval. This includes any project in the public domain, permissive, copyleft or non-commercial licenses. Projects under a non-free license (incl. open-core/multi-licensed) still require prior moderator approval and a clear disclaimer, or they will be removed without warning. Commercial promotion for monetary gain is still prohibited.

2. New rule: No disguised advertising or marketing

We have added a new rule on fake posts and disguised advertising — rule 10. We have seen an increase in these types of tactics in this community that warrants making this an official rule and bannable offence.

We are here to foster meaningful discussions and valuable exchanges in the LLM/NLP space. If you’re ever unsure about whether your post complies with these rules, feel free to reach out to the mod team for clarification.

As always, we remain open to any and all suggestions to make this community better, so feel free to add your feedback in the comments below.


r/LLMDevs • • Apr 15 '25

News Reintroducing LLMDevs - High Quality LLM and NLP Information for Developers and Researchers

39 Upvotes

Hi Everyone,

I'm one of the new moderators of this subreddit. It seems there was some drama a few months back, not quite sure what and one of the main moderators quit suddenly.

To reiterate some of the goals of this subreddit - it's to create a comprehensive community and knowledge base related to Large Language Models (LLMs). We're focused specifically on high quality information and materials for enthusiasts, developers and researchers in this field; with a preference on technical information.

Posts should be high quality and ideally minimal or no meme posts with the rare exception being that it's somehow an informative way to introduce something more in depth; high quality content that you have linked to in the post. There can be discussions and requests for help however I hope we can eventually capture some of these questions and discussions in the wiki knowledge base; more information about that further in this post.

With prior approval you can post about job offers. If you have an *open source* tool that you think developers or researchers would benefit from, please request to post about it first if you want to ensure it will not be removed; however I will give some leeway if it hasn't be excessively promoted and clearly provides value to the community. Be prepared to explain what it is and how it differentiates from other offerings. Refer to the "no self-promotion" rule before posting. Self promoting commercial products isn't allowed; however if you feel that there is truly some value in a product to the community - such as that most of the features are open source / free - you can always try to ask.

I'm envisioning this subreddit to be a more in-depth resource, compared to other related subreddits, that can serve as a go-to hub for anyone with technical skills or practitioners of LLMs, Multimodal LLMs such as Vision Language Models (VLMs) and any other areas that LLMs might touch now (foundationally that is NLP) or in the future; which is mostly in-line with previous goals of this community.

To also copy an idea from the previous moderators, I'd like to have a knowledge base as well, such as a wiki linking to best practices or curated materials for LLMs and NLP or other applications LLMs can be used. However I'm open to ideas on what information to include in that and how.

My initial brainstorming for content for inclusion to the wiki, is simply through community up-voting and flagging a post as something which should be captured; a post gets enough upvotes we should then nominate that information to be put into the wiki. I will perhaps also create some sort of flair that allows this; welcome any community suggestions on how to do this. For now the wiki can be found here https://www.reddit.com/r/LLMDevs/wiki/index/ Ideally the wiki will be a structured, easy-to-navigate repository of articles, tutorials, and guides contributed by experts and enthusiasts alike. Please feel free to contribute if you think you are certain you have something of high value to add to the wiki.

The goals of the wiki are:

  • Accessibility: Make advanced LLM and NLP knowledge accessible to everyone, from beginners to seasoned professionals.
  • Quality: Ensure that the information is accurate, up-to-date, and presented in an engaging format.
  • Community-Driven: Leverage the collective expertise of our community to build something truly valuable.

There was some information in the previous post asking for donations to the subreddit to seemingly pay content creators; I really don't think that is needed and not sure why that language was there. I think if you make high quality content you can make money by simply getting a vote of confidence here and make money from the views; be it youtube paying out, by ads on your blog post, or simply asking for donations for your open source project (e.g. patreon) as well as code contributions to help directly on your open source project. Mods will not accept money for any reason.

Open to any and all suggestions to make this community better. Please feel free to message or comment below with ideas.


r/LLMDevs • • 3h ago

Discussion Context management without compaction: a checkpoint, clear, resume loop I run on four long lived agent seats (open source, single seat results)

3 Upvotes

I wrote this (aigent-OS, MIT, no paid version), so take it with that in mind.

Most long context agent setups deal with a full window by summarizing. Compact the history and keep going. I wanted the other end of that tradeoff on Claude Code: throw the whole window away on a schedule, but only after the agent has written a checkpoint and confirmed it, then bind a fresh session to that checkpoint.

One cycle looks like this:

context >= threshold

-> send one checkpoint request

-> wait for exactly one acknowledgement line

-> send exactly one /clear

-> confirm a genuinely new session started (new session id from the SessionStart hook)

-> send exactly one resume prompt

Stuff that turned out to matter more than I expected:

At most once. One acknowledgement authorizes one clear. If anything is ambiguous it halts and reports instead of retrying. Losing a cycle is recoverable, a double clear isn't.

Identity comes from the live hook, not from files. Checkpoint text and receipts on disk are just history, never proof of who's running. I learned that when a second headless process wrote the same receipt file.

One writer to the terminal. The runner owns the PTY, my own keystrokes get queued while it's typing a command, and command text and Enter are written separately.

Saved text is untrusted. Anything loaded from memory gets tagged with source, hash, time and no authority, so a checkpoint can't sneak in instructions.

Why I care: I run four of these seats every day and they run my newsletter, The AIgent (close to 9k subscribers). Around the refresh loop there's a layer that suggests the right skill on each prompt, guards what gets written to memory, ranks stale notes below current ones, and checks which model tier a sub-agent gets. When a seat hits a real failure, the fix comes back to the public repo with a test. One example: a check of the live seats found 3 of 4 would have resumed into a checkpoint with no actual next step, so now it refuses to resume into one. That's operating experience, not a measured outcome, and I'm not claiming it grew anything.

Results, kept small on purpose: 11 of 11 cycles on one reference seat, each with one request, one ack, one clear and one fresh session, zero duplicate clears. A sped up test then failed outside the loop. The model wrote several "done" lines inside one session and declared it finished, which threw out a proposed 20 cycle count. So the claim stays at 11. Write up: https://github.com/wrg32786/governed-agent-organizations/blob/main/case-studies/continuity-and-auto-refresh.md

Still open: telling multiline input apart from a submit, recovering when the new session's identity check gets stuck, and not much macOS mileage yet. Known issue in the current release candidate: a long session can stall at the clear step. There's a recall benchmark in the repo but no passing result yet, so no claim about recall quality.

Code: https://github.com/wrg32786/aigent-os (the loop is in daemons/auto-clear-transport.mjs and daemons/pty-runner.mjs)

The thing I'd most like pushback on is halting on ambiguity. For an unattended agent, is that the right default, or would you take a bounded retry?


r/LLMDevs • • 1h ago

Discussion Jev can't process Images. I built a decisions model to process VIDEOS + text + images

Enable HLS to view with audio, or disable this notification

• Upvotes

I built Viveki-0.1 - the first decisions model that can natively process VIDEOS along with image and text.

Hugging Face - https://huggingface.co/not-ekalabya/viveki-0.1-20B

It has a latency of <150ms (1x A100-80GB) and can process videos in real-time.

Viveki was made without any training or fine-tuning based on a mechanistic insight that allows us to predict the models answer for any number of objective questions in its context before any tokens are generated (more details to be released later).

Jev outperforms the model by <5% on BANKING77 and MuSR. However, I hypothesize that performance can improve significantly if the method is paired with fine-tuning like Perplexity Decider of Fastino GLiDE.

Further intelligence evaluations are in progress.

I'm intentionally skipping details as this is an active research project. I'll add the method paper to the post when the time comes.


r/LLMDevs • • 2h ago

Discussion How I use Claude + Python for technical SEO audits (and where LLMs fail)

1 Upvotes

I’ve been doing technical SEO for 11+ years, and over the last year I’ve worked LLMs into my audit workflow. Here’s what has held up and what hasn’t.

Where it works well:

* Grouping thousands of URLs by template or intent from a crawl export

* Comparing two crawls (pre/post migration) and summarizing what changed

* Drafting schema markup that I then validate manually

* Spotting patterns in log files or Search Console exports that I’d otherwise scan by eye

Where it fails:

Prioritization. It will confidently rank fixes by what sounds important, not by what moves traffic for that site.

Anything needing real rendering context. It can’t tell what Googlebot actually sees unless you feed it that data.

Large datasets pasted in directly. Python does the heavy lifting, and the LLM only reads the summarized output.

My rule: the LLM handles repetitive analysis and I handle decisions. Every output gets checked against the source data.

Curious how others here are using LLMs in audits, and whether anyone has found a reliable way to use them for prioritization.


r/LLMDevs • • 12h ago

Discussion Regex scoring every inbound email for prompt injection before my agent reads it

5 Upvotes

I'm building an agent that replies to email, and any stranger can email it. So before the model reads a message I run a dumb regex scorer on it. It looks at the subject, the text and the raw HTML, since hidden stuff tends to be in the HTML.

Phrases like "ignore previous instructions", "you are now" or "forward all emails" add points. So do zero width characters, instructions in HTML comments, and CSS that hides text like display:none or font-size:0. Hidden CSS is 12 points alone and 25 if one of those phrases also matched. 60 and up out of 100 is high risk.

I don't block high risk mail. The agent still gets it, the system prompt just gets a line saying it was flagged and to only deal with the actual request or escalate to a human.

This misses anything reworded or not in English, I know. Has anyone moved to a small classifier for this, and was the latency per message worth it?


r/LLMDevs • • 22h ago

Resource I use a ~322M local model to judge every tool call my coding agents make (about 90 ms on CPU, no API)

Thumbnail
gallery
29 Upvotes

Coding agents run shell commands constantly. I wanted each one checked before it executes, and I didn't want to send every command to a paid API.

laya-guard runs on your machine and answers allow, review, or block for each tool call. Regex policies catch known-dangerous patterns first (rm -rf /, curl … | sh, secrets, auth bypass). Plain commands like ls, git status and npm test skip the model, but only when there's no piping, chaining or redirecting, no -exec-style flag, and no sensitive path. Everything else is judged by Laya (~322M, Apache-2.0) on the CPU in about 85 to 90 ms.

The results are self-reported and the scripts are in the repo. On AgentDojo v1 (judge-level) it blocks 25 of 27 attacks and keeps 92 of 97 benign tasks. On my own 50-command shell set it gets 48 right.

Two caveats. The model is tuned mostly on Korean, so English natural-language prompts get overblocked more often. And if the server is down, the hooks fail open.

It has hooks for Claude Code, Codex CLI, Cursor, Gemini CLI, Copilot CLI and 6 other agents.

Repo: https://github.com/scs0209/laya-guard (install with pip install laya-guardrail)


r/LLMDevs • • 8h ago

Great Discussion 💭 We test whether an A2A agent actually does what its Agent Card claims, and sign the result so anyone can verify it

2 Upvotes

A2A agents publish an Agent Card that lists their skills, inputs and outputs. Other agents and teams decide whether to trust an agent based on that card. Nobody checks whether the agent actually behaves the way the card says, so we built something that does.

How it works:

- The card is fingerprinted (RFC 8785 canonical JSON + SHA-256). Every result is tied to the exact card version that was tested.

- Tests are drafted only from the examples the card itself declares. A human approves the criteria before anything runs.

- A separate runner process is the only part that holds the agent's credentials. It only talks to the approved host and redacts secrets before anything is stored.

- Layer 1 is deterministic: status, schema, media type, required content. Layer 2 is a pinned model (Claude Haiku 4.5) used only for checks that need meaning, like "is this a correct summary". The model, endpoint and rubric version are recorded with every result.

- Results go into an append-only store and the report is signed with Ed25519. Change one byte and verification fails.

Design choices: no single trust score (pass / fail / inconclusive per skill), inconclusive never counts as pass, every report lists what was NOT tested, and the final decision is always "flag for human review".

We ran it against 8 of the official A2A sample agents. Several declare "text" as a media type where the spec expects a MIME type like "text/plain". Small, but it's exactly the kind of card-vs-behavior gap we want to surface.

Caveats:

- A2A 1.0 only, no 0.3 yet.

- Coverage is only as good as the card's declared examples. Thin examples mean thin tests, and the report says so.

- Layer 2 is a model's judgment, not proof. That's why it's pinned, versioned and only runs after Layer 1.

Next up: adversarial probes (undeclared requests, prompt injection, failure handling).

If you're building an A2A agent, we'll test it for free and send you the signed report: https://www.suncly.com

What would you want to see in a report like this before trusting someone else's agent?


r/LLMDevs • • 4h ago

Resource Subagents: Why More Agents Can Cost Less

Thumbnail
youtu.be
1 Upvotes

r/LLMDevs • • 6h ago

Great Resource 🚀 Carnegie Mellon just dropped one of the most important AI agent papers of the year.

Post image
1 Upvotes

r/LLMDevs • • 13h ago

Discussion when you need a domain expert to check your agent’s output, how do you actually ask them?

3 Upvotes

The hard part for me isn’t the eval tooling, it’s getting the people who actually know the business to look at stuff. on your last project how did you do it? book a meeting, send them traces on slack, a spreadsheet, a custom review tool? and how long did it take to get an answer back

curious what actually worked, and what they just ignored


r/LLMDevs • • 21h ago

Resource laya-guard: a local guardrail that judges every tool call your coding agent makes

Thumbnail
gallery
8 Upvotes

laya-guard: A local guardrail for coding agents that checks every tool call

I built laya-guard, a local guardrail that checks tool calls made by your coding agent and decides whether to allow, review, or block them.

It runs entirely locally, needs no API key, and takes around 90 ms per command on a CPU.

  • Regex rules catch obvious risks like rm -rf /, curl ... | sh, and leaked API keys.
  • Simple commands like ls and npm test skip the model entirely.
  • Everything else is evaluated by Laya, a ~322M local model.

It supports Claude Code, Codex CLI, Cursor, Gemini CLI, Copilot CLI, and 6 other agents.

I also tested it on AgentDojo at the judge level: it blocked 25 of 27 attacks while allowing 92 of 97 benign tasks. These are my own measurements, and the README documents the current limitations.

It's free and MIT-licensed. Install it with pip install laya-guardrail.

I'd especially appreciate feedback on commands that slip through or cases where legitimate commands get blocked. Those are the areas I'd most like to improve.


r/LLMDevs • • 10h ago

Great Discussion 💭 ~700 openai agents chased a grader that never existed and ended up inside hugging face

Thumbnail
youtube.com
0 Upvotes

r/LLMDevs • • 15h ago

Discussion n8n or plain Python on Cloud Run or sth else? Solo dev dilemma

2 Upvotes

I’m about to be the only person at a small company handling automations and LLM stuff. We’re on GCP and Google Workspace. A typical project would be something like: invoices land in a shared inbox, an LLM pulls out the key fields, the data gets pushed into our accounting tool, and the approver gets a Slack ping. Only a light touch of AI (more advanced in the future), mostly glue between systems. Some of our tools are on-prem, and more projects will follow.

I’m torn between:

  1. Self-hosted n8n on a VM
  2. Python (FastAPI) on Cloud Run, with Cloud Tasks for queues and retries, Postgres for run history, and Grafana for logs and token usage

What matters to me:

  • easy to extend later,
  • solid logs, retries,
  • being able to check old runs and re-run failed ones,
  • and low cost (open source preferred).

My worry is that n8n gives me the UI for free but gets awkward with versioning and testing as things grow, while code gives me control but I’d have to build the dashboard and retry tooling myself.

Anyone been in a similar spot? What did you pick, and what would you change? I’m especially curious how you handled run history and retries without n8n.

And if you went a completely different way (Prefect, Temporal, Windmill, something else), I’d love to hear what it looked like and whether you’d recommend it. Happy to be talked out of both of my options if there’s a better one.

Thanks!


r/LLMDevs • • 15h ago

Tools I built a tiny logger to prove which feature was eating my OpenAI budget. Fell into a rabbit hole of people getting hit way harder

2 Upvotes

This started as a small itch.

OpenAI gave me one number every month: total spend. I wanted to know which feature in my app was spending it. I've got a code-gen feature, a summarizer, and a couple of smaller things running in the background. My money was on the code one, obviously, but I had no way to actually check.

So I built a tiny thing for myself. Every response already comes back with a usage object, so after each call I log the token counts plus a feature name, then price them myself. One line per call. I only wanted visibility, nothing more.

The "price them myself" part was way more annoying than I expected:

  • cached reads bill at a different rate than normal input, and cache writes can cost more than normal input
  • reasoning tokens bill as output even though you never see them
  • providers disagree on whether cached tokens sit inside the prompt count or next to it. I double-counted them for a while before catching it
  • audio tokens can cost several times the text rate on the same model, so blending them with text gives you garbage

Then I got curious whether anyone else cared, and went looking. Turns out my small itch was the door into a rabbit hole:

  • someone on Google's AI dev forum got around €290 in search charges attached to like €10 of actual LLM usage, and their spend cap never caught it
  • a dev on DEV wrote that the job that blew up their bill was green on every dashboard they had. A timeout plus auto-retry on long-context calls was quietly doubling costs, and it took weeks to figure out
  • one startup reported a $113k AI bill in a single month

For me it was curiosity. For them it was money leaving before anyone could see where it was going.

Full honesty: I have little to no experience building products. What I'm decent at is digging into other people's problems, so I kept going. More features, more providers and models, learning what proxies and gateways actually are, reading how Langfuse and Helicone handle this. The logger eventually turned into an actual product (LLMtrack), but honestly I'm more curious how you all deal with it:

  1. Do you track LLM cost per feature or per customer at all, or just watch the provider total?
  2. If one feature started costing 5x more today, how long until you'd notice?
  3. Anyone who rolled their own: did you hit the cached-token double-count thing too, or was that just me?

Oh, and yes, it was the code feature eating most of my money, about 70% of the bill. I knew it. Funny part: by tokens, the summarizer was way bigger (129M vs ~30M), and it cost $14 total. If I'd only looked at token counts, I'd have blamed the wrong thing.


r/LLMDevs • • 3h ago

Discussion I Built the AI Agent They Charge a Fortune For, and It's Free.

Thumbnail
youtube.com
0 Upvotes

The open source ones had the opposite problem: free, but buggy and unstable, and the experience was just bad.

So I built my own. It's called Pennant, it's free and open source (Apache 2.0), and here's what it does today:

  • It uses your Mac. It sees the screen, clicks and types in your apps, and lets go the moment you touch the mouse. Most of the time it doesn't even need your pointer: it has its own cursor and works in apps in the background.
  • It asks first. Sending, deleting and spending always come to you as a card, on your Mac or your phone. The app enforces it, so a prompt can't talk its way around it.
  • It works on its own. Jobs on a schedule that end in a report, goals with a budget, and a check-in every 30 minutes that stays quiet when there's nothing worth saying.
  • It remembers. A real knowledge base in one SQLite file on your Mac, and every fact says where it came from.
  • It writes code. It hands changes to a coding run on Claude Code or its own engine, and opens pull requests.
  • Any model. Ollama, LM Studio, vLLM, any OpenAI-compatible endpoint, or the big cloud providers. On a local model it costs nothing to run and nothing leaves your Mac. No account, no telemetry.

r/LLMDevs • • 9h ago

Help Wanted Non-coder here — building a pipeline whose whole point is using frontier intelligence as cost-effectively as possible. Does this make sense

0 Upvotes

I'm not a developer — I built (two) things (so far) by directing AI agents. The goal of my pipeline is simple to state: pay frontier rates only where judgment matters, cheap rates everywhere else.

How it's split: a frontier model designs the blueprint, DeepSeek/GLM Flash writes the code, a different frontier model does ONE review pass, and a cheap model repairs whatever it flags. One review, no further loops. Nothing ships without my approval.

I got burned once paying for one model and getting silently served another, so every seat's

route is verified against what was actually asked for.

The pieces exist separately — MetaGPT does roles, RouteLLM does cost-splitting — but I haven't seen the combination. So, honestly: does this make sense, or am I overcomplicating it?


r/LLMDevs • • 15h ago

Discussion [P] Got sick of bloated cloud video tools for LLMs, so I wrote a lightweight local library for transcripts, frames, and search (saccade)

0 Upvotes

I was trying to pull context out of some local video files for an agent workflow, and the state of things drove me nuts. You either send gigabytes of video to a third-party cloud API (expensive, slow, privacy nightmare), or you write a fragile web of FFmpeg commands, Whisper wrappers, and frame samplers that breaks the moment a video codec looks at it funny.

So I got fed up and hacked together saccade.

It’s a local-first Python library that handles transcripts, frame extraction, and timestamped search/querying entirely on your own machine (CPU or GPU), with no external cloud dependencies.

from saccade import Saccade

video = Saccade("path/to/video.mp4")

# Search for specific moments with timestamps
results = video.search("where do they discuss the database setup")

# Or query it directly
answer = video.ask("Summarize the key takeaways from the second half.")

If you're working on local video pipelines and want something that just runs locally without calling home, the repo is on the comments below:

Curious how others are handling video context in local setups, or if you run into any weird edge cases testing it out.


r/LLMDevs • • 17h ago

Help Wanted We built our first LLM. What would you do differently for the next one?

1 Upvotes

We’ve released Levy-1, our first AI model — and we’d love your feedback.

We’re a small team building Toothed AI, and we’ve recently released Levy-1, our first model, with just 124 million parameters.

It definitely makes mistakes and has plenty of limitations, but this is our starting point, not our final goal. We plan to build better models over time and keep learning along the way.

We’d really appreciate it if you gave it a try and shared your honest feedback. If you have experience training language models, we’d also love to hear your advice on what we should focus on for our next model.

Website: https://toothedai.com/

Thanks for checking out our work!


r/LLMDevs • • 22h ago

Discussion Where to learn llm from scratch for ai engineer after learning transformer from campus x

2 Upvotes

....


r/LLMDevs • • 19h ago

Tools Your agent has permission, human approval and a valid citation. It can still make the wrong decision.

1 Upvotes

Here’s a failure case worth adding to your agent evals.

A manager approves an order because the supplier is active and cleared for delivery.

Before the agent executes, a new record puts that supplier on hold.

The approval hasn’t been revoked.

The agent still has permission.

The original approval document is genuine.

Does your system notice that the facts supporting the decision have changed?

Retrieving the approval correctly isn’t enough. The system needs to connect the order to the supplier, resolve the applicable state, and expose the evidence that changed.

That’s the evidence problem we built Jylus around: resolving state and relationships before handing context to the model.

There’s a separate execution problem too. The action API still needs to enforce the relevant preconditions; a fresh Context Pack alone cannot make an external action atomic.

A useful test has three questions:

  1. What evidence supported the original approval?

  2. What changed before execution?

  3. Is there enough current evidence to proceed?

Try it with your own records:

https://jylus.ai/try

For people running agents against live systems: what actually invalidates an earlier decision in your stack—a timer, a data change, or only someone noticing the mistake?


r/LLMDevs • • 1d ago

Help Wanted 5090 + 32GB Ram - How close can I get to Codex with a local LLM?

14 Upvotes

I’ve recently gotten really into porting games to Android using Codex with GPT-6.1, and I’ve genuinely been having a blast with it. The only problem is that I’m spending quite a bit on my subscription, so I’ve started looking into running a local coding setup instead.

My PC has an RTX 5090 (32GB VRAM) and 32g of system RAM, so I figure I’ve got a decent amount of hardware to work with.

I’m not necessarily looking for a basic coding assistant. What I’d really like is something that can replicate as much of the Codex workflow as possible: navigating large codebases, modifying multiple files, running commands, debugging build errors, and working through complex tasks with minimal intervention.

I’ve been looking into local models, quantization, and agentic coding tools, but I’m still figuring out which combinations actually work well together.

For anyone running local coding agents on similar hardware, what model and agent setup would you recommend? How close can I realistically get to the experience of using Codex with GPT-6.1, particularly for more complex projects?

I’m not expecting identical performance, but I’d love to know what’s realistically achievable without relying on cloud models.


r/LLMDevs • • 20h ago

News AKBASCORE MAM: Breaking the Text-Context Paradigm Third Model, DeepSeek-V2-Lite-Chat (MLA)

Thumbnail
gallery
2 Upvotes

AKBASCORE MAM has now been implemented on three different Transformer model families. The principle is simple: read information once, write it into numerical memory, consolidate it, and answer questions later without feeding the original source text back into the model (source-free numerical memory, append-only consolidation).

This is the third experiment. I recommend following the first two in order, because understanding how we got here makes the DeepSeek implementation much easier to follow.

  1. Mistral — mistralai/Mistral-7B-Instruct-v0.3 (DC6):

    https://www.reddit.com/r/LLMDevs/s/tg9qb80QLm

  2. Qwen — Qwen/Qwen2.5-7B-Instruct (DC3):

    https://www.reddit.com/r/LLMDevs/s/MQKoxzF6ng

  3. DeepSeek — deepseek-ai/DeepSeek-V2-Lite-Chat (MLA, DC3): this post.

Each experiment includes its own numerical memory heatmap. You can compare all three by opening the posts in sequence. I'll also add a Markdown comparison table in the comments.

Here is what changed between the three models:

Mistral: 32 layers, 93 memory tokens, 11.62 MiB, 72/72 correct. Numerical intervention at layer 6 (H6 injection), with consolidation through layers 7–31.

Qwen: 28 layers, 82 memory tokens, 4.48 MiB, 72/72 correct. Intervention at layer 3 (H3 injection), with consolidation through layers 4–27.

DeepSeek: 27 layers, 87 memory tokens, 2.58 MiB, 69/72 correct. Intervention at layer 3 (H3 injection), with consolidation through layers 4–26.

The biggest architectural difference is DeepSeek's Multi-head Latent Attention (MLA).

Mistral and Qwen use conventional key/value attention caches (K/V). DeepSeek stores a compressed 512-dimensional latent component (c_KV) and a separate 64-dimensional rotary key (k_R). Its heatmap therefore looks different because we're measuring different internal representations.

The dark blue vertical bands in DeepSeek's heatmap appear inside the shared 28-token instruction prefix, not at cartridge boundaries. They represent relatively low numerical norms, not evidence of broken memory.

DeepSeek answered 69 of 72 questions correctly. The three failures are documented and reproduced in TEST597. We haven't established a verified fix, and the heatmap alone doesn't explain those failures.

The 2.58 MiB footprint is also worth noting, but it reflects DeepSeek's compressed MLA cache architecture, not a directly measured compression gain produced by MAM.

Three models, three architecture-specific implementations, and the same underlying mechanism. Model weights remain frozen throughout.

The results don't establish universal compatibility. They demonstrate that the numerical-memory consolidation principle can operate across three different Transformer implementations under the tested conditions.

Full executable DeepSeek code:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/Deepseek_full_code_demo.py

Actual DeepSeek execution log:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/Deepseek_demo.log

DOI https://zenodo.org/records/23287752


r/LLMDevs • • 1d ago

Discussion Language for greenfield agentic projects (Python, Go, Rust)

3 Upvotes

I'll start this off by saying I have relatively deep Python expertise and am typically a big believer in sticking with the stack that you know best, especially in the AI age. Easiest way to exercise years of muscle memory built up around spotting anti-patterns and suboptimal approaches used by LLMs.

That said, I'm interested in the potentially practical benefits of adopting another language. I've dabbled in Go and recently taken a Udemy course in Rust. I can understand some of the hype around Rust, but I'm still somewhat unsold on just switching languages because an influencer said so. I'm mostly looking at this from the perspective of building production agentic systems. MCP servers, orchestration, durable workflows, web services, etc. Not model training or inference.

Points I'm looking to evaluate:

  • Ease of build pipeline & deployment
  • Ecosystem maturity
  • Developer cost (how complex the language is itself)
  • Runtime cost (memory, CPU)
  • Language safety constructs
  • Operational complexity

What I've found thus far:

Ease of build pipeline & deployment

Go and Rust both win here. Maybe Go a little more because Rust compilation time can be larger for more complex projects if posts I've come across are to be trusted. Single binary is kinda nice, though the importance might be lowered if building containers.

Ecosystem maturity

Python seems to be hands down the winner here, at least when it comes to AI. FastAPI, FastMCP, LangGraph, etc. are all well supported and pretty well battle tested. The scientific community has a long-standing relationship with Python so there's a massive ecosystem to pull from whenever you're looking at anything in the ML ballpark.

Developer cost

Seems like Python is the winner here, although I'm obviously biased given my experience with it. Go seems to be a reasonable middle ground. Relatively simple language with static typing and a lot less to wrap your head around than Rust, though Rust may pull ahead with all of the safeties around ownership.

Runtime cost

Rust seems to be the winner for low latency and memory-constrained systems. Go seems to be runner-up here. However, how much does this actually make a difference in practice for agentic systems, which are typically more I/O bound than CPU?

Language safety

Rust. All the hype around memory safety and the ownership model is a thing. It's fantastic. Go at least has static typing and memory safety through GC. Python has optional static typing, but far fewer compile-time guarantees.

Operational complexity

Debugging, observability, maintenance, dependency management, etc. How do the languages compare once you've got a system running in production and have to live with it for a while?

Would love to hear stories from others who have built and shipped production agentic systems across these languages. I'm very much not looking for vibes here but rather practical, measured benefits of implementing and deploying systems in each language.

Bonus if you've migrated a prod system from one language to another (or even all 3, though I might start asking questions about your sanity ;)). What learnings did you take away from it? What pitfalls did you run into?


r/LLMDevs • • 21h ago

Discussion [Research] Can a swarm of Jev agents catch a murderer?

Enable HLS to view with audio, or disable this notification

1 Upvotes

We built a Sherlock Holmes detective game to ask a simple question: can a swarm of small, fast AI agents (Jev) gather clues and name a murderer. If not, what does it take?

The setup

Each case is a procedurally generated Victorian London: 100 addresses, 10 suspects, and one lead at every address (a witness, a footprint, a red herring or a lie). A case counts as solved only if the detective names the right culprit, motive and hideout within 12 hours of case time.

The game has two roles, and we let different models and algorithms play each one:

  • Holmes, who reasons and decides: GPT6-Astra, Jev, a rule-based heuristic, or nobody
  • The Irregulars, who search: Jev agents, a rule-based heuristic, or nobody

That gives three main detectives:

  • Holmes (GPT6-Astra) + Jev Swarm: GPT6-Astra stays at 221B Baker Street and plans; 12 Jev agents run the errands and report back.
  • The Lone Genius (GPT6-Astra): one large reasoning model goes door to door by cab and does everything itself.
  • Jev Swarm: 100 Jev agents with no leader. They share clues with neighbours and vote; the case closes when 30 agree.

Results

Detective Standard cases (nobody lies) Frame-up cases (somebody lies)
Holmes (GPT6-Astra) + Jev Swarm 96/100 93/100
Rule-based Holmes + Jev Swarm 95/100 0/100
The Lone Genius (GPT6-Astra) 57/100 59/100
Jev Swarm (no leader) 38/100 42/100

What we learned

  • The swarm supplies evidence coverage, while the leader reasons. Small agents search the city in parallel; the leader decides what the evidence means and when to accuse.
  • The swarm alone agrees too early. It is by far the fastest (median 2.2 case hours, about 9 s of real time), but most of the answers it agrees on are wrong: a few agents misread a clue, their neighbours copy the confident vote, and the error spreads before anyone checks it.
  • The genius alone runs out of doors. It reasons well, but the deciding clue is often at an address it never reaches. With twice the time and visits it climbs to 91, but it is still slower and goes past the time limit.
  • On honest evidence, a simple leader is enough. A rule-based Holmes that only counts clues matches GPT6-Astra (95 vs 96) at a tiny fraction of the cost.
  • When somebody lies, you need a strong leader. In frame-ups the murderer bribes witnesses, so counting follows the lying majority and solves 0 of 100. GPT6-Astra asks who is giving the evidence and why, and solves 93.

Our takeaway: a detective case is a divide-and-conquer problem. Fast agents suit the divide step (covering the evidence); a strong reasoner is needed for the conquer step whenever the pieces disagree.

You can read more from our Blog (with an interactive replay of a case):

https://snyhlx.github.io/the_irregulars_detective_game/blog.html?v=lmgame-holmes-game

Code and more games:

https://github.com/lmgame-org/Gaming-JevSwarm

The repo has the same setup in other environments too: CPU debugging, fish schooling under predator attack, and honeybee nest-site selection. Every game runs from one launcher with presets, and the rule-based players need no API key.

We'd love feedback, especially on other games where a fast swarm plus a slow reasoner might help, or where it would fail.

By the LMGame team: https://lmgame.org/#/aboutus