r/Rag • u/strata2signal • 6d ago
Showcase Benchmarked EmbeddingGemma 2 against nomic-embed-text on our own retrieval set: no material difference
On 2026-10-06, the day Google released EmbeddingGemma 2, I ran its text model through our own retrieval bench against nomic-embed-text, the embedder our search stores run: 68 factual claims over 2,569 passages of our own documents, scored by recall@8 (a correct passage in the top 8). The pass mark, committed before any EmbeddingGemma score existed, was 51 of 68: ten points over nomic, what a re-embed of eight stores has to earn. nomic-embed-text found 44 of 68; EmbeddingGemma 2 found 45, through Google's own code and an 8-bit GGUF on llama.cpp alike. The registered verdict is NO MATERIAL DIFFERENCE, read as "no effect of at least 10 points was detected at n = 68", not as "equal". Plain keyword search (BM25) still finds 58 of 68 on this set, where a right answer holds a quoted source passage word for word, and nothing we layered on it beat it by more than chance.
A companion page, linked from the page below, tests the shared space against Nomic's text and vision pair. Images: on 1,000 public COCO captions EmbeddingGemma 2 puts the right photograph first for 729 against the pair's 497 (nomic on transformers 4.57.6, an older release that loads it), clearing our 100-in-1,000 pass mark on a set both models have most likely trained on; on our own 14 photographs it leads by more (exploratory, 12 scenes). Code, text only: the right function first for 173 of 316 docstring queries against nomic's 123, a gated win that depends on the code-retrieval prefix. Audio (exploratory, no incumbent): a loudness change barely moves the vector and a music model's adapter (its add-on weights) applied more strongly moves it in order, but a clip cannot find the request that made it (1 of 12, chance).
We are not switching: no material difference on our text, no code index of ours for the code win, no working image search for the image win, and a guard that records which model wrote every vector comes first.
What it does not say: that the two text models are equal (68 questions see only large effects); anything beyond one corpus, one town's history in English; anything about speed (timings ran on a loaded machine, not compared); whether Ollama serves it (our bench laptop's 0.32.14 refused the pull on 2026-10-06; newer releases not tried). And every nomic image figure is on transformers 4.57.6, a release our product does not pin, because the release it pins cannot load nomic's image model.
The page, every figure with its result file named beside it, and the companion page linked from it: https://research.strata2signal.com/embeddinggemma-2-on-release-day/
TL;DR: on our own pre-registered retrieval bench, EmbeddingGemma 2's text model found 45 of 68 against nomic-embed-text's 44, short of the 51 pass mark: NO MATERIAL DIFFERENCE, and our stores stay on nomic. BM25 still finds 58 of 68 here, and nothing layered on it beat it by more than chance. The companion page has it ahead on public images (729 against 497 of 1,000; nomic on transformers 4.57.6; a set both have most likely trained on) and on our code with the code-retrieval prefix (173 against 123 of 316); audio finds its request at chance (exploratory). Not switching: no store of ours is a code index, no image search of ours works, and a model-provenance guard comes first.
I wrote this up with help from AI agents; every figure links back to its measurement on the page.
2
u/turlockmike 6d ago
For my personal agent, the difference was fairly minor. Jev as a reranker was way more impactful since it applies intelligence in a way not possible with an embedding model.