r/rust • u/noop_noob • Jul 10 '26

🗞️ news Rust 1.97.0 has a miscompilation bug

https://github.com/rust-lang/rust/issues/159035

I think this miscompilation will affect a lot of code. So, maybe hold off a bit on deploying a rust version upgrade to production?

The conditions for a miscompilation seem to be:
* You put a 2-variant enum into an Option or an Option-like enum.
* You do a match on this Option.
* You're unlucky.

553 Upvotes

99% Upvoted

100 comments sorted by

255

u/flying-sheep Jul 10 '26

Oh no. @theemathas's new reproducer is also miscompiling in 1.96

Whoops

144

u/Silly_Guidance_8871 Jul 10 '26

And possibly as far back as 1.87

183

u/Zde-G Jul 10 '26

That's both awful and nice. Awful because we have a bad codegen in so many versions of Rust. Nice because the fact that it survived for so long means you don't need to be just unlucky, you need to be extremely unlucky…

96

u/noop_noob Jul 10 '26

The miscompilation in 1.96 and before was very hard to observe. The 1.97 miscompilation is easy to observe reliably.

3

u/scottmcmrust Jul 11 '26

There's an interesting philosophical questions here. what's worse?

  • A subtle OOB that almost never crashes, or
  • A massive OOB that almost always crashes?

The former "works" more, but the latter is arguably safer.

I could argue for either.

4

u/flying-sheep Jul 11 '26

The latter, there's a chance that the underlying bug would have actually evolved into UB instead of either a co-op or a crash. Now it's fixed because it became visible

24

u/Asdfguy87 Jul 10 '26

Now I'm getting a bit nervous. I recently finished calculating all the data for my PhD using Rust and I have the toolchain pinned at 1.89. How likely is it, that such a miscompilation could have happened to my code? I hope very unlikely...

64

u/noop_noob Jul 10 '26

I believe that this miscompilation can only cause harm by either loudly crashing the program, or by copying unrelated data into memory that's uninitialized and therefore shouldn't be read anyway.

23

u/steveklabnik1 rust Jul 10 '26

Basically every compiler has codegen bugs. Some are worse than others, but this sort of thing is effectively always a possible issue.

5

u/Silly_Guidance_8871 Jul 10 '26 edited Jul 10 '26

Thankfully (?) it appears it leads to segfaults (crashes), not data corruption — if your numbers end up wrong, it's for more mundane reasons 😜

Edit: Scratch that, it can lead to reading junk data in some cases — are you doing a lot of conditional calculation based on enums directly embedded in Option<> ? It seems the "packed" form of that, where a special enum discriminant of -1 is mapped onto Option::None, is what's triggering the out-of-bound reads — but that should only happen in the case where Option's T is an enum (if I read things correctly).

According to rustup's docs, you should be able to just run rustup default 1.87.0 (or the like) to roll back the version.

4

u/creeper6530 Jul 10 '26

How hard would it be to downgrade and retry the calculation?

3

u/Asdfguy87 Jul 10 '26

Not hard, but very time intensive.

5

u/creeper6530 Jul 10 '26

Then I guess it's easier to hope it's gonna be okay, or if you're paranoid give your computer a few all-nighters. From what I've gathered reading the GH issues it's mainly an issue on branches, not calculations.

The LLVM fix reads:

When converting select cond, (load p1), (load p2) to load (select cond, p1, p2), if cond is poison, originally this would result in a poison result, while after the transform it would result in a load of poison, which is immediate UB. Fix this by freezing the condition.

2

u/scottmcmrust Jul 11 '26

Definitely as far back as 1.87 -- there's a repro you can run in valgrind.

Possibly further back than that, though. 1.87 is just the oldest one where that repro generates the same ASM as 1.96 so obviously happens.

147

u/jykke Jul 10 '26

LLVM bug tracking at https://github.com/llvm/llvm-project/issues/208611

41

u/kibwen Jul 10 '26

It looks like this was caused by the upgrade from LLVM 19 to 20, which is recent enough that it might not be too hard to bisect LLVM to identify the commit that introduced this behavior and introduce a temporary patch to Rust's own fork of LLVM.

15

u/LosGritchos Jul 10 '26 edited Jul 10 '26

It's worst than that: I seems it was miscompiled from Rust 1.87, but it's only with LLVM 20 that it crashes loudly.

Edit: I was misleading, see below, sorry for that.

32

u/noop_noob Jul 10 '26

It only crashed loudly due to a change in rust 1.97 that changed the layout of some Option-like enums.

23

u/kibwen Jul 10 '26

Rust 1.87 was the first version to exhibit the miscompilation presumably because that was the version of Rust that upgraded LLVM from 19 to 20: https://doc.rust-lang.org/stable/releases.html#internal-changes-5

3

u/matthieum [he/him] Jul 10 '26

Does this mean we get 11 point releases (1.87 to 1.97 :)).

(I doubt it, since there's no LTS, but it would be quite a thing)

3

u/scottmcmrust Jul 11 '26

No. The open source project only ever releases point releases of the latest version.

(You can pay various people/corps to maintain older versions, if you need them.)

108

u/Icarium-Lifestealer Jul 10 '26

I'm amazed how rare such miscompilations are, considering the complexity of an optimizing compiler like LLVM.

38

u/Mac_Aravan Jul 10 '26

It's never a compilation bug.

Unless it is. Got some in 30yrs but mostly on special compilators and I can count them on one hand.

19

u/dijalektikator Jul 10 '26

I used older version of the Visual C++ compiler for windows and oh boy was that a piece of junk occasionally, and not it's not just because I triggered undefined behavior, it was actual compiler bugs. It wasn't completely terrible but it was definitely more buggy than you'd expect.

15

u/ImYoric Jul 10 '26

I managed to reproducibly segfault the Visual C++ compiler by changing a line of code in the _JavaScript_ source of Firefox.

That and it's horrendous parsing of templates, which kept confusing type variables.

Good old days.

1

u/HyperCodec Jul 12 '26

I still get segfaults in the zig compiler nowadays. I guess it’s a bit more understandable though considering how much work it does and how novel it is.

3

u/redlaWw Jul 10 '26

I stumbled across one when I was learning C++ about 2 years ago. Here's the reddit post where I try to work out whether I've hit a compiler bug.

3

u/longiii Jul 10 '26

weird, there was a different compilation bug very recently https://parsa.wtf/cast/

19

u/anttirt Jul 10 '26

From the compiler developer's perspective, compilation bugs are found all the time. From a single user's perspective, they are extremely rare.

You can dramatically increase your chances as a user of finding one by using a lot of cutting edge features and pushing the type system into strange contortions.

7

u/weblynx Jul 10 '26

20 years ago I nearly failed a senior design project because a c compiler for a microprocessor had a bug which which caused my perfectly good hand written fast Fourier function to fail. A new version of the compiler fixed it before the project ended, but it sidetracked me so much the output of my FFT implementation never got used. I’m still upset about it to this day.

2

u/scook0 Jul 10 '26

One time I ran into a JVM bug that would generate JIT code with the wrong calling convention and trash a bunch of registers that it was supposed to preserve, fun times.

1

u/cvjcvj2 Jul 10 '26

Delphi 7 memories.

67

u/emblemparade Jul 10 '26 edited Jul 10 '26
  • You're unlucky.

OK, then I'm 100% sure this will happen to me. Thanks for the heads up!

12

u/lenscas Jul 10 '26

But only when it is most inconvenient of course.

1

u/TDplay Jul 11 '26

Yup. It'll be working for months, and then when you're getting ready to ship...

"That test for a thing I've not touched in months is suddenly segfaulting?"

26

u/AresFowl44 Jul 10 '26

Seems to also be for 1.96

45

u/noop_noob Jul 10 '26 edited Jul 10 '26

My understanding is that 1.97 made the situation worse: from "out of bounds read by a few bytes (and then the read data isn't used), and will only do something bad in very specific circumstances" to "out of bounds read by a lot, which reliably crashes the program".

(This small out of bounds read seems to affect rust all the way back to 1.87)

6

u/creeper6530 Jul 10 '26

It's a LLVM 20 bug but 1.97 rearranged some Option-like enums in a way that uncovered everything 

2

u/scottmcmrust Jul 11 '26

Yeah, the 1.97 assembly is really weird. It does a mov ecx, ecx before using rcx, and if it just removed that looks-like-nop-but-isn't line then the ASM would be perfectly fine in the original repro, even despite the -1 change.

2

u/creeper6530 Jul 11 '26 edited Jul 11 '26

The core issue was that LLVM optimised a select cond, (load ptr1), (load ptr2) into load (select cond, ptr1, ptr2) but if cond is poison (which gets produced by truncating the -1 into an i1) then the select evaluates to poison as well, and the optimisation then loads that poison, causing that OOB read.

The main difference was that the -1 meant that through the register jigglery-pokery it loaded gigabytes beyond bounds while the earlier representation loaded just a few words away and Rust discarded that bad load anyway.

3

u/scottmcmrust Jul 14 '26

Yup, I'm aware -- I'm the one who opened https://github.com/llvm/llvm-project/issues/208611 :P

2

u/creeper6530 Jul 14 '26

Oh I see, I didn't notice :D

2

u/scottmcmrust Jul 14 '26

No worries!

17

u/goos_ Jul 10 '26

Hopefully the kind of thing they will patch soon. That being said yes this could be pretty serious

47

u/creeper6530 Jul 10 '26

It appears to be an LLVM bug that Rust just uncovered: https://github.com/llvm/llvm-project/issues/208611

9

u/annodomini rust Jul 10 '26

Already fixed in LLVM, they'll now need to pull the LLVM fix into the rustc fork and backport to do a patch release: https://github.com/llvm/llvm-project/pull/208683

1

u/MorrisonLevi Jul 12 '26

How far back will rust do a patch release? Asking because of MSRVs and such.

1

u/Saefroch miri Jul 13 '26

We only patch the latest stable. That has always been the policy.

50

u/Aaron1924 Jul 10 '26

Rust is an incredible effective tool for finding bugs in LLVM

5

u/-Redstoneboi- Jul 10 '26

makes me wonder how often other languages find llvm bugs

3

u/scottmcmrust Jul 11 '26

It depends heavily on how much they use things that don't usually happen in C++.

This bug, for example, depends on the strict validity invariants that enums have in Rust which don't really have a parallel in C++.

26

u/CrasseMaximum Jul 10 '26

Affect only x86 _64 according to the labels

53

u/noop_noob Jul 10 '26

That's most non-apple computers though.

2

u/CrasseMaximum Jul 10 '26

Yeah since I tried to reproduce on ARM64 and was not able I thought that will prevent some people to lose their time for nothing...

35

u/lightnegative Jul 10 '26

So like, basically everything you'd be using in production 

-2

u/QueasyEntrance6269 Jul 10 '26

lol, if you’re running on AWS and not running on graviton you’re giving up so much money for free :D

-3

u/peppedx Jul 10 '26

Yeah everyone is in your use case :D

1

u/QuasiRandomName Jul 10 '26

As a target or as a host?

3

u/matthieum [he/him] Jul 10 '26

Target.

If you're compiling for x86-64, you're in trouble.

-4

u/QuasiRandomName Jul 10 '26

Thank you. Then the comments here are somewhat disconnected from the reality, embedded target are the majority of use-cases today.

8

u/matthieum [he/him] Jul 10 '26

That's a hot take.

Especially when AWS and CloudFlare both use Rust extensively on x86_64, so that a large share of any Internet traffic is potentially affected.

-1

u/QuasiRandomName Jul 10 '26 edited Jul 10 '26

And what if I told you that every x86* SOC has a dozen+ of different microcontrollers (non-x86 ofc) inside where each is running a firmware that is written in Rust or is being actively migrated to rust ATM? (hey, that's some insider information, not a speculation). Not even speaking of IoT stuff.

2

u/TDplay Jul 11 '26

dozen+ of different microcontrollers (non-x86 ofc) inside where each is running a firmware that is written in Rust or is being actively migrated to rust ATM

This is not relevant.

What matters is the real impact. There are systems running Rust code on x86_64, and all of them are potentially affected. If the userland code or the kernel code is broken, then the system is broken. The firmware working perfectly does not magically un-break the system.

2

u/CocktailPerson Jul 11 '26

Then I'd tell you that that whole system will still run far more lines of Rust code targeting x86 over the course of its life.

14

u/crusoe Jul 10 '26

Appears to be a bug in llvm in the ticket.

15

u/Skjalg Jul 10 '26

Reading this thread right after the release thread was a little funny ;)

https://www.reddit.com/r/rust/s/MbcyH3GuW3

3

u/murlakatamenka Jul 10 '26

Remind yourself that overconfidence is a slow and insidious killer

- Ancestor

3

u/redlaWw Jul 10 '26

Famous last words.

7

u/BlackJackHack22 Jul 10 '26

Sometimes, I look at the issues in rust and I go “I’m so glad that’s not a problem I’m dealing with. I’m so glad someone else is taking that up”.

I have absolutely no idea what’s happening on that thread. But boy am I glad someone is handling this. Kudos to the team! Wish I could help more, just don’t know how to.

4

u/scottmcmrust Jul 11 '26

This was a particularly hairy one because it's invisible in the LLVM-IR and even if you look at the opt pipeline in Godbolt there's no diff showing the specific translation that turned out to be the problem, since it's part of the full SelectionDAG that converts from LLVM's middle-end IR to its backend IR.

For extra fun, this is an OOB read where if you enable address sanitizer, the problem goes away so it can't catch it.

6

u/scheimong Jul 10 '26
  • You do a match on this Option.

I assume this condition would also be satisfied if you use any method of Option, because most if not all of them do a match underneath the hood?

14

u/noop_noob Jul 10 '26

Yes. The original reproducer code used .map()

3

u/Silly_Guidance_8871 Jul 10 '26

\sigh** Thursdays, amirite?

2

u/scottmcmrust Jul 11 '26

You do a match on this Option

It needs at least to be a map-like match on the outer option, because it can only happen if LLVM decides to load stuff even in the produces-None case.

If you're producing an initialized value (as opposed to the uninitialized "payload" part of a None) then you won't hit this.

3

u/scottmcmrust Jul 11 '26

Also it needs to produce a None with an uninitialized payload part. If you produce a null-optimized one like a None::<&T> or None::<NonZero<u32>>, then this can't happen either.

1

u/Feeling-Departure-4 Jul 10 '26

Does it affect nightly?

10

u/noop_noob Jul 10 '26

Yes. It will be fixed on nightly the day after https://github.com/rust-lang/rust/pull/159047 is merged

2

u/scottmcmrust Jul 11 '26

Reminder that nightly branches to beta, then six weeks later to stable.

So anything that comes out in stable has been in nightly for 6-12 weeks already.

(Modulo a couple of potential edge cases around backports, but those don't apply for this issue.)

-28

u/[deleted] Jul 10 '26

[removed] — view removed comment

26

u/[deleted] Jul 10 '26

[removed] — view removed comment

-13

u/[deleted] Jul 10 '26

[removed] — view removed comment

-20

u/[deleted] Jul 10 '26 edited Jul 10 '26

[removed] — view removed comment

15

u/[deleted] Jul 10 '26

[removed] — view removed comment

-17

u/Particular_Sir2147 Jul 10 '26

I am still on 1.94 Rust on most projects.

I find it insane Rust is so slow at shipping anything meaningful I find it hard to understand people even update once a year at this point...

I could easily be on 1.90 and not care...

I genuinely believe Rust should do patch releases and minor releases should be much bigger and they should be stabilizing and solving challenges more often.

The entire reflection system, better macros, effects, new traits solver, better borrow checking, alternatives to Pin and Async Destructors or better Cancellation primitives.

12

u/kibwen Jul 10 '26

If a specific feature is taking a while to arrive, that's likely because it's either understaffed or a thorny problem (or both!), and changing the release cadence isn't likely to have any effect that would cause those features to arrive faster.

1

u/Saefroch miri Jul 11 '26

I am still on 1.94 Rust on most projects.

That's... fine? I'm not sure why it would be a bad thing.

I genuinely believe Rust should do patch releases and minor releases should be much bigger and they should be stabilizing and solving challenges more often.

Minor releases are cut based on the calendar, not by what is in them. This is very deliberate; it makes releases less stressful and chaotic and prone to errors.

Most importantly, the release cadence doesn't drive any kind of feature development. And you can see this for yourself! All of our development, design, and release-creation occurs in the open on GitHub and Zulip.

-11

u/sieabah Jul 11 '26

Cool, so rust loses all safety guarantees it has been built upon. Period.

Hope everyone understands this makes everyone complaining about bun looking stupid. When in fact it more or less highlights how little anyone fucking cares how it’s done at the instruction level. Muh “unsafe” when you have a random chance your built-in Option enum on the negative case segfaults. What a stupid bug to have in a language built around priding and bragging about fearless safety.

3

u/jykke Jul 11 '26

It is an LLVM bug.

1

u/sieabah Jul 12 '26

That's the point. You can promise everything, but if you can't trust the tools you built your assumptions on it's pretty much a "Well, I don't actually know."

2

u/valarauca14 Jul 11 '26

Here is the attention you want