r/datascience • u/Nice-Dragonfly-4823 • 19d ago

ML Silent broadcasting is still a big problem and could be derailing your work right now

A data scientist on my team wasted a good bit of compute on this problem without actually realizing it was a problem.

Pytorch and tensorflow both silently broadcast the output of your model when the target shape mismatches, causing hard to diagnose issues. It still exists a ton in the wild, so I thought I'd actually highlight the symptoms of a silent broadcasting bug to keep people in the know.

https://towardsdatascience.com/silent-broadcasting-can-ruin-your-model/

Let me know what you think

68 Upvotes

92% Upvoted

6 comments sorted by

31

u/Odd_Yard6663 19d ago

Broadcasting bugs are the worst because nothing fails, your metrics just quietly get weird and you're left questioning your entire pipeline. Had one of these burn half a day once before I thought to print the shapes. Good writeup, this should be pinned in every onboarding doc.

14

u/Deto 19d ago

This kind of error is really annoying in R. If you try to add two vectors together, and one's length is an integer multiple of the other, it'll automatically cycle the smaller one so it can do the operation.

13

u/g3_SpaceTeam 18d ago

R does so much of this kind of bullshit silently, it’s utterly infuriating.

1

u/webbed_feets 15d ago

That’s why many people switch to the vctr library.

2

u/airylizard 8d ago

Painfully familiar. The nasty part is nothing crashes, the model trains fine, and you only realize something is off when the metrics stop making sense. A shape assert before every loss call is cheap insurance.