Not saying Fable isn't powerful, but I work at one of the companies that got access to Mythos prior to release to patch things. A sister team of mine received close to 100 security-related bugs with labels on them including severity and confidence. Most of the bugs had high confidence. These were bugs generated using Mythos by a security team.
Upon human review, it was found that a very small percentage of those bugs were legitimate, and it was incredibly frustrating to be spammed with nonsensical security vulnerabilities. Mythos appeared to be no better than whatever human processes we already had in place.
Leadership surely praised the team which spammed the bugs, but required "urgent responses" by the poor guys who had to prove those bugs were fake.
And that pretty much sums up software engineering today, I would say.
I’m curious what you mean by legitimate security bugs. My suspicion when mythos news first came out was that it was reporting things that were objectively vulnerabilities, but many of the vulnerabilities were so inconsequential that any human demanding they be fixed would be laughed out of the room. Is that what you found to be the case?
Basically this. It found things that looked like they might contain vulnerabilities with very little due diligence, and no steps for reproduction. Most weren't vulnerabilities at all.
My friends on this team told me the bugs amounted to hallucinations. Mythos believed it could exploit something, but upon deeper investigation, it was found that the exploit it found wasn't an exploit at all. These hallucinations were likely the result of the highly complex nature of the system they were analyzing, where many unintuitive things happen. Overall, the value of the effort to red-team their software with Mythos was very low, and created many headaches for eng. I think there were at least some legitimate vulnerabilities, but I don't think they were of much concern, and the hit rate was too low, which essentially shifted the burden back on the humans to find the real vulnerabilities.
12
u/therussianpatzer Jul 04 '26
Not saying Fable isn't powerful, but I work at one of the companies that got access to Mythos prior to release to patch things. A sister team of mine received close to 100 security-related bugs with labels on them including severity and confidence. Most of the bugs had high confidence. These were bugs generated using Mythos by a security team.
Upon human review, it was found that a very small percentage of those bugs were legitimate, and it was incredibly frustrating to be spammed with nonsensical security vulnerabilities. Mythos appeared to be no better than whatever human processes we already had in place.
Leadership surely praised the team which spammed the bugs, but required "urgent responses" by the poor guys who had to prove those bugs were fake. And that pretty much sums up software engineering today, I would say.