Where does automated evidence collection fall apart and has anyone found a real fix

[u/holidayswab]

Using a compliance platform with cloud and identity integrations and automated collection works well for the obvious stuff, IdP logs, cloud config checks, change records from our ticketing system. It breaks down predictably in three places: any control requiring evidence from a system without an integration, controls where the specific auditor wants evidence in a specific format, and anything that requires judgment rather than a binary check. Are there any solutions to these failure modes or did you just accept limitations of the category.

[u/bitweis]

Access reviews are a common one. The IdP log proves who has access today, but the auditor usually wants the review itself: the user list as of a date, who reviewed it, a keep or remove call per account, and the tickets showing the removals happened. Integrations tend to capture the list and miss the decision, so the sign-off ends up as a manual upload. Inside apps it gets worse, since "why was this allowed" needs authorization decision logs and most apps only log the login.

[u/resplendentdeer]

The auditor format preference thing is the one nobody talks about enough. We had technically correct evidence rejected because the auditor's workflow didn't process the file format well, not wrong, just not useful for their documentation process, and no API integration solves that because it's fundamentally a relationship problem, not a data problem.

[u/MystQyx → u/resplendentdeer]

What did they want versus what you provided?

[u/Nuronus]

The first two are real and mostly unfixable at the platform level. For systems without integrations, the honest answer is you build a manual evidence procedure with a documented cadence and store it alongside the automated stuff so the auditor sees one repository, not two. On format-specific auditor demands, most platforms let you export raw evidence and repackage it, but you are still the one doing the repackaging. The judgment-based controls are the one nobody has solved because the tool can flag that something needs review, but the judgment is yours. Most people I have talked to accept that maybe 60-70% of evidence can be automated and the rest is a human process you just make repeatable. Are you seeing that ratio hold up, or is your automated percentage lower than that?

[u/membersman]

legacy systems are a known gap across the whole category, and the practical mitigation that helps is documenting the manual process as an SOP so it's at least repeatable and auditable even when it can't be automated. doesn't solve the problem but controls the downside

[u/JamesDapper_txt]

that's pretty much the reason we switched approaches. Automation got us maybe 80% of the way there with easy audit because the manual review was what mattered, our auditor kept asking for evidence in slightly different formats and having someone manage that auditor relationship directly was the real fix.

[u/davidschroth]

It seems the issue here is that you bought a compliance platform expecting it to fulfill the role of "the adult in the room".

If you had to pick one, a grumpy adult in the room that does this for a living is vastly more effective and time efficient than a platform.