error vs malice

Safety people refuse to consider malicious actions, security people refuse to consider mistakes.

This is a paraphrase of a comment that came up in discussion following the REdeploy resilience conference last year (I could not find the original this morning for attribution).

Both of these worldviews have something important in common - the context here is people working on process or policies which need to catch, recover, or prevent bad things from happening. The systems designed from these different perspectives will be quite different in two important ways

• The systems will have different levels of effectiveness at stopping different types of threats.
• They will feel very different for the people working in the system. When there is no trust of good intent, it bleeds through at every interaction and rapidly drains the goodwill of the people working within the system. What is lost is any initiative of the people right in the thick of things to improve, adapt, to help correct the system when it’s not working perfectly.

For example, consider a light barricade placed in front of a section of sidewalk that has been removed, and compare that with a steel barrier around an opening in a military facility wall that is being repaired.

When we are working on software and business processes, can we take the best from both perspectives, and design systems that encourage participation while also having some backup checks and failsafe measures to detect and limit malfeasance?

One of my favorite examples of this approach is with providing admin access in AWS accounts. I often advocate giving the operators full admin access to the account so that they have full power to fix unanticipated problems, while also having tamperproof audit logs and backups stored safely out of reach of someone acting maliciously.

I’d love to know if you have good examples of these different perspectives in system design. Hit reply and let me know!