Innovation is about taking risks, but we waste a lot of time taking unnecessary risks with known failure modes and then acting surprised when the inevitable happens.
Failures that were not only predictable, but predicted, happen all the time. They are enacted and then not removed because those who built them are unwilling or unable to admit they made a mistake. The sunk cost fallacy rules! We tinker at the edges, trying in vain to fix the unfixable and repair what is broken by design, rather than making the bold decision to turn it all off.
What if the most innovative thing to do was to stop?
When new systems are developed and deployed, we are conducting an experiment on human beings in real time. We should, at the very least, consider what might go wrong and then set some basic expectations of what too much failure looks like. And then we should write it down, in advance.
Currently, those without systemic power are expected to simply put up with poorly designed systems that hurt them. The hurt may not be intentional, but sufficiently advanced incompetence is indistinguishable from malice, especially when you are on the receiving end.
It is particularly galling when the harm was not only predictable, but predicted, and when you are powerless to stop the machine that continues to hurt you. If those in charge of deploying the system fail to act when the failure conditions are met, there must be some other group with sufficient power to compel them to act.
When designing new policy, we should explicitly think about, discuss, and come to some form of agreement about how much harm a new system will be allowed to cause before it is turned off. This would require policymakers to confront difficult questions about risk before new systems are put into operation.
Failure is inevitable
“If you find yourself at the bottom of a hole: first, stop digging.” – Anonymous
IT projects fail. A lot. To assume that success will be the inevitable outcome is, at best, hopelessly naïve.
For any system of consequence, and certainly for any system that is destined to affect a great many human beings, the assumption that at least a few things will go wrong should be the baseline. We should not accept any less from those proposing new systems.
Given that failure is expected, we must act defensively, and that should start with the simplest defensive measure of all: ensuring the system can be turned off.
Any new IT system should have a clearly established way to simply turn it off if things go badly wrong. Instead of continuing to cause damage, turning the machine off and stopping the damage should be a clear and readily available option. Ideally, it will never be needed, but the discipline of ensuring that there is an off switch somewhere means we can at least establish a way to stop making things worse.
If the functions of the machine are so vital that we can no longer do without them, there should be an alternative, a workaround, another way of achieving the same result. The alternative may be more expensive or time-consuming, but it should be a known quantity, something that is at least safer than the out-of-control machine that was turned off.
Without an alternative, we have deliberately removed a ‘less good’ option and can now only choose between ‘bad’ and ‘worse’. This is simply capitulation, and we should expect more of ourselves and our leaders.
Risk imbalance
When new ideas are deployed by powerful groups, there is often an imbalance in who suffers the consequences of failure. Ideally, incentives should be aligned: those who take the risks receive the rewards, but they also suffer the consequences. This is accepted as a principle of good governance, and many senior executive compensation plans are based upon this principle.
Yet in many cases, the risks are taken by those with power to arrange matters so they reap the rewards of success but outsource the consequences of failure onto others. Both governments and large corporations behave in this way, as we have seen far too often in recent times:
- Robodebt actively hurt hundreds of thousands of Australians, yet no one in government was fired for designing and operating an unlawful system for years on end
- Multiple law enforcement agencies routinely violated the law and the privacy of Australians when they unlawfully accessed people’s metadata, but the powers remain in place
- Adtech platforms seem unable or unwilling to address the breathtaking levels of fraud
Our response has been to permit these systems to continue operating while those in charge tinker with the settings. The harm is not substantial enough, apparently, for anything much to be done about the problem.
What if, instead, we enabled those on the receiving end of harm to win when others gamble with their rights and lose? What if we rebalanced the risk equation by asking those designing a system to put their money where their risk plans are?
The power to harm must be balanced by the power to resist.
The harm will continue until compliance improves
Such a design would create an in-built governor, a brake, on systems that run out of control. If the rewards of failure accrue to those in charge of the system, what incentive is there to change it when failures occur? But imagine, instead, if the hundreds of thousands of Robodebt victims automatically received compensation for each unlawful act by the government as part of the system design? Would it have been allowed to run for so many years, or would it have been stopped once the compensation bill passed A$50 million or so?
Explicit compensation for harm should be defined upfront, requiring policymakers and system designers to quantify the risk they are willing to take. This would create an inbuilt economic incentive to reduce harm, and a counterbalancing system of power to resist the harmful system by those harmed by it. This contrasts with the current attitude that mistakes are inevitable and of essentially no consequence, which sets up perverse incentives to take untethered risk with other people’s data, money, and rights.
Some systems may be deemed too expensive to run. If the cost of a major data breach outweighs the benefits of assembling a vast database of sensitive information, then the risk of a data breach is reduced by simply not building the database in the first place. What’s not to like?
Right now, for many new systems, nebulous future benefits are assessed as having near infinite value that is deemed to outweigh any and all risk. Actual harm caused to individuals is relatively (and often actually) cost-free to the powers that operate these systems. This prevents any sane cost/benefit analysis from being undertaken, resulting in policy that is made on ideological grounds rather than being grounded in evidence and real-world effects.
And sometimes a system cannot be made safe and should simply be destroyed.
How much harm is too much?
It seems radical today to suggest that some systems should simply be turned off or not built at all. Why? Focus requires choosing what not to do, and creativity thrives under constraints. To not even consider the conditions for when a system should be turned off is both a failure of imagination and a failure of governance.
For any complex system, figuring out a good answer to ‘how much harm is acceptable’ is far from easy. But it should be a conscious policy decision made carefully in advance, not a knee-jerk tactical response to a poorly designed system after the fact. Wrestling with the complexity of this question is what we should expect of our leaders.
When designing policy, we should explicitly determine – in advance – how much harm, and of what kinds, will be tolerated before the system is deemed too dangerous to keep running. We should have the hard conversations, as a society, of what and who we value, and who is simply expendable.
It seems impossible to imagine that we could, for once, just stop.
It may not always be possible, but we should at least consider it, and define when turning off the machine would be the best plan, and explicitly designing in how to do that.
If we must decide to enact unsafe systems, let us be honest about what we are really doing. Bring the sociopaths into the open where we can see them.
Killing the machine
It may not be possible to make the machine safe. Such a machine should be destroyed.
We take this approach with any number of other systems: a dog that bites too many people is declared dangerous and is destroyed. A batch of contaminated food is removed from sale and thrown out. We don’t try to muddle on as best we can, accepting that the dog might maul up to nine toddlers a year, or that a dozen elderly people might die from eating poisonous tinned tuna every quarter. There are some areas of life where the cost of safety is deemed worth paying.
Why not for IT systems? Why is disabling an online form and building a new one seen as unpalatable while killing a living animal (though poorly trained) is just fine? Why is changing software – a highly malleable abstraction of electrons and maths – seen as more difficult than un-poisoning contaminated food? Shouldn’t the bar for destroying a computer system be much lower?
The constraints we have erected for ourselves are artificial, and act as a needless barrier to creativity. Giving up without a fight is a failure of imagination as well as courage.
Any new machine should be designed to be destroyed, and serious thought given to the circumstances under which destruction is correct answer, and how it should be done.
Conclusion
Before a machine is deployed, we should always make sure there is a way to turn it off. If we can’t turn the machine off once it starts, we must be much more careful about turning on such a machine in the first place, because once it’s turned on, we will be stuck with it for some time.
A careless disregard for risk is not innovative, no matter how many ‘tech bros’ breathlessly exclaim it is while setting huge piles of other people’s money on fire. The most innovative people understand that constraints are what drive creativity. Finding ways to fail gracefully, rather than catastrophically, is how resilient systems are built.
Perhaps after the past couple of years experiencing just how brittle systems can be there is a refreshed appetite for robustness of design, and a willingness to simply say “No, we won’t do that.”
We should stop letting the thoroughly mediocre be the enemy of the good. We deserve far better, and it’s well past time we expected broken systems be turned off so they can be fixed or dismantled. We shouldn’t be forced to use broken systems because those in charge refuse to let us build something better.
We should turn more things off and focus our energies on building new, better ones rather than insisting that a few minor tweaks around the edges will provide the systemic changes we actually need.
Justin Warren is founder and chief analyst at PivotNine, a specialist IT consulting firm based in Melbourne, Australia. He is an independent technology analyst and consultant with over 25 years experience across a range of industries. Justin’s writing has appeared in Forbes, CRN, and The Register, and he is a regular media commentator on IT issues. Justin holds an MBA from Melbourne Business School, and is a graduate member of the Australian Institute of Company Directors. He is currently Chair of Electronic Frontiers Australia.
Do you know more? Contact James Riley via Email.