On the logic of enforced order and the law of unintended consequences
Recently I was doing a talk on AI Ethics for a government organization, and the department head said, “We have a mandate to do no harm.”
I said, “I have bad news for you.”
All technology has impacts. If the impacts get large enough, it causes harm somewhere.
I’m not talking about computers, or Silicon Valley, or even AI… I’m talking about the more root question of technology that encompasses agriculture, writing, the wheel, and systems of government: This is the suite of technologies that are sufficiently advanced that they are not only indistinguishable from magic, they are indistinguishable from reality.
Thought of this way, technology has its own logic, which sits on a single assumption that is never questioned or made explicit: We make order (in some place) at the expense of causing increased disorder somewhere else.
We embody and enact that logic
We take water from the ground and place it in irrigation channels at the expense of the downstream users of the water (or the buildings sitting on top of the aquifer).
We build cars that require an entire systems of roads and extraction of power to give them the ability to drive.
We create business models that require ever-larger CPUs and databases and ever-reduced privacy expectations to “extract value” for the organizations that figured out that we leave traces of ourselves everywhere we go.
In each case, we focus our collective efforts on one part of a vast system, say, “This is worth making bigger,” and neglect the impacts on the surrounding components.
Although we couch it in more complex considerations, at root it is that blunt: Every technology prioritizes a particular form of the world and by doing so makes other forms less likely, even if they already exist.
It concentrates energy/matter in particular places by taking it from other places.
But it gets worse.
Because of the second law of thermodynamics, it is worse than a zero-sum game.
There can be no technology that “does no harm” because (at least) the energy to create order must come from somewhere. There is no truly clean energy, because we must count the entirety of the system, not just the end point if we are to reason fully about the impacts of our work.
There are no isolated systems. There is no “away,” there is only “where we are not looking.” And that is where unintended consequences arise.
Some Good News
Well, that’s bleak.
Fortunately, I also have some good news. When we look directly at the logic of a system, it is easier to come to terms with it. If we are wrong about the compromises, we make them accidentally and with no consultation. If we bring them into the field of “what we can think about,” we can make those choices more intentionally and add an element of mitigation.
If the solution to pollution is dilution (a debatable claim) at least, “The solution to disruption is time to accommodate.”
In addition to being open, all systems are evolutionary. It’s only when the “contaminants” pouring across the boundaries overwhelm the surrounding systems that the harm grows unbounded. The tiny amount of background radiation from naturally occurring Uranium can be absorbed; the concentrated amount from a spent fuel rod will kill you in moments. (1) A lot of our modern technologies create a lower-dosage equivalent of spent fuel rods… they use the power of computing or industrialization to concentrate the impact into smaller and smaller and more and more concentrated packages, so the harm is more immediate and harder to survive… for surrounding institutions, for societies, and for the environment.
We do have the chance to do less harm, to monitor and mitigate, and to make better choices about how and when to deploy technologies. We have the ability to choose human scaled systems that people can understand and respond to. We can prioritize (for example) bicycles and public transit over cars… but to do that, we first have to recognize that there is a choice, and not allow the logic of the dominant technology to remain invisible and dictate itself as the only “reality” that is possible.
- “For example, 10 years after removal from a reactor, the surface dose rate for a typical spent fuel assembly exceeds 10,000 rem/hour – far greater than the fatal whole-body dose for humans of about 500 rem received all at once.” – https://www.nrc.gov/reading-rm/doc-collections/fact-sheets/radwaste.html
The Practical Dilettante 