In late 2021, researchers spotted an opportunity for a natural experiment. At four clinics, a group of endoscopists (the doctors who examine the body from the inside with a camera-tipped scope) were using a new AI tool that automatically alerted the operator when it spotted a precancerous growth during a patient’s examination. The issue was that scheduling constraints meant it only ran on some days.

The researchers conducted an observational study comparing the AI-free days with the months before the tool was introduced. What they found was alarming: the same doctors, in the same rooms, now found growths in fewer procedures than before the AI tool arrived, falling from 28.4 percent to 22.4 percent. With the AI’s introduction, the doctors had become worse at finding the growths.

As co-founder of SparrowHub, I build software for pharmacists, so findings like these aren’t purely academic to me. I feel the pull of the obvious response: if using AI makes doctors worse, in any way, abandon it until we can design these systems safely. Patient safety demands nothing less.

The only problem is that the counterexamples are remarkable. In Sweden, the MASAI trial randomized 105,934 women between radiologists paired with an AI tool and radiologists without. The AI’s job was to pick the cases needing a second pair of eyes. The AI-supported doctors found 29 percent more cancers, and the number of scans they had to read fell by 44 percent.

There is clearly something worth pursuing here. So how do we, as designers and engineers, make the design decisions that lead to safer outcomes for patients?

In 1983, Lisanne Bainbridge, a cognitive psychologist who studied the operators of automated systems, published “Ironies of Automation”, five pages that read like a review of this last year in tech. People, she argued, make poor monitors.

However motivated, a person watching a screen for an event that rarely comes struggles to maintain attention and focus. She predicted exactly what the endoscopist study’s authors fear today. Requiring a person to monitor an automation, she warned, will over time lead to a loss of proficiency, a de-skilling effect:

Bainbridge’s vision for a solution is in the design of automations. To avoid this pitfall, the automation must “maintain the effectiveness of the human operator by supporting his skills.” Keep the person in live control, keep the skill in use. The MASAI trial, knowingly or not, followed this principle. The machine absorbed the watching, and the radiologists went on reading every scan.

The difference was the job the system’s design left us humans to do.

Designing systems that perform well is easier said than done, and the risks are hard to spot. For example, in one study, researchers intentionally biased a diagnostic AI tool, then asked hundreds of clinicians to use it to work out why patients had been hospitalized with acute breathing problems. The systematically biased AI dragged clinicians’ accuracy down eleven points, explanations and all.

This is an example of what anthropologist Madeleine Clare Elish calls a “moral crumple zone”. The human becomes the part of the system that absorbs the blame when it fails, even when they have little real power to prevent it.

“Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions.”

This is the trap we keep falling into. When we build AI that leaves people to absorb the blame, we squander the good the technology could do in healthcare. We pour attention into what AI can do, and almost none into designing what the human does beside it. That is precisely what design research knows how to do: bring people’s personal experience of their work into the design process.

In one clear example, researchers used co-design to build an AI tool that verifies dispensed pills. Pharmacists were asked to sketch the interface they wanted from the machine. They didn’t ask for faster, more accurate AI decisions. They asked to see the AI’s confidence in its analysis. They asked to see when it was “unsure.” They asked to see which pills the machine most often confuses with the one in front of them, so they would know where to look hardest. They intuitively identified where to keep human judgment in the loop.

The requests share a shape, a machine that augments, extends, and enhances human decision making rather than dictating conclusions. Medicine’s emerging consensus calls this type of machine a “digital copilot”, not an autopilot. Bainbridge asked for much the same in 1983.

The public, it turns out, agrees. Pew found 65 percent of Americans would welcome AI screening their skin for cancer, yet 60 percent don’t want their provider relying on AI. People aren’t asking the machines to leave. They’re asking for the professionals to still be there, and still sharp, when it matters.

Bainbridge’s finest irony is that the more successful the automated system, the greater the investment it may need in the role the human plays. Designing the human’s role well may be one of the strongest levers we have for AI adoption. At the pharmacy counter, the AI tool flags a script and the pharmacist looks up. Whether that look is real judgment or a rubber stamp was decided earlier, by whoever designed the loop.

So which one are you building? A copilot keeps the person skilled and in command. A crumple zone keeps them only to absorb the blame. Right now, we mostly build the difference by accident. The opportunity is to change this. By design.