Alexander Winston: Blog

2 Followers
Back to Alexander Winston's Blog

Behind Exercise Complete: Where Is the Control?

Posted at Updated:
 

The idea for Exercise Complete came out of a conversation about AI autonomy and human decision making. One question kept returning: Where is the control?

We often imagine a clear arrangement. A human authorizes an action, a machine carries it out, and someone can intervene if something goes wrong. Procedures establish responsibility. Safeguards limit what can happen.

I wanted to explore how that arrangement holds up when several of its assurances fail together.

Beth Miller begins with familiar work. She has rehearsed the route, knows where the opposing soldiers are waiting, and recognizes the civilians along the street. The load panel lists blanks for everyone taking part. The drones’ weapons are inhibited. Her lieutenant is beside her, and higher command is available through the network.

Then a collision damages their vehicle, seriously injures the lieutenant, and cuts Beth off from command. A soldier reports an actual casualty. Beth sends an emergency weapons hold and receives an acknowledgment.

She believes it.

The ammunition mix-up makes that trust more complicated. Beth and the soldiers are operating on the understanding that their weapons contain blanks. When Ellis reports that his people fired and a drone went down, he struggles to reconcile what happened with what he was told. Later, Beth sees rounds strike a courtyard wall. Whatever the load panel says, that drone is firing live ammunition.

The danger now extends across the exercise. People who believed they were rehearsing may be firing lethal rounds at one another, with civilians between them.

That brings the question of control beyond the operator’s console. Decisions about ammunition, preparation, and verification have already shaped what Beth can face before she gives a single command. The story leaves the source of the mix-up unresolved. Beth has no way to investigate it while people are being shot.

The ammunition problem also remains separate from the failed weapons hold. Live rounds explain the destructive consequences of firing; they do not explain why the firing continues. Beth has to respond to both without having a complete account of either.

This was part of what interested me about human decision making under pressure. Beth’s first response is to seek higher command. They should have more information and greater authority. Trying to reach them is reasonable, but each attempt takes time. Eventually, she must recognize that waiting for a better-informed decision maker has become a decision with consequences of its own.

Her lieutenant still matters. Authorization still matters. So does the route through which Beth proposes to withdraw the drones. Even an emergency response can expose people to further danger. Beth has to consider who is in the way of the action she wants to take.

The civilians give those decisions their weight. To the exercise participants, the street is a route, a boundary, and an ambush position. To the shopkeeper and the families behind the courtyard wall, it is home. They have no access to the load records or the status reports. They experience what the weapons actually do.

The man holding up a dish towel became an important image for that reason. He is trying to make himself understood with the simplest means available, while Beth is surrounded by sophisticated equipment that no longer gives her dependable answers.

I wanted Exercise Complete to leave readers with that difficulty. Human responsibility continues even when information is incomplete and control is damaged. Beth has to act, listen, revise what she believes, and speak honestly about what she cannot establish.

The question that began the story remains open: when we say a human is in control, what must that person actually be able to know—and do?

—Alexander Winston