Wait a minute, I said. If you’ve been around Washington long enough, you know that the “decisionmakers” are often puppets of their own staff, who determine which options make it to the boss’s desk in the first place and then put a heavy thumb on the scales of which is best. In this paradigm you’re proposing, the computer plays the role of the savvy staff, the human the hapless principal. How do you make sure the commander isn’t just a rubber stamp for the computer?
“You’ve put your finger on one of the biggest issues,” Prabhakar said frankly. “As we enhance the abilities of these machine systems, [it] is about our trust and confidence in what they tell us, about what they think is happening, or what courses of actions they’re proposing.”
“There’s this powerful new wave that’s happening today in AI,” she continued, and the Pentagon needs to exploit it, “but I think it’s really important to just put on the table the fact that a lot of what’s happening in deep learning doesn’t yet have [a] rigorous theoretical foundation….We all see these systems come up with solutions that violate common sense because they lack the context.”