In the conclusion of his series on AI Risks, Ethics, and Liability, John Sumser provides questions to ask your AI Tech vendor and his thoughts on how to manage the new wave of algorithms you’ll encounter with intelligent tools.

AI Risks, Ethics, and Liability


Questions for your vendor

If you are considering the utilization of intelligent machines in your HR/Operations processes, here are some questions you might consider.

  1. Tell me about your views on product liability
    Be sure to have a long conversation about how the tool works and how the vendor is monitoring the impact of machine learning curves. You’ll learn a lot by raising the topic of product liability. Most vendors still imagine that we are in the first generation of software where liability is not really a possibility. The key question here is ‘what if your tools recommendations cause damage to our people or our business?
  2. How do we make changes to the historical data?
    Most machine learning systems are ‘black boxes’. If you ask the designers how they work, they can only explain about 80%. That means that you are likely to want to modify the results that the machine produces. It is likely that the answer to this question is ‘you can’t.’ Having the conversation is what’s important. It will give you a window on your real risks.
  3. What happens when we turn the “it” off? How much notice will we receive if you turn it off?
    Imagine that you are using a tool that does the job of several employees (sourcers who review resumes, for example). If the tool fails in a way that requires a shutdown, what sort of advance warning do you get. Since most providers are in experimental stages, the answer to this question also matters if the project ceases to operate. In a very real way, these are digital employees and it is best to have a replacement plan.
  4. Do we own what the machine learned from us? How do we take that data with us?
    Part of the way that these systems operate is that they learn in both the aggregate and individual case. Most vendors guarantee that your data is ‘anonymized’. You still may not wish to have your operating practices be a part of some larger benchmarking process after you change suppliers. Being very clear about whether the system will retain evidence of your participation after you go is of strategic import.
  5. What are the startup costs, resources and supervision?
    We know precious little about the behavior of intelligent machines. There is good reason to expect that their impact on your resource consumption is greater than anyone thinks today. Like any employee, they require training, supervision and discipline. Make sure you have a very clear picture of the Total Cost of Ownership of any leaning machine you enable.

The age of human-machine integration is in its infancy. It is inevitable. In the transition, it is important that we move forward carefully with a clear picture of the risks and ethical issues. This note is a starting point.

Managing your algorithm

We’re driving into Healdsburg, a chi-chi enclave of movie star getaways in the north end of wine country. They are in the process of adding a roundabout to the main street headed into town. Like any traffic circle, there are four or more arms of the circle. It’s where at least three streets intersect.

There is a ton of construction flotsam and jetsam. Traffic cones, concrete dividers, odd fences, steel plates on the ground. It looks like the working model for the idea that the plan never survives contact with reality.

As we come upon the traffic circle, our navigation tool says: Turn Left on Vine St.

The problem is that the directions worked before the city installed the brand new roundabout (or traffic circle if you’re from the East Coast). For the moment, it is impossible to see the left turn in the jumble of construction detritus. There are no road signs.  While the streets are the still the same, the directions are impossible to understand.

My partner asserts, “That’s Vine St. over there, turn there!” As I begin to utter a small disagreement, I am overruled by the urgency in her voice.

It turns out that it wasn’t Vine St. My urge to grump and say, “I told you so” runs high. (I may have even given in to temptation ever so mildly 😉  )

Here we arrive at the next layer of challenge in the design of machine-led decision making. And, it has two parts.

Part one involves the clarity of the recommendation. Like any sort of direction or delegation, specificity in the guidance provided by the machine is important. When the guidance is unclear and the people receiving the guidance have to debate its meaning, all of the advantages of machine insight is lost. There are going to be some very expensive court cases involving the question of whether the instruction was clear. We will see debates about whether a person who thinks they are following an instruction are, in fact, following the instruction.

One way of saying this is that machines are inexperienced at managing people. In human to human communications, one judges the amount of required specificity very carefully. It is a measure of trust. Being very explicit is exactly what you do along the way to building that trust. Without variability in levels of specificity, machines will over-manage some and under manage others. Undermanaged workers produce liability. Overmanaged workers begin to ignore their bosses.

Undermanaged workers produce liability. Overmanaged workers begin to ignore their bosses. Neither situation is useful.

The second part involves keeping the flow of recommendations relevant to current circumstances. As much as their creators would like to believe otherwise, machines that learn work in environments that shift rapidly. Recommendations that worked yesterday may fail today. When the machine issues direction that turn out to be irrelevant or mistaken, their utility declines. 

It may be that the largest expense in owning a machine learning tool is monitoring the relationship between real circumstances and recommendations. This is a critical part of the task of supervising an algorithm. It’s sort of like the quality control required at the end of the line. So far, there do not appear to be any examples of tools that allow for this kind of quality control.

Read the entire series

Read previous post:
Ducking Risk and Shifting Blame in Hiring

“Hiring is the single riskiest responsibility for most managers. Like the Kobayashi Maru, hiring appears to be a no-win scenario...