Chatbot Musings

Topics: HRExaminer, John Sumser, by John Sumser



“Here are the tidbits from my side of a long facebook conversation about chatbots, their relative effectiveness and the actual cost of ownership.” – John Sumser

Here are the tidbits from my side of a long facebook conversation about chatbots, their relative effectiveness and the actual cost of ownership. You can see anything I post on facebook here (to get all sides of the story). It’s a shame that Facebook doesn’t allow greater access to the material people generate.

These snippets are a direct result of the research I am doing into what I am calling ‘machine led decision making’. 

I’m learning the most interesting things about ‘chatbots.’ They range from tools that simply read FAQs to systems that attempt to parlay semantic analysis into broad useful conversations.

The core technical challenge is different than I expected.

It turns out that it’s pretty simple to define and understand all of the answers an intelligent machine conversationalist might give.

The hurdle is figuring out what all of the questions are.

The second biggest challenge involves what you do when the company makes a significant change. It turns out that much of the intelligence that is based on transactional experience must be rediscovered under a set of predictable circumstances.

Mergers, acquisitions, stock price fluctuations, scalar growth, executive changes, strategic direction changes (pivots) and disruption all conspire to invalidate what the system has learned to date.

Those are the things that businesses tend to do.

…the real expense for existing content comes when you don’t start with a good set of data. I’m afraid that many customers will be disappointed by the initial cost to get started.

The total cost of ownership is non-trivial. The cost to license the technology may be small (although there is a huge ‘caveat emptor’ attached to that assertion).

What you can have for a small investment is most likely to produce low-quality results. The first real expense is getting the answers right (what you call internal content). The second (and really difficult) bit is getting the questions right. The tool has to understand which question elicits which answer. That’s proving to be very difficult for some chatbot companies.

I don’t think that anyone has done an exhaustive inventory of the questions that candidates are going to ask. That means that humans will be required to assist the Chabot until there are no more questions that t doesn’t cover.

In other words, it’s very difficult to quantify the total cost of ownership.

I think it’s way too early to tell what an effective installation actually costs. Total Cost of Ownership (TCO) should account for acquisition costs, internal labor costs and predictable maintenance. My guess is that, like anything technical, TCO will depend on the level of quality you find acceptable. 

As I said in the beginning of this piece, we don’t really understand what happens when the organization makes a change (layoffs, M&A, overall strategy, outsourcing, disruption, technical innovation). Those factors can invalidate all of a chatbot’s utility and need to be accounted for. They happen very routinely.

In other words, the TCO of a chatbot is entirely dependent on organizational stability (once you’ve solved the initial hurdles).

Latency is the degree to which an AI’s learning matches organizational reality. Over time, latency will vary in response to organizational demands. Users will experience this as a performance defect. TCO has to include the means necessary to get the system back into shape.

I’m hearing no more than 50% to 60% before you hit the investment hurdle. As Jonathan mentioned above, that’s when you have to start figuring the question of intent. Right now, I’m imagining that a chatbot is like a small child that has to be trained by the entire organization to hit anything like 90% and then, that gets disrupted by predictable pivots and changes.

Machines learn. Their capacity to adapt to sharp turns is not very well understood. They are happier in stable circumstances with measured incremental change.

90% is optimistic, I think. In other words, the work required to have a chatbot (or the savings from having one) is different than currently imagined. That’s not to say that the work isn’t valuable, just that the way we think about the investment has not accounted for all of the facts.

The cost of managing the bits of intelligence we unleash on our organizations will be significant.

Read previous post:
HRExaminer v8.33

In An Open Letter to Job Seekers, Mary Faulkner writes, “We, the recruiting community, understand your frustration. We know that...