2020-02-04 HR Examiner article john sumser AI and Intelligent Software Implementation in HR part iii photo img cc0 via pexels board game 207924 part 4 544x305px.jpg

“From data quality to regulatory compliance, there are 28 key parameters to evaluate when considering a purchase of intelligent tools for HR.” - John Sumser

Catch Up on the Series

Haven’t read parts 1-5? Catch up on the series by clicking on the links to each article at the bottom of this post. Click here to navigate to the links.
 
Today, in part IV of the series we look at how to evaluate vendors in the AI and intelligent software space. We’ll finalize the series later this week with our vendor evaluation guide that provides 40 Key Questions for Evaluating your Prospective AI Vendor.
 

How To Evaluate an AI or Intelligent Software Vendor Solution:

Readiness, Functionality, Change and Improvement, Maintenance, Support, and Compliance, Liability, and Warranties.

 
From data quality to regulatory compliance, there are 28 key parameters to evaluate when considering a purchase of intelligent tools for HR.

Readiness
  • What are the format and volume requirements that your data must meet? What are the differences in the vendor’s output based on the volume and quality of your data?
  • What is the process for acquiring and integrating your data into the solution?
  • Ask the vendor how they think about improvements in their models and algorithms. How close are they to perfection? What are the looming obstacles and limits? (Quickly eliminate vendors who don’t have clear notions of the collaboration and investment required to finish their work.)
  • How many people does the vendor believe are required to manage the process (account relations, maintenance, change management, exception reports, additional service people)?
  • How much time is required to change existing processes and train team members in the new technology?
  • How does this particular tool fit into the larger HR Department intelligent tools strategy and vision?
  • What are the processing demands this purchase places on internal systems (API calls, data submittals, security)?
  • Can the vendor operate at the scale of your company? Ask for examples.
Functionality
  • What, exactly, does the new tool do? How does that improve current operations? Where did the data that trained the model come from? How is it updated or replaced?
  • What becomes possible once the tool is installed? What can we do with the tool that we haven’t done before? What can we do with the tool that we’ve never imagined doing?
  • How does the tool handle privacy and security concerns? Is it GDPR compliant? What about other standard security issues? How does the vendor manage the security of our data? Have they had breaches? What is their policy about breach notifications? How do we know if our data has been compromised?
  • What is the product roadmap? How are we kept abreast of changes to the roadmap?
  • Do you store all of our user transactions and their outputs so that we can survive an audit?
Change and Improvement
  • Technical debt is the set of ‘to be kept’ promises that a vendor has made to its customers. How does the vendor communicate the details of its technical debt? How do they plan to balance the elimination of technical debt with forward progress?
  • If the vendor pivots into a new direction, what are our rights to preserve the service we originally contracted for?
  • 2017-04-21 HRExaminer photo img sumser john bio pic IMG 3046 black and white full 200px.jpg

    John Sumser is a Principal Analyst for HRExaminer.

  • How are we informed when the user interface changes? Do we have the capability to defer changes in order to maintain operational consistency?
  • How are we informed when the API changes? Do we have the capability to defer changes in order to maintain operational consistency?
  • How are we informed when the data models and algorithms change? Do we have the capability to defer changes in order to maintain operational consistency?
  • How do we ensure results consistency?
Maintenance
  • How do we monitor the tool’s performance at a top level? How do we get visibility on the way that changes are affecting our results?
  • What is the process for identifying and fixing flaws in the models, the algorithms, or the NLP processes?
Support
  • What are the parameters and limits of customer support? Is there a response time guarantee?
  • Is there routine reporting on the status of open support tickets?
Compliance, Liability, and Warranties
  • Do you have experience with regulatory compliance audits? Give examples.
  • Do you have evidence that your system is compliant with all of the regulations we have to comply with?
  • Exactly what is the vendor’s liability when the system makes an error?
  • What is warranted and how is it covered?
  • At what point (where is the line) does the vendor expect the customer to take responsibility for results?
  • Who owns the data? If we terminate the arrangement, is our data removed from the system? How about the things that were learned from our data?

Later this week we’ll finalize the series with our vendor evaluation guide that provides 40 Key Questions for Evaluating your Prospective AI Vendor.
 

Catch Up on the Series


 
Read previous post:
2020-02-03-hrexaminer-article-john-sumser-AI-and-Intelligent-Software-Implementation-in-HR-part-iii-photo-img-cc0-via-pexels-board-game-207924-sq-200px.jpg
AI and Intelligent Software Implementation in HR – Part III

In part III of this series, we'll be looking at the role of data cleanliness, data models, and process governance....

Close