Video with full transcript

While scientific and psychometric measurement for “fit” has existed for decades, it remains very hard to sell into large organizations. This panel led by John Sumser debates the tried and true vs. the innovative and unproven in assessing talent for the Future of Work. Panelists include, Alina von Davier, PhD, Senior Vice President, ACTNext, and Josh Jarrett, Chief Product Officer and Co-Founder at Koru, Inc.

I’d love to hear your thoughts about the discussion we had in the interview. Post your comments on or my YouTube Channel »



Panel with full transcript


Moderator: John Sumser, HRExaminer — Link to this video and transcript »


  • Alina von Davier, PhD, Senior Vice President, ACTNext Link »
  • Josh Jarrett, Chief Product Officer and Co-Founder at Koru, Inc Link »




Important: Our transcripts at HRExaminer are AI-powered (and quite accurate) but there are still instances where the robots get confused and make errors. Please expect some inaccuracies as you read through the text of this conversation. Thank you for your understanding.


File Length: 00:50:01


JS – John Sumser, Principal Analyst, HRExaminer
AV – Alina von Davier, PhD, Senior Vice President, ACTNext
JJ – Josh Jarrett, Chief Product Officer and Co-Founder at Koru, Inc


00:00:09;00 – 00:00:12;27 – JS
I’m John Sumser, I run the HR Examiner, thanks for coming and we’re going to do some introductions that’s going to include everybody in the room. We’ll start with Alina. Why don’t you introduce yourself.

00:00:28;02 – 00:00:39;18 | AV

Sure, thank you John, Hello everyone, I’m Alina von Davier and I lead ACTNext, which is the business innovation and research and development unit at ACT. And my background is in mathematics, and building mathematical models for learning, for assessment, for classification, and for prediction. I think that’s good for now.

00:00:40;29 – 00:01:01;03 JJ
That’s great, I’m Josh Jarrett, I am the co-founder and chief product officer of a company called Koru, K-O-R-U, we’re a Seattle based predictive hiring company. So I have I have one-thousandth of the mathematical training of Alina but I try and connect the dots really down to the users and how people use data to make better decisions.

00:01:03;22 – 00:01:33;10 | JS
So one of the things that’s interesting about about about these two is this conference has been traditionally in edtech conference over the years, so there’s a lot of warmth and inclusion over the ed tech side and this is part of a stretch out into the human capital marketplace.

These two represent where that Venn diagram overlaps. It’s a pretty interesting thing. So we’ve got an academic view of what assessment is and the commercial view of what assessment is. And. That’s what we’re going to mine over the course of the conversation.

For starters would you mind because we’ve got time. Tell me your name and what company you represent or how you got here. Because, the audience is small and we may as well know who we’re talking to. So let me start with you. Would you tell me your name and who you work for. Thank you

00:01:58;20 – 00:04:37;24
Audience members introduce themselves to the panel.

00:04:38;14 – 00:04:59;07 | JS
So Alina, what does your company do?

00:04:59;08 – 00:05:14;09 | AV
So, ACT is known for its college admission test primarily but ACT does more than that. Then some of you might have heard the announcement we made today ACT and ASU are partnering, so that was the announcement at 1 o’clock when I left you guys.

04:59 | JS
What does partnering mean?

00:05:00;10 – 00:05:41;07 – AV
So we plan to collaborate on research projects and new products. So we will work together as an extended group. We want to put together a center. And we will work bringing in students postdocs coming up with new research projects. And as I said before products hopefully.

It has not yet been all you know outlined. It’s going to be some work to do to figure that out. But this is just an example about how ACT as a company is changing so ACT has been around for about 59 years. It started as you know as a disruptor, pretty much, Lindquist who was a professor at the University of Iowa invented the automatic reader of, for multiple choice test. And with that he pretty much disrupted the way the assessment took place. He also proposed that the test for automation at the university should be based on what students learn so that the students can actually do something with the information they receive from the test not only a score so they can know what we improve on as opposed to other tests that were more aimed as aptitude. And called aptitude at that time. Meanwhile the S.A.T. itself transitioned quite the lot. And there’s also more curriculum oriented but at that time it was not done for many many many years. There was this difference between the ASU details that was more curriculum oriented and S.A.T. that was more aptitude oriented. So over the years the company became quite successful to the point of becoming the lead in the country of admission polish. That’s about 60 percent or more of the market is of a city. Now we got the new CEO about two years ago and he is changing the direction of the company and is moving it into learning and into becoming an ad tech company. And that’s of course a strong claim to make. And but he definitely. Talks the talk and walks the walk pretty much one of the things he did among other things of course but since it’s about me now he hired me 18 months ago to lead this innovation group and I put together a team in this 18 months and two at this conference as well as at last week’s conference in New York City at they are in CMG. We had our first presentations based on the work and results that we actually conducted in the group. The approach that we take for learning is to based everything on our. On our strengths. So we what we know to do best is measurement and assessment. So for us the question is how can we differentiate ourselves from all this mass as you can see here of people who act in this learning environment. What do we bring to the table. So our learning components are within the evidence that design framework which is quite familiar for those who have been working in assessment for years. So that’s pretty much what we do. We work our we have I have a team on artificial intelligence and one an advanced cycle metrics and another one on learning solutions. And together we try to build these systems and we do foundational research in developing models for how people learn how people forget. How people deal with scaffolding and with hints then we have an artificial intelligence team that is working on packing content instructional content as well as. Assessment content and the learning solution person is driving the how and the what and the why. So so is your work still focused on. The sort of.

00:09:29;16 – 00:09:32;19
Academic environment or the issue of structuring.

00:09:32;19 – 00:10:25;22
We are definitely stretching out. I mean in our group we also develop new products. We have several research based prototypes at the moment. We build an educational companion. The idea is to have an app that accompanies a person from younger age to the workforce and can provide support responses and advice with respect to everything educational which also includes social emotional learning skills. So yes the short question the short answer is we are also developing prototypes. However it’s an at the moment in the structure of the company. It’s not us. We’ll bring those to market it is the new product development team that is working on that. We also develop capabilities that we provide to other existing products.

00:10:25;22 – 00:11:07;07
For example we have a patent now on a new cognitive back gnostic model that accounts for the behavior analytics based on the data that we collect from the interactions of the student with the system. So this became in itself a capability that can be used by other existing products in the company for example cognitive diagnostic model for the ACP test or for the work is test or aspire and so on. So roughly the the mark of excellence of your work is so rural that we have to measure that shows you the assessment target correctly

00:11:07;29 – 00:11:14;15
Or learning state. So if we work through yeah.

00:11:14;16 – 00:11:34;03
So I think the I think your story illustrates your Venn diagram and that we start from the business problem first and then work backwards so the core business probably work on is that hiring is fundamentally broken which is that it’s sort of the definition of insanity we do the same thing over and over and expect different results.

00:11:34;06 – 00:11:52;11
Think of campus hiring for example we go to the same 12 schools we fight with our competitors over the bottom third of candidates in those classes. We screen out 80 percent of people because they don’t go to the right school with the right GPA the right major didn’t work at Google Facebook or Amazon before

00:11:53;07 – 00:11:58;23
Then we are surprised when we have turnover we have performance issues and we have diversity issues.

00:11:58;23 – 00:12:24;23
So we really took that as a problem both of access opportunity from the candidate side and business problem with real or ally and financial incentive to solve it. And as we look to how to solve that we talked to employers and hiring managers and said OK tell me about the best hires you’ve made. What were the attributes how did you know what an experience they had work with. Where they studied.

00:12:24;23 – 00:14:12;17
It kept coming back to. It wasn’t about that. It was do they have the grit in the work ethic. Do they have the teamwork skills. Can they think on their feet. Are they actually curious. Are they accountable. They follow through. If they have that stuff will teach them the tactical skills if they’re that sort don’t care where they went to school. We use all these things for proxies. Increasingly people are figuring out or predictive. So Google did a big research study. They determined that GPA was worthless as a hiring criteria. So using that information we’re really the banks. Okay. How do we start to look at bio data which is a fancy word for. Stuff about people being you know where do you work. What classes did you take. What leadership positions you hold. What prior work did you have. That’s that gives you some predictive power. But there was a piece that was missing that’s humanistic traits we call them impact skills. And there’s seven that we focus on grit rigour impact teamwork curiosity ownership and polish. And we said that’s what hiring managers want to measure. Can they be measured. So then we get back to the research. Eighty five did research studies other people validating work in terms of sub competencies that roll up into what those hiring managers were saying. Leveraging I spent seven years at the Gates Foundation before co-founded crew. So I spent a lot of Bill on what there’s money and a lot of my time I try to answer these questions of what are the things that that could predict success beyond GPA and where you score etc.. So we mind the best that research and married it up through a 20 minute online instrument. That is one of the inputs into a machine learning driven predictive hiring. So what’s excellence when you just say one thing because you mentioned the Google stuff. And

00:14:13;02 – 00:14:56;21
You and I have talked about wrath on the phone. So at the end when you talk and I think it’s fascinating what Carl does. And now at the end you said you are looking at all these skills that are necessary in addition to the GPA. And I want to emphasize that in addition to and here is why the Google study unfortunately is flawed psychometric study. And the problem is the following. The people that they invest in that field for which they checked the team war skills their creativity and so on we’re already very high on their STEM qualities. So psychologically speaking that is. It’s called a ceiling effect.

00:14:56;21 – 00:17:12;03
So there was no body Asian in that particular body. But that plus reflective of the technical skill. So when you do a traditional value did this study if. You know people have not had the exposure to psychometric analysis they might tend to see no evidence that that particular body up one is relevant. But the truth is that’s the most important valuable and only after that you’ll have the financing at once. So actually I did. Comment on that and I recommended Google to hire psychometric shots because. It’s a problem because you know people will want to see that. Oh always come from Google. So it must be correct. So I wouldn’t comment. This happened. I am I work with a group from at the heart truly which is a university activity that. Is extremely it’s an elite university and they’ve been using profiling injury to screen for their graduate students. And it turns out that the Jerry Moss didn’t show up to be relevant to talk but once we stopped looking at it it was because everyone. Had an almost perfect score on the Jerry moth so if everyone scores very highly or very low or whatever the variance is very low on a particular variable then that variable won’t appear as a valid and predictive tool. And that’s a you know that’s that’s why you need psychometric shots to help look at conditional distribution and conditional predictive value. So we found out that everyone had a great math score. Therefore we had to look at other variables that would distinguish them. And. Colorful scores. In that case what extremely relevant. But if you only chose by total score because that plus the variable that showed the highest predictive value then they wouldn’t have been able to do the programming for which they were brought into school. So personally I’m very concerned and this is about the title that we have here with not all glitter is gold. That’s a good example for that. Yeah there’s Google.

00:17:12;06 – 00:17:24;02
I think it’s a great point. And I think that they’re here if. We trust data it’s the truth. Despite imperfections in the quality of data selection bias and survivorship bias.

00:17:24;10 – 00:17:36;13
And we have to be very careful drawing absolutism spin and making these large leaps of inference however.

00:17:36;26 – 00:18:04;04
One of the things we like to say is grit over grades and what that is reminding people is that relying on singular variables particularly ones that carry a significant amount of bias is a bad way to do hiring. All right. So if you were to select people only on there where they went to the exclusive school right. You probably that actually would be illegal in the United States to use that as a single parent variable because that carries disparate

00:18:06;06 – 00:18:10;05
Against productive classes. All right. That’s just it’s a that’s a mathematic I mean it’s just a

00:18:10;20 – 00:19:20;02
Fact. So what it reminds people is how do we think more holistically about candidates and holistically about how we hire. So we say great of a great school I love that. I love that. You know and I had breakfast with Angela Duckworth this week and she was like remind remember Josh grit and grades are called right. Not surprisingly I say yes. But what we’re what we’re saying is reminding people to look more broadly and then we build predictive models that are context specific and that gets to the point of these broad assumptions. You know there’s really dangerous. We make generalizations from one data set to another one context to another but we build local models on that set and I can say that grit is not always predictive and in some cases grit is negatively predictive. You can have too much grit. You can be too narrowly focused you mean not accepting of other people’s opinions. Can we not willing to be flexible and grades sometimes are predictive. And then we work with our clients. Do you want grades to be you college. U.S. News college report ranking is predictive here. Do you want that to be included in your model going forward. You want to give Harvard kids credit an advantage or you want to take that out of the model and just use these other variables.

00:19:20;02 – 00:20:01;06
So I think it’s an art and a science and it’s not for the faint of heart. I think so. So as I listened to your talk I think here part of part of something that every startup is going to face that every practitioner is going to face here and that is that is there there’s a there is a theoretical purity that you could attain and there is a practical application and anytime you do for the practical application you students or theoretical purity and anytime you step in the world of zero purity you end up with a whole lot of people who don’t understand what

00:20:01;14 – 00:20:17;21
You’re talking about. And so so as you take your work out into the world both of you how do you deal with that. Well you see it because because there are vastly more people who don’t understand what you’re talking about to. Do

00:20:17;28 – 00:20:32;01
Right. Well it’s also what the bar what do you consider that the quality success. I think that a leading product because I love everything I mean is doing and I’m a big supporter of it but I’m not being provocative because why are we here at 4:00.

00:20:32;10 – 00:20:35;27
Let’s get the rocks here.

00:20:36;01 – 00:21:20;04
But I would say I think our bar is very different right. My bar is. Can I help influence a hiring process so that it creates economic value for the organization a better economic opportunity for the candidate and is less biased than the process has been used today. And anything that moves in that step in the right direction is good. And if imperfect process gets people saying you’re right we’re all higher with our gut. This is ridiculous. I hire athletes. You hire people who work or you know. Head of the marketing club. And we can get away from that start using data and start holding data back up to us and start moving the needle slightly better than it was yesterday.

00:21:20;04 – 00:21:25;05
That’s my threshold to get over that’s probably different than us.

00:21:25;06 – 00:23:56;11
Well I mean communicating about what we do is definitely challenging with some groups of people and straightforward with other US as always. So I’m trying to. You know do to communicate with those that understand the way we talk directly and then work with communication experts to help me translate some of our technical verbiage into something that is digestible for the other people in terms of stunned that I’ve been actually my dream is to create capabilities that will help people like you like cordwood with companies like Horowitz and many others here too without the need of getting into all these details but to be able to use this capabilities for a more accurate predictions as our CEO also said today. One of the things we work on and that again differentiates us because we are a large and not for profit company. We want to be very transparent. So what we do we will publish our methodologies. We don’t propose black boxes with John Dowd you Well just trust me because I’m Lena. What we have tried to do is to write papers that are peer reviewed and that published out there. And once we come to any of you with a commendation and that capability we can say well check it out. They are out there and you can form yourself an opinion. But we also try to be very careful at building interfaces. So we work with you designers so that even if Al Gore is this really complex they use up doesn’t need to know that that is what they’ve set up to second order in order to use that capability. So we want to make it as self-contained as possible. Our dream is actually to become part of the people solutions because we realize that what we do is I don’t know what else can I compare with. Take your phone for example I have seen it. It’s the same thing. You real the the algorithm for Siri to speak with you but you don’t need to know the mathematics behind that algorithm. So it’s exactly for us. We want to create those. I’ll go and give it to companies like us and work with any one of you will be your first testers.

00:23:56;15 – 00:24:04;11
Wonderful let’s say and I would say to your point on your design I know John this is an area of a lot a lot of passion around too.

00:24:04;13 – 00:25:29;02
I think we learned very early that year that a black box doesn’t help a process where humans are engaged. So if we could outsource the decision a hundred percent of black box is fine. Let’s run it see whose black box is better at the end of the day we pick the one that performs better. These are human processes and most things that apply machine learning AI it’s going to be the human cost the machine that beats the machine alone or the human element. And so that means it’s a it’s a human it’s a human computer interface challenge. And so we will even even ship less predictive models because they’re easier to understand. So just saying Josh is maybe two it’s not helpful. I need to be able to unpack the black box and say because remember we did the research six variables were most important. And Josh is is hitting four of those six as potential weaknesses on the remaining two. And then you continue to follow that down all those weaknesses. Here’s how to probe those in an interview kind of close the loop and meet that decision maker where they are. We like to say is that our goal is to help make these people better deciders not ever make a decision for them. And that’s a high bar when you have a distributed process moving very quickly a recruiting team is a it’s more like more like a factory floor than you know than a job shop.

00:25:29;05 – 00:25:32;22
And so it’s a big big design challenge. Absolutely.

00:25:32;22 – 00:25:48;20
And like you said it’s going to the data is going to drift further and further to the background when you log into Netflix to get your recommendation feed. You don’t see it a oh here’s you know here’s Jumanji. It’s got a thirty two point seventy five three to fit with you.

00:25:48;20 – 00:25:55;09
It says Hey because your kid watched something else you know we recommend you much.

00:25:55;23 – 00:26:09;22
So so assessment has almost universal acceptance of the education market in industry. It hovers somewhere around 40 or 50 percent and the rest of the people who do

00:26:10;10 – 00:26:19;20
Use assessment think that it’s food. What do you suppose explains the difference between the two worlds. It’s more than just that.

00:26:20;09 – 00:26:25;26
Well there’s a couple of things I think. One is there’s legal ramifications. We

00:26:25;26 – 00:26:52;16
We have a set of employment laws that are very clear about protecting employment rights and they’re actually stiffer on the front end in many ways than firing on the back end. And so in places like the EU and Australia other places where it’s much more difficult to get rid of employees that aren’t a good fit and lower bar in terms of the protections on the front end. You see a lot more assessment adoption because the cost of a bad hire you’re stuck with it.

00:26:52;22 – 00:27:33;03
I think that’s one I think the candidate experience is another one particularly as we get much more in passive candidates and linked in. I’m reaching out to you. Everybody’s kind of constantly in this search like you know 80 percent of a candidate at 80 percent of employees are. You know. Of open and available the idea of throwing up a old school assessment makes you feel like you’re back at high school. The bubble chart a number two pencil is pretty off putting. It’s really to create a win win experience and then I think there’s a sense of I know what I need. It’s the hippo versus the geek. I’m the expert. I’m a higher expert for my team.

00:27:33;10 – 00:27:43;04
And so yeah we should reduce bias it’s all time. Hey I’m really glad we’re working this project will help reduce bias. Yeah all those harmony is their bias. You should definitely work with them right. They need to figure it out.

00:27:43;04 – 00:27:48;05
No not me right. So anything you’re implicitly telling somebody they’re not good at their job.

00:27:48;08 – 00:29:36;06
There’s resistance and I agree with everything you said. I just want to point out a few examples where the assessment is actually used and you wouldn’t want it any other way. So think about. The case that just happened yesterday the pilot. Who. Brought the plane in you know under those circumstances just saved the life of all those people that were left there. So in order for her to become a pilot she had to be tested and you wouldn’t want it any other way. You would want your pilots to be tested. And you want to feel comfortable that they passed. Similarly you want your medical doctors to be tested. And they are a lot. Not only that they are tested once they start but they are tested regularly. I know every five years every eight year. Again you would not want it any other way when you go out to choose your doctor you would rather want to know that those doctors have been certified. Similarly with other men the military if you got into the military you need to pass a test. It’s actually one of the oldest tests we have in the country is to ask for the military entrance test. And I can come up with other examples but I guess I already communicated that there are jobs that have a really high. Level of scrutiny. And that of course the legal requirements are attached to that. But not only the legal one is just a professional level that they require is so high and it needs so much precision that you. Do expect them to be to be tested and you want people to be certified.

00:29:36;13 – 00:29:45;20
So what do you think John we observe this space. Let’s see what else did we miss. Sure there’s a couple of things that you would you have quietly accounted for the 50 percent of adoption

00:29:46;29 – 00:29:58;24
In places where the lawyers will get you when the plane falls out of the sky. If the if the welder isn’t certified properly that’s all. That’s all very well done.

00:29:58;29 – 00:30:28;21
When you’re talking about people in non-life threatening non safety situations the process of installing assessment means that the organization has to make a mindset change and it has to adopt a way of thinking about people and the language associated with that way of thinking about people. And that is an extraordinarily difficult organizational transformation to make happen. It is by leaps and bounds.

00:30:28;28 – 00:30:41;06
The biggest failure point in the expansion of assessment as a tool because assessment isn’t just a test assessment is a framework.

00:30:41;06 – 00:30:56;03
It’s a way of talking about things. It’s a way of thinking about things. And if you and I agree on this and Josh doesn’t understand what we’re talking about the assessment adopted because Josh is going to perpetuate it if you say that

00:30:56;14 – 00:31:26;29
Grid is the thing in our organization and you don’t evolve processes to cultivate and reward and incent that and really make the assessment system stick it doesn’t it doesn’t. So it builds off the urges and it can be perfectly wonderful science but perfectly wonderful science that isn’t designed to be held by an organization with its organizational processes so it fails.

00:31:27;17 – 00:32:24;15
I think it’s a good point. There are jobs for which probably we the interview that’s not quite qualified that’s an assessment. Of. It. It probably also depends on the type of job. Even for a company like ours if you hire a scientist for example I’m sure you don’t put that person to a test with multiple choice but you respect that person to give a presentation and that presentation is being rated so it is in a way a performance assessment if you want at least at the research level. So for other jobs probably where you don’t have a lot of risk you you can be content with recommendations and assume. But it depends on how. I still believe that it depends on how much. Responsibility there will be on that person and how much how. What type of legal consequences will be sure.

00:32:24;18 – 00:33:02;25
I think many of the people who were in this room during storybooks are concerned about organizational fit did organizational fit sort of transcends those arguments. It’s the assertion that you can somehow without perpetuating discriminatory biases figure out what makes somebody stick in an organization versus what what doesn’t stick inside of an organization. So it’s less of a I think that’s the heart of what you’re doing is trying to. Make sense about whether or not I’m going to get fired two weeks into the job because I don’t belong there.

00:33:03;03 – 00:33:04;14
Yeah I think that’s right.

00:33:04;14 – 00:33:12;06
I think you’re right on both those issues. One being that culture fit is better a shield for

00:33:12;27 – 00:33:21;29
This person is not like me. And hurting our diversity inclusion and belonging work.

00:33:21;29 – 00:33:34;08
And so I think that we try and tie these things not as abstract constructs of how people are or they like me but instead do they demonstrate the behaviors that are required to succeed in the job.

00:33:34;11 – 00:34:02;09
So while you know teamwork is we’d like people who play nice with others it’s like what is like this specific action the observable behavior that this person may have done at a previous job that they will they will report being good at. How do they resolve conflict on a team. Do they draw others out into a conversation. Are they the person that gets asked to present findings. Right. Those are tied to the job itself.

00:34:02;09 – 00:34:31;26
And I think if you if you hear people talk about cultural fit things and you can’t map them to bullet points on the job description you’re at this abstract level which is at risk of staying up on the big letters on the wall or letting itself go down into a culture fit place. We actually get people talking about values of it. Which is a different way to try and get beyond this black box of culture and in hiring scientists. I’m definitely more

00:34:32;18 – 00:35:30;13
Focused on value fit. And then. By any means fit they find a different way. I was wondering what do you think about transplant because in my job as a manager of sorts of lead that often research is set really high end such as. I was often in the position of hiring people from all different disciplines and bringing them into education. Brilliant people and with many of them after many many conversation I still had the impression that they could then make the transfer form. Even in terms of start these things. Talking about the data on the models instead of talking with the application in the same methodology and talking about the application into the learning process. So many people don’t seem to be able to make that. Jump. I was wondering that

00:35:30;19 – 00:35:36;06
Does coral look at transplanting or do you have other body stuff that get a transplant.

00:35:36;16 – 00:37:16;21
Well I think we would. We would want to see that show up in some of the variables that are correlated with success. Sort of I think that the data would tell us is only three things that predicts someone’s success a job correctly or wrong as you probably know that’s a much better idea. But what is cognitive ability. Right. There’s good connection at least to a point. And for a lot of jobs to his technical skills you do have the hard skills the stats or whatever. The third is the all else bucket right. We call those impact skill soft skills personnel whatever those things are. And so the question to be if somebody is not making that transfer. What is it. Was it A. Was it a horsepower issue was it the SATs or there’s there’s other areas. So. We’re looking for proxies for our assessment is really trying a proxy. It says it’s you know how close can you get to direct assessment is still a close proxy as opposed to a far proxy. And so if people been successful there in the past have all worked in education before right or having worked in education increase the probability that being a fit here we should see that come through in the bio data which is things we haven’t talked much about is this Let the machine find the find the correlation without any research based explanatory power. And I think we try to combine both of those because to try to take advantage of the big data sets but not just say that know left handed people that post on Facebook on Tuesday nights are better researchers so therefore that’s the profile. So are they right. I don’t know. I think I think there’s that would be one way that we do either of you work with a variable that is sort of the capacity to drop one frame of reference for. The

00:37:16;24 – 00:37:30;04
Reference one of the things I know in this transfer process is that all organizations have a point of view and they me harness the same technical skills and really harness the same softer skills.

00:37:30;04 – 00:37:47;08
But that point of view may have customers first it may have employees first it may have the value of cost savings for us but there’s a whole range of things inside of the inside of the new environment that are different from the old environment.

00:37:47;09 – 00:38:05;02
That’s where you get the hiccup. You get people who can’t make the shift for maybe genetic reasons and maybe Shorten should be the worst firing was too brutal. You don’t really know. You do have variables that address that.

00:38:05;09 – 00:38:06;28
They do change.

00:38:07;19 – 00:38:38;22
So yeah. So the two that are relevant here in the literature. One is openness to change. So that is that willingness to listen to different points of view being OK. You know who moved my cheese. You know that that’s not going to totally derail you. And then the others quick learner. If you have those two attributes that idea of kind of learning agility the willingness to take on a new challenge. To have your preconceived notions challenged and change them and then quickly adapt.

00:38:38;23 – 00:38:53;19
So I was on a panel with a woman from Cisco a couple months ago and she said learning agility is the new smart. It’s not about horsepower it’s how quickly you can take in these new things. So those are two of these things that we look at that are connected. Yeah

00:38:54;15 – 00:39:19;28
I mean I definitely is is a challenge for many companies. Everyone is talking about change and we are changing all types of things around us and we expect the employees to notice to actually hear that. That is why it change in the way they work. And then the skills that they need to build there is an expectation that people will do that quickly.

00:39:19;28 – 00:41:27;26
And I believe it’s hard for some. I haven’t quite figured out why. I mean I don’t I really don’t think maybe it’s a different construct. It’s not pure ability. Maybe this this agility of learning that. Stands out as a separate construct in my case I was actually talking about something that’s even more difficult. I was looking for people who came from other disciplines such as physics particularly those who studied physics have been exposed to modelling very complex large datasets and working with dynamic models and working in within this house theory. So where do you you know you develop a model that may change its state and my not my never converge to. And that can leave state and that type of thinking and modeling was important is still important I believe for what we are trying to do with learning because the learning process sometimes is completely unsuccessful and we need to figure out what external factors to come and interact and destabilize that process. So I’ve been looking for people with B HD physics to come and work with me in application to start modelling this type of problems and many of them had difficulties making the transfer between the data that they were using for physical experiments or from astronomy into the data that would come out from application as text for example from I know from the essays or as state in intelligent tutoring system but as a mathematician if you stand back and you just write it as random variable it shouldn’t be such a big deal. I thought well it was I was wrong. So for many people that was a problem. So it turned out that in order to hire. I and I ended up hiring really really brilliant very good people.

00:41:27;29 – 00:42:39;04
But I spent as a hiring manager I spent a lot of time in pre hiring screening that were pretty much working on the board with them and it was nothing that if you would read in any book about how you’re hired at this level you don’t you can’t ever find that that you would actually view on working with them on the board like a school. But that’s what we ended up doing. And it turned out that a number of our side put for hiring at that level was an incredible amount of time so I would love to hear more from companies like yours that would help me hiring at that level. I don’t think it’s something we can make automatic at this point. I think it’s one of those places where artificial intelligence and human intelligence you know can Why that official one can. Quite compete with what the human brain should table. But that would be something I would really love to. See from code all that. Up high and helping the hiring manager hiring up high end expertise put on the product please.

00:42:39;29 – 00:42:41;14
Let’s do a last word.

00:42:41;18 – 00:43:44;17
One of the things that concerns me about about the ways machines are starting to control standards for hiring. When I include this is a broad category of standards for hiring. Is it the living rates the kind of serendipity that actually makes organizations work so. So if you say here is this profile and you gotta hit this profile what you do is just establish a bar. That eliminates in practice the possibility of somebody who doesn’t fit the criteria. Coming in and excelling right and that is. Maybe a primary source of innovation in organizations. It’s certainly a primary source of learning inside of inside of innovations and so how do you how do you prevent. Good assessment testing from becoming a kind of a lead beyond which you can’t take anything yeah.

00:43:44;21 – 00:44:22;26
I think there’s really good research on positive deviants they they are deviant from the way they about the operating system in the organization but for good. It’s like the Robin Hood you know. In the company. So it’s something it’s something we think about we get questions a lot on that and I don’t have a perfect answer. But I’ll say a few things that we’ve done there is one is there’s more than one way to get a high school. Right. There is not a singular you fit this you get a good score set. As I said May 6 or maybe 8 and 10 twelve variables and very few people are actually hitting the bet on all 12.

00:44:22;29 – 00:45:23;24
So that you know there’s multiple ways being eighty two is we’ve actually built something into our candidate profiles this is why is this person unique. So it actually calls out things that are rare among the dataset. This person is a triple major and then hears of this three majors. This person than ninety ninth percentile in communication polish and then nobody was ever taken this test. So we try and flag that as a way to catch the attention. But wait a second I’m going to put them on this team where the whole thing is they’ve got to stand up and present the findings if they can’t be believed that authentic that no one’s going to take them. So I think that’s you know I think the other thing is that we also say look the assessment needs to be guardrail in terms of how much of the decision it’s taking on. Sometimes people have said you should never use an assessment for more than a third of the data in your decision making. So it it’s true you give people a tool and yet be careful how they’re going to they’re going to wield it. I absolutely concur with everything that

00:45:25;29 – 00:45:30;08
I find on this one. I think so.

00:45:30;09 – 00:45:35;11
Let me you a you which is which is where we’re in this. He wants us to fight normally.

00:45:36;09 – 00:45:40;19
And we are learning every day. What we knew yesterday was wrong

00:45:41;13 – 00:45:54;29
Copying assumptions that don’t make it so. So what do you think the odds are. That there’s going to be a variable that pops out of the data other the big data flow that upsets your apple

00:45:55;29 – 00:45:58;13
Pie. It’s very likely.

00:45:58;19 – 00:46:15;26
I think it’s quite likely especially now with the help of technology if we are indeed able to observe people in action. I think we have learned much more about them. I mean sure the privacy and legal aspect might intervene and control a bit what they could do.

00:46:15;26 – 00:46:23;29
But definitely I think that it’s a good thing that we will discover a new new feature.

00:46:23;29 – 00:47:23;29
I mean even this one that we’ve been talking about. I still hope that we would be able to find what would be a good proxy to transfer what is a good proxy for creative thinking and I didn’t like what you said about the positive deviants because this is one of the scoring for creativity. So you want off the stand the test for creative thinking if you will ask people to come up with a number of solutions for using say one boring object that frankly is very common and then you would look to see what people came up with and the ones that have this positive deviance. So that has that hasn’t showed up many times since they’re out there by itself. That is all fun. See that when I was. Getting a higher weight for scoring creative creative thinking. So maybe transfer is part of creative thinking. So while. I’m sure that we will have four responses to some of these questions in the next year years with that technology.

00:47:24;17 – 00:47:45;12
So I think the other thing is that there’s a lot of inputs that a lot of outcomes. What that means is it creates multiple sources of truth and that’s really confusing and destabilising. So nobody read the article weeks ago. A couple months ago now where somebody said their DNA to twenty three me. They said it’s like all the five major ones and they got back like you’re twenty five percent Native American and the other was like you’re twenty five yet.

00:47:45;17 – 00:47:50;13
So the results were totally different. So as long as you only ask one of those you feel confident.

00:47:50;13 – 00:48:07;25
But as soon as you have you send your DNA to two places you feel like you have no idea. And so it’s the same kind of thing you look at the day is it great. What are the predictors of retention. Here’s the that’s the surgery. What are the pretenses of hitting quota on a sales team. Oh wait those are different. And then the next cohort next year. So what what predicts quota next year. Oh what’s.

00:48:07;25 – 00:48:40;08
Well that’s different too. So I think one of the things is we’ve been playing with this how do you how do you smooth the data. How do you create a rolling average. So I think we like stock price. It changes a lot and that sort of destabilising we like what’s the trend. And so rather than how do you add the data in and look at it and frequent enough standpoints so that you see the evolution so that you don’t have these giant shocks where you say yeah we added you know 100 more records and things have started to merge a thousand more records and so you know you don’t

00:48:43;05 – 00:48:48;05
Close your eyes for three years and then things merge and then you realize you screwed yourself last year.

00:48:48;22 – 00:48:51;17
So thanks. Thanks. I’m going to take my

00:48:52;16 – 00:49:14;24
Call with information to try to summarize the thing that makes me happiest about this conversation. is we we’ve talked about assessment in the workplace for an hour and nobody claimed to eliminate bias, and that makes me happier than you could ever know because that is that is a singularly nonsensical thing that’s been percolating around the environment, you cannot do that.

00:49:15;14 – 00:49:29;22 | JJ
Which you can never do. You can reduce it.

JS You can mitigate it you can reduce it you can do all sorts of things but human beings without bias are not interesrting.

They’re tables. JJ

JS They’re tables, that’s exactly right. They’re furniture. So, So I want to thank you for taking the time to do this if you would join me in (applause). Thank you. Thank you. It was wonderful. Thank you. That was

JS Thanks for coming in. If you are in the business if you would give me your card that would be phenomenal. We will get you into the report.


Watch more of John’s talks on his YouTube Channel »


Tagged with:  
Read previous post:
HRExaminer Radio Executive Conversations Badge Podcast Logo
HRExaminer Radio – Executive Conversations: Episode #317: Bonnie Tinder, Founder & CEO at Raven Intel

John Sumser speaks with Bonnie Tinder, the founder & CEO at Raven Intel. She is a 20+ year veteran of...