AI Literacy: Why Confidence ≠ Capability in Skill Assessment

Most of the engineering leaders I talk to believe they have a pretty good read on their people. What Dustin Clinard has found, repeatedly, is that they don’t — and the gap isn’t random. It’s systematic. The data shows people overestimate their communication skills and underestimate their analytical thinking, almost every time. The leaders who are most confident about their self-awareness tend to score the worst.

Dustin is the CEO of Ignis AI, a platform that produces verified, confidence-weighted skill profiles based on scenario-based observation rather than self-report. The research roots run through Harvard, MIT, Microsoft, and the OECD. He’s spent close to two decades watching organizations make resourcing decisions on data that was never reliable in the first place.

  • 31 min

AI Literacy: Why Confidence ≠ Capability in Skill Assessment

Keep Moving Forward

Play

Your Data Disagrees With You

Dustin’s platform doesn’t ask people how good they are. It observes them. And what it keeps finding is that the people who rate themselves highest on communication are frequently the ones scoring lowest. The people who think they’re weak analytically tend to be stronger than they believe. Self-assessment doesn’t reveal skill level — it reveals confidence. Those two things don’t just differ; they often point in opposite directions.

The senior leader who runs a billion-dollar business and assumes that means they communicate well — Dustin sees this constantly. “Maybe you run a billion dollar business for different reasons other than your communication skills,” he told me. It’s not a comfortable conversation. But once the data walks them through it, most agree.

The Measurement Problem That’s Been There the Whole Time

L&D leaders have known for years that they couldn’t prove their programs were working. What Dustin argues is that this wasn’t a prioritization failure — it was a tooling problem. Observing how someone actually thinks and communicates, at scale, required human raters. Lots of them. So organizations substituted self-assessments and peer reviews, ran a snapshot before a training program and another one after, and called it measurement. It wasn’t.

The urgency to fix it also wasn’t there when skills were built to last a decade. Now they go stale in months. “The rate of skill decay changes the ROI calculation entirely,” Dustin told me. The organizations still running pre- and post-training snapshots are making investment decisions on data that’s outdated before anyone acts on it.

Treat Skill Like a Live Variable

The shift Dustin kept pushing toward is a staffing question as much as a development one. When project teams get assembled based purely on what someone has done, organizations consistently pass over people who are close to a skill threshold — people who, with the right environment, would outperform someone who’s already been there. “This isn’t some soft HR question,” he said. “This is a resource allocation question that a lot of engineering teams are still not getting right.”

What Ignis AI is building is visibility into that gap — not as a performance management tool, but as a way to make better decisions about where people go and what they’re actually capable of next. The organizations that get that right aren’t just building better teams. They’re making smarter bets.

Transcript

Dustin Clinard:

Don't treat skill like it's a fixed input. Treat it like it's a variable. It's a live variable and it can change. So when you think about project staffing or capacity planning or client resourcing or whatever your business happens to be, don't use what people have done as the skill.

I think you need to factor in what people have done. But you have to factor in what people are capable of.

Gemma Versace:

Hey everyone, and welcome to Keep Moving Forward, the podcast from X-Team for tech professionals who are passionate about growth, leadership, and innovation. I'm your host, Gemma Versace, Chief Client Officer at X-Team. In every episode, we sit down with leaders who are redefining how technology teams work, grow, and lead. People who understand that performance begins with connection.

Now that AI is handling so much of the technical load, what does the excellent human contribution actually look like? We talk a lot about headcount, output, and tools. We talk far less about the judgment, influence, and systems thinking that no model can replicate.

Today's guest is Dustin Clinard, CEO of Ignis AI. At Ignis AI, he's building a platform that uses multimodal AI and natural language processing to produce verified skill profiles for individuals and teams. Not based on hard skills or self-reported data, but on the power skills: communication, analytical thinking, creative thinking, collaboration, and AI fluency.

In this conversation, we get into what the data actually shows when you compare how people think they communicate versus how they actually do, why L&D budgets have been so hard to justify, and what genuine AI fluency looks like beyond just using the tools.

Let's get started.

Well, welcome Dustin. Thank you so much for joining us today here at Keep Moving Forward.

Dustin Clinard:

I love it. Nice to, nice to see you, Gemma. Thanks for having me.

Gemma Versace:

Yeah, likewise. We always love to start off with asking our guests to tell us a little bit about who you are and the work that you do.

Dustin Clinard:

I'm Dustin Clinard. I'm the CEO of Ignis AI. My background is engineering HR tech, HR leadership. What we're building here is a platform to help intelligently understand human capabilities of people and teams. We combine kind of a multimodal AI model with some NLP architecture to produce verified confidence-weighted skill profiles for people.

So if you think about human capabilities as things like the ability to communicate, the ability to reason, the ability to understand and influence, we're measuring things called power skills as a skill, meaning it's something that can improve if you focus on it. And so our whole business is built around helping to improve these human capabilities or power skills of people in businesses.

Gemma Versace:

Fantastic. It seems like a very interesting development and journey that you guys are on so far. AI is eating routine technical work pretty fast, and what does that actually leave for human contributors or engineering teams? And are most organizations even thinking clearly about what that means yet? What's your thoughts on that, Dustin?

Dustin Clinard:

I agree with you. I think AI, you know, it's moving so fast and I think it's exposing some issues that have always mattered. When it comes to human contributions, the challenge has been, they've always been very difficult to measure. You know, AI is really good at pattern matching and speed. It's not so good at navigating ambiguity. If you have to hold kind of ethical tensions when you're making decisions. Reading a room, knowing when to push back, you know, on a product decision versus a ship decision.

So what's left for humans in this case is judgment, influence. You know, systems thinking, like the ability for us to use our brains to make these consequential calls that aren't so logical. And I think to answer your other question, I don't think organizations are thinking that clearly about this yet, because what we hear is, you know, how do I use AI with fewer people?

I mean, you can see that happening in the news. What we should be framing this around is, you know, what does the excellent human contribution look like in a world where AI handles so much of the rest? I think those are very different questions.

Gemma Versace:

Yeah, absolutely. And I think you hit the nail on the head there too, that there's a lot of — we've seen in the news recently, over the last weeks and months, so many businesses using AI as the rationale or the main contributor for layoffs and reduction in staff, rather than kind of highlighting potential bloat that has come within the business.

But I agree with you as well that, you know, you want to be able to have the human in the loop, being able to be the value add and the force multiplier, with all of the different AI tools at their disposal as well.

You've built an AI-enabled assessment platform, as you gave us a bit of an intro at the start, and it observes how people actually think and communicate rather than just asking them to rate themselves, and moves beyond the technical skills that workers might possess. What is the most surprising thing that gap reveals when you compare what people think they're capable of versus what the data shows? Because obviously people have an understanding in their own mind, or they have their own truth as to how they believe their capabilities and skill set is. What's the most surprising thing that the gap does kind of reveal?

Dustin Clinard:

Yeah. Yeah. I think self-aware leadership is one of those things that is often very difficult to really find. People, from the results that we've seen, consistently overestimate things like communication skills and they underestimate analytical thinking. That's what people would say. I'm a really good communicator, I could use a little bit of work on the kind of the logical stuff.

When we observe people in practice, which we do as part of our validation approach, we see the opposite. People who tend to think of themselves as great communicators are not that great. They're scoring — I won't give numbers — they're scoring lower on our assessment than you think. And then we're validating that with observation. Same thing on the analytical thinking side. People who think they're not so good tend to be better than they think. And from an observation standpoint, they tend to be better than they've indicated.

I think what it tells you is the self-assessment model is failing us. The ability for people to self-assess measures confidence and maybe self-awareness, but not capability. We have a version of what we do that's for senior leaders, and a lot of our senior leaders that have taken it say things like, of course, I'm a good communicator, I run a billion dollar business, and then they score poorly. And it causes this interesting discussion, which is, maybe you run a billion dollar business for different reasons other than your communication skills. There are other factors that lead to success. If you parse this one apart, the skill level's not so great.

And when you break it down into some of the, we usually get agreement from that point forward. It's not what you would've come in saying. You would've come in saying the opposite. So there are some really interesting counterintuitive results that come out of this.

Gemma Versace:

Yeah, I can imagine there would be some fascinating results. And also that whole human element of how people perceive themselves versus actual capability, and then also how others perceive them, is a really interesting point too, that no doubt you and your team have some fascinating results that you'd be able to see and dive into.

At X-Team we have a 98% developer retention rate, which is pretty enviable for our industry. And we would argue that culture and engagement are really the engine behind that, that it's in our DNA. Where does culture end and measurable skill development begin? And why does that distinction matter to you in the work that you're doing?

Dustin Clinard:

Well, first of all, congratulations. That's an amazing retention rate. I may push back a little bit on the separation between those two. Let me give you a slightly different take. I think culture, as an environment, is an environment where skills, where people feel comfortable letting skills develop and grow and to test.

And if you have a 98% retention rate, what that tells me is that you've built an environment in the context where people feel safe enough to stretch and to try to grow new skills. And it's very measurable through something simple like a retention rate, which is pretty amazing.

I think the distinction between engagement and growth is very different. You can have people who love their job, but they're stuck. And the goal is to make development visible inside culture, not to replace it with some, you know, kind of performance management measure, so that the measurement is serving the person and not just the organization.

So I think your retention rate's high because people feel like they can develop skills in the context of your culture, which I wish everybody would take that seriously.

Gemma Versace:

It's such a, a lot of businesses out there preach about culture being such an integral part, but to really put the time and effort, investment, and, you know, make it a deliberate practice is difficult. And it really, I think, does help separate, you know, businesses that are good from the businesses that are great as well.

Your team has research roots at Harvard, MIT, Microsoft, and the OECD, including the global PISA Creative Thinking Assessment across 70 countries. What did studying creativity and collaboration at that scale teach you about evaluating talent beyond hard skills or self-reported data?

Dustin Clinard:

Our co-founder Al Rosen has been integral, prior — even prior to Ness — in doing a lot of this work. And one thing that's fascinating is that you kind of touched on it: scale forces honesty. When you go at scale and you start looking at country comparisons, the data does not lie. And you can't hide behind these kind of cultural assumptions about what good looks like.

One thing we learned is that creativity and collaboration are genuinely measurable skills, and they're relatively stable across cultures at the construct level, when expressions may differ from culture to culture.

We also found that scenario-based observations dramatically outperform any sort of self-report or even peer ratings. But it's very difficult to observe that many people. I mean, it would take an army. And so what I mean — that's kind of the roots for us — is building something that starts to replicate what an army of observers would look like.

And maybe the last point is that the skills that we're talking about are learnable skills. You know, generally speaking, they're soft skills, but they're not characteristics. They're not genetic traits that you can't get yourself out of. They are learnable skills. And once you believe they're learnable skills, you can learn them in an improving direction. You could learn them, you could decide not to focus on them, they decline. But the point is they move and they're not the same today as they were yesterday, as an example.

Gemma Versace:

Yeah, fantastic. Thanks for that.

Dustin Clinard:

Okay.

Gemma Versace:

AI fluency is one of the seven power skills that you measure. What does genuine AI fluency actually look like in a person? And what does that mean in terms of each employee being productive and effective in a team dynamic?

Dustin Clinard:

That's a fascinating question. We get it a lot right now. There's a lot of interest from companies that we're talking to about measuring AI fluency. There's some that are building it into KPIs and bonus plans. The challenge almost universally is, we don't know how to measure it though.

And if we use an observation or a measure, AI fluency and AI usage get confused. I'm on ChatGPT all the time, therefore I must be AI fluent. Oh, this person moved to Claude, they must be AI fluent. But the folks that are evaluating that often aren't so fluent themselves. And so it becomes this really interesting conundrum for companies that want to measure this, but they don't really know how.

For us it's not about the tools. It's about knowing when to use judgment in solving a problem, and AI as an output mechanism. You know, AI has hallucinations and bias and has overconfidence, but a person, an AI fluent person, can calibrate that accordingly. They can treat it like a brilliant, you know, maybe unreliable partner.

But somebody who is AI fluent is able to use tools as an accelerant without creating additional technical debt. It's not, I don't know what this program was, but it did the thing I wanted. No, I know exactly what problem I'm trying to solve, and I'm looking for creative ways around the edges to get to it, or something more efficient, you know, or something faster.

So I think, you know, we have seven power skills that we look at. AI fluency is one of them, but if you think of the other six — collaboration and creative thinking, things like that — productivity, true AI fluency is a multiplier on the other six. To be able to amplify, you know, communication and analytical thinking when you're doing work.

Gemma Versace:

Well, as a quick side question: of the seven power skills that you measure, just out of curiosity, when did AI fluency become the seventh? Was there always seven? Did it replace something? How long has AI fluency been a really key measurement that businesses have been wanting to measure?

Dustin Clinard:

It's a good question. The inside baseball answer for us is that our company is only a year and a half old, so we are not that old of a business. AI fluency has always been something that we have measured. Interestingly, some of the prior research that's been done over the last several decades — AI, and if you go back to a machine learning construct or an automation construct — there's always been some form of it there, which is measured a lot like the creativity and the communication. Because AI fluency, we have AI fluency as a skill, but you could argue that AI fluency also encompasses several of the skills.

And so they've been here the whole time in some form or another. We are, you know, lucky enough that we're new enough and AI — and the way we think about it now — has been around since we've been around. So it's always been there for us. But some form or another has been there in the past.

Gemma Versace:

Amazing. Amazing. Thank you. L&D leaders are spending an enormous amount on development programs with almost no way to prove those programs are working. How has that problem gone unsolved for so long? And what are you doing at Ness that helps make it solvable now?

Dustin Clinard:

Yeah, I mean, I've been somewhere around the L&D world for the last — probably too close to two decades. And I think there's a couple reasons that we're in the state that we're in.

One, the tools to measure this didn't exist. Frankly, they didn't exist. You could observe and score complex human behavior at scale with a whole bunch of observers, which does not scale. And so you'd need this army of human raters. That therefore turns into, I'll have peer reviews or I'll have self reviews, and then you run into, well, how accurate is that? How frequently can we do that?

So great training has some observation at the beginning, some observation at the end. Did I observe a difference? Did somebody else observe a difference? And then maybe an observation later. What we really need is the observation constantly to see if it changes. And if you're using human observers, you're multiplying. It's time-consuming. And so at some point you stop.

I think the second thing is, the urgency to measure this at the skill level is different when you think of some of the most enigmatic leadership development programs. You come in and you get trained and you're expected to hold those skills for a decade. These are the fundamental skills that are gonna be useful for decades in our business.

With the decay of skills — like technical skills decay — I mean, even that is rapidly increasing. I don't actually know what skill I'm gonna need next month. What I'm doing now is probably gonna be outdated really soon. So I think the rate of skill decay changes the ROI completely on how we're measuring L&D programs.

And what we're doing here — building true ROI — is very company dependent. So we can show that a skill improved as a result, and we can show that your skill individually or on average moved from point A to point B. Where we need the help of our customers is always, what is the observable way that skill improvement is showing up in the business? Is it, you know, commercial pipeline is growing? Is it project development time is dropping? Is it whatever the measure is?

But now we have a skill measurement that corresponds to the business outcome, and we can use it as a proxy, but we can also actually calculate the ROI of that particular training or learning experience or whatever it is. So it's not self-reported evidence. It's truly outcome evidence from that point.

Gemma Versace:

Yeah, fantastic. And that would also help make for a really compelling business case, why people would come to you at Ness to be able to do these types of assessments, because of that level of greater detail that you can help provide and help make sense for them around what the investment in additional training for individuals or teams is going to help drive greater ROI across their business as well, which is quite powerful.

You're measuring skills through scenario-based observation and with the support of NLP in your platform rather than personality tests or self-assessments, as you mentioned. For a technical audience that thinks a lot about signal quality and model reliability, how do you know your scores are actually measuring what you think they're measuring?

Dustin Clinard:

A technical audience would be quite at home with our kind of psychometrician background. The scoring, the validation, the consistency, the bias or the lack of bias — those things are essentially important if you're making any sort of decision.

You know, this sounds like a very simple assessment. I'd like to know my communication skill. But that may inform how L&D resources are directed at you. It may inform how you decide to assemble teams. If you think of breaking down all these skills, what is a great high-performing team in our business look like?

And I think there's a couple, like in our business, that really matters in that context. Are there certain skills that are minimum requirements for people to have to be on a great team? And there are certain skills where we want a big variation and a big diversity of those skills. We want everybody to have a minimum threshold of collaboration, but we want creative thinking to be a little bit all over, combined with analytical thinking and communication. Like, there's some assembly of that that makes the greatest team for us. All that stuff becomes possible if you believe the numbers.

And so from a psychometrician background, we look at the consistency of scores. We have a paper that explains transparently all of this. We look at the consistency of validation responses, which is very strong by academic standards. Our most recent study was 0.82 to 0.91 on variation or consistency, which is really high.

We look at when we did a follow-up study with tech companies, the consistency was in agreement with the human raters. So if you kind of think of what we're doing, you could follow the path of developing the skill framework, creating and validating the questions, having somebody take an assessment, and then taking their open constructed response paragraphs and evaluating — a psychometrician evaluating those and saying, based on my expertise, this is the score. That just takes like two hours, what I just told you.

And so if you're doing this for hundreds and thousands of people, you either need a lot of time or you need a lot of those people, and you don't have a lot of time and there aren't that many of those people. And so what we're measuring is, does this give you a proxy for what a human would do?

And so we have a human in the loop check. We see what the response is, what the score is, what the rationale is, and then the psychometrician can say, I agree with this, the model's working. And you can check that at any point in time. You can check that on everything. You don't need to, once you get belief that it's working. But that's the science. That's the part that is crucial for us to get right.

Gemma Versace:

Perfect. So there's definitely a comfort level there for them to be able to, as you mentioned, believe in the science. Developers often push back hard on being assessed, and there's a real resistance to anything that feels like, you know, a performance review dressed up as a quiz. How do you design for that in a way that really helps to get buy-in from the talent completing the assessments and being a part of the response?

Dustin Clinard:

Are you saying your developers don't like gamified performance reviews?

Gemma Versace:

Maybe. Yes. Yes.

Dustin Clinard:

Good. Good.

Gemma Versace:

We do hear it quite regularly, so yes. Help us fix that, Dustin.

Dustin Clinard:

I mean, the design matters. The design matters enormously. The scenario needs to feel like it's a real situation. It's not some gotcha question. We build a combination — we have simulated videos, we show cross-functional conflicts. There are creative challenges that have real constraints around them like you would in a real business.

We use a combination of selected response, so like a multiple choice — we use that minimally, it's what people are used to. But we use a lot of constructed response, so open text answers. And so it's, you know, your boss wants this, their boss wants this, these two things are very different. You're in a meeting with both of them in a half an hour, how are you gonna frame up this problem? Because you need to get to a solution. And it'll be like, what are you gonna say? Are you gonna say I agree with one? Are you gonna kind of waffle on the edge? Like there's a lot of ways that you can answer that, but that's a real scenario that you're gonna face.

And so the more you can build real scenarios into this, the more people want to figure out how they would deal with it, understand if their response, like, how does that actually score? And what we find is people want to talk to us after they take our assessment. They want an interpretation of how some of their, you know, not so much the scores, but how some of their responses compare. Like, what did good really look like there?

So I think that's the signal that you're doing something that both people care about and businesses will value. And the people caring about it is an important factor to us in our kind of founding roots.

Gemma Versace:

Yeah, definitely. I definitely agree with you that, you know, them wanting to be able to understand what the responses look like, and as you said, what good looks like, kind of gives you the signal that you've got the buy-in from them, which is really positive.

If you could get every engineering leader to change one thing about how they think about human performance on their teams, just one thing, what would that be?

Dustin Clinard:

It's a discussion that comes up a lot for us also. And I think the biggest thing that we hear, and I agree with personally, is don't treat skill like it's a fixed input. Treat it like it's a variable. It's a live variable and it can change.

And so when you think about project staffing or capacity planning or, you know, client resourcing or whatever your business happens to be, don't use what people have done as the skill. I think you need to factor in what people have done. But you have to factor in what people are capable of, and when they're close to being capable of a skill, with the right type of support, are they actually gonna accelerate faster than if they had some fixed skill coming into it?

This isn't some soft HR question. You know, this is a resource allocation question that a lot of engineering teams are still not getting right, because they're treating, this is the skill of this person, this is the skill of that person, we need these skills, therefore we're gonna pattern match it like that. And you're missing out on the capability of people who want to grow something.

And if they have the right motivation and they have enough technical background to be capable of it, you'll see better performance out of a group like that than somebody who's done it all before, whose expertise is in that area and they've done it a lot.

Gemma Versace:

Yeah, that's fantastic insight for those technology leaders and engineering leaders listening as part of our audience to take note. Thank you so much for sharing that.

With each guest, we have reached the end of the podcast today, but we do like to ask each and every guest that joins us here today: what keeps you moving forward, Dustin? What gets you out of bed in the morning to keep you moving forward?

Dustin Clinard:

I've built a lot of proxy measures for training and ROI and HR initiatives, and I truly believe we're at an inflection point where if organizations get this right — if they can get true visibility into the skill level and the potential of their people in an unbiased way, in a measurable way, and something that happens in real time or near real time, at a price that everybody can handle — companies that get that right are gonna have a durable advantage over competitors.

The world is not as black and white as everything is going this way or everything is going that way. I don't fundamentally believe in that in almost any case. The organizations that I get the most excited with are HR teams that say, I have an inherent belief that if we go to human capabilities, if we go to the soft skills, the stuff that we've always known is important but never shows up quite as technically important as some of the technical skills, if we build a team of people whose collaboration ability improves and creative thinking ability improves and AI fluency ability improves and, you know, the other skills we just happen to measure, that's what's gonna drive us to success in the future.

And if our competitors don't focus on that, they will eventually be asking, why do we have these people? What are these people doing? Or the people are gonna get tired and they're gonna leave.

And so I think we're at this interesting point where team dynamics, personal dynamics, company success — like, what are the characteristics of the best leaders in your business? We know what their outputs are. Do we really know why? What went into that? And for somebody to say to us, I've never been able to see this level of measurement, either in myself or in my people, before, because of your approach — that's the thing, that is the holy grail of what we're doing here. So it's really exciting and I think all the tools and tech have enabled us to be at that point right now.

Gemma Versace:

Yeah, fantastic. And for those businesses and engineering leaders and just leaders in general that are listening today, you know, have a look, check it out, because it is something that, you know, when you think about how to really make sure that your employees know their value, know their worth, know just how much you want to continue to invest in them, doing this type of partnership with Ness would, I think, be such a powerful tool as part of your professional development, as part of your reviews.

To be able to give them the insight that not only they can take to their everyday life, but also some of the powerful investment and return on investment that businesses can get by being able to have this level of understanding of their workforce.

So thank you so much for joining us today, Dustin. It's been a fabulous chat and thanks so much for sharing so many wonderful insights with myself and our audience.

Dustin Clinard:

I appreciate being here, Gemma. If anybody's still listening to this, get in touch with me, tell them you're friends with Gemma, and we'll find some way for you to experience the assessment and get your own scores as well. So I really appreciate it, Gemma. Thank you.

Gemma Versace:

Amazing. Everybody, take Dustin up on that. Thank you so, so much. It's been great.

One thing Dustin said that I keep coming back to is the idea that we've been treating skill like a fixed input. We may say that this person has these skills, that person has those skills… and slot them in accordingly. That may be efficient, but it’s the wrong approach.

Skills are live variables. They grow with the right environment and support, and they decline when neither exists. How you staff a project, how you assemble a team, how you invest in someone's development: all of those decisions look different when you stop treating capability as a snapshot.

The data is also more honest than we are about ourselves. Self-assessment measures confidence, not capability. Those are not the same thing.

The organizations that invest in genuine visibility into who their people are and what they're capable of are going to have a real advantage. Not because they deployed the right software, but because they chose to treat people's growth as something worth measuring.

Join us next time for more conversations with technology leaders who inspire us to grow, lead, and innovate. You can find us on Apple Podcasts, Spotify, or YouTube Music. If you enjoyed this episode, please share it with your network. We'll see you next time.