This is how AI influences assessment scores

The world has changed dramatically because of AI; we use AI for all kinds of applications. Ixly asked ChatGPT how AI affects assessments.

How will ChatGPT influence the practice of selection assessments?

ChatGPT: “ChatGPT can influence the practice of conducting assessments for selection in various ways. For example, it can be used to quickly and efficiently screen a large number of applicants based on their responses to standardized questions. This allows recruiters to make faster decisions about which candidates are suitable for further selection. However, it's important to realize that ChatGPT can only generate responses based on the data it was trained with and isn't always able to consider subjective or contextual factors. That's why recruiters should always perform human evaluations to verify the accuracy of ChatGPT's responses.”

Will ChatGPT cheating and fraud in intelligence tests play a role in screening applicants?

“It is possible that cheating and fraud using ChatGPT in intelligence tests will play a role in the practice of screening applicants. Since ChatGPT is based on artificial intelligence and large amounts of data, it is able to generate responses that can be manipulative in one way or another. As a result, it can be used to fraud during intelligence tests. That's why it's important to remember that the results of these tests should not only be based on ChatGPT responses, but should always be combined with human evaluations and other assessment methods. It is advisable to take adequate measures to prevent cheating and fraud, such as implementing advanced anti-fraud technologies and conducting regular audits,” says ChatGPT.

What are effective ways to prevent cheating and fraud in assessments?

ChatGPT: “There are several effective ways to prevent cheating and fraud in selection assessments:

• Anti-fraud technologies: This includes, for example, advanced technologies such as proctoring, where the subject is monitored while taking the test.
• Encryption and Security: Provide highly secure systems that can detect and prevent fraudulent activity.
• Human evaluations: Perform regular human evaluations, for example by psychometric experts, to review and verify the results of the assessments.
• Regular audits: Perform regular audits to ensure that the processes and systems used for selection assessments are reliable and effective.
• Education and awareness: Provide training and training to recruiters, assessors and candidates about the implications of cheating and fraud and how to deal with them.
  
  It's important to realize that no solution is 100% effective and that a combination of different tools is needed to prevent cheating and fraud in pre-selection assessments.”
  

Which statistical measures can provide insight into the extent of fraud and impression management?

“There are various statistical measures that can provide insight into the extent of fraud and impression management in assessments:

• Statistical anomalies: If certain scores or answers differ significantly from the expected results, this may indicate fraud or impression management.
• Comparison with standard groups: By comparing the test subject's results with those of a standard group, it can be determined whether the results are abnormal and whether there is fraud or impression management.
• Respondent behavior: The subject's behavior while taking the test, such as the number of attempts to change answers, may indicate impression management.
• Respondent validity: Specific questions may also be included in the test that aim to obtain information about the subject's real intentions and motivations.
• Multiple-choice analysis: In multiple-choice questions, the subject's response patterns can be analyzed to determine whether there is impression management.

It is important to realize that these measures are only clues and that it is necessary to combine them with other methods, such as human evaluations and audits, to get a more complete picture of the level of fraud and impression management,” ChatGPT replies.

Conclusions and insights generated by human intelligence

Of course, AI is a great invention! A chatbot with self-insight, where can you find that? And this one will be even better and will be able to do more. It does have an impact on the practice of selection assessment. Within the Ilxjy R&D team, we arrive at the following insights.

The end of taking knowledge tests online without observation?
With the availability of these types of resources, you can no longer take knowledge tests online. The temptation to take a peek with ChatGPT is just too great. All of that will have to be under observation again. This is possible with online proctoring, but we actually think we can only do this under observation. It's too tempting to hide a cell phone under your table and search for answers. We are also seeing a revival of oral exams and written exams without 'devices'. If you don't want or are unable to organize them, you could use physical exam centers, such as Lamark's.

And what about intelligence tests?
Fraud already occurs when taking intelligence tests online, for example by letting someone else watch. Proctoring is actually necessary, online or live. There is a lot of resistance among candidates and clients against online proctoring, because it is rather “intrusive”. After all, you are observing someone in their home situation.

We work with an Approve test, a retest based on someone's results of a test completed at home. When you use the Approve test and communicate it clearly, we see limited fraud attempts. And then people will certainly slip up when they are fished out for a complete retest under observation.

Does the same apply to personality tests?
Personality tests are less affected by AI applications. In addition, impression management has of course always played a role in questionnaires. Observation has no meaning here, because there are no right or wrong answers. You will be asked to paint a picture of yourself.

Statistical measures for detecting impression management
As ChatGPT points out, and has found in everything it has read, there are some measures we can use to detect fraud, impression management, and “normal” social desirability. Person-fit statistics offer that possibility. Smart combinations of ipsative and adaptive tests will also provide insight. We are experimenting with AI to detect these in response patterns. Here, we also take into account the time it takes to fill out a question. However, these statistics are partly context-specific and we will therefore have to develop partly per client. What is socially desirable in one context may not be socially desirable in another context.

In short, it's good to ask yourself how you want to deal with this. In the test structure, Ixly takes into account the influence of AI and runs pilots with fraud detection.

‍