The Big Problem with Using Personality Instruments for Giving Feedback

The Big Problem with Using Personality Instruments for Giving Feedback

Organisational Success Podcast
Personality Instruments
Personality Instruments are they valid for giving performance feedback?

Personality instruments, indicators or tests are used in many organisations and by coaches and consultants around the world for giving feedback to employees and executives about work-related performance and for developmental purposes.

The idea appears simple. Understanding your personality can give you insights into your performance and how you engage with work. But is it true? Can personality indicators be reliably used to give people feedback for work-related performance issues and are there any dangers in using them in this way?

If you or your organisation are using personality indicators or instruments for giving employees work-related feedback, you are going to want to listen to this interview about some just published research.

Personality instruments like the MBTI, Enneagram, Type Finder, DiSC etc. are used routinely for such purposes, but is there any good research evidence to support the use of the findings from such personality tools for giving performance and work-related feedback? Apart from the fact that a number of such personality instruments are considered to be suspect (not valid or reliable from any evidence-based research perspective), what is the evidence for using them for giving people feedback based on any form of personality indicator or test?

Interview with Professor Blake Jelley

In this podcast David Wilkinson, Editor of the Oxford Review, talks with Blake Jelley, Professor of management in the University of Prince Edward Island’s Faculty of Business in Canada. Professor Jelley recently published a paper looking at what evidence there is for using personality instruments for work related performance feedback (reference below).

Podcast

Subscribe here

Professor Blake Jelley

Blake is a professor of management in the University of Prince Edward Island’s Faculty of Business in Canada. He holds a PhD in industrial-organisational psychology from Western University and is a Chartered Professional in Human Resource. He is also a  Fellow of the Center for Evidence-Based Management, a former Chair of the Canadian Society for Industrial and Organizational Psychology (CSIOP), and recipient of the Joan Finegan Award for Outstanding Contributions to the society.

Blake Jelley
Blake Jelley

You might also like:

2 things predict if you will get work burnout: your personality isn’t one of them

Transcript

– Hello, and welcome back. Today we’ve got Professor Blake Jelly. Blake is a Professor of Management at the University Prince Edward Island Faculty of Business. He holds a PhD in Industrial and Organizational Psychology from Western University is a chartered professional in human resources. He’s also a fellow of the Center of Evidence-Based Management, a former chair of the Canadian Society for Industrial and Organizational Psychology and a recipient of the Joan Finegan Award for Outstanding Contributions to Society. Dr. Jelly’s scholarly work exceeds in excess of a hundred publications and presentations, more than 20 of which have appeared in peer-review journals. His interests include things like evidence-based management, management education, the appraisal, prediction and improvement of human performance, and his teaching assignments have included courses in organizational behavior, human resource management, management skills, resource methods, evidence-based management and global leadership and ethics. And if that was not enough, what are the reasons why I’m talking with Blake today is to look at a paper that he published earlier this year in the Canadian Journal of Behavioral Science that was titled, “Using personality feedback for work-related development and performance improvement a rapid evidence assessment.” So welcome Blake.

– Thank you, David.

– Great, pleasure to have you here. Would you just start off by telling us a little bit about your background and what kind of led up to the publication of the paper?

– Certainly and thanks again, David, for inviting me to speak with you. My PhD is in a field that focuses on psychology applied to work, my dissertation involved personality research and during graduate school, I was an intern at a reputable test, psychological test publishing firm. When I worked in an applied research capacity at a Police Training Institution, I encountered at least one personality inventory that despite my background, I had not heard of before it was used by members of our leadership training unit as a development tool, as a Management Professor, I’ve had various discussions over the years with students and organizational stakeholders about personality assessments, especially with people who are concerned about the quality of a particular instrument that their organization was using or the manner in which it was being used. So when I decided to try doing my own Rapid Evidence Assessment as a guide for my MBA and executive MBA students, my curiosity out the developmental effects of what I eventually called Personality Feedback Interventions, or PFIs seemed to provide a good topic to investigate.

– Yes, yes and it is. It’s a very interesting paper. So many organizations use these personality instruments, inventories or indicators. Some people call them tests, having the psychologist jumping out of the windows at the moment for a variety of reasons, kind of including kind of recruiting personnel development and things like that within organizations. But what are we talking about here? What is a personality inventory and why do you think they’ve become so popular?

– Well in my review, I focused on interventions that use data from participants’ self-reports on one or more standardized measures of personality. And when we’re talking about personality we’re referred to individuals tendencies or patterns of thoughts, emotion, and behavior that are kind of consistent or characteristic of them. In coaching, training, or team building contexts, the data are thought to help identify areas for improvement and motivate people to change. You are right though, that organizations use personality inventories for purposes other than employee development. My review wasn’t meant as an attack on Personality Assessment or its thoughtful use by organizations well-developed, job-related, scientifically validated personality measures can compliment other predictors of job performance and be integrated in a valid hiring process. There’s been a resurgence of research in that respect since two influential meta-analysis were published in 1991. My focus was different though. I wanted to examine the evidence that using a personality assessment as part of development program enhance subsequent performance or other indicators of development. It seems like both critics and proponents of personality assessments at work regularly assume that using personality inventories for development is helpful, but he seemed to clash more regularly about using personality inventories for high stakes decision-making maybe part of the reason these tools are popular in development contexts is that people think or assume they’re useful. One of my former colleagues posted a reply to my LinkedIn article on this topic. She used a potato chip analogy that PFIs may be immediately satisfying but have no nutritional value, maybe she’s under something or maybe there is real value in personality feedback in her actions. It’s also possible that the extent to which there are demonstrable benefits might depend on the quality of the instrument used as well as the quality of facilitation,

– The required variety such personaility instruments some of the more famous and widely used instruments like Myers-Briggs or MBTI Disc, Hogan Personality Inventory, and things like that. Can you just comment on the validity and reliability of some of these indicators? Because some of them may not be exactly what people think they are.

– Yeah well, I came to realize that in the development arena, people like me tend to take on the role of psychometrics knob, criticizing lower quality instruments and advocating for practitioners to use higher quality tools. There certainly are differences in terms of the approaches to content and measurement quality of different inventories. So psychometric critiques make sense to a point more recently though, I’ve been arguing that psychometric critiques are insufficient. We need to also examine direct evidence that offering feedback from a particular personality assessment has beneficial effects. So I didn’t devote much space in my review to detailed psychometric critiques of different instruments. Thankfully though Matthew Pruitt and colleagues had published a 2013 chapter in the handbook of “Personality At Work” and in it they reviewed and compared 12 selected popular personality inventories. So I was able to use that source. In my Canadian Journal of Behavioral Science Article, I just grabbed their chapter as a sober well-reasoned guide. Sometimes the language used in this area, it can be quite inflammatory at times, but with respect to the specific instruments you mentioned, I believe they classified the MBTI in their lowest or overall modest evaluations category and the Hogan Personality Inventory in their highest or overall positive evaluations category. If I recall correctly, they wanted to include the Disc in their review, but they couldn’t get a copy of the test manual from the publisher, which prompted them to advise caution in its use until a potential user can get relevant information to evaluate its quality. Certainly not all personality inventories are equally well-developed or validated.

– Yes, and we also find some of the publishers of these indicators are how can we say less forthcoming with their data, particularly to researchers and inquiring people. So the papers primarily about, as you were saying before, about the use of what are known as Personality Feedback Interventions, can you just explain what you mean by that? And what type of interventions are we talking about?

– Sure, so my focus was on workplace related programs that provide feedback to respondents of a Standardized Nonclinical Self-Report Personality Assessment, proponents believe that personality feedback as I mentioned before, can help people identify areas to focus on and energize a desire to change. Typically, the idea seems to be, to increase people’s awareness of their tendencies so they can adapt their behavior in functional ways rather than to change personality per say. But I did learn that personality change is an area that seems to be getting more attention recently, a Personality Feedback Intervention often involves a trained coach or feedback provider, and can be conducted individually in group workshops with other people or with intact teams and organizations, and with the intact teams sharing and comparing results is thought to help team members learn about and appreciate one another similarities and differences to improve communications and relationships. So I started using the term Personality Feedback Intervention, or PFI to help differentiate between this kind of employer personal development program, versus when personality measures are used as predictors and correlated with outcomes of interest.

– And your papers what’s known as a Rapid Evidence Assessment. What does that actually mean? And what purpose do they serve and how useful are they really?

– So advocates for an evidence-based approach to management prefer practice oriented literature reviews that are transparent and were available evidence is considered regardless of whether it supports or contradicts one particular point of view, a so-called systematic reviews that’s the biggest variety they can be quite involved in resource intensive. The Center for Evidence-Based Management has developed guides for abbreviated versions of the systematic review process with Rapid Evidence Assessments as a mid range form. In this case, I restricted my database searches to peer-reviewed journal articles to keep the scope of the project manageable. But I did expand my search beyond the ARI guideline in some other ways. So anyone can find and use my search procedures to critique, replicate, or enhance the search to be more exhaustive. At this point though, I suspect the more valuable research efforts in this area will focus on new studies or sharing existing data that practitioners have gathered, but not yet disseminated. In terms of usefulness a former executive MBA student once told me that during our program, meta-analysis or quantitative syntheses of available evidence had become her favorite reading material, that material, like so she likes when a credible summary of research is available to inform her making. Research syntheses do not always need to involve meta analytic methods or other quantitative summaries to be similarly useful. The basic idea is to make use of the cumulative knowledge base by figuring out what we know and what we don’t know yet. If someone hasn’t done that for you doing or commissioning a Rapid Evidence Assessment can compliment or replace primary data collection to support decision-making, you don’t necessarily have to start from scratch. I like to tell students that there are armies of nerds all over the world that have thought carefully approach problems similar to the ones you face. So finding those insights and considering how they might apply in your context could be a source of competitive advantage.

– I love a good meta-analysis. I like a good lit review even can be very, very useful for kind of scanning across the piece. So many of the personality instruments or personality indicators that we’re actually talking about here that are being used to give feedback to individuals, as well as teams and organizations in a way that suggests that there’s a strong connection between an individual or a team’s personality profile or indicator results and their performance, and is therefore used as the basis for developmental activities. Many such instruments are actively sold and used in organizations on that express basis. And we’ve probably both seen that kind of activity. What did you find with regards to the current research evidence about the validity and reliability of using PFIs in this way?

– Yeah, well, so the publishers have some inventories that are popular in developmental context, sometimes see more guarded in terms of their tools ability to predict performance, and they may even recommend against those tools being used for decision-making purposes, probably because of the heightened scrutiny and technical expectations for decision-making applications. Still, as I mentioned earlier, well-developed, job-related personality skills have been related to important outcomes like job performance ratings, but I’m not sure it would be accurate to describe those relations as strong. The user really needs to consider the available evidence for a given tool in terms of reliability and validity. My point though, is that such evidence is insufficient for PFI evaluation anyway, we need to consider directly what happens as a result of a PFI like that feedback giving process. So I found some indications of possible benefits, like a manager who mitigated some maladaptive tendencies and became clear with expectations and vented anger or less anger toward others. Unfortunately, the research designs use so far do not allow for a clear cause effect conclusions about PFI effectiveness. So for example, the coaching and counseling that manager received may explain the observed improvements. It’s not clear that the PFI was an active ingredient. Likewise PFIs are often paired with multi-source performance feedback, training, or other features that may improve performance. And as a result, make it difficult to find out how well PFIs work or how much value they add to other development interventions. Based on my review, I concluded that so far anyway, the evidence that PFI has improved performance is unclear.

– If fact what I’ve also found interesting in your paper is that you reference someone’s work. I think we both know Rob Briner who’s Director of the Center of Evidence-Based Management and Professor of Organizational psychology in the School of Business and Management to Queen Mary’s University in London. And you study now, Rob’s pretty well known for his views on such instruments. And he stated in a 2018 article, that inaccurate feedback based on a psychometrically inferior tool could be misleading on helpful and even harmful. Can you explain the problems of both using tools that the got a little validity and reliability particularly for this kind of purpose or being used for a purpose or in a context actually hadn’t been tested in? So empirically tested in?

– Yes Rob has been encouraging me to share the results of my review, and he’s helped draw attention to my study. I think the quote you mentioned was from my paper, and hopefully I paraphrased Rob’s concerns appropriately.

– Yes.

– His 2018 article may have been news for some people and a reminder for others that inventory is focused on identifying distinct personality type groups are controversial. Many personality researchers prefer to focus on trait-based inventories in which different characteristics are each measured on a continuum from low to high with most people scoring in the middle. But in terms of PFI evaluation research, I found that neither proponents of type-based nor trait-based approaches to personality assessment have yet offered much in terms of strong evidence that PFIs enhanced performance, the type-based versus trait-based approach may turn out to be an important contingency factor of PFI effectiveness but it’s too early to know. I didn’t review in a systematic way studies of participants’ reactions to PFIs, but some studies of that sort reinforce warnings, like Rob’s about type based tools in terms of negative side effects, such as people feeling confined, or the term is pigeonholed by a type label. More generally PFIs have sometimes given participants the sense that there’s something wrong with them, even though the tool used is not appropriate for clinical diagnosis. Some of the side effects could be due to inappropriate messages from poor facilitators. Some participants find PFIs emotionally taxing and stressful, including if they have privacy concerns related to sharing personality information with others, and also inaccurate feedback from a tool that’s low on reliability and validity if that is the case that may cause people to focus on problems that don’t exist and divert their attention away from activities that are more promising from a developmental perspective, like how they can improve performance on job tasks.

– I’ve seen situations where managers have been using the results from these to make decisions about people and even give them feedback in a way that’s not being easy them to deal with. So what are the implications for this for practitioners? People who are using PFIs in organizations, or as coaches or consultants, for example?

– Well, this had been popular for many years and some users firmly believe in their benefits. So the inconclusive evidence of PFI effectiveness does not necessarily mean they should be stopped, but I think the uncritical assumption that they work well needs to be set aside so we can learn more about them, several of the PFI evaluation pioneers that I cited in my review describe the difficulty of conducting studies in this area. So practitioners should be aware of the limited evidence-base with respect to PFI effectiveness, and hopefully gather or provide access to data in applied contexts so that we can help clarify which tools work for whom under what conditions and for what purposes, one place to start is identifying the outcomes, thought to be improved through a PFI. So what are you hoping to improve by using these tools and measuring associated impact?

– Yeah, and it’s not that easy, particularly in a work context to actually isolate individual areas is it anyway, and certainly you’re going to have to move into something a little bit more empirical for that kind of process to happen, and that hasn’t been done yet. So what advice would you give to anyone using personality indicators for feedback aimed at performance and development poses?

– So it was that the same. So this is for the practitioners or for people?

– Yes yeah what advice would you give to anyone who who’s actually using personality indicators?

– That’s an interesting question, a difficult one for me to answer from the participant’s perspective based on my rapid evidence assessment. So my advice is speculative since I don’t think we know yet how to make PFIs work most effectively still there are some things in my review that I think could be useful for the participants in these programs. So first I learned that the generic positive depiction of self-awareness that comes across in much of the PDFI literature is not shared by self-awareness scholars. They distinguish styles of self-awareness with self-reflection being a positive form, on the other hand, rumination or dwelling on negatives can be detrimental. So being aware of those two different styles and being alert to the fact that you don’t wanna be ruminating too much and reflecting and looking for constructive responses that might be useful. The second major thing is that feedback intervention theory warrants us that self-focused feedback can be detrimental versus feedback more closely aligned with task performance. So putting those two things together it might be helpful for PFI participants to reflect on their tendencies in terms of how to leverage and mitigate weaknesses, but with a clear focus on functional behaviors, such as those that are likely to enhance your performance at work. Likewise, try not to ruminate, as I said before on negativity, especially if it’s coming from a facilitator with suspect qualifications using a tool that may well problematic.

– Yeah and I think there are a few of those. So just to kind of sum up your evidence assessment of the use of PFIs Personality Indicators for performance and development feedback is that there really isn’t enough good research evidence that they actually help, I’ve got that right?

– Yeah exactly. So I think the main thing is that the assumed effectiveness of PFIs needs to be investigated so we can learn about whether they work or how they could be made to work most effectively. I also think that the popularity of the practice combined with some intriguing theoretical issues makes the scenario where practitioners and researchers can partner in meaningful ways so that in 10 years or so, we could have a much better understanding of PFIs and their effects.

– Yeah, nothing that needs to be done pretty urgently. I certainly suspect that some of these instruments are being used in ways that aren’t exactly helpful to the individuals and has been mentioned, can actually do some harm. Thank you very much, Blake. I really appreciate your time taking to talk to us. This has been absolutely fascinating.

– Oh, well, thanks very much for your interest, David. Appreciate it.

– That’s has been good.

Be impressively well informed

Get the very latest research intelligence briefings, video research briefings, infographics and more sent direct to you as they are published

Be the most impressively well-informed and up-to-date person around...

Powered by ConvertKit
Like what you see? Help us spread the word

David Wilkinson

David Wilkinson is the Editor-in-Chief of the Oxford Review. He is also acknowledged to be one of the world's leading experts in dealing with ambiguity and uncertainty and developing emotional resilience. David teaches and conducts research at a number of universities including the University of Oxford, Medical Sciences Division, Cardiff University, Oxford Brookes University School of Business and many more. He has worked with many organisations as a consultant and executive coach including Schroders, where he coaches and runs their leadership and management programmes, Royal Mail, Aimia, Hyundai, The RAF, The Pentagon, the governments of the UK, US, Saudi, Oman and the Yemen for example. In 2010 he developed the world's first and only model and programme for developing emotional resilience across entire populations and organisations which has since become known as the Fear to Flow model which is the subject of his next book. In 2012 he drove a 1973 VW across six countries in Southern Africa whilst collecting money for charity and conducting on the ground charity work including developing emotional literature in children and orphans in Africa and a number of other activities. He is the author of The Ambiguity Advanatage: What great leaders are great at, published by Palgrave Macmillian. See more: About: About David Wikipedia: David's Wikipedia Page

>