Anonymous Evaluations Team Initiative
Hi All!
I wanted to post each announcement under a category called Announcements, but it would've gotten too messy with everyone replying to each announcement in the same thread, so I will be making them all separately.
Secret Shoppers are a popular quality tool that businesses often use. They go into the store, and then they buy items while evaluating the service. Once upon a time back in 2016, 7cups had an Anonymous Evaluations Team that would evaluate listeners to identify which could use some training (not because they were "bad", but they were new and inexperienced), and which people were here for reasons that don't fit with the site's mission.
We've been wanting to restart it for many years, but due to various reasons, were unable to. Quality Track leadership team and I are proud to say we will be restarting this initiative. We may call it Secret Shoppers instead as more people know what that is, but the name is flexible. It will restart once we get all the pieces in place.
I've created a draft of what I'd like to do. The basic run down this:
1. We will make it part of the Quality Track projects, so those who apply for Quality Mentor/Teen Quality Star and are accepted can sign up to join. They will have to do a certain amount per week for this project to be effective.
1. Team member reaches out to a non-verified listener via personal request and does a pretend chat of medium dificulty based on a topic from their profile. It would last about 20 minutes.
2. They would evaluate based on criteria such as did they respond, empathy, professonalism, giving advice, etc. Those that are here for the wrong reasons would get reported to admin. Those that are just inexperienced, but genuine, would be informed they were evaluated.
3. The listener would be paired with a quality mentor to do skill building.
Here's where I am getting stuck, and I need your help as a community. I am not sure how to best relay the evaluation infomation in a non-punitive way. I feel like a email might be too cold, but I also know receiving feedback that you need to improve can be difficult and hurt. I want there to be a way to balance compassionate feedback with the reality that every listener (including myself) has areas they can improve in to be more effective as a listener. We're all growing and that's okay. It doesn't make us a "bad" listener.
All feedback is appreciated, and if you have other ideas you think might be good for this, please feel free to share!
That being said, please keep in mind three things:
1. I am new. I've been here less than week in an admin role, so I'm still learning. This won't be perfect right off the bat. It's a draft, and it can be tweaked and changed. I am putting it out to the community because I think communication and transparency goes a long way, and I want this to be something that works. More brains means more ideas that I might not have thought of. We'll start the project once we have all the pieces in place.
2. Please try to keep any negative feedback constructive. I have noticed some forum posts receive feedback that looks like "That's a stupid idea", and that is not very helpful feedback. I'm doing my best to learn the ropes and plan ideas that will benefit the community openly and transparently. Tactful constructive criticism is always more valuable. If you post a criticism, please also add a solution that you think might work better.
3. I may not be able to use all the feedback received. It just depends on what fits together. I still appreciate all of it and may be able to incorporate an idea in the future.
Thank you for your time everyone, and i look forward from hearing from you!
Edit: A lot of great ideas have been brought up! A good point that was brought up is what kind of reward could we give for doing a good job. The idea of a Verified Listener badge (if they meet criteria) or leaving a review as a "reward" is intriguing. Please let me know if you think of any ideas for that.
@EvelyneRose
I've been employed with companies who use a secret shopper method to evaluate employees. One bonus is that we did get a certificate that we'd been 'caught' doing an excellent job. Those who needed addition support were mandated to accept the support, as you would expect most employers might do. So, I do like the idea of receiving some incentive for a job well done even if it is an email saying you did well.
I am curious about the VL process as it related to new listeners. As of today, listeners can take a proactive approach to better their skills through the Mock Chat Master badge. A listener completes an additional 5 mock chats over and above the one required for VL. These chats (for me) were much longer than the standard 20 minuets and the evaluation of skills was much more in depth than the initial VL chat. Could that badge be modified in such a way that new listeners complete a series of mock chats before becoming verified? It might be a way to help improve listener quality for new listeners.
More practically speaking, I am a listener who sets appointment times with members. Would a secret shopper set an appointment with me using the times I have available to chat? I'm also curious to know what a 'moderate' chat might be since moderate is subjective based on the topic and the pace/style of the shopper/member.
@EvelyneRose
This sounds like an amazing idea! How can I participate :P
@Psychstudies13
Awesome! Let me see how things pan and I will get back to you!
@EvelyneRose
That sounds perfect <3
@EvelyneRose
This is great! All the best, Rose! You'll do awesome! And yes, the reward review is something I would personally love, I usually read my reviews when I'm feeling down. It makes me feel so much better. Will try and be a part of this initiative! 🙋♀️
@EvelyneRose this is amazing! As for feedback reports, how about designing a template, having the shopper fill it out but have our mentors accompany the report with suggestions such as examples of how they could have responded -- basically just like a post-mock tips thing, and as for delivery of the report, it would be nice if the Mentor who would be helping this listener could share the report and then the initial conversation could be having both parties agree to goals they would work on. Secret Shoppers would then stay anonymous.
@EvelyneRose Honestly, I've just joined recently and while I was giving my mock chat for the verifiers badge, I actually thought of this idea about conducting an anonymous mock chat. As you mentioned in the edit, it would be a great idea to leave a review and also, I think a new forum pist could be made where the listeners could be tagged and it can be mentioned that an evaluation on the basis of a mock chat was conducted and so and so were the areas of improvements noticed and so and so were their strengths. This could also help keeping the identity of the person taking the mock chat anonymous.
@Affliction1
Hi! Thank you for your feedback! My only worry about that is listeners might feel called out in a public forum, but I definitely appreciate the ideas. please keep them coming!
@EvelyneRose Oh yeah, good point. I'll think more about it.
@EvelyneRose
Love this! Was literally talking about this idea the other day with someone!!! Would love to participate so bust i work towards Quality Mentor 😝 I definitly think providing a badge of somesort to the listeners that are check will be benificial to avoid 3 people doing a chat with the same person! It is definitly something that is needed... Instead of an email i guess you could always have an outreach team that could try contact the listener but then if they dont reply in a week then an email is sent possibly? Just an idea!
Awesome project! Cant wait! You are amazing Evelyne!
-Joy <3
@EvelyneRose
How anonymous/secretive will this be? I mean, if Quality Mentors/Quality Teen Stars can participate then it'll be obvious if the listener that's being evaluated checks the badges of the QTS/QM. Also, it will be obvious since the listener being evaluated will be chatting with another listener.
A VL mock chat is obvious because both parties are aware that it is happening. If there's gonna be an anonymous mock chat, the listener must have no way of finding out that the person they're talking to is actually evaluating them.
I think this thing will only work if the QTSs/QMs/volunteers are given a member account to work with or something. Or if only members are allowed to evaluate (this is highly unlikely). But then again, this all depends on how anonymous/secretive you want it to be.
Oh and about the method of giving feedback, I don't think you need to sugarcoat anything. If they did awful, tell them they did awful. If they did okay, tell them they did okay. List where they can improve upon. Be blunt and straight to the point. Maybe let them know by email or let them know through PMs. Maybe make it somewhat similar to the coaching emails. Of course, it will hurt. I know I was definitely devastated when I got my first coaching email but without it, I most likely would've never worked on my empathy skills. The focus here should be on the quality of help the members are getting not the feelings of the listeners because they got some constructive criticism. This is pretty harsh but I think it's the truth.
Anyways, this is a pretty good idea and I hope it goes through. Good luck with it ❤️
~ Lex ☀️ | Take careo and eat oreos 👋🏿❤️🍪
@LexIris
I think it'll be done from their member accounts, because @EvelyneRose mentioned that a personal request would be sent. I dunno if they'll have separate member accounts or the pre-existing member accounts would be used, though. And it'd be helpful to have some members in on the project, as it expands, I guess.
@bouncySalamander26 for anonymity sake, it is most likely that the people on this project will be using separate members accounts for this (not their personal member accounts)
@MidnightRaven999 @EvelyneRose
Yes, that makes sense! Thank you for clarifying!
Just a bit of a thought though, as these member accounts would be anonymous-if a new listener who chats with one of the accounts, does decide to check up on them (a few days after the mock, believing it to be a legitimate member account), there needs to be an agreed protocol on how to handle the check-up chat, especially as there is a possibility it may come up as this member account is conducting another mock chat (member availability statuses can't be modified, and the listener would know if the member account was online).
Just putting out something that may need to be thought through too!
@EvelyneRose
Hi Evelyn
I don't really like the idea of having someone deciding that they should pretend a scenario in order to assess me without my knowledge:
1. mock chats aren't real and someone pretending they have issues which are distressing is going to react completely different to someone who actually has real issues at that specific moment: there is no way to have enough scripts available to be able to predict a series of different reactions to different responses
2. many of the people that would make up this "assessment" team aren't psychologists, they would be working on their personal experience and subjective opinions and would not be able to fake reactions that can only be real - what position would they really be in to assess what had only been fake anyway?
3. if you wanted to acheive real quality control, don't you think you would be far better off to monitor real chats with real situations and real listener responses and real feelings involved? That would actually feel much less intrusive and much less of a betrayal of my good faith and commitment than finding out that I had just wasted my time and effort and my care on someone who wasn't even qualified in psychology who had only pretended so they could trick me and give me an amateur judgement on how I responded to their hypothetical remarks and what they "think" a member might say in response to my replies.
4. if you insisted that you would go on with this deception without permission, why do think the reviews from your team of tricksters come up with should be mixed up with the reviews of members who come for support and care, not judgement and deceit. I give my members my best and treasure their reviews - why do you think I'd want a review from someone who had tricked me on purpose to be sitting amongst the reviews from people I'm proud that I've actually helped?
5. I've been a secret shopper. You might not be aware, but secret shoppers is also a program that retail outlets actually not only sign up but pay for and actually are fully aware of: the stores that take part not only refund purchases, but also dictate what the shoppers buy, how long they stay in the store, what they should be observing and how much they can spend - it's not as if secret shoppers just choose shops at random and then come back later and ask if the manager would like to pay for a review and refund their purchases. If you had a program like this, why do you think the listeners should not have to sign up with permission to be assessed without their knowledge - how are you going to get signed permission from 300 000 listeners? How are you going to change listeners' terms of committment to include this ruse?
6. What are you going to tell a listener who gets upset by being confronted with what you call a "medium difficulty" chat by one of this team of people who want to deliberately fool listeners and who ends up needing peer support from being betrayed or judged, or from the team members being deliberately uncooperative so that they can mock people in real distress - "Oh don't worry that you couldn't console the member - it was all just pretend anyway" . . . ? What is a peer supporter supposed to tell a listener who is utterly distressed and traumatised because she or he feels entirely humiliated by a bad review, when the chat wasn't even real and the review comes from the sort of people who like to fool listeners?
7. Do you really think that all listeners will do their best and put their heart and soul into their volunteering if they think that each time the next listener might be someone who wants to trick them so that they can judge them and then print a fake review to go with their fake chat? I think that this scheme risks making listeners not even want to log on to volunteer to chat - on the other hand, at least if the chats which are assessed are real, the listener's time wouldn't have been wasted on purpose.
8. There are 300 000+ listeners at 7 Cups - how many of them do you think should have to write in to say they approve of secret fake listeners with fake issues tricking them so they can be assessed without their knowledge before you have enough positive feedback to go ahead and start deceiving listeners and then putting your fake comments where people have had real support used to put feedback? Wouldn't the fair number be at least half?
9. What's wrong with getting that team of people who want to do mock-chats and get them to do mock chats by appointment and just raise the grade for verified listener (VL) and have VL 1, 2, 3 etc - then at least you wouldn't be deliberately betraying good faith and intruding on listeners who don't ever want an opinion from the sort of person who would think that sort of behaviour was either respectful or appropriate.
I really hope that you don't do this - I used to like being a listener. At least if you do start a program like this, please let everyone know so I can go find a site to volunteer on who respects their listeners.
Thanks at least for asking for feedback.
@CintaBali
I have to say, that was well written and well thought out in my humble opinion! It was so very well written, I almost feel as if I can feel your passion about this topic through my very computer screen! I think you have brought some very constructive thoughts and ideas to the table that should be taken into consideration. At least to read what you had to say and I want you to know that I for one, read every line. I am not a part of this program, but I'm a part of this community.
In addition to what you said about the "secret shoppers" program as it works in the real world, isn't this a third party company that is hired to do this as a service for the retailer? With that type of program, it's independent third party and it's much more likely that the "secret shopper" has no knowledge of or affiliation what so ever with the employee being graded. Here, it is much different as we would have people doing this internally that could potentially have prior knowledge of the listener, past interactions, a bias.
@CintaBali
Firstly, let's remember that I'm human and I'm not out to trick people. It was just an idea that I was gathering feedback on. As I stated in my disclaimer, I am new and interested in constructive feedback. That being said, I will take your feedback into account as I understand you dislike the idea, and perhaps we will alter or tweak some things. How it will look depends on what the best way is to move forward. Thank you.
To answer #5, the listeners would be informed when they sign up.
You raise an interesting point though with #9. How would you judge quality? Do you feel like knowing you are being graded like with VL affects a listeners behavior? Basically, what would you consider to be respectful while also assuring quality?
@EvelyneRose
Thanks Evelyn. I'm sorry if anything I wrote about your idea made you think that I considered you as not human (and therefore certainly capable of making mistakes). I think you're very human. However I think your idea is inhumane, and when I say that your team of secret shoppers would be tricking listeners, I mean that they would be deliberately deceiving listeners who volunteer in good faith to help and support people: A secret shopper would never be there for help or support, they would be there to betray that good faith and make judgements about fake conversations based on limited and uneducated experience of a limited number of conversations which they have no qualified opinion to state is representative of either an "average" out of 42 000 000 messages or of an "average" person with e.g. anxiety issues, PTSD, depression etc.
I'm not just saying I dislike the idea, I'm saying that it's unfair, unsafe, unhealthy, disrespectful, intrudes on the good faith of listeners and if artificial, and also assumes that there is someone there who is capable of convincing listeners without any actual psychology qualifications that they are "qualified" to invent mock conversations on the spot which are sufficient to use to judge other listeners. Now, say if someone had 1000 chats under their belt, that would be 500 hours, which adds up to 12.5 whole weeks of actual full time experience in active listening - enough to get credit for one semester of an entire 4 year college degree (without the actual tertiary training that would applied during such work experience, where students spend years learning what experiments have been designed to prove that certain responses are likely in certain situations). Why would someone with 3 whole months of experience and training that amounts to less than a third of that think that meant that they could train other people to not only always know what a person with a "medium difficulty" chat issue would say, but to also always know how other listeners should respond to each individual comment?
There are algorythms that could easily measure and provide feedback on, say, listeners' response time to member comments, and that would need no one to either betray fellow listeners' good faith or to impose uneducated unsolicited opinions.
A really big consideration is safety - how are you going to make sure that the people you ask to pretend they are upset for x number of hours per week are able to do that without endangering their own wellbeing - how could you possibly assess their fitness to such a task responsibly without a clinical qualification ? How are you going to make sure that no listener is safe from being traumatised by these fake "middle difficulty" conversations? How do you presume someone without even a psychology degree, (let alone the doctorate or masters + 2 full years experience that real psychologists need to even begin to be able to claim they "know what someone would say") is able to train people to use to trick listeners with - and yes I say trick, because that's exactly what you're doing if the listener has not given express permission for the next conversation to be faked.
In answer to your question, any assessment of a listener without their express knowledge and permission is intrusive and disrespectful, which means that there is no way to do this secret shopper idea respectfully. That is why I had already said, if you want to recruit a team of people who want to judge other listeners, then the only way to respect either them or the listeners they assess is to set up a scheme of levels of "verified listener" badges, and use a system where listeners ask for the assessment and make an appointment for their mock chats. Obviously in that far more respectful situation, an assessor in a mock chat appointment would judge quality by not only monitoring listener response time, but also by noting whether the listener demonstrates empathy by reflecting, summarising and validating member comments, asks open-ended questions, avoids giving advice or making diagnoses and, if asked for advice offers instead to brainstorm several options, and by noting whether the listener can direct the member to useful and relevent pages both within the site and at reputable sites with further relevant free online information.
It is obvious that knowing that they are being graded affects listeners' behaviour - all you need to do is visit the site where the feedback for the mock chats is published: so many of the listeners are extremely nervous. What you're doing therefore, with a secret shopper scheme, is risking extending that exposure to stress to every single chat.
The thing is, that you presume that these secret mock chats will actually assure any quality - there is no actual proof that's the case, or that 3 months of full time experience is anywhere near enough to start training listeners to start betraying their peers. The staff who currently volunteer for the mock chats work really hard, and subject themselves to a huge risk of becoming upset by pretending to be upset, and this whole forum thread seems to diminish the hugely valuable contribution that they make and risk they subject themselves to, and mislead the reader that anyone can do what they already do. I have already put it to you that secret mock chats do nothing to assure quality at all, and that you have not proven that they do. The likelihood is instead that such betrayal of good faith will decrease quality, because listeners will suspect members of being fake, and instead that this scheme will highly risk affecting the morale of the entire organisation: If this system was abolished in the past, it was probably due to a number of complaints, which don't even seem to have been published or addressed along with this resurrection idea.
As to your idea of getting new listeners to sign to agree to this, I still think that it's a bad idea and risks not only their health and morale, but also the health and morale of the listeners who take part as assessors in such a program, and that by trivialising my response to "I don't like it" you have failed to address this important issue at all - Where is your written testimonial from a professional that they agree with you and can guarantee that this idea won't cause any harm ? If you managed to get a professional to reply contradicting the points I have made about the fact that this idea is not safe, and claiming that people would not be hurt because of it, then the people that are hurt by such a "human" scheme could at least make a claim against that professional's public liability insurance. Putting permission to conduct mock-chats in the fine print of the agreements signed by new listeners doesn't really seem to constitute informed consent: unless you warn them that consenting to mock chats may also lead to them suffering symptoms of anxiety, and either exacerbating any mental health issues they have or risking them encountering new issues with stress and anxiety, which the mock chat feedback page already seems to prove will occur.
In the meantime, the 300 000 people who already have become 7Cups listeners, even if 30 000 people write in and say that they don't mind worrying about whether the next chat they bother to do might be some fake "expert" wasting their time and judging them, that will only be 10% of the actual volunteer workforce that has agreed to this health hazard.
@CintaBali all the listeners posing as members will be highly trained to be these secret shoppers, we will not chose hard topics, and the listener is always welcome to tell us that they cannot help us with the particular chat topic we choose. Also, most of the issues (at least in my experience with mock chats) are based on past experiences that I have had, and worked through; so i hope that others will do this, and that they will be real in a sense that the listener has gone through whatever issue they are talking about. We are not trying to trick anyone wtih this program, this is meant to help listeners improve their skills and become better at active listening (and better at helping with different chat topics too). The reason anon eval has been an idea for so long, is that with the verified listener team, the listener knows that they are being tested, and a lot of listens may 'fake' their listening skills. i have seen this in my personal experience, i have gotten listeners who i think are highly qualified, because they have the verified listener badge, they list the topic i want on their profile, etc, but when i contact them, they act like they are brand new. thats not to say all verified listeners are like this of course, i have also gotten brand new listeners who are fantastic and help me a lot with my issue, they have all the active listening skills of a seasoned listener even tho they are new. So the anon eval team is just a way for us to see their real skills, in real time, and help them in areas they could improve in, or give them the credit they deserve for being a wonderful listener already.
@MidnightRaven999
Thanks for your comments, Raven
My point is that this "high level" of training is not really high at all - even if an assessor has 4000 chats, this actually adds up to less than one year of full time active listening experience. An entry degree is 4 years and a psychologist needs at least a 2 yr. masters plus 2 years full-time supervised active listening experience, and even a fully licensed practising psychologist would not consider their own experience enough to justify tricking listeners that they were members, so how could it possibly justify that for you? What I don't get is why listeners who know they are not even qualified to give advice think that they would be qualified to trick people and force listeners to be part of fake chats without even providing their express permission.
It's all well and good that you "hope" that listeners will "do this", i.e. relive symptoms that they have already actually experienced, but how can you claim that is a safe idea - to ask people to do this with listeners who don't even know they are faking? Okay, maybe it's never happened with you, that reliving symptoms you've actually had has caused those symptoms to appear for real, but why would you think that means that couldn't happen with anyone else? Just because you can do mock chats without ending up as an inpatient or back on a couch doesn't mean that everyone can, or for that matter that you personally can design training for other people to do that safely. I'd be happy to bet a dollar, that if you get a team of even only 10 people with lived experience (who haven't even got an entry degree in psychology) to deliberately fake symptoms, that eventually at least one of them will end up reliving those fake symptoms for real.
If you've had lived experience, maybe you've even had lived experience enough to know, even if therapists have doctorates and bundles of training, some of them are just not compatible with some patients: why hasn't it occurred to you that some of the listeners you've talked about who are just beginners might just be more compatible with you personally? Yes sure we have slack listeners here on this site and listeners who don't do as well as those beginners who helped you, but why do you think that deliberately fooling them you are logging in for help and then humiliating them with your "special" bad reviews would actually make any difference whatsoever to overall quality?
This position where you've maybe assessed a few hundred people in mock-chat situations and that means that you think you are therefore qualified to deceive and judge listeners in real chat situations seems entirely illogical and without basis: all that actually seems to mean is that you've been able to tick off a list of training objectives like "reflecting" and "open-ended questions", etc, and provide feedback as to whether those objectives have been exhibited or not.
There's a phenomenon in social psychology with people who get great feedback all the time becoming convinced that they are right or justified because no one ever disagrees with them. It seems like you actually think that you are better at working out whether you've been reflected or asked open ended questions than any other member would be: it sounds like you think that being able to identify those objectives means "highly trained", but that is not actually the case. The mock-chat pages are full of consistently complimentary and flattering comments, but that doesn't mean you are qualified to deceive listeners and pretend that you should be assessing them instead of logging in as a member in good faith - to get help and support for yourself just like everyone else.
What's to say that there couldn't be a form for all members to choose to do "technical listener reviews", which list the same elements of active listening training, like validation, (saying that your feelings were reasonable in that situation), reflection (the listener told you in their own words what it sounded like you were feeling, thinking or saying at the time), open-ended questions (the listener asked you questions that needed more than a yes-no answer most of the time) etc: after all, it's not as if objective assessment of active listening is anywhere near rocket science or actually requires a "high level" of training at all, is it - really, those great reviews can really distort your perception of how difficult it is to actually assess a set of basic rudimentary skills, now can't they?
Something like this form could actually not only improve the quality of the site, it would take far less volunteer effort, would not be deceiving or disrespecting anyone, and would lead to member education about quality they should expect from professional therapists as well as 7 Cups listeners, and might even therefore spill over for a positive effect on the entire profession, which certainly has its bad apples as well. More to the point, it would be HONEST, which this idea of anonymous spying and deception is not.
If that sort of form was there as an option for real members to fill out, where's the justification for you or anyone else to either start fooling people or make these dubious claims that you're "highly trained" at all? Just because you've been disappointed once or twice if you've logged in as a member, why would that mean you should be allowed to write youself a license to spy on everyone, or anyone for that matter? Any listener who wants to be able to direct members about where to go on the member's site, or to make contributions in some of the chats needs to make a member account, and we all know that there is a vast range in the standards of listeners' skills . . . but just because you've gotten nearly all flattering reviews doesn't mean that you deciding to personally fool listeners with fake chats meanswould be the only way to fix it, or even any way to fix it at all. Where's some evidence of the justification for anyone to disrespect listeners or their time or for anyone to pretend that assessment of active listening either needs a "high level" of training or is beyond the ability of any member to do for us in any case? All we have as evidence is that this system was abolished in the past.
I don't get why you don't think that you would be tricking a listener when this scheme would literally be deliberately deceiving a listener that you have an issue that they should donate their time and effort to listen to you about. Either you'd be faking the issue that you had at that time, or having the issue for real - If you had the issue for real, why would you think that your review should be considered with any more weight than any other member? If you were faking the issue, why do you think that having a few hundred mock chats under your belt would mean that you have a right to deceive human beings who are donating their time or that could ever be ethical? As I said above, mock chat assessors do really great and difficult work, but the difficult part is risking reliving symptoms for real. Mock-chats have an appopriate time and place and that's with listeners who have provided informed consent and who are fully aware of the assessment taking place, and just because you've had a lot of flattering reviews in a role that attracts that doesn't mean you are "highly trained" or that you are able to "highly train" other people, or that you should use member chats for anything other than for what they've been designed and portrayed.
A little bit of volunteering, or even a big bit of volunteering, does not give any of you the right to deceive volunteers nor to consider your feedback as a member any more important than any other feedback by any other prankster who wanted to use the site to pretend that they had issues in order to waste someone's time. Most especially it does not give you the right to risk hurting either listeners or the people you set up to fool them.
@CintaBali
Oof yeah you do have a lot of valid points. I put myself into the member chats I'm in and I would be pretty hurt, disappointed, frustrated and angry if I find out that one of those reviews or members is actually just someone testing me to see how well I can be a listener and not an actual human looking for support with what they're going through
~ Lex ☀️ | Take careo and eat oreos 👋🏿❤️🍪
Just wanted to outline/summarize the ideas and concerns @CintaBali raised (as I understand them) for future reference.
1) Ethical concerns: There are ethical concerns in using deception to have listeners chat with secret shoppers while believing they are supporting real members.
2) Possible alternatives: There could be other possible alternative quality initiatives that are less ethically controversial, more effective, or require less level of effort to implement (e.g. utilizing existing feedback from real members, offering members the option of completing a technical listener review form, expanding or improving existing listener verification, algorithmic evaluation of response time).
3) Training (evaluation): Secret shoppers may not have enough training or expertise to evaluate listeners.
4) Training (realism): Secret shoppers may not have enough training or expertise to know how to make mock chats realistic enough that they could be reliably used for listener evaluation.
5) Secret shopper well-being: Secret shoppers could potentially experience mental distress in their participation, particularly if they use personal issues in order to try to create realistic chats.
6) Global listener anxiety/distrust: The presence of secret shoppers might put cause listeners as a whole to feel anxious or less invested in chats based on knowing that any member chat they take might potentially be a fake/evaluation chat.
7) Past issues: If a similar program existed in the past but was discontinued due to issues, those issues might potentially recur in the present program.
8) Informed consent: Listeners would need to give informed consent that by using the site as a listener, they may potentially be subject to anonymous evaluation.
9) Distracting from real chats: Secret shopper chats could potentially pull listeners away from chats with real members who need help.
10) Secret shopper bias: Secret shoppers might provide biased evaluations if they have previous relationships or familiarity with listeners.
11) Mixing real and mock feedback: It is better to only have real/legitimate member reviews posted on a listener's profile or used for rating purposes; otherwise, listener reviews become contaminated with phony data and lose their genuineness/trustworthiness.
******
Re: #1 (most worrisome/serious concern), here's what I was able to find on that:
- Secret shopper programs are used in health care settings.
- Secret shopper programs have been proposed for use in mental health care as well.
- The American Medical Association (AMA)'s Council on Ethical and Judicial Affairs ( CEJA) studied ethical dilemmas associated with secret shopper programs in healthcare ~10 years ago. AMA ultimately declined to take an official stance in support or opposition, but CEJA's initial findings were that secret shopper programs were beneficial and ethically justified as long as certain conditions were met. Quote from CEJA's report:
Physicians have an ethical responsibility to engage in activities that contribute to continual improvements in patient care. One method for promoting such quality improvement is through the use of secret shopper patients who have been appropriately trained to provide feedback about physician performance in the clinical setting. A sound secret shopper program should include the following elements:
(1) All relevant parties, especially those to whom secret shoppers will be making unannounced visits, should be notified that this mechanism is being implemented in their practice setting.
(2) The information collected by secret shoppers should be used only to identify areas of improvement and not as a basis for punitive actions. Third parties should not have access to information collected by secret shoppers that includes personally identifying data.
(3) Feedback from secret shopper patients should not be relied on as the sole source of data for evaluating clinical performance.
(4) The use of secret shopper patients should not be implemented in a manner that adversely affects access to medical care by legitimate patients. For example, the need for urgent care (such as in the emergency department setting) must always take precedence over secret shopper patients.
Here's what these guidelines might look like if applied in a 7 Cups context. (I think 7 Cups would be able to check off these boxes.)
- Make sure all listeners are aware that anonymous evaluations are happening.
- Use anonymous evaluations only to help listeners improve rather than to harshly berate them.
- Have other ways of checking listener quality besides anonymous evaluations.
- Make sure anonymous evaluations don't get in the way of real members receiving support.
@CintaBali I have a serious issue with this, and any indication of people listening In on conversations requires peoples permission. I try and tell mebers that the conversations are private and confidential of that is not the czars then someon better tell me pronto. So that I can warn members accordingly.
@CintaBali
@CintaBali
As you stated that you were a Secret Shopper previously-I am hoping that it would be fair to assume that you do not necessarily condemn the original idea, or how it works-but are rather, disliking the application of it in the manner that was proposed on the main post. Developing a bit on:
"Physicians, nurses and staff members will be forewarned of a secret shopper before they come in,"
I would like to imagine this scenario, as to how this would play out in reality-and to see if we have the same understanding (I haven't gotten a chance to be a part of the actual program, my bad!).
From my understanding-how this works is that, at a specific instant, the employees in the shop are let know that one (or more) amongst the people shopping, is/are secret shoppers. If I have understood correctly, this does not mean that which of the shoppers is a secret shopper is revealed (
That'd be a bit of a chaos with all employees rushing over to help that one person alone).
Anyway, the point is that-the employees do their best to assist everyone shopping in for that specific time period. As they do not know who the secret shopper is, they provide service at a good standard, to every customer they meet, during the time period.
(I'm using relevant terms in the analogy in brackets)
1. In the system that you are proposing of VL1, VL2, etc.- the employee (listener) would know exactly which of the persons in the shop, is/are the secret shoppers (Evaluation team member)-which was why I believe the subject of anon evals came up.
2. I do agree that this should not be a case of evaluator's feedback holds greater value than member's feedback-which is why I would like to suggest that evaluations in this manner, be done specifically for listeners who are getting constant negative reviews, and more so, for whom the reviews do not specify where the listeners are lacking. This feedback, along with member reviews could definitely provide mentors/Listener Coaches more information to work with-(if you believe so differently, please let me know).
3. That being said, I also think the listener (with the negative reviews) should know that one of their PRs, in the coming days could be a mock chat-however which PR, they do not know. (This was the closest I could come to the analogy of the secret shopper. Does that sound good to you?)
4. With regards to evaluators experiencing the symptoms themselves, as a result of partaking in a mock chat-I do think that's a bit of a risk we take everyday, in empathizing as a listener, on topics we have previously experienced. That being said, I do think some topics like PTSD, should not be used in a mock chat. Lighter topics like Anxiety (Anxiety is lighter, only in my subjective opinion) could be used, and I believe this was what, was meant by "a chat of medium difficulty". It should also be well-known amongst the evaluating team that they ought not to take topics that are triggering/have the capability to be triggering for them-and that they can leave the mock chat midway if they feel triggered.
5. As for consent, I do think that this initiative applies only to new listeners (those who join in after an official announcement, if there is one) and that they would be let know of this program while signing up-essentially, in signing up, they would consent to it?
I would love to know your thoughts, but I also understand that you have multiple people to reply to (wink), so, I would not mind a delay~ If you have a different idea, which fits with the idea of "not knowing who the secret shopper is", I would love to know of it!
6. Finally, I would like to acknowledge that yes, feigning a scenario does not nearly work as well-but you would be hard-pressed to prove that a member could never present you with the same scenario. As listeners, both you and I have a fair idea of how a member could respond, or can sufficiently imagine plausible responses, and I believe that is what this project is working on. It does not mean to demean legitimate member feedback, but complement it. However, a review being published based on the mock chat-is not something I would be in favour of, either.
(And with regards to reading actual chats to provide feedback, as you had originally proposed-that is actually quite a lot more intrusive, especially as we make the promise that all chats are confidential)
@EvelyneRose
Just a little food for thought. When rolling out the secret shopping program, perhaps leadership should be shopped first. Let me say, I'm not trying to pick on leadership, especially since I hold leadership roles myself. It seems to me that a short member chat with me, for example, will not give you a clear picture of who I am in relation to my roles here at Cups. If I am chosen to be shopped, I would like my shopper to see me as a listener and in my roles as a leader, too so that my shopper could offer feedback to me on areas of quality that you wouldn't get to see in a 1-1 chat. A lot of focus seems to be on new listener quality but, I think, a focus should also be pointed at those of us who are offering direct support to listeners to ensure we are providing outstanding support to our fellow listeners and to point out areas where we might improve our sills as supporters to the community. Just my 2 cents.
@PeaceLoveandPaws
I like this idea a lot, to have secret shopping translate into even area of leadership outside of member 1-1 chats, such as Peer and Chat Supporters, Mods and Room Supporters, Discussion Leaders, etc... to ensure that even those who are no longer newbies and are moving up and supporting the new folks are functioning in their roles and being good examples as well. Like many earlier have said, some may act differently during the VL mock chat or even maybe during trainings for leadership roles than they actually would behave in those roles or in 1-1, and it would be good to try to ensure quality across the board, not just in member 1-1s.
@PeaceLoveandPaws I am with you on this, from what I have seen, most of us have more qualifications and hold leadership positions thst are irrelevant for the reasons as to why we are here, but I reiterate this is a seriously bad Idea - we chose to be here to give something back and for our personal reasons, had we wanted to be subjected to these shaningans we would of continued on our own paths. This Idea reminds me of those pathetic team building courses we used to have to attend by those horribly unqualified managers who had a problem with their self esteem and where you ended up being with people you did not want to be. There is a big difference between quality control and forcing people to do things that they are currently doing because they are doing out of love and charity. I realise that there must be training and that not everyone now equipped to deal with people no matter how great their desire may be, but be careful of turning something that people want to do into something that ppl no longer want to do because the see it as an obligation or some other enforced act. Just my 2 cents worth
