Anonymous Evaluations Team Initiative
Hi All!
I wanted to post each announcement under a category called Announcements, but it would've gotten too messy with everyone replying to each announcement in the same thread, so I will be making them all separately.
Secret Shoppers are a popular quality tool that businesses often use. They go into the store, and then they buy items while evaluating the service. Once upon a time back in 2016, 7cups had an Anonymous Evaluations Team that would evaluate listeners to identify which could use some training (not because they were "bad", but they were new and inexperienced), and which people were here for reasons that don't fit with the site's mission.
We've been wanting to restart it for many years, but due to various reasons, were unable to. Quality Track leadership team and I are proud to say we will be restarting this initiative. We may call it Secret Shoppers instead as more people know what that is, but the name is flexible. It will restart once we get all the pieces in place.
I've created a draft of what I'd like to do. The basic run down this:
1. We will make it part of the Quality Track projects, so those who apply for Quality Mentor/Teen Quality Star and are accepted can sign up to join. They will have to do a certain amount per week for this project to be effective.
1. Team member reaches out to a non-verified listener via personal request and does a pretend chat of medium dificulty based on a topic from their profile. It would last about 20 minutes.
2. They would evaluate based on criteria such as did they respond, empathy, professonalism, giving advice, etc. Those that are here for the wrong reasons would get reported to admin. Those that are just inexperienced, but genuine, would be informed they were evaluated.
3. The listener would be paired with a quality mentor to do skill building.
Here's where I am getting stuck, and I need your help as a community. I am not sure how to best relay the evaluation infomation in a non-punitive way. I feel like a email might be too cold, but I also know receiving feedback that you need to improve can be difficult and hurt. I want there to be a way to balance compassionate feedback with the reality that every listener (including myself) has areas they can improve in to be more effective as a listener. We're all growing and that's okay. It doesn't make us a "bad" listener.
All feedback is appreciated, and if you have other ideas you think might be good for this, please feel free to share!
That being said, please keep in mind three things:
1. I am new. I've been here less than week in an admin role, so I'm still learning. This won't be perfect right off the bat. It's a draft, and it can be tweaked and changed. I am putting it out to the community because I think communication and transparency goes a long way, and I want this to be something that works. More brains means more ideas that I might not have thought of. We'll start the project once we have all the pieces in place.
2. Please try to keep any negative feedback constructive. I have noticed some forum posts receive feedback that looks like "That's a stupid idea", and that is not very helpful feedback. I'm doing my best to learn the ropes and plan ideas that will benefit the community openly and transparently. Tactful constructive criticism is always more valuable. If you post a criticism, please also add a solution that you think might work better.
3. I may not be able to use all the feedback received. It just depends on what fits together. I still appreciate all of it and may be able to incorporate an idea in the future.
Thank you for your time everyone, and i look forward from hearing from you!
Edit: A lot of great ideas have been brought up! A good point that was brought up is what kind of reward could we give for doing a good job. The idea of a Verified Listener badge (if they meet criteria) or leaving a review as a "reward" is intriguing. Please let me know if you think of any ideas for that.
Hello,
I think it sounds like a very good initiative that might help improve the overall quality of listeners on the platform. When it comes to feedback I think that it would be most helpful if someone would reach out to this listener, told them that one of the recent chats was this 'test' chat and give some overall impression starting with something rather general and preferably positive like I feel the chat went pretty well, but I would like to give you a few suggestions that might be helpful for you in the future. And later it would be best in my opinion if the person would be given specific quotes from the talk in which there is an area for improvement with a proposition of how to change it and why to change it. But the main focus should be on rules, guides, links, people to refer to, etc. Also the response time etc. But personal input should be treated with more caution because everyone reacts differently. If needed these listeners should be suggested to go into training for example in asking questions or if that sounds like a good idea, they could be proposed to get into more mock chats to practice their skills with someone more experienced, in which they could ask questions as they go through it.
@AnnieMA @EvelyneRose
I'd like to bounce off this idea, as I had very similar thoughts. If after the test chat was run, the inexperienced listener would be paired with a mentor to do skill building, maybe it would be worth it to have that mentor be who reaches out to the listener to inform them of their evaluation? They could be given the "evaluation report" from whoever the secret shopper was that details things that they felt could be improved upon, maybe some examples of text or responses as Annie suggested above, and then the mentor would have it to read over and then would contact the listener to introduce themselves, kinda initiate mentorship in a sense, and then would ease them into the fact that they were anonymously evaluated and then go over the evaluation with them (or schedule a time to if need be).
The evaluation form could have general ratings (like a 1-5 Likert scale, something simple) and comments for each basic active listening component, some overall feedback like how was the chat in general and maybe an overall rating, some suggestions for improvement the evaluator may have, as well as maybe some quotes/examples, and then the mentor would have a period of time to review the eveluation and kinda get a grasp on what they want to say and how to nicely present it in a compassionate and understanding and encouraging way to their new mentee. Because saying "You were rated a 1/5 in Empathy" makes you feel really bad about yourself, whereas "The evaluator mentioned that maybe working on communicating empathy over text would be a good area for us to start with" comes across a little nicer and doesn't just say that they failed, like sending an email of a raw report might come across as saying. And if a mentee was interested in raw scores and such, then that's up to them to request from the mentor like "I know I didn't do too well with open-ended questions. Can I ask the exact rating so I know how much I might need to work on this?" And then the mentor could share those numbers with them.
But this would give mentors a solid foundation on where to start with mentees, and it would also give mentees a softer reception of the anonymous evaluation, as I feel like having a listener test out your skills without you knowing it would be a bit shocking and uncomfortable to hear about right off the bat, so receiving that information in a compassionate manner from a mentor and then the mentor kinda jumping in and taking you under their wing and such would mostly likely soften that blow.
@LuLuRings
I think that mentorship after such a test sounds like a great idea. I generally speaking think that any kind of feedback is most helpful if it comes in a form of a discussion so that the person can ask questions and get a full understanding of what did they do wrong and how they can get better.
Personally I am not sure how I would feel about star ratings, because I feel that everyone has their own scale in their head, like for some 3/5 is an okay chat, it's good enough, while for someone else 3/5 is rather bad and there is quite some room to improve. That's why I think that comments and description based ratings can be more profitable and informative.
@AnnieMA
That's a really good point about the star ratings! I think in my head I was more envisioning like a feedback form? If you've participated in discussions in the group rooms before and filled out the feedback form, you are often asked to rate the discussion on some scale and then are prompted immediately after to explain your rating.
And rating guidelines could even loosely follow like how this Mock Chat Guide shared when preparing for the Verified Listener Mock Chat presents it (https://www.7cups.com/forum/ListenerLearningJourney_149/VerifiedListenersProject_1839/HowtoAcetheMockChatExpectationsandExamples_196827/1/), as it has like Awful, Mediocre, Good, and Exemplary, which, instead of using words that could invoke some type of feelings, the feedback form could just have say 1, 2, 3, 4, with 1 kinda being the "Awful" end of the mock chat guide and 4 being the "Exemplary" end. And then the evaluator could defend their rating with examples and such. So if the person just said "yo what's up" (like in the guide example), the evaluator would select a "1" rating and then leave example of "Listener greeted me with 'yo what's up'." So the ratings would just be there to kinda say where along a scale the listener may fall in certain active listening components, but the longer-response item would give the examples.
@LuLuRings
Great idea! 
This sounds like a very kind and compassionate way to do things! 
@EvelyneRose
This is such a wonderful idea!! I had actually just been thinking lately that I wish there was a way to get more practice and feedback from other Listeners. My only concern though is that I feel like 20 minutes might not be enough. I know that due to time limitations and much fewer Mystery Shoppers than Listeners this wont be possible for everybody, but maybe it would be helpful to have some way to request a longer follow-up mock chat focusing on the skills mentioned in the feedback from the first?
And as far as how to send the Mystery Shopper feedback, maybe a combination of a message sent directly to the listener thanking them for the time they spend volunteering and letting them know that they had been randomly selected for a Mystery Shopper chat and to check their email for more information and for a full review of the chat. It might be helpful to frame everything as just a bit more of the wonderful training provided to us Listeners and as way that we on 7 Cups are working as a community to improve our Listener Quality.
@EvelyneRose
How exciting! I can't wait to see the reintroduction of this project back into the community! Currently, I think the majority of feedback that new listeners receive are through coaching emails, so restarting Anonymous Evaluations would be a great way to increase the number of opportunities available for listeners to learn. Also, I can see how this project can be a great way to get listeners to understand the site from a member's perspective, especially if they do not have a member account or often use one.
In terms of feedback, I am in favor of providing the feedback directly to the listener in PMs. I think this gives the listener a greater chance of interacting with and responding to the feedback, and reaching out to various resources on the site which can help them improve, such as the quality coaching team. By providing the feedback directly, I think we can also be more supportive of listeners and encourage them to continue to grow and forge ahead in their journeys as listeners, while an email can simply be ignored and brushed off.
As a final thought, perhaps we could also let listeners know if they have been evaluated and performed well? (I don't know if this is in the plan yet or not) I think this could serve as a great way to boost listeners' confidence in their abilities and also signal that they are doing something correctly. On top of that, if a listener has done well, we could also direct them to the Verified Listener project or the Active Listening tests so that they could further their knowledge and expand their listening skills.
Ivy 🐑 Through the Fire, You will Prevail
Sounds good.
I have few questions that I think it is important to look after.
1. Evaluating a listener based on a single chat which is about 20-30 minutes?
2. Sometimes listener may feel uncomfortable with the topic of chat in that case how it will be evaluated?
3. Analogy to shoppers may seem good but we are dealing with listeners who are volunteering and feedback may be interpreted in subjective ways.
4. New listener may take a time to understand and navigate the website and it may be dependent on listener's availability and support. Evaluation in early stages may be viewed differently I believe.
These is my personal opinion and since I am new to 7 Cups I may don't have complete picture in case If I am missing any point please consider it as a lack of understanding. Thank you.
@Saquib16 for the topics, im pretty sure the ones evaluating listeners would choose a topic they list on their profile, at least its how i would proceed if i was evaluating listeners for this project. so listeners should be comfortable on those topics
@TortueDesBois
That's a good way. What if listener is not having any topic listed. Or even if it mentioned listener may feel that at that moment he or she is in not mood for that topic to liste. Or may be listner could say that he or she is occupied with another member and having same topic. (I am just looking for various possibilities that may happen.)
@Saquib16 if no topic listed, then the topic they taken the trainings about, or in last ressort a tooic light to moderate in content (nothing very triggering such self harm or sexual issues). If by the general request queue, you select a topic prior as members can do, so the topic visible for the listeners. If they take topics they are uncomfortable with, it may be something they can get coaching about. (Its a reality that we see a lot as a member and is frustrating so adressing this is good). If a chat cant be completed then you write it on the form (that included abandonned chats that we see A LOT)
@Saquib16
I would say Tortue and Bee covered it! We can try our best to accomodate, but we can only account for so many situations!
@EvelyneRose
Ok
@EvelyneRose
I think this is a nice idea and even more importantly I'm glad that you have displayed care and empathy for all of your team mates in knowing how cold one of those emails can feel. If we want listeners to display empathy then we can lead by example. I also agree that there does come a time to share constructive feedback and in a non-punitive or scary way.
Not sure if there could be a leader that could take the results of the "secret shopper" chat and simply pm the listener. And just chat, in a kind way? Maybe there is someone that could be identified on the team that shows great warmth, compassion and leads in this way to get the listeners "buy in". Something that would be important to the process is sharing the information and giving concrete examples as opposed to vague if the true aim is to have learning take place and quality increased. Hoping this helped.
@EvelyneRose You would probably want to get this gonig with listeners first, but at some point would you include adding in members?
We could have real chats but get a bit of training in terms of feedback. Maybe not all of our chats would be appropriate for this type of listener feedback, but we could be selective with when to do it.
Or maybe there is too much bias when the situations are real to us and not made up, but it's something that could at least be considered.
@AffyAvo i think it would be good.
@AffyAvo Working with the long term members here would be a good idea considering that I've had certain members telling me how many listeners have been rude and unhelpful. Also, you could always ask some members to volunteer because I've come across many members who gave been feeling perfectly fine currently and are here just to talk to someone. This could also help create detailed and varying situations for the listeners to deal with.
@EvelyneRose
This sounds like a superb initiative but i gotta ask, will evaluating a listener based on a single chat for about 20 minutes really enough? I mean there's got to be perhaps an additional evaluation method or more time to better know whether the listener is fit for the role.
@Asruu
I agree with your point.
@EvelyneRose
This sounds amazing--thanks for the announcement and for asking for feedback. 
I have some general questions. I don't really know the answer to any of these, but just brainstorming.
1) My understanding is that this is targeting new listeners. Is it that once a listener joins the site and passes the basic listener training, they will be added to a queue of potential listeners for Secret Shoppers to reach out to?
2) Assuming we're not able to reach all new listeners, should certain listeners be prioritized over others (e.g. listeners with serious negative reviews, listeners with a high proportion of negative reviews)?
3) What happens if a new listener we want to reach out to doesn't respond to the personal request or doesn't list themselves as being online?
4) Would all listeners who complete a Secret Shopper chat be paired with a mentor, or only those that were less effective in the mock chat?
5) Is there anything from the Verified Listener program that can be recycled/tweaked for use in Secret Shoppers?
@QuietMagic
Having peeked through some stats here, I second your second point.
Here are the numbers:
119,257-Newbie (Listener who have been here for <1 month)
44-Quality Mentor
17-Quality Mentor Leader
4-Teen Quality Leader
19-Teen Quality Mentor
4-Teen Quality Star
Even if you were to add all the listeners with Quality roles' numbers, you'd get 88 people (some people have multiple of the above badges, so the number drops lesser. But ignoring that-). For this program to cover every listener, each Quality Mentor/Teen Quality Star, would have to take a whooping 1355.19318182 chats per month. I would definitely like to see prioritization.
@bouncySalamander26 Good point.
On the same page we can see there are 10,258 verified listeners, but 180,872 listeners have Loyal Friend for one or more positive reviews, and are therefore around or beyond the stage where they could be verified, and 44,850 have Jester of Smiles for going on to receive 8 or more positive reviews. So we can see from those numbers that the verified listener scheme isn't working in terms of coverage.
That doesn't make it seem likely that diverting resources into another mock chat type of scheme is the right thing to do. Seems to me it would be better to focus on one scheme and try to provide full coverage at some level—probably a higher level than the current verified listener scheme.
I think we need more data on very new listeners in their first few weeks to understand what's really going on with our very inexperienced listeners. I'm still convinced better training would be far more cost-effective than mock chats at that level.
Charlie
This data on new listeners from long ago is probably still relevant.
I'd like to conduct similar research again, but I'm not in a position to do that for a few weeks and of course it takes a couple more weeks to get the data.
Charlie
@RarelyCharlie
I see where you are headed. In my early days, I used to only pick up GRs, without visiting the chatrooms- and it was a long time before I got to hear about the Verified Listener project. I think it comes down to whether or not the VL project is visible/promoted to listeners joining in-as you said "coverage", although that is not the only factor, I suppose.
I'm inclined towards the AET Project as the evaluation is anonymous, and there is a lesser probability of the listener being on their "best behaviour" for the one mock chat alone (as is possible with the VL Project). It's definitely a more natural environment to test the listener's skills, and accurately evaluate them. I guess, I'm looking for something of a merge between the VL project and the Anonymous Evaluation Team- because the Verifier's team (131 members currently) looks to be already equipped with necessary resources to conduct a mock, and the AET is anonymous and mocks are held randomly (without prior information). It also seems redundant to hold two mock chats (AET & VL, as would be currently) for new listeners looking to be verified.
But I can also notice logistical problems due to the volume of new listeners we have joining in.
This however brings me to the interesting point you noted 4 years ago (I was about to link it, but you beat me
) on how many listeners who joined, continue to be a listener here (actively). We do need more data on how new listener behaviour is panning out. I'm guessing that, it would be a much better way to use existing resources-if we were to conduct the anonymous mock chat for those listeners who have been active for more than a particular number of weeks? Of course, this does not address the issue of Listener Quality for those new listeners for whom the mock hasn't been conducted yet, but it is a step-forward~
And I believe that this should be implemented in tandem with improved listener training, rather than one excluding another.
This sounds for an interesting idea, and I think the idea of email can be too cold, and also some people don't check their email at all. The suggestion I would give is to have the mentor DM the person and give them pointers. For example, say there did really well on building rapport with someone but may need to work validating the feelings of the user more. Sometimes for listeners, it's mostly working on a skill that they know they need to work on. Also, it would allow them to build a relationship with that mentor where it might lead to improved quality of the listener. This would build a tighter community and may also increase the quality rating we see weekly.