Some conversations are not evaluated by AutoQA categories, why?
After setting up AutoQA categories, only new conversations that are imported with "closed" status are being evaluated, historical data will not be evaluated by AutoQA.
"Trash" conversations (spam or not enough context for conversation) are not being evaluated by AutoQA.
Some categories have N/A as an evaluation, why?
Not enough information was present to give an accurate rating for this category in the analyzed conversation. Azure categories have a requirement that at least 2 messages are sent by both agent and customer.
I don’t agree with a score, what can I do?
You can always change the AutoQA rating to a rating you think suits your process more.
I would like some evaluation cases to be stricter/less stricter, what can I do?
We can’t change the auto score but we would love to hear any feedback that you have. To help gather feedback about category accuracy, please try mentioning context, category name, given score, suitable score, conversation link, and reasoning for long-term score change. We cannot guarantee the change but we will improve the accuracy over time based on the feedback.
For some categories, the highlight toggle does not appear, why?
All Klaus based autoQA categories have highlighting capabilities. As for Azure based autoQA categories, the highlighting will be present soon. When any category is selected that is present in both Azure and Klaus list, Azure one is preferred (for better accuracy) and highlight toggle and highlights are not present at the moment.
Some categories are not highlighting a specific place in conversation, why?
When highlighting is present, it can be in the form of highlighting words (e.g. Grammar & Spelling) or messages (e.g. Greeting).
Are auto scores counted towards IQS of a conversation?
Auto category scores are provided as helpful information, which only counts towards IQS after a reviewer has submitted the review.
Can AutoQA detect and evaluate chatbots?
Currently we are evaluating Intercom's bot Fin, but other bots are not evaluated.
Can we add words to have them whitelisted for grammar category, so they are ignored?
Let us know which words cause any problems (or are inaccurate) in grammar category and we can assess whether we can change the model based on these.
Can we add custom rules for any category?
Currently no.
What languages are support for the AutoQA categories?
Category name | Languages/OpenAI-enabled | Languages/OpenAI- disabled |
Closing | All | English, German, Spanish, French, Italian, Dutch, Polish, Portuguese |
Empathy | All | English |
Grammar & Spelling | English (UK & US), German, French, Polish | English (UK & US), German, French, Polish |
Greeting | All | English, German, Spanish, French, Italian, Dutch, Polish, Portuguese |
Solution | All | Not available* |
Tone | All | Not available* |
*Solution and Tone are only available if you opt in to OpenAI processing.