Still, as we are recruiting real humans for our studies, the problem of fraudsters may occur. Some of them are not in the mood to give proper answers about the topic, while others want to check the limits of a moderator.
The number of valid interviews in a conversational study is more than 95% (the remaining 5% are manually culled).
We detect fraudulent interviews to eliminate them from analysis using the comprehensive approach, which includes the following steps:
1. Analyses the response for meaningfulness. Allows us to identify outright nonsense or gibberish, such as "qweroijwoirgjn" or "idk".
2. Detection of threatening, hate speech, harassment, sexual excitement, or violence. Interviews with such content are sent to a special status for manual cross-checking before they can be included in the analysis.
3. Moderation of the dialogue development scenario according to the situation. Before generating the next message, the agent tries to understand if there is anything unusual in the respondent's answer to care about, such as a counter-question, a negative reaction or a refusal to answer. It could also be a sarcastic note, a joke or even flirting with the agent that requires extra attention.
This sophisticated conversation management logic allows us to enrich the content of the response and minimise the risk of collecting unqualified data.