Chatbots are taking the world by storm: they can assist in customer service queries, book medical appointments, recommend books or movies, and find our missing parcels. In order for AI to do its job, the models they use need to be trained on data. However, data brings quite a few obstacles to the table and the unconscious bias pressed by conversational AI is one of them.
AI tools are not free from bias because they learn from their human makers and inherit their biases. This phenomenon is confirmed by numerous studies performed on AI tools. This goes on to show that AI is not foolproof. It is just an extension of human nature and culture.
What is an unconscious bias?
Unconscious biases, also known as implicit biases, are the underlying attitudes and stereotypes that people unconsciously attribute to another person or group of people,affect how they understand and engage with them. Biases in how humans make decisions are well documented.
Some researchers have highlighted how judges’ decisions can be unconsciously influenced by their own personal characteristics, while employers have been shown to grant interviews at different rates to candidates with identical resumes but with names considered to reflect different racial groups or genders.
In many cases, AI can reduce humans’ subjective interpretation of data, because machine learning algorithms learn to consider only the variables that improve their predictive accuracy. At the same time, extensive evidence suggests that AI models can reflect human and societal biases and deploy them at scale.
Sources of bias in conversational AI
The sources of unconscious bias in AI:
- Misrepresentation in data
- Lack of diversity in the training dataset
- Not enough data
- Using third-party training datasets
- Failure to account for biases
The main source of problems is misrepresentation in data. AI chatbots can only learn based on the input provided by humans. If our training data ignores a certain target group, type of accent or minorities, it will skew the chatbot’s ability to understand their questions, intents and motivations. How can we ensure that we have well-rounded and well-represented content to create a training dataset? Even if we take data from real life conversation transcripts and historical messages from customer service database we may still underrepresent certain groups.
Researchers from University of Massachusetts reported that accuracy of several common NLU (natural language understanding) tools was dramatically lower for speakers of “non-standard” varieties of English, such as African American Vernacular English, slang or those with strong accents.
Second of all, someone needs to judge the quality of content for the training dataset. Again, this is crucial for the success of any chatbot as human bias can easily creep into AI through algorithms and data. Hidden bias is present in both people and data, and oftentimes bias is transferred to data because of people.
If you do not have enough data, or you want well-rounded data, you can go shopping around for it. However, that data may have a bias in it that you don’t even know about. There were two famous examples of well-known chatbots that have quickly misbehaved, purely due to their training dataset.
Facebook trained their chatbot, Blender, on Reddit data, and it quickly learned abusive language, consequently ended up being offensive and vulgar. Another example? In 2016 Microsoft’s AI chatbot, Tay, was withdrawn from the market just after 24h as it started tweeting racists comments. What happened to Tay? It was simply trained on conversations from Twitter and replicated human bias.
Another limitation to AI is that machines often don’t know what they don’t know. While AI is fantastic for interpreting large volumes of information, there is no guarantee that the technology will understand all the data.
Again, a flawed chatbot is either a result of skewed data or an algorithm that does not account for it. It is crucial for AI engineers and chatbot designers to be aware of those limitations so they can prevent them, or at least mitigate the risk at the development stage.
The consequences of unconscious biases
The end result of biases is less reliability in bots. This can have disastrous effects in different AI applications. For instance, facial recognition algorithms used in law enforcement have been found to be much less accurate towards women and people of colour. The use of such AI applications without accounting for these biases leads to these biases being reinforced, rather than dispelled.
The problem is made worse because of the enormous pace of AI applications being deployed. Machine learning algorithms play a crucial role in areas such as hiring programs, healthcare, law enforcement, and more.
On a more mundane note, our biases reflected in training data can make conversational AI much less effective. This has a direct negative impact on customer experience. If your bot can’t understand a certain accent, for instance, customers with that accent will just bypass the bot and contact your customer service department directly. There’s nothing worse than seeing all your effort and investment in a chatbot go to waste.
How can we minimize unconscious bias in conversational AI?
Encourage a representational set of users, content and training dataset. Create a diverse development team that will have an eye on unconscious bias of the others.
Establish processes within the organization to mitigate bias in AI such as additional testing tools or hiring external auditors. Some AI researchers suggest that the way forward in making algorithms more equalized is to be aware of humans’ cognitive biases and how they could affect the algorithms. However, such a process is neither straightforward nor quick.
There are ongoing efforts to mitigate bias in AI. The Excavating AI project aims to try and fix the ImageNet neural network by providing it with more well-rounded data. Organisations such as AI4ALL host initiatives aiming at introducing more diversity to the field – not just in terms of datasets. Finally, companies such as Google publish guidelines which help companies avoid common biases and mistakes in implementing AI.
We’re doing our part, too: if you’re thinking of implementing your own chatbot, we have you covered. Read our article about avoiding common mistakes when building your bot and our exhaustive guide to chatbots and make sure your bot is built upon solid foundations.
