Skip to content

Knowledge that lights the world

What Is Bias in AI? Where It Comes From, Real Examples and Solutions

AI systems can learn and amplify the inequalities in the data they are trained on. Real examples of bias from hiring to facial recognition, its sources, how it is measured, and steps developers and users can take.

What Is Bias in AI? Where It Comes From, Real Examples and Solutions

AI systems may look like “neutral machines”, but they learn from data produced by people. If that data contains social inequalities, under-representation or past wrong decisions, the system learns them and sometimes amplifies them. This is called bias in AI.

Where does bias come from?

  • Unbalanced data: If some groups are under-represented in training data, the system works worse for them.
  • Reflecting the past: If past decisions were discriminatory, a model imitating them will produce discriminatory results.
  • Measurement and labelling: Defining “success” wrongly, or the labellers’ own biases.
  • Context of use: Using a system developed in one country or group in a different setting without checking it.

Real examples

The 2018 “Gender Shades” study by Joy Buolamwini and Timnit Gebru of MIT showed that commercial gender classification systems were almost error-free for lighter-skinned men but had much higher error rates for darker-skinned women. Tests by the US National Institute of Standards and Technology (NIST) in 2019 also found different error rates across demographic groups in many facial recognition algorithms. According to a 2018 Reuters report, Amazon scrapped an experimental hiring tool trained on past CVs that disadvantaged women candidates.

Large language models and image generators can also show bias, such as linking jobs to particular genders or portraying some cultures in stereotyped ways.

Why does it matter?

Bias in a film recommendation is annoying; but in hiring, credit scoring, health care, social benefits or the justice system, bias can directly affect people’s lives. That is why the EU AI Act sets data quality, documentation and human oversight requirements for high-risk systems.

Ways to address it

  • Use diverse, representative data sets.
  • Test systems separately across different groups before release and publish the results.
  • Design auditable systems that can explain how decisions are made.
  • Leave the final say on important decisions to a human and offer a way to appeal.
  • Include different perspectives in development teams.

Tips for users

Use text and images produced by AI tools critically; watch especially for generalisations about people, groups and cultures. When an automated decision affects you, find out about your right to ask how it was made and to challenge it.

How is bias measured? Fairness metrics

To understand whether a model is biased, its results have to be compared across groups. The metrics researchers use most often are these:

  • Demographic parity: It checks whether the rate of positive decisions (for example, loan approval) is similar across different groups.
  • Equal error rates: False positive and false negative rates are expected to be close between groups.
  • Calibration: When the model says "70 percent probability", that prediction should actually be right about 70 percent of the time in every group.

An important point: Alexandra Chouldechova's 2017 study showed that when groups have different base rates, calibration and equal error rates cannot both be fully satisfied at the same time. This is why a "fair model" is not a single number but a deliberate choice made according to context, and the reasoning behind that choice should be written down openly.

Legal framework and standards

In its SP 1270 report published in 2022, the US National Institute of Standards and Technology (NIST) divides AI bias into three main sources: systemic (institutional and societal), statistical/computational (coming from data and models) and human (coming from design and usage decisions). The report stresses that the problem cannot be solved by technical fixes alone.

The European Union's Artificial Intelligence Act (Regulation 2024/1689) requires that, in high-risk systems such as recruitment or credit assessment, training data be examined for possible biases and that appropriate measures be taken to prevent them. For organisations, this means bias testing moves from an optional good practice to a legal obligation.

Sources and further reading

  1. Buolamwini, J. and Gebru, T., “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification”, FAT* (2018).
  2. NIST, “Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects”, NISTIR 8280 (2019).
  3. Reuters, “Amazon scraps secret AI recruiting tool that showed bias against women” (2018).
  4. European Union, Artificial Intelligence Act (Regulation 2024/1689).
  5. NIST SP 1270 – Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022)
  6. EUR-Lex – Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 10
  7. Chouldechova, A. (2017). Fair prediction with disparate impact. Big Data, 5(2)
Share
Follow YulaHubGet the day's most important stories first.
RSS

Leave a comment

Your email address will not be published. Required fields are marked *