The expected benefit of automated screening needed to be weighed against development, testing and potential withdrawal costs. Public information does not establish that the project was profitable.
REAL CASE · 2014 – 2018
Amazon — the AI recruiting tool
An artificial-intelligence tool built to screen résumés, developed by one of the world’s strongest companies — and scrapped because it disadvantaged women. A textbook case for the question HUMICS asks: should this project exist this way?
01 / What happened, in order
What happened, in order
- 2014
Amazon builds an AI tool to rank applications: 100 résumés in, the top 5 out (as reported by Reuters).
- The mechanism
the tool learns from ten years of résumés received by the company, most of them from men. It infers what a “good” profile looks like: like yesterday’s.
- 2015
internal finding: the system is not neutral. It penalizes résumés containing the word “women’s” (for example “women’s chess club captain”) and downgrades graduates of two women’s colleges.
- Early 2017
the team is disbanded, the tool abandoned.
- October 2018
the story is made public by Reuters.
What this case raises
Business Readiness Score (BRS) examines whether the project could work commercially. HUMICS examines its human alignment and effects on people. Their intersection informs one decision: a project could be viable and still be rejected as it stands.
The ranking disadvantaged women applicants. Before deployment, reviewers needed to examine effects on people, ways to challenge a ranking and recruiters’ responsibilities.
The intersection
A supposed economic benefit does not cancel this human risk. Even if viability had been demonstrated, a committee could have rejected the tool as it stood, requested changes and recorded its reasons.
What HUMICS would have put on the table, beforehand
For an applicant-screening tool, reviewers should examine how rankings are explained and challenged, how effects on different groups are monitored and who remains responsible for hiring. Later facts clarify these questions; they do not prove that a short meeting would have prevented the project’s development.
IN ONE LINE
Technical feasibility alone is not enough. Even if commercial value were demonstrated, a committee could reject this ranking as it stood to protect applicants, record its reasons and set conditions for reconsideration.
Sources and limits
Dastin, J., Reuters, “Amazon scraps secret AI recruiting tool that showed bias against women,” 10 October 2018. Case featured in the BRS & HUMICS casebook:
Retrospective reading of public data — an illustration of method, not an official evaluation.
A retrospective reading of public information. The matrix informs deliberation; it does not decide for a committee.