• by dreamfactored on 10/23/2024, 3:41:33 PM

    While developing an AI tool to help hiring managers prepare for interviews, we stumbled upon what seems to be a novel method for detecting bias in Large Language Models.

    By comparing how LLMs (Claude, GPT-4, Gemini, Llama) interpret anonymized vs. non-anonymized versions of the same content, we can measure and quantify bias reduction. The interesting part is that this technique could potentially be used to audit bias in any LLM-based application, not just recruitment.

    Some key findings:

    - Different LLMs show varying levels of bias reduction with anonymization

    - Llama 3.1 showed consistently lower bias levels

    - GPT-4 performed better in specific tasks like interview question generation

    We've published our methodology and findings on arXiv: https://arxiv.org/abs/2410.16927

    We're a boutique AI consultancy, and this research emerged from our work on building practical AI tools. Happy to discuss the technical implementation, methodology, or real-world applications.