maki · Paris
Skills: Python
ABOUT THE SCIENCE AND AI TEAM The Science and AI team is at the heart of Maki People. It builds the science and AI behind our platform, from job analysis and assessments to AI interviews and predicting performance. Led by our Head of Science, Aiden Loe, and working closely with our COO, Paul-Louis Caylar, the team brings together business psychologists, psychometricians and AI scientists. Together, they make sure every hiring decision on our platform is accurate, consistent and fair. As Maki grows, the Science and AI team’s job is to show, with evidence, that our approach leads to better hiring decisions. ABOUT THE ROLE The AI Research Scientist works at the intersection of AI, psychometrics and research, ensuring that Maki’s automated scoring systems are scientifically robust, fair, and continuously improving. 1. Evaluating and testing LLMs for scoring - Assess the accuracy and reliability of LLMs used for automated scoring, such as structured grids, job-specific skills and multilingual proficiency (written and spoken). - Benchmark closed- and open-source models across assessment types and languages. - Test robustness to prompt variation, input noise and adversarial behaviour such as prompt injection. - Explore prompt engineering and model optimisation, and recommend which models and prompts go into production. 2. Building our own scoring models - Move our scoring onto self-hosted open-weight models, and fine-tune them on expert-rated data where they can beat frontier models. - Check every model switch keeps accuracy, reliability and fairness at least as good as today's. 3. Comparing AI with human judgement, and calibrating psychometric models - Design studies that compare AI scores with expert ratings, against both individual experts and expert averages, to establish agreement and validity. - Develop hybrid scoring pipelines with engineering that combine human oversight with AI scoring. - Evaluate and calibrate psychometric models for validity and comparability ac…
Sign in to see the original posting
Browse more developer jobs on RoleBurst