LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will analyze AI‑generated content for accuracy, coherence, factual reliability, bias, safety, and alignment with our guidelines, working remotely with a global team.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines for ambiguous cases.
• Participate in cross‑checking sessions to ensure inter‑rater reliability.
• Analyze underperforming outputs, hypothesize root causes, and flag novel behaviors.
• Contribute to a living library of unique model outputs and failure modes.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP/AI quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Strong understanding of prompt engineering and model limits.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection.
• Proven ability to monitor and improve evaluation consistency across teams.

Required skills

• Prompt engineering
• RLHF data collection
• Dataset sourcing, cleaning, and annotation
• A/B testing for AI models
• Experiment design for model comparison