AI Data & RLHF Reviewer: Response Comparison and Annotation

About the work

Models learn from human judgements, so those judgements need to be careful and consistent. We produce preference data, annotations and reviewed training examples that follow written guidelines and can be audited.

How it works

You compare model responses side by side, choose the better one against a written guideline and record why. You also annotate and correct training examples. Quality reviewers sample your work and share calibration feedback so decisions stay consistent across the team.

Who this is for

Careful readers and writers who can apply a guideline consistently over many items. Experience in content review, editing, data annotation or quality assurance is a strong fit, as is a technical background for code-related review.

What you’ll do

• Compare model responses and record preferences against a written guideline
• Write short, specific justifications for each decision
• Annotate and correct training examples
• Flag unclear guidelines and edge cases to the project lead
• Take part in calibration rounds to keep decisions consistent

Who should apply

• 2+ years of experience in review, editing, annotation, QA or a related field
• Strong reading comprehension and written English
• Ability to follow detailed guidelines consistently
• Comfortable explaining your judgement in a few clear sentences

Contract & payment terms

• You will be engaged as an independent contractor, not as an employee.
• This is fully remote work, open to applicants in any location.
• Work is project-based. Availability varies, and an application does not guarantee an assignment.
• Pay, scope, review criteria and access are confirmed at onboarding, after you sign a Non-Disclosure Agreement (NDA) and an Independent Contractor Agreement (ICA).
• Please do not share confidential material from a current or former employer or client.