Senior Data Annotation Jobs in Montreal

Overview

Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full-time role.

Key Responsibilities

• Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling
• Design, maintain, and enforce annotation guidelines compliance and taxonomy standards
• Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication
• Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits
• Conduct prompt evaluation and rubric-based scoring aligned to model requirements
• Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality
• Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines
• Document edge cases and escalation paths to reduce guideline drift
• Support onboarding and calibration using examples, gold sets, and retraining plans
• Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy

Required Qualifications

• Senior data annotation or data labeling operations experience with QA evaluation ownership
• Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution
• Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows
• Familiarity with NLP tasks such as named entity recognition and text classification
• Comfortable working remotely with cross-functional stakeholders

Nice to Have

• Content safety labeling or policy-driven evaluation experience
• Inter-annotator agreement metrics and quality sampling methodologies
• Computer vision annotation exposure (bounding boxes, polygons, segmentation)
• Multilingual evaluation or locale-specific guidelines

Work Model

Remote, Full-time.

Apply with relevant annotation experience and examples of QA evaluation or guideline development work.