Job Description: Research Evaluation Specialist (PhD, Master's, or Researcher)

Role Type: Contractor (Approximately 15 hours per week)
Location: Remote, Worldwide

Job Summary

We are seeking experienced Research Evaluation Specialists with advanced academic qualifications or substantial research experience to contribute to the development of rigorous evaluation materials for advanced AI models.

In this role, you will leverage your subject-matter expertise, research skills, and analytical reasoning to create challenging questions and evidence-based answers that assess AI model capabilities. You will conduct in-depth research, verify findings using authoritative sources, and refine evaluation materials to ensure accuracy, complexity, and clarity.

Prior experience in AI training, data labeling, or model evaluation is required, along with strong research expertise in your respective discipline.

Scope of Work

1. Develop original, high-difficulty question-and-answer pairs that assess advanced AI model capabilities within your area of expertise.
2. Research, verify, and thoroughly document answers using primary sources, scholarly literature, and authoritative references, including clear citations and supporting reasoning.
3. Design questions that require advanced reasoning, methodological understanding, and synthesis across multiple sources while avoiding oversimplified solutions.
4. Test questions against AI systems, identify items that do not adequately challenge the models, and iteratively increase complexity while maintaining factual accuracy.
5. Write questions and answers with exceptional precision and clarity, eliminating ambiguity and ensuring that responses are logically defensible.
6. Incorporate reviewer feedback and consistently follow project guidelines and quality standards.
7. Maintain high standards of research integrity, documentation, and analytical rigor across all deliverables.

Preferred Qualifications

1. A completed Master's or PhD, active doctoral candidacy, or equivalent research experience as a specialist, researcher, assistant professor, or professor in the relevant discipline.
2. A demonstrated record of scholarly research, advanced subject-matter expertise, and familiarity with primary academic literature.
3. Exceptional analytical thinking, attention to detail, and strong written communication skills in English.
4. Experience locating, evaluating, and cross-verifying information from authoritative sources.
5. Strong ability to work independently, manage priorities, and deliver high-quality research within a remote working environment.
6. Expertise in developing original, challenging, and methodologically sound questions that assess deep subject-matter understanding.
7. Required: Previous experience in AI training, data labeling, or AI model evaluation.

Compensation Structure

Compensation is output-based, with payment provided for each task that meets the project's specifications and quality requirements. The time required to complete tasks may vary depending on individual expertise and workflow.

Minimum submission requirements apply, and contributors are expected to complete a specified number of tasks each week.

Start Timeline and Availability

• Hiring Timeline: Positions are typically filled within 48 hours.
• Availability: Candidates should be prepared to begin promptly.
• Project Start: Selected candidates are expected to start their first tasks within 24–48 hours of completing onboarding.

Ideal Candidate Profile

The ideal candidate combines advanced academic or research expertise with practical experience in AI training or evaluation. They can develop sophisticated evaluation questions, support answers with authoritative evidence, identify weaknesses in AI-generated responses, and deliver precise, well-documented work independently while maintaining rigorous research and quality standards.