Job Description
- *Project Overview:**
Join a growing community of professionals advancing the next wave of AI. As an AI Trainer, you’ll play a hands-on role by analyzing and providing feedback on data to improve LLM performance, helping ensure that the next generation of AI technology is accurate and trustworthy.
- We are seeking a skilled Power Platform Governance Architect to work as a project consultant in our AI Labor Marketplace. This is not a full-time employment position — you will be engaged as an expert project consultant on a contract basis.
- *Location:**
- U.S.-based experts only
- *Engagement:**
- Part-time, project-based expert evaluation work
- *Work Type:**
- Remote
- *Project Summary:**
- Design and produce structured evaluation datasets to train and benchmark an AI Advisor Agent operating on Microsoft Power Platform tenant-state data. The consultant will define evaluation schemas, scoring logic, and governance-aligned validation standards, and create approximately 400 structured JSON-based evaluation test cases across Metrics, Insights, Action Plan Step, and Action Plan Generation components.
- *Consultant Engagement Terms:**
- This is a project-based consultant role. Consultants will be paid on a per-project basis; hourly rates are estimates based on anticipated completion time. Consultants control their own schedule, provide their own tools, and may simultaneously provide services to other vendors/employers (subject to those vendors’ allowances).
- *Responsibilities:**
- *Contributors will:**
- Design structured tenant-state schemas for evaluation purposes
- Define evaluation task structure, including expected outputs and scoring criteria
- Develop governance-realistic synthetic Power Platform scenarios
- Author structured JSON inputs and expected outputs
- Create scoring rubrics and validation logic for agent evaluation
- Ensure logical correctness, schema integrity, and compliance alignment
- Produce ~400 gold-standard evaluation test cases over 12 weeks
- Document assumptions and edge-case handling standards
- *Expected Outcomes:**
- A defined evaluation framework and scoring methodology
- A structured tenant-state schema for agent evaluation
- ~400 validated evaluation test cases
- Documented scoring and validation standards aligned to governance best practices
- *Qualifications:**
- 6–10+ years of experience as a Power Platform Administrator or Architect
- Deep experience with tenant governance, DLP, environments, connectors, and ALM
- Hands-on experience with CoE Toolkit or tenant telemetry analysis
- Strong proficiency authoring and validating structured JSON
- Prior experience designing QA frameworks, evaluation rubrics, or scoring guidelines
- Ability to reason at enterprise governance depth
Apply Now
Apply Now