Job Description
Note: The job is a remote job and is open to candidates in USA. STAND 8 Technology Consulting provides end-to-end IT solutions to enterprise partners across the United States. They are seeking an AI/ML Prompt Scientist/Engineer to design, evaluate, and optimize prompt-based systems that drive real-world impact across cutting-edge AI applications. Responsibilities • Design, optimize, and evaluate prompts and orchestration pipelines to enhance LLM performance and scalability, supporting both Data Scientist and engineering workflows • Implement programmatic prompting using frameworks such as LangChain, LlamaIndex, DSPy, Guidance, and LangGraph • Define and manage evaluation frameworks, including A/B testing, human and automated scoring, and model quality assessments using tools such as Ragas, UpTrain, DeepEval, TruLens, Promptfoo, and OpenAI Evals • Engineer and maintain context management systems, including conversation memory, retrieval logic, and context compression strategies (chunking, summarization, prioritization) • Design and optimize retrieval and RAG pipelines, integrating vector databases (e.g., Weaviate) and rerankers (e.g., Cohere) to enhance relevance and accuracy • Establish observability and traceability mechanisms (e.g., Langfuse) to monitor performance metrics, latency, and operational costs • Implement guardrails, safety mechanisms, and bias mitigation policies using frameworks such as Guardrails AI, NeMo Guardrails, Outlines, and Rebuff • Collaborate with cross-functional engineering and research teams-including Data Scientists, ML, and AI engineering groups-to ensure high-quality, compliant, and well-documented system delivery Skills • Proven experience in LLM and NLP model operations, including tokenization, context windows, and safety considerations • Strong technical expertise in prompt engineering, orchestration frameworks, and context management • Hands-on experience with retrieval-augmented generation, evaluation frameworks, and observability tooling • Understanding of key evaluation metrics such as faithfulness, groundedness, coherence, toxicity, latency, and cost • Experience with compliance and ethical AI practices, including bias and hallucination mitigation • Excellent analytical, documentation, and communication skills, with the ability to collaborate effectively across teams • Model fine-tuning and alignment (e.g., LoRA, DPO/RLHF, preference tuning) • Experience developing AI agents and integrating tool-use capabilities Benefits • Medical coverage and Health Savings Account (HSA) through Anthem • Dental/Vision/Various Ancillary coverages through Unum • 401(k) retirement savings plan • Paid-time-off options • Company-paid Employee Assistance Program (EAP) • Discount programs through ADP WorkforceNow Company Overview • STAND 8 is a global technology consulting and staffing leader that helps businesses transform and scale through tailored IT solutions. It was founded in 2009, and is headquartered in Long Beach, California, USA, with a workforce of 201-500 employees. Its website is Company H1B Sponsorship • STAND 8 Technology Consulting has a track record of offering H1B sponsorships, with 1 in 2024, 5 in 2023, 3 in 2022, 6 in 2020. Please note that this does not guarantee sponsorship for this specific role. Apply tot his job