Job Summary
We are seeking an experienced Senior Consultant – AI Product / Business Analyst to support the development and validation of AI-powered products and solutions. The ideal candidate will have strong experience working as a Business Analyst, Product Analyst, Product Owner, or similar role, along with hands-on experience in AI/LLM evaluation, prompt engineering, AI testing, and product validation.
The candidate will work closely with Business Stakeholders, Engineering, UX, and Data Science teams to translate business and AI requirements into actionable product features, user stories, workflows, and acceptance criteria. Experience in the Healthcare domain is highly preferred, along with hands-on AI experience.
Key Responsibilities
- Gather and analyze business requirements for AI-powered products and solutions.
- Translate business requirements into detailed Jira user stories, acceptance criteria, workflows, and product requirements.
- Manage and prioritize product backlogs in collaboration with Product Owners and stakeholders.
- Design and execute specialized AI/LLM test cases and evaluation scenarios.
- Validate LLM-generated outputs for accuracy, quality, completeness, relevance, consistency, and reliability.
- Evaluate different prompt variations and identify opportunities for prompt optimization.
- Develop and execute test scenarios for AI agents and agentic workflows.
- Perform AI model/product validation and analyze AI performance metrics.
- Identify defects, inconsistencies, and quality gaps in AI-generated content and workflows.
- Collaborate with Engineering, UX, Data Science, and Business teams to define and prioritize AI capabilities.
- Support Agile ceremonies including backlog refinement, sprint planning, reviews, and retrospectives.
- Drive continuous improvement of AI product quality and user experience.
- Document requirements, test results, evaluation findings, and product recommendations.
- Partner with stakeholders to ensure AI solutions meet business objectives and user needs.
Required Skills & Qualifications
- 8–10+ years of experience as a Business Analyst, Product Analyst, Product Owner, Product Consultant, or equivalent role.
- Hands-on experience working with AI-powered products, Generative AI, LLMs, or AI/ML solutions.
- Strong experience creating Jira user stories, acceptance criteria, product requirements, and workflows.
- Hands-on experience with AI/LLM testing and evaluation.
- Experience evaluating LLM outputs and comparing prompt variations.
- Strong understanding of prompt engineering and prompt optimization.
- Experience validating AI-generated content for accuracy, completeness, consistency, and quality.
- Experience testing or evaluating AI agents and agentic workflows.
- Strong Agile/Scrum experience.
- Excellent communication and stakeholder management skills.
- Ability to work effectively with Engineering, UX, Data Science, Product, and Business teams.
Preferred / Nice-to-Have Skills
- Healthcare domain experience is highly preferred.
- Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions.
- Experience working with AI-powered enterprise applications.
- Knowledge of AI evaluation methodologies and performance metrics.
- Experience with Jira and Agile product development environments.
- Experience contributing to UX/product discovery and user journey analysis.
Top 3 Responsibilities
- Requirements & Product Management: Gather business requirements, create detailed Jira user stories, define acceptance criteria, and manage product backlogs while translating AI product requirements into actionable features and workflows.
- AI/LLM Evaluation & Testing: Plan, execute, and monitor AI/LLM evaluations by designing test scenarios, validating model outputs, evaluating prompts and agentic workflows, analyzing metrics, and driving continuous product quality improvements.
- Cross-Functional Collaboration: Collaborate with Engineering, UX, Data Science, Product, and Business stakeholders to prioritize AI capabilities and ensure successful delivery of AI-powered products.