Program Analyst Sr – Generative AI, Prompt Engineering & Validation

Company:

Qualcomm India Private Limited

Job Area:

Operations Group, Operations Group > Program Analyst

General Summary:

Assists with program development and implementation through managing processes, procedures, and tools that improve efficiencies. A Program Analyst coordinates across teams and monitors timelines, budgets, risks, and priorities to achieve program progress. Typically, a program needing a Program Analyst will be of significant size and will require expertise related to the development of project management mechanisms.

Minimum Qualifications

• Associate's degree in Business Administration, Management, Computer Science, Engineering, Computer Science, or related field.

OR

High School Diploma or equivalent and 2+ years of relevant work experience.

Job Title

Program Analyst Sr – Generative AI, Prompt Engineering & Validation

Location: Bangalore

Experience: 2–6 years

Employment Type: Full-time

About the Role

We are looking for a proactive and detail-oriented Program Analyst to support Generative AI initiatives focused on prompt engineering, feature validation, test execution, output quality evaluation , and program tracking . The role will support internal GenAI platforms and tools, including search, summarization, RAG-based workflows, analytical agents, and domain-specific AI assistants.

The ideal candidate should combine strong program coordination skills with a practical understanding of GenAI systems, structured prompt-based testing, multi-source data validation, and clear reporting for engineering and leadership stakeholders.

Key Responsibilities

1. GenAI Program Tracking & Coordination

• Track GenAI feature validation plans, deliverables, owners, dependencies, and closure status across teams.
• Maintain dashboards, trackers, action item logs, risk registers, and validation status reports.
• Coordinate with engineering, product, platform, and domain expert teams to drive timely completion of testing and feedback cycles.
• Consolidate weekly progress updates, blockers, risks, and recommendations for program reviews.
• Ensure validation documentation, prompt libraries, and issue logs are accurate, current, and accessible.

2.

Prompt Engineering & Prompt Test

Design

• Create, refine, and maintain prompts for GenAI features, search workflows, summarization use cases, and analytical agents.
• Develop zero-shot, few-shot, role-based, multi-turn, edge-case, and negative test prompts.
• Build reusable prompt templates and feature-wise prompt libraries.
• Compare prompt variants and document impact on accuracy, relevance, completeness, consistency, and latency.
• Translate domain expert feedback into improved prompts and updated test scenarios.

3. Feature Validation & Test Execution

• Design and execute structured test cases for GenAI features, including functional, sanity, regression, negative, and exploratory testing.
• Validate end-to-end behavior from user prompt to retrieved context, generated response, and final output format.
• Map feature requirements to prompts, expected outputs, acceptance criteria, and validation evidence.
• Identify defects, usability gaps, hallucinations, incomplete answers, inconsistent responses, and low-confidence outputs.
• Support release readiness assessment using clear status indicators such as Go / Watch / Risk.

4. Data, Search & RAG Validation

• Validate AI outputs against structured, semi-structured, and unstructured data sources such as SharePoint, Confluence, Jira, documents, JSON/XML, and internal knowledge bases.
• Check search relevance, context quality, source completeness, and data freshness before summarization.
• Support validation of RAG-based workflows, semantic search, keyword search, global filters, and domain filters.
• Identify data gaps, indexing issues, retrieval failures, and source-quality problems impacting AI response accuracy.
• Maintain golden datasets, expected-answer sets, and benchmark queries for repeatable validation.

5. Output Quality Evaluation & Defect Analysis

• Evaluate GenAI responses for factual correctness, relevance, completeness, clarity, format quality, and consistency across repeated runs.
• Log defects with clear reproduction steps, prompt used, expected output, actual output, severity, and owner.
• Perform first-level root cause analysis to distinguish prompt issues, data issues, retrieval issues, model limitations, and product defects.
• Work with SMEs to validate technical correctness and define acceptance thresholds for complex responses.
• Track quality improvements over time and report trends in accuracy, hallucination reduction, and feature readiness.

6. Reporting, Documentation & Stakeholder Communication

• Prepare weekly validation summaries, risk updates, action item trackers, and executive-ready status reports.
• Document prompt libraries, test suites, validation evidence, known issues, and improvement logs.
• Facilitate review meetings, capture minutes of meeting, document decisions, and follow up on action items.
• Communicate clearly with cross-functional stakeholders across engineering, platform, product, and program management teams.
• Support leadership reviews with concise summaries, dashboards, and recommendations.

7.

Automation & Productivity

Support

• Use Excel, Power BI, Jira, Confluence, SharePoint, or similar tools to maintain trackers, dashboards, and reporting views.
• Support basic scripting or automation for batch prompt execution, response capture, comparison, and reporting where applicable.
• Help improve validation efficiency through reusable templates, structured checklists, and repeatable test processes.
• Track productivity metrics such as test coverage, closure rate, defect aging, prompt reuse, and feature readiness.

Required Skills & Competencies

• 2–6 years of experience in program analysis, project coordination, validation, QA, data analysis, or technical operations.
• Hands-on exposure to Generative AI tools, LLM-based applications, prompt engineering, or AI-assisted workflows.
• Strong ability to create structured test cases, validation reports, action trackers, and executive summaries.
• Good understanding of prompt-response behavior, hallucinations, context limitations, and output evaluation methods.
• Working knowledge of search, summarization, RAG concepts, information retrieval, or knowledge-base validation.
• Strong Excel skills; Power BI, Jira, Confluence, SharePoint, or dashboarding experience is preferred.
• Basic Python, SQL, or scripting knowledge is preferred but not mandatory.
• Excellent written and verbal communication skills with strong attention to detail.

Preferred Qualifications

Education & Background

• Bachelor’s degree in Engineering, Computer Science, Electronics, Data Science, Information Systems, or a related discipline.
• Prior experience in semiconductor, SoC, embedded systems, platform engineering, enterprise AI, or technical program management is preferred.
• Experience supporting validation of internal tools, AI platforms, search systems, summarization tools, or workflow automation platforms is a plus.
• Exposure to responsible AI, data privacy, model evaluation, or enterprise GenAI governance is desirable.

Success Metrics

• Completeness and accuracy of GenAI feature validation coverage.
• Quality, reusability, and effectiveness of prompt test suites.
• Timely closure of validation defects, risks, action items, and stakeholder feedback.
• Improvement in output accuracy, relevance, consistency, and hallucination reduction.
• Clarity and usefulness of weekly reports, dashboards, executive summaries, and readiness recommendations.

Applicants

Qualcomm is an equal opportunity employer. If you are an individual with a disability and need an accommodation during the application/hiring process, rest assured that Qualcomm is committed to providing an accessible process. You may e-mail [email protected] or call Qualcomm's toll-free number found