Interaction Designer

Our client,

a leader in financial and tax software,

is seeking an

Interaction Designer/AI Evaluation

to join their team!

**This is a 5-month W2 contract with benefits and paid holidays**

Team focused on

digital help experiences and generative AI, helping improve how AI-powered assistants support customers across products

. In this role

, you will evaluate AI-generated conversations, establish quality standards, coordinate human evaluation efforts, and translate findings into actionable improvements.

Responsibilities

• AI Evaluation & Quality:

Lead evaluations of generative AI outputs, ensuring responses meet established quality guidelines and deliver effective customer experiences.

• Conversational Design:

Assess conversational flow, language quality, relevance, clarity, and the overall effectiveness of AI-generated customer interactions.

• Human-in-the-Loop Operations:

Coordinate evaluators and subject-matter experts, providing guidance, feedback, and training to improve annotation quality and consistency.

• Cross-Functional Collaboration:

Partner with product managers, engineers, AI teams, and designers to generate and evaluate AI responses across different use cases.

• Data Analysis & Insights:

Summarize evaluation results, identify trends and recurring issues, and recommend actionable improvements to AI experiences.

• Prioritization & Planning:

Coordinate evaluation and training data requests with product managers, balancing priorities and business needs.

• Quality Management:

Develop and refine evaluation processes, communicate quality status and risks, and support continuous improvement of generative AI experiences.

Required Qualifications

• 3+ years of relevant experience

in conversational design, AI evaluation, interaction design, or a related field.

• Experience evaluating chatbot interactions, AI-generated responses, or conversational quality.
• Foundational understanding of

LLMs, generative AI, and NLP

.

• Experience with

human annotation, data labeling, or AI quality evaluation

.

• Familiarity with Claude, AWS SageMaker Ground Truth, or similar evaluation tools.
• Understanding of RAG and AI quality improvement processes.
• Experience in customer support, fintech, or digital help experiences.

Motion Recruitment Partners is an Equal Opportunity Employer, including disability/vets. All applicants must be currently authorized to work on a full-time basis in the country for which they are applying, and no sponsorship is currently available. Employment is subject to the successful completion of a pre-employment screening.

Accommodation will be provided in all parts of the hiring process as required under Motion Recruitment Employment Accommodation policy. Applicants need to make their needs known in advance.