Mid-Level Generative AI and LLM QA Automation Engineer

Exciting Opportunity for a Mid-Level Generative AI and LLM QA Automation Engineer!

Join a forward-thinking, innovative team shaping the future of AI-powered features. Bring your expertise in automation and testing to drive quality, safety, and performance in cutting-edge AI systems.

Location & Work Model

Portugal, Porto — Hybrid (up to 4 remote days per week)

About the Role

As a Mid-Level Generative AI and LLM QA Automation Engineer, you will be working for an international IT company delivering advanced AI solutions. Your focus will be on building and maintaining the testing infrastructure that ensures AI features—such as conversational AI, AI Rater, and productivity tools—are accurate, reliable, and compliant. This role offers an excellent opportunity for growth in a dynamic environment at the forefront of AI innovation.

Your main responsibilities:

• Develop and sustain automated test frameworks using tools like Playwright, Cypress, Selenium, or similar.
• Write custom automation scripts in Python or JavaScript/TypeScript to enhance testing capabilities.
• Design evaluation datasets and create automated evaluation pipelines, including LLM-as-judge methods.
• Analyze AI outputs for hallucinations, errors, biases, and help optimize model behavior.
• Integrate AI and functional tests into CI/CD pipelines for continuous validation.
• Utilize AI observability and tracing tools to debug model behavior in production environments.
• Create comprehensive test cases for conversational AI, covering dialogue flows, edge cases, and adversarial prompts.
• Support testing for fairness, bias mitigation, and compliance to ensure ethical AI operations.

You’re ideal for this role if you have:

• 3+ years of experience in QA/SDET roles, with hands-on automation development.
• Proficiency in automation tools such as Playwright, Cypress, Selenium.
• Strong scripting skills in Python and JavaScript/TypeScript.
• Knowledge of AI-specific testing challenges, including prompt regression, hallucination detection, and output evaluation.
• Experience working with evaluation datasets and AI observability tools.
• Understanding of fairness, bias testing, and compliance considerations in AI systems.
• Familiarity with CI/CD pipelines and automation integration.
• Excellent root cause analysis skills for complex AI behaviors.

It is a strong plus if you have:

• Experience with AI-assisted test generation and self-healing test scripts.
• Knowledge of LLM evaluation strategies.
• Relevant certifications in QA or AI safety.

Language Required for the role:

Fluent English (verbal and written)

Eligibility to work in:

Candidates must have the legal right to work in Europe.

Interested? Apply now by submitting your CV (preferably in English) along with a brief statement confirming your consent to data processing and storage.

Apply here

Contacts and Address

Twitter Linkedin Facebook