Job at a glance
- Stack
- Python
- Location
- Porto
- Work setup
- On-site
- Posted
- Over a month ago
Description
Exciting Opportunity for a Mid-Level QA Automation Engineer for Generative AI and LLM!
Join an innovative and forward-thinking team that shapes the future of AI-powered functionalities. Bring your automation and testing expertise to drive quality, security, and performance in cutting-edge AI systems.
Location and Work Model
Portugal, Porto — Hybrid (up to 4 remote days per week)
About the Role
As a Mid-Level QA Automation Engineer for Generative AI and LLM, you will work for an international IT company that provides advanced AI solutions. Your focus will be on building and maintaining the testing infrastructure that ensures AI functionalities—such as conversational AI, AI Evaluator, and productivity tools—are accurate, reliable, and compliant. This position offers an excellent growth opportunity in a dynamic environment at the forefront of AI innovation.
Your Key Responsibilities:
- Develop and maintain automated testing frameworks using tools like Playwright, Cypress, Selenium, or similar.
- Write custom automation scripts in Python or JavaScript/TypeScript to enhance testing capabilities.
- Design evaluation datasets and create automated evaluation pipelines, including LLM methods as judge.
- Analyze AI outputs for hallucinations, errors, biases, and help optimize model behavior.
- Integrate AI and functional testing into CI/CD pipelines for continuous validation.
- Use AI observability and tracing tools to debug model behavior in production environments.
- Create comprehensive test cases for conversational AI, covering dialogue flows, edge cases, and adversarial requests.
- Support testing for fairness, bias mitigation, and compliance to ensure ethical AI operations.
You are a great fit for this role if you have:
- More than 3 years of experience in QA/SDET roles, with hands-on automation development.
- Proficiency in automation tools, such as Playwright, Cypress, Selenium.
- Strong scripting skills in Python and JavaScript/TypeScript.
- Knowledge of AI-specific testing challenges, including request regression, hallucination detection, and output evaluation.
- Experience with evaluation datasets and AI observability tools.
- Understanding of fairness, bias testing, and compliance considerations in AI systems.
- Familiarity with CI/CD pipelines and automation integration.
- Excellent root cause analysis skills for complex AI behaviors.
It's a big plus if you have:
- Experience with AI-assisted test generation and self-healing test scripts.
- Knowledge of LLM evaluation strategies.
- Relevant certifications in QA or AI security.
Language required for the role:
Fluent English (verbal and written)
Eligibility to work in:
Candidates must have the legal right to work in Europe.
Interested? Apply now by submitting your CV (preferably in English) along with a brief statement confirming your consent for data processing and storage.
