How to Set Up AI QA Testing in Your CI/CD Pipeline

Adding AI QA agents to your CI/CD pipeline is one of the highest-leverage changes you can make to your development workflow. Instead of waiting for manual QA cycles or maintaining brittle test scripts, autonomous agents run on every push and surface bugs before they reach staging. This guide walks you through the setup for GitHub Actions, GitLab CI, and Jenkins — with working configuration files you can copy into your repo today.

How the Pipeline Fits Together

The architecture is straightforward. Your CI job fires on a push or pull request and makes a single authenticated HTTP call to the P1·QA engine. The QA Lead reads the target, decides which agents are relevant, and runs them in parallel — security, accessibility, SEO, end-to-end flows, API checks and more. The call returns a JSON report containing an overall grade, a pass rate, and every bug found with its severity and reproduction steps. A targeted run typically completes in two to five minutes.

Every request goes to the same endpoint: POST /api/suite/smart, authenticated with the X-Quantacore-Key header. Your engine URL and API key are both issued during onboarding — store each one as a CI secret and reference it by name, exactly as the examples below do. Never inline either value in a committed config file. The three examples that follow are the same call expressed in three CI syntaxes.

GitHub Actions Setup

Create .github/workflows/qa-agents.yml. The workflow listens for push and pull_request events and calls the engine against your deployed staging URL. Because the engine tests a running application rather than your source tree, point it at an environment that is already deployed and reachable — a preview deployment URL works well here.

# .github/workflows/qa-agents.yml
name: P1·QA Scan

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  qa-scan:
    runs-on: ubuntu-latest
    steps:
      - name: Run P1·QA smart suite
        run: |
          curl -sS --fail-with-body \
            -X POST "${{ secrets.P1QA_ENGINE_URL }}/api/suite/smart" \
            -H "Content-Type: application/json" \
            -H "X-Quantacore-Key: ${{ secrets.P1QA_API_KEY }}" \
            -d '{"url": "https://staging.example.com", "maxPages": 10, "timeout": 30000}' \
            -o qa-report.json

      - name: Show grade
        run: |
          jq -r '"Grade: \(.summary.grade)  Pass rate: \(.summary.passRate)"' qa-report.json

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: p1qa-report
          path: qa-report.json

The report is uploaded as a build artifact so you keep a historical record of every run. To gate merges on quality, add a step that reads summary.grade from the JSON and exits non-zero below your chosen threshold.

GitLab CI Setup

For GitLab, add a qa stage to .gitlab-ci.yml. GitLab supports merge request pipelines natively, so you get automatic triggering with no extra webhook configuration. Store the key as a masked, protected CI/CD variable under Settings → CI/CD → Variables.

# .gitlab-ci.yml
stages:
  - build
  - qa

qa-agents:
  stage: qa
  image: alpine:3.20
  before_script:
    - apk add --no-cache curl jq
  script:
    - |
      curl -sS --fail-with-body \
        -X POST "$P1QA_ENGINE_URL/api/suite/smart" \
        -H "Content-Type: application/json" \
        -H "X-Quantacore-Key: $P1QA_API_KEY" \
        -d "{\"url\": \"$STAGING_URL\", \"maxPages\": 10}" \
        -o qa-report.json
    - jq -r '"Grade: \(.summary.grade)  Bugs: \(.allBugs | length)"' qa-report.json
  artifacts:
    when: always
    paths:
      - qa-report.json
    expire_in: 30 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Jenkins Setup

In a declarative Jenkinsfile, a single sh step triggers the suite and archives the JSON report as a build artifact for historical trend data. Use the Credentials plugin to inject the API key rather than hardcoding it.

// Jenkinsfile
pipeline {
  agent any

  environment {
    P1QA_ENGINE_URL = credentials('p1qa-engine-url')
    P1QA_API_KEY    = credentials('p1qa-api-key')
    STAGING_URL     = 'https://staging.example.com'
  }

  stages {
    stage('QA Agents') {
      steps {
        sh '''
          curl -sS --fail-with-body \
            -X POST "${P1QA_ENGINE_URL}/api/suite/smart" \
            -H "Content-Type: application/json" \
            -H "X-Quantacore-Key: ${P1QA_API_KEY}" \
            -d "{\"url\": \"$STAGING_URL\", \"maxPages\": 10}" \
            -o qa-report.json

          jq -r '"Grade: \(.summary.grade)  Pass rate: \(.summary.passRate)"' qa-report.json
        '''
      }
    }
  }

  post {
    always {
      archiveArtifacts artifacts: 'qa-report.json', fingerprint: true
    }
  }
}

Which Agents to Run on Every PR

The smart suite picks agents automatically, but you can constrain the run when speed matters. On every pull request, favour the fast, high-signal agents: end-to-end flows, functional checks, and security. Reserve the slower, browser-heavy work — full performance profiling, visual regression baselines, and exhaustive accessibility sweeps — for a nightly run against your main branch. That split keeps PR feedback under five minutes while still giving you full coverage once a day.

On failure handling, start conservative. Block a merge only on critical and high-severity findings, and treat medium and low as warnings for the first few weeks. Tighten the threshold once your team trusts the signal — a gate that cries wolf gets disabled, and a disabled gate catches nothing.

What You Get Back Today

It is worth being precise about the output, because this is where CI integrations usually over-promise. The engine returns a JSON report; your CI job decides what to do with it. Every run also appears in your P1·QA dashboard with the full graded report, each bug categorised by severity, and reproduction steps with screenshots. If you have configured them, Slack and email alerts fire on completion. Setup for both is covered in the onboarding guide.

What P1·QA does not do today is post results back into your pull request as native GitHub check annotations. That integration is on the roadmap, not shipped. In the meantime the pattern above — fail the CI step on a bad grade, archive the JSON as a build artifact, and link to the dashboard report — gives you the same merge-blocking behaviour using tooling your CI already has.

Common Pitfalls

Most setup problems come from one of four places. First, network reachability: your staging environment has to be publicly reachable from the CI runner, and private or VPN-only environments will simply time out. Second, scope: running an exhaustive crawl on every pull request pushes feedback past the point where anyone waits for it — keep maxPages modest on PR runs. Third, secrets: the key belongs in your CI secret store, and a key committed to the repository should be treated as compromised and rotated. Fourth, timeouts: give the CI step a ceiling comfortably above your expected run time so a slow scan fails loudly instead of hanging the pipeline.

Once configured, the setup largely maintains itself. The end-to-end agent keeps its own selector map and re-resolves locators when your UI changes, and the bug reporter deduplicates findings across runs so the same known issue does not reappear as new noise every morning. If you want a working baseline before wiring any of this up, run a free audit against your staging URL and compare it to what the pipeline reports on your first green build.