Run Automatic Integration Tests

With Coherence, you can configure your cnc.yml to automatically run integration tests against your deployments to a non-production environment.

Navigate to your environment homepage in the Coherence UI and add the following configuration section to your cnc.yml:

integration_test:
  type: integration_test
  command: [“npx”, 'cypress', 'run']
  image: 'cypress/included:10.9.1'

Modify command: and image: to match your test script requirements.

Coherence will set a COHERENCE_BASE_URL environment variable that describes the URL of the Coherence environment that you are running in. You can make requests to this URL from within your tests.

Any environment configuration variables required for testing can be set in the Variables tab on your environment homepage, and will be injected into the integration_test step in your builds.

Full Playwright example

Consider the example of running integration tests for a Coherence application with a service named "my-next-app" and the following cnc.yml:

my-next-app:
  type: backend
  local_packages: ["node_modules"]
  dev: ["npm", "run", "dev"]
  prod: ["npm", "run", "start"]

This application also contains the following file tree in its repo:

.
|-- Dockerfile
|-- src
|   |-- ...src here
|-- e2e
|   |-- README.md
|   |-- globalSetup.ts
|   |-- package.json
|   |-- playwright.config.ts
|   |-- state.json
|   |-- tests
|   |   `-- smoke.spec.ts
|   `-- yarn.lock
|-- index.html
|-- package.json
|-- vite.config.js
`-- yarn.lock

All you need to do to run the integration tests is add the following configuration information to the cnc.yml:

integration_test:
  type: integration_test
  command: ["cd", "e2e", "&&", "yarn", "install", "&&", "yarn", "playwright", "test"]
  image: "mcr.microsoft.com/playwright:v1.36.1-focal"

This YAML causes integration tests to run in each Coherence pipeline to a branch environment after the deploy step.

Optionally, you can also:

Add baseURL: process.env.COHERENCE_BASE_URL to the use block of your playwright.config.ts to use the smoke.spec.ts script to read the COHERENCE_BASE_URL variable from each environment and identify which URL to hit in each environment.
In a workspace, the COHERENCE_BASE_URL will be set to the appropriate local port or path for the application.
In a preview environment, the COHERENCE_BASE_URL will be set to the URL of the environment.
Use reporter: 'list' to output failure information to the terminal instead of a local HTML server (this should also be set).
Replace the process.env.CI check with the Coherence-provided COHERENCE_CI variable by using process.env.COHERENCE_CI in your plawright.config.ts file.
Run yarn playwright test --trace on to get trace information out of your workspaces (because the port for the debug web server isn't forwarded). When test completes:
Refresh your file browser.
Open the test-results folder and find the test you want.
Download the trace.zip file by right-clicking it and selecting Download.
Go to test.playwright.dev and drag the zip folder onto the screen.
Specify where to save the traces by adding outputDir: 'test-results/' to your plawright.config.ts file.
Use reporter: 'list' to disable the default debug web server from starting after the test run.

The complete updated cnc.yml for the example application would look as follows:

backend:
  dev:
    - npm
    - run
    - dev
  prod:
    - npm
    - run
    - start
  type: backend
  local_packages:
    - node_modules
integration_test:
  type: integration_test
  image: mcr.microsoft.com/playwright:v1.36.1-focal
  command:
    - cd
    - e2e
    - '&&'
    - yarn
    - install
    - '&&'
    - yarn
    - playwright
    - test

And the playwright.config.ts would look as follows:

import { defineConfig, devices } from '@playwright/test';

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
// require('dotenv').config();

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
  testDir: './tests',
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.COHERENCE_CI,
  /* Retry on CI only */
  retries: process.env.COHERENCE_CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.COHERENCE_CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: 'list',
  /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
  use: {
    /* Base URL to use in actions like `await page.goto('/')`. */
    baseURL: process.env.COHERENCE_BASE_URL,

    /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
    trace: 'on-first-retry',
  },

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },

    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },

    /* Test against mobile viewports. */
    // {
    //   name: 'Mobile Chrome',
    //   use: { ...devices['Pixel 5'] },
    // },
    // {
    //   name: 'Mobile Safari',
    //   use: { ...devices['iPhone 12'] },
    // },

    /* Test against branded browsers. */
    // {
    //   name: 'Microsoft Edge',
    //   use: { ...devices['Desktop Edge'], channel: 'msedge' },
    // },
    // {
    //   name: 'Google Chrome',
    //   use: { ...devices['Desktop Chrome'], channel: 'chrome' },
    // },
  ],

  /* Run your local dev server before starting the tests */
  // webServer: {
  //   command: 'npm run start',
  //   url: 'http://127.0.0.1:3000',
  //   reuseExistingServer: !process.env.CI,
  // },

  outputDir: 'test-results/'
});

An advanced note on the use of `yarn install`

The example test uses the playwright image from mcr and uses Dockerfile to install the package.json.

The playwright image is useful because its browser OS and package dependencies are correctly installed to be used as expected by Playwright. Similar project-provided images are available for other popular testing tools like Cypress.

The project-provided image requires yarn install to be run in the cnc.yml integration test command, as it installs the dependencies in the Playwright container.

However, the use of yarn install is not ideal, as the installation is a slow process that may result in occasional network errors.

This can be avoided by building your own image based on the testing tool and the packages installed at build time.

Please contact Coherence support if you need further assistance.

Run Automatic Integration Tests

Full Playwright example

An advanced note on the use of yarn install

An advanced note on the use of `yarn install`