E2E Testing with Playwright

Background

The Playwright test suite contains system tests verifying the most important features of Artemis. System tests test the whole system and therefore require a complete deployment of Artemis first. CI selects affected modules, their declared consumer workflows and smoke tests for eligible internal pull requests. Integration-branch pushes and other full-run events retain complete E2E coverage.

Artemis uses GitHub Actions as its CI platform. E2E tests run on self-hosted runners because they require substantial compute resources, access to a PostgreSQL database, and a locally deployed Artemis instance. Selections include cross-module consumers: lecture changes also exercise competency progress and course tabs, while programming changes also exercise exams. Shared runtime, helpers and unclassified changes require the full suite. Each spec's runtime module dependencies are recorded in the selection contract. See the CI Pipeline section for details.

ℹ️

Information:

Testing Iris (the AI tutor) end to end against a real Pyris microservice — architecture, the mock LLM, and the nightly cross-repository monitor — is covered separately in E2E Testing Iris (Real Pyris).

Set up Playwright locally

To run the tests locally, developers need to set up Playwright on their machines. End-to-end tests test entire workflows; therefore, they require the whole Artemis setup - database, client, and server to be running. Playwright tests rely on the Playwright Node.js library, browser binaries, and some helper packages.

The recommended way to run E2E tests locally is the fast local runner — a single script (./supporting_scripts/e2e/run-e2e-tests-local-fast.sh) that handles database, server, client, and Playwright setup automatically. It auto-kills conflicting processes on ports 8080/9000, keeps services running between re-runs, and supports filtering individual tests.

Alternatively, you can use the Docker-based scripts in supporting_scripts/playwright (see the README there), or set up everything manually as described below. The manual setup relies on fully configuring your local Artemis instance following the server setup guide and is useful when you need IntelliJ integration for debugging.

Supporting scripts overview

The following helper scripts live in supporting_scripts/playwright and can be combined as needed:

• runArtemisInDocker_macOS.sh / runArtemisInDocker_linux.sh: start the Artemis server/client plus database in Docker.
• setupUsers.sh: create the Playwright test users.
• startPlaywright.sh: run the full Playwright suite headless in the terminal.
• startPlaywrightUI.sh: launch Playwright in UI mode for debugging.
• prepareVSCodeForE2ETests.sh: install dependencies and patch the Playwright config so VS Code can discover and run the E2E tests directly from the Testing view.
• checkFlakiness.sh: repeatedly run tests to investigate flaky behavior.

The three steps for setting up Playwright with these supporting scripts are:

1. Start Artemis by running runArtemisInDocker_macOS.sh or runArtemisInDocker_linux.sh (depending on your OS). This starts the database, server, and client; keep the client running and use a separate shell for the remaining steps.
2. Create test users by running setupUsers.sh (skip if users already exist).
3. Run tests either headless with startPlaywright.sh, in UI mode with startPlaywrightUI.sh, or via VS Code after running prepareVSCodeForE2ETests.sh.

Fast local runner (recommended for development)

For the fastest local development experience, use the supporting_scripts/e2e/run-e2e-tests-local-fast.sh script in the repository root. This is the recommended way to run E2E tests locally for both developers and AI agents. Unlike the Docker-based approach above, this script runs only PostgreSQL in Docker while running the server and client directly on the host. Services stay running between test runs, so re-runs only take seconds.

The script automatically kills any processes occupying ports 8080 (server), 9000 (client), and 7921 (local VC SSH), so you don't need to manually stop conflicting processes before running it.

Prerequisites: Docker, Java 25+, Node.js 24+, pnpm (activate via corepack enable).

Architecture:

• Docker: PostgreSQL only (port 5432)
• Host: Spring Boot server via ./gradlew bootRun -x webapp (port 8080)
• Host: Angular client via pnpm start (port 9000, proxies API calls to 8080)
• Host: Playwright tests connect to http://localhost:9000

First run (starts everything):

./supporting_scripts/e2e/run-e2e-tests-local-fast.sh

This starts PostgreSQL, boots the server (~30-90s), starts the client (~10-20s), then runs all Playwright tests.

Run specific tests:

Use --filter with a pattern (supports regex) to run only matching tests:

# Run tests with "Quiz" in the name
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --filter "Quiz"

# Run tests matching multiple patterns
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --filter "ExamAssessment|SystemHealth"

Re-run tests (services already running):

After the first run, services stay running. Skip startup for faster re-runs:

# Run all tests (skips starting services that are already running)
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --skip-server --skip-client --skip-db

# Run only specific tests
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --skip-server --skip-client --skip-db --filter "Quiz"

# Open Playwright UI for debugging
./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --skip-server --skip-client --skip-db --ui

Stop everything:

./supporting_scripts/e2e/run-e2e-tests-local-fast.sh --stop

Available options:

Flag	Description
--stop	Kill server, client, and database; exit
--filter <pattern>	Run only tests matching the pattern (supports regex, e.g., "Quiz" or "Quiz|Exam")
--skip-server	Reuse already-running server
--skip-client	Reuse already-running client
--skip-db	Reuse already-running PostgreSQL
--headed	Run Playwright in headed mode (visible browser)
--ui	Open Playwright UI mode for interactive debugging
--video	Enable video recording (off by default to save CPU)
--coverage	Enable coverage collection (off by default, requires extra memory)
--debug	Show server and client output inline (normally only in log files)

Logs and PID files are stored in the .e2e-local/ directory (gitignored).

If you want to manually install playwright, you can follow these steps:

Install dependencies

First, navigate to the Playwright folder:

cd src/test/playwright

Then install the dependencies (activate pnpm once via corepack enable if you haven't already):

pnpm install

Customize Playwright configuration

We need to configure Playwright to match our local Artemis setup and user settings. All configurations are stored in the playwright.env file. The default configuration for an ICL setup looks as follows:

ADMIN_USERNAME=artemis_admin
ADMIN_PASSWORD=artemis_admin
BASE_URL=https://localhost
EXERCISE_REPO_DIRECTORY=test-exercise-repos
FAST_TEST_TIMEOUT_SECONDS=45
SLOW_TEST_TIMEOUT_SECONDS=90

ℹ️

Information:

BASE_URL=http://localhost:9000 is correct for manual setup where the Angular dev server runs on port 9000.

The committed playwright.env defaults to BASE_URL=https://localhost (no port, HTTPS) for the Docker-based setup.

Make sure BASE_URL matches your Artemis client URL and ADMIN_USERNAME and ADMIN_PASSWORD match your Artemis admin user credentials.

Configure test users

Playwright tests require users with different roles to simulate concurrent user interactions. If you already have generated test users, you can skip this step. Generate users with the help of the user creation scripts under the supporting_scripts/playwright folder:

setupUsers.sh

You can configure user IDs and check their corresponding user roles in the src/test/playwright/support/users.ts file. Usernames are defined by support/users.ts: the user with ID n is artemis_test_user_<n>, and its password equals its username. The user creation script does not support other names.

Setup Playwright package and its browser binaries

Install Playwright browser binaries, set up the environment to ensure Playwright can locate these binaries. On some operating systems this might not work, and playwright needs to be manually installed via a package manager.

pnpm run playwright:setup-local
pnpm run playwright:init

Open Playwright UI

To open the Playwright UI, run:

pnpm run playwright:open

This opens a graphical interface that allows you to run individual tests, test files, or test suites while observing the test execution in a browser window.

Another way to run tests is through the command line. To run all tests in the command line, use:

pnpm run playwright:test

To run a specific test file, use:

pnpm exec playwright test <path_to_test_file>

If you want to run a specific test suite or a single test, add the -g flag to the previous command, followed by the test suite name or test name. For example, you can run the test suite "Course creation" located in the file CourseManagement.spec.ts using the command:

pnpm exec playwright test e2e/course/CourseManagement.spec.ts -g "Course creation"

Test parallelization

Running tests in parallel may speed up test execution. We achieve this using Playwright's built-in parallelization feature. By default, tests are configured to run in fully parallel mode. This means that all tests in all files are executed in parallel. Test execution tasks are divided among worker processes. Each process runs a separate browser instance and executes a subset of tests. The number of worker processes can be adjusted in the playwright.config.js file.

⚠️

Warning:

Using more worker processes divides the available computing resources, giving each worker fewer resources. Using too

many workers can lead to resource contention, slowing down individual test execution and potentially causing timeouts.

To run tests sequentially (one after another), set the workers option to 1. To run tests within each file sequentially, while running test files in parallel, set the fullyParallel option to false.

Test projects: fast and slow tests

Playwright is configured with two test projects that determine the per-test timeout budget:

Project	Tag	Timeout (local)	Timeout (CI)	Use for
fast-tests	@fast or no tag	45 s	60 s	Tests that complete quickly and do not involve long-running background jobs
slow-tests	@slow	90 s	90 s	Tests that wait for CI/CD builds, large data processing, or other asynchronous operations

Tag a test suite with { tag: '@slow' } when it involves build pipelines or polling for background jobs; otherwise omit the tag (defaults to fast-tests):

// Fast test (default — no tag needed)
test.describe('Competency Management', { tag: '@fast' }, () => { ... });

// Slow test
test.describe('Programming Exercise Participation', { tag: '@slow' }, () => { ... });

Best practices when writing new E2E tests

Understanding the System and Requirements

Before writing tests, a deep understanding of the system and its requirements is crucial. This understanding guides determining what needs testing and what defines a successful test. The best way to understand is to consolidate the original system's developer or a person actively working on this component.

Identify Main Test Scenarios

Identify what are the main ways the component is supposed to be used. Try the action with all involved user roles and test as many different inputs as feasible.

Identify Edge Test Scenarios

Next to the main test scenarios, there are also edge case scenarios. These tests include inputs/actions that are not supposed to be performed (e.g. enter a too-long input into a field) and test the error-handling capabilities of the platform.

Write Tests as Development Progresses

Rather than leaving testing until the end, write tests alongside each piece of functionality. This approach ensures the code remains testable and makes identifying and fixing issues as they arise easier.

Keep Tests Focused

Keep each test focused on one specific aspect of the code. If a test fails, it is easier to identify the issue when it does not check multiple functionalities at the same time.

Make Tests Independent

Tests should operate independently from each other and external factors like the current date or time. Each test should be isolated. Use API calls for unrelated tasks, such as creating a course, and UI interaction for the appropriate testing steps. This also involves setting up a clean environment for every test suite.

Use Descriptive Test Names

Ensure each test name clearly describes what the test does. This strategy makes the test suite easier to understand and quickly identifies which test has failed.

Use Similar Test Setups

Avoid using different setups for each test suite. For example, always check for the same HTTP response when deleting a course.

Do Not Ignore Failing Tests

If a test consistently fails, pay attention to it. Investigate as soon as possible and fix the issue, or update the test if the requirements have changed.

Regularly Review and Refactor Your Tests

Tests, like code, can accumulate technical debt. Regular reviews for duplication, unnecessary complexity, and other issues help maintain tests and enhance reliability.

Playwright testing best practices

Use page objects for common interactions

Page objects are a design pattern that helps to abstract the details of the page structure and interactions. They encapsulate the page elements and their interactions with the page. This makes the tests more readable and maintainable. Page objects are stored in the support/pageobjects folder. Each page object is implemented as a class containing a Playwright page instance and may have instances of other page objects as well. Page object classes provide methods performing common user actions or returning frequently used locators. Page objects are registered as fixtures to make them easily accessible in tests without caring about their initialization and teardown.

Use fixtures

Test fixture in Playwright is a setup environment that prepares the necessary conditions and state required for your tests to run. It helps manage the initialization and cleanup tasks so that each test starts with a known state. We use fixtures for all POMs and common test commands such as login. Fixtures are defined in support/fixtures.ts.

The page fixture enables navigation recovery by default. Tests that assert exact network request counts for one navigation use test.use({ recoverNavigation: false }) and wait for the actual page content to render. This prevents a recovery reload from adding cancelled requests to the count. Keep the exact request assertions and existing timeouts.

To create a fixture, define its instance inside a corresponding existing type or define a new one:

export type ArtemisPageObjects = {
    loginPage: LoginPage;
};

Ensure the base test (base) extends the fixture type. Define a fixture with the relevant name and return the desired instance as an argument of use() function as below:

export const test = base.extend<ArtemisPageObjects>({
    loginPage: async ({ page }, use) => {
        await use(new LoginPage(page));
    },
});

Inject the fixture to tests when needed as an argument to the test() function as follows:

test('Test name', async ({ fixtureName }) => {
    // Test code
});

Use uniquely identifiable locators

Use unique locators to identify elements on the page. Playwright throws an error when interacting with a locator that matches multiple elements on the page. Reach for locators in this order:

1. data-testid, through page.getByTestId() or [data-testid="..."] when it has to be combined with another attribute. This is the default attribute Playwright looks for and the one to add when an element has no hook yet.
2. An id, but only when that id already exists for a production reason of its own, such as a for on a label, an aria-describedby, or an anchor target. An id that exists only so that a test can find something is a data-testid wearing the wrong name.
3. Text-based and role-based locators, for the rare case where the user-facing text is the thing under test.

Prefer data-testid over text-based and role-based locators as the primary strategy. Text-based and role-based locators break whenever display text or component structure changes — turning a UI rename into a test failure unrelated to any regression. A data-testid attribute survives UI refactors as long as the attribute itself is preserved.

❗

Danger:

Never bind a locator to a class that exists for styling. Bootstrap, PrimeNG and Tailwind class names describe how an element looks, so restyling it silently breaks every test that hunted for it that way, and nothing in the diff points at the tests.

Both archive tests used to find the download button as button.btn-primary[data-mode="Course"]. When that button became a TUM AET UI button, the class disappeared and both tests waited about eleven minutes each for an element that could no longer exist.

Avoid using the nth() method or the nth-child selector, as they rely on the element's position in the DOM hierarchy. Use these methods only when iterating over multiple similar elements.

A data-testid is a contract: it tells whoever edits that template that a test depends on the element, which an id or a class never does. Keep the value stable and kebab-cased, and name it after what the element is (archive-download-button) rather than where it currently sits.

Markup you do not own

PrimeNG, Monaco and Apollon render their own DOM, so there is no template to put a data-testid in. That is not a reason to fall back on their class names: those are as much a styling detail as Bootstrap's, and they change between major versions.

PrimeNG answers this with the pass-through API. [pt] names each internal section of a component, so the attribute lands where the class hunt used to be. Declare it as a component field rather than a template literal, so change detection does not hand the component a fresh object on every cycle:

protected readonly passThrough = {
    listContainer: { 'data-testid': 'taxonomy-select-overlay' },
    option: { 'data-testid': 'taxonomy-select-option' },
};

<p-select [pt]="passThrough" ... />

Two details are worth knowing. A section whose name starts with pc forwards to a whole PrimeNG component, so the attribute belongs on that component's root: { pcToggleButton: { root: { 'data-testid': 'example-solution-toggle' } } }. And where one section renders more than one element, PrimeNG's own structural attributes tell them apart, without reaching for a class: icons carry a data-p-icon attribute that names them.

For a helper that means "whichever dialog is currently open", ask for the role instead of naming a framework: page.getByRole('dialog') matches the PrimeNG and the ng-bootstrap dialog alike.

Monaco is the one documented exception. It builds the editor, the code lines and the suggest widget itself, and its decoration API accepts a class name and nothing else, so .monaco-editor, .view-line and .suggest-widget have to be named directly. Scope them through a data-testid on the component that hosts the editor rather than through more Monaco classes, and say in a comment why the class is there.

Asserting a state class is not the same as finding an element by one

The rule above is about finding elements. Asserting that an element carries a state class is a different thing, and it is fine when the library exposes that state in no other way. Find the element by its own hook first, then assert the class on it, and re-check the assertion when the library is upgraded, since the class is an implementation detail.

For markup Artemis owns, do not settle for that: give the element an attribute that says what the test wants to know, such as data-selected or data-invalid, and assert on it. The TUM AET UI date picker does this: its calendar marks the selected day with aria-selected and gives every day button its date, so a test asserts on those:

await expect(dialog.locator('td[role="gridcell"][aria-selected="true"] button')).toHaveAttribute('data-date', '2026-12-20');

Consider actionability of elements

Checking for the state of an element before interacting with it is crucial to avoid flaky behavior. Actions like clicking a button or typing into an input field require a particular state from the element, such as visible and enabled, which makes it actionable. Playwright ensures that the elements you interact with are actionable before performing such actions.

However, some complex interactions may require additional checks to ensure the element is in the desired state. For example, consider a case where we want to access the inner text of an element that is not visible yet. Use waitFor() function of a locator to wait for its visible state before accessing its inner text:

await page.locator('.clone-url').waitFor({ state: 'visible' });
const urlText = await this.page.locator('.clone-url').innerText();

⚠️

Warning:

Avoid using page.waitForSelector() function to wait for an element to appear on the page. This function

waits for the visibility in the DOM, but it does not guarantee that the element is actionable. Always prefer the waitFor() function of a locator instead.

In some cases, we may need to wait for the page to load completely before interacting with its elements. Use waitForLoadState() function to wait for the page to reach a specified load state:

await page.waitForLoadState('load');

⚠️

Warning:

Waiting for the page load state is not recommended if we are only interested in specific elements appearing on

the page - use waitFor() function of a locator instead.

Layout contract tests

Some rules about a page are not about what a control does but about how big it is and where it sits: every page of a running exam starts with a title row of the same height at the same position, a button in that row is 30px high with air above and below it, a card keeps 12px to its content, the panels of a page are set apart by the 6px divider of the application, the toolbar of the code editor sits in the row where there is room for it, the sidebar and the content end on the same line, the summary scroller ends inside the window instead of growing with its content, and no page scrolls sideways. A layout contract test states such a rule as numbers and checks it in the browser, at several window sizes. The exam has two: e2e/exam/ExamLayoutContract.spec.ts walks a running exam, and the text complaint test of ExamAssessment.spec.ts checks that the Complain button leaves the page where it is, that the complaint is in a box and the rating is quiet, and ends by checking the summary with a result and a complaint in it.

Measure instead of comparing screenshots

A screenshot comparison fails on anti-aliasing, fonts, a date that changed or any other pixel nobody cares about. It needs a tolerance and a reviewer to decide whether a difference is a bug, and it does not say which rule broke. A measured contract fails with the element, the viewport and the numbers, and it needs no reference images to review or to update:

title row of the quiz page at 1280x800: height is 41px, expected 40px (tolerance 0.5px)

The helpers

support/layout.ts measures in the browser and retries until the client has finished laying out, so a test does not sleep after a resize or a navigation. The assertions retry a violated expectation until it holds or the timeout runs out, which defaults to the expect timeout of the project (ten seconds). Where they apply, they take { name, tolerance, timeout }: name says what the element is for and replaces its selector in the message, tolerance defaults to half a pixel, which covers sub-pixel rounding and nothing more. A helper refuses to compare what is not there: an element that never comes fails with its name, and a comparison of fewer than two elements or values, or of no edges, throws instead of passing.

Helper	Expects
forEachViewport(page, check)	the check to hold at every viewport of the matrix; each runs as a test.step, and the viewport is restored
expectHeight(locator, height)	an element to have the given height
expectAligned(locators, property)	the elements to share a top, right, bottom, left, width or height
expectFillsParent(child, parent, edges)	the child to reach the given edges of its parent, for example a scroller that ends at the bottom of its card
expectInside(child, container, edges, { margin })	the child to lie inside the container and keep at least margin px of space to the given edges, for example the air above and below a button
expectInset(child, container, edge, px)	the child to start exactly px inside the container at one edge, for example a title 12px from the edge of its card
expectGap(first, second, axis, px)	the space between two neighbours to be px, side by side or one above the other, for example the divider between two panels
expectBelow(locators, reference)	every element to start at or below the bottom of the reference, for example a toolbar that wrapped below its title row
expectNoOverlap(locators)	no element to cover another, for example a title and the buttons next to it
expectInsideOrBelow(locators, container)	the elements to be all inside the container or all below it, never half in; returns which, for a layout that depends on the room
expectScrollPositionKept(scroller, action, options)	a scroll container to stay where it is while the action runs, for example a click that must not carry the page away; smooth scrolls are waited for
expectWithinViewport(locator, edges)	the element not to reach beyond the given edges of the window, for example a scroller that ends above the bottom of the window
measure(locator), expectSameValue(page, what, values)	a position to stay the same while the page changes: record it page by page, then compare the recorded values
expectNoHorizontalOverflow(page, scrollers)	neither the document nor the given scroll containers to scroll sideways; the failure names the elements
expectNoHorizontalScrollAround(locator)	none of the scroll containers the element sits in to scroll sideways, for a page in a card that scrolls on its own while the document keeps its width
expectComputedStyle, expectSameComputedStyle	a CSS property to have the given computed value, or elements that look alike to compute the same value

A scroll container takes the room of its scrollbar at the right and bottom edge, as much as the platform draws, and always with scrollbar-gutter: stable. That room is no inset of the content, so expectInset takes the scroll containers between the child and the container as options.scrollers and does not count their scrollbars, which keeps the number the same on every platform. expectGap fails for elements that overlap, so a divider that is too narrow is told from one that is missing.

import { expectAligned, expectHeight, expectNoHorizontalOverflow, expectNoHorizontalScrollAround, forEachViewport } from '../../support/layout';

await forEachViewport(page, async () => {
    await expectHeight(page.getByTestId('exam-bar'), 40, { name: 'exam bar' });
    await expectAligned([page.getByTestId('exam-sidebar'), page.getByTestId('exam-content')], 'bottom', { name: 'the sidebar card and the content card' });
    // A page that is too wide widens the document, or the card it sits in when that scrolls on its own: check both.
    await expectNoHorizontalOverflow(page);
    await expectNoHorizontalScrollAround(page.getByTestId('exam-start-header'), { name: 'the title row' });
});

What belongs to the exam rather than to layout in general lives in support/examLayoutAssertions.ts: the sizes the exam agrees on (the 40px title row and exam bar, the 30px button in a title row, which includes the hand-in early button of the bar, and the 34px small button elsewhere, the 12px inset between a card and its content on every side, the 6px divider), the checks of the summary, of the complaint boxes, of the rating and of the programming actions, which two specs share, the checks of the dividers (also with the problem statement collapsed, and by dragging them) and of the file upload row, and expectPageNotToScrollSideways, which pairs the two sideways-scroll helpers for a page of the exam.

The viewport matrix

LAYOUT_VIEWPORTS holds the five window sizes a contract has to hold at. A rule that holds at some of them only says so in its test. The welcome page and the summary sit directly in the course shell, which keeps its sidebars on a phone and leaves them a 200px column, so their sideways-scroll contract, and the back button of the summary, hold from 800px (COURSE_SHELL_MIN_WIDTH). The complaint of a student and the response to it are side by side from 1200px and one above the other below it. The Code button, the result, Refresh and Submit of the programming page are in its title row from 1280px and below it at 480px; in between they are in the row where the title keeps its room next to them and below it where it does not, so the contract only asks for one of the two there. The file input of the file upload page and its Upload button are in one row from 1280px; below it the panel has no room for both and the button is below the input, at the same edge. The solution and the problem statement are 6px apart at every viewport, although below 993px the divider has no handle to drag.

Viewport	Stands for
1728x1000	a large laptop screen
1280x800	a small laptop screen, still wide enough for two column layouts
1024x768	a tablet in landscape, just below the 1200px breakpoint at which two columns stack
800x700	a tablet in portrait
480x800	a phone

Add a contract

1. State the rule as a number or as a relation between elements: "the title row is 40px", "the scroller ends where its card ends". A rule that cannot be put like that is a matter for a reviewer, not for a test.
2. Find the elements by data-testid, and add one to the template when the element has none. Name it after what the element is, such as exam-bar or exam-summary-scroll.
3. Pick the helper from the table and give the assertion a name.
4. Run it inside forEachViewport. In a flow that visits several pages, wrap each page in a test.step, so that the report names the page that broke the rule.
5. Add it to a flow that has already reached the state. A spec that creates data through the API requests, and does not wait for builds, is cheap; a new exam is not. ExamLayoutContract.spec.ts already builds one with an exercise of each type, so extend it instead of creating another.
6. Tag the spec like any other (see Test projects: fast and slow tests): @slow when the flow does not fit the time of a fast test, as the exam flow does, otherwise @fast.

A new or changed helper gets a case in e2e/shared/LayoutHelpers.spec.ts: a small document of known geometry for which the helper passes, and one for which it must fail. A helper that cannot fail makes every contract that uses it worthless, and that file proves they can. It renders the documents itself with page.setContent, so no case needs a login or a page of the application.

What not to do

• Do not compare screenshots, and do not assert pixel positions as absolute numbers. Compare elements with each other, or record a value on one page and compare it on the next.
• Do not sleep after a resize or a navigation. If a helper needs more than its retry, the layout is still changing after it has loaded, and that is the finding.
• Do not widen the tolerance until a contract passes. A difference of one pixel is a different layout.
• Do not find elements by Bootstrap, Tailwind or PrimeNG classes. Ask for what the class stands for with expectComputedStyle.
• Do not write a contract for what a unit test can state. jsdom has no layout engine, so a contract about sizes and positions is exactly what needs a browser; whether a template binds a class is not.

AI-assisted test authoring with a Playwright MCP server

You can optionally use a Playwright Model Context Protocol (MCP) server as an external tool to expose the live application DOM to AI coding assistants such as GitHub Copilot and Claude Code while Artemis is running locally.

Without access to the running page state, AI assistants can still generate test structure and boilerplate, but they may produce incorrect locators. When a separately configured Playwright MCP server is active, the assistant can inspect the current DOM, retrieve the data-testid attributes attached to elements, and generate locators that reference those IDs directly.

To use this workflow, start Artemis locally (via the fast runner or any other method) and configure a Playwright MCP server in your editor or AI assistant environment. The assistant can then inspect page structure and generate accurate data-testid-based locators for the workflow under test. Review the generated test by running it locally, confirm that the interaction sequence matches the intended workflow, and correct any steps where the assistant misread the application state.

CI Pipeline

Overview: PR selection and full-suite runs

For eligible internal pull requests, CI sets the reusable E2E workflow's run_full_suite input to false. Phase 1 runs the affected modules' consumer specs, changed specs and Login, Logout, and SystemHealth smoke specs. Phase 2 is intentionally omitted, including when Phase 1 passes. A green result establishes coverage of that selected scope; it does not establish full-suite coverage.

Configuration, shared runtime infrastructure and unclassified changes require the full suite in a single PR job. Fork PRs do not start E2E because they cannot publish the required container image. The umbrella's change filters also skip E2E for drafts and changes limited to server/client unit tests or documentation.

Full PR mode remains available to callers of the reusable workflow with run_full_suite=true, its default. For mapped changes, Phase 1 runs first, then Phase 2 covers the remaining tests after Phase 1 passes its gate. A blocking Phase 1 verdict skips Phase 2; positively classified known-flaky failures can pass the gate. The selector returns the same affected set and disjoint complement in both modes; the workflow decides whether the complement is required. The diagram below describes this full PR mode.

Full PR mode only: Phase 1 runs affected workflows first, then Phase 2 runs the remaining tests after Phase 1 passes its gate. Eligible selective PRs omit Phase 2. Global-input fallbacks and non-PR runs use one full-suite job.

Selected spec counts describe scope, not a proportional runtime saving. Stack startup and fixtures remain, and programming/build tests can dominate execution. Compare physical runner job durations when evaluating capacity; omitting Phase 2 removes its additional stack setup as well as its test execution.

Non-PR runs use the full suite: pushes to develop, main, and release branches, published releases, merge-group events when a merge queue is configured, and manual CI runs. These runs use one full-suite job. Full-suite execution before every merge depends on which events the repository uses; a selective PR result alone makes no such guarantee.

To request full CI and E2E for a branch, dispatch the CI workflow with force_build=true:

gh workflow run ci.yml --ref <branch> -f force_build=true

Non-PR E2E uses the multi-node stack (three Artemis nodes behind nginx) with Redis (Valkey) as the distributed data provider, so no Hazelcast instance is created. The suite runs once, not once per provider, because E2E cleanup stops every container on its host. Locally, run-e2e-tests-local-multinode.sh --middleware redis (or the -fast variant) runs the same configuration; the default hazelcast middleware uses the embedded cluster.

The isolated E2E stacks exempt their loopback and private Docker client addresses from server-side rate limits through ARTEMIS_RATELIMITING_EXEMPTADDRESSES. Parallel workers share an address and otherwise exhaust the authentication budget, causing HTTP 429 responses during fixture setup. The nginx exemption alone does not bypass the server's limiter. These settings belong only to test stacks; production deployments retain their configured rate limits.

Selective test execution and e2e-test-mapping.json

The determine-tests CI job reads supporting_scripts/ci/E2E-tests/e2e-test-mapping.json to decide which tests to include in Phase 1. Module mappings identify changed production inputs. The specModules contract records every discovered spec's runtime consumers, including setup, browser actions, HTTP calls and cleanup. For each changed module, the selector includes all specs declaring that dependency, even outside the module's own folder. All spec paths are relative to src/test/playwright/ (the Playwright test root).

The mapping has five selection inputs:

Field	Purpose
mappings.<module>.sourcePaths	Production paths owned by an eligible module.
specModules	Runtime module dependencies for every spec; a new unassigned spec forces a full run.
catalogModules	Explicit module ownership for translation catalog filenames; global or unknown catalogs require full validation.
alwaysRunTests	Authentication and health smoke specs included in every selection.
runAllTestsPatterns / ignoredPaths	Global runtime inputs that require all tests, and known non-runtime inputs that do not expand the selection.

Do not infer runtime ownership only from TypeScript type imports or a spec's folder. For example, StudentCompetencyProgressView.spec.ts consumes lecture and quiz behavior in addition to atlas behavior. The contract audit also checks observable API and helper usage to detect missing dependency declarations.

An audited client surface can declare replacesModule for exact source files. Its recorded consumer set then replaces the broader module only for those files. For example, exam-management UI files include exam-management and quiz-editor workflows, while shared exam participation code also includes the student course container's child routes. Other files keep the broader module selection, and a mixed PR unions both sets. These exceptions require explicit audit notes and non-empty consumer contracts; they are not directory-wide exclusions.

The determine-tests job compares the complete committed PR diff from its merge base against the actual base branch. Renames classify both the old and new paths; deleted specs are not passed as Playwright arguments. Module consumers, changed specs and alwaysRunTests form Phase 1. Accompanying documentation, Java tests and client unit specs do not expand E2E scope. The selector expands every selection to exact spec paths and computes the remaining discovered specs without overlaps. The workflow's run_full_suite input determines whether that remaining set runs as Phase 2.

Selection fails closed to one full-suite job when a runtime file has no mapping, a mapped module has no consumers, the mapping is invalid, a discovered spec has no contract, a required smoke spec is missing, or the base comparison cannot be computed. An empty committed diff also selects the full suite. Shared Playwright helpers and infrastructure require all tests, including helpers placed inside e2e/; only changed spec files can select themselves.

Shared runtime changes such as authentication, configuration, base components and editor infrastructure require the full suite. A module-local bugfix uses its declared consumers. Explicit module translation catalogs use the same consumer selection; unknown or global catalogs retain full coverage. Entries in runAllTestsPatterns require the full suite; unknown runtime paths never degrade to smoke-only coverage.

Adding tests: record each new spec's runtime modules in specModules, including setup and cleanup dependencies. For a new production module, add its sourcePaths to mappings. Run the selection contract tests; an unassigned spec or module without consumers must retain full fallback.

PR comment reporting

The pipeline posts a single comment on each pull request and updates it progressively as phases complete. A typical comment shows a phase table with status (✅ / ❌), test count, passed, skipped, failed, and wall-clock time per phase; the test file paths assigned to each phase; links to the workflow run and full HTML report; a collapsible section listing each failed test name and its duration; and a flakiness score table for any failed tests when Helios data is available.

The comment and job summary state the requested scope. Selective runs explicitly mark Phase 2 as intentionally omitted and report Selected E2E tests passed when successful. Full-suite fallback runs report all tests; full PR mode reports both phases or the blocking Phase 1 verdict that prevented Phase 2. Missing or cancelled jobs in the requested scope remain blocking rather than counting as a successful skip.

Helios flakiness scores

After each phase, the pipeline queries the Helios API to retrieve flakiness scores for any failed tests. A flakiness score reflects how often a test has non-deterministically produced different results over recent CI runs. A high score suggests the failure may be a flake; a low score suggests a genuine regression. Scores appear in the PR comment alongside failed test names.

Reports Dashboard

The Reports Dashboard is a web interface that stores and serves all E2E test artifacts from CI runs, accessible from the link in the PR comment. It provides:

• Aggregate view: total runs, 30-day pass rate, average flakiness rate, pass rate trend chart, and runtime-by-phase chart
• Per-run view: phase breakdown, test-level results with status and duration, links to the full HTML and coverage reports
• Failure detail view: the full assertion error, stack trace, and an embedded video recording of the browser at the moment of failure

Artemis Deployment on GitHub Runner

Every execution of the Playwright test suite requires the Artemis server and client to be deployed and started. The runner starts all required services using Docker Compose and executes Playwright tests on the host.

Hardware-software mapping of the E2E testing infrastructure. The Developer Machine pushes to GitHub, which triggers the Actions Orchestrator. The Self-Hosted CI Runner executes tests and sends results to the Reports Dashboard and PR comments via the Report and Communication Handler. The Helios node provides flakiness scores.

E2E tests run on self-hosted runners (not GitHub-hosted) because they require substantial compute resources. Self-hosted runners do not start from a clean state between jobs; the pipeline explicitly cleans up leftover test result files between phases to prevent workspace pollution from inflating test counts.

In total there are two Docker containers started on the GitHub runner, plus Playwright running on the host:

PostgreSQL

This container starts a PostgreSQL database and exposes it on port 5432. The container automatically creates a new database Artemis and configures it with the recommended settings for Artemis.

Artemis Application

The Docker image for the Artemis container is created from the already existing Dockerfile. When the build of the Playwright test suite starts, it retrieves the Artemis executable (.war file). Upon creation of the Artemis Docker image the executable is copied into the image together with configuration files for the Artemis server.

The main configuration of the Artemis server is contained in the Playwright environment configuration files. Security-relevant settings are passed to the Docker container via environment variables and GitHub secrets.

The Artemis container is configured to depend on the PostgreSQL container and uses health checks to wait until the database is up and running.

Playwright

Playwright runs directly on the runner host (not in a container). The base URL is configurable via the BASE_URL environment variable read from src/test/playwright/playwright.env (defaults to https://localhost). The necessary configuration for the Playwright test suite is passed in via environment variables. Playwright only starts once Artemis has been fully booted.

Maintenance

The Artemis Dockerfile as well as the PostgreSQL image are already maintained because they are used in other Artemis Docker setups. Therefore, only Playwright and the Playwright Docker image require active maintenance. Since the Playwright test suite simulates a real user, it makes sense to execute the test suite with the latest browser versions. If you run Playwright inside Docker locally (using deployment/docker/playwright.yml), the Playwright Docker image has browsers at specific pinned versions. The docker-compose file should be updated monthly to pull the latest Playwright image with up-to-date browsers. This step does not apply to CI, where Playwright runs directly on the host (not in a container).

When a new Artemis module is created or an existing module is significantly reorganized, update supporting_scripts/ci/E2E-tests/e2e-test-mapping.json:

1. Add a new entry to mappings with the module's sourcePaths.
2. Audit affected specs, page objects and HTTP helpers, then update their runtime dependencies in specModules.
3. Add module-owned translation catalogs to catalogModules where their scope is known.
4. Run the selection tests and review both selected and remaining exact spec sets.

The verifier imports the production selector and checks that selected and remaining specs form an exact, disjoint partition:

./supporting_scripts/ci/E2E-tests/verify-test-determination.sh tutorialgroup exam-management-client
./supporting_scripts/ci/E2E-tests/verify-test-determination.sh --all

A module without audited consumers keeps the full-suite fallback. The contract needs maintenance when an existing spec starts exercising another module, as well as when a new spec is added.

Functionalities Covered

Category	Functionality	Description
Atlas	Competency Management	Creating, editing, and deleting competencies; setting taxonomy levels and mastery thresholds; soft-deleting with linked exercise warnings
	Competency–Exercise Interactions	Linking and unlinking exercises to competencies; verifying that exercise progress contributes to competency mastery
	Competency–Lecture Unit Interactions	Linking lecture units to competencies; verifying lecture unit completion updates competency progress
	Competency Import	Importing competencies from other courses; verifying imported data integrity
	Learning Path Management	Enabling learning paths in a course; verifying the learning path activation flow
	Student Competency Progress View	Student competency overview, mastery indicators, judgment of learning ratings, and progress tracking across lecture units and exercises
Courses	Course Management	Creating, editing, deleting courses; adding/removing students from a course
	Course Exercise	Tests filtering exercises based on their title
	Course Communication	Messaging within courses, including channel creation, group chats, student participation, and message interactions
Exams	Exam Management	Manage exam students and exercise groups
	Exam Creation & Deletion	Creating, editing, and deleting exams
	Exam Participation	Early & normal hand-in, exam exercise participation for text, modeling, quiz, programming (Git SSH/HTTPS) exercises, exam announcements
	Exam Assessment	Assessing modeling, text, quiz and programming exercise submissions in exams, including complaint handling
	Exam Checklists	Exam setup checks, including student registration, exercise groups, and exam publication
	Exam Date Verification	Exams appear/disappear based on visibility dates
	Exam Results	Exam result overviews for text, quiz, modeling, and programming exercises
	Exam Test Runs	Creating, managing, and deleting exam test runs
	Exam Statistics	Exam statistics is displayed correctly
	PlantUML Diagram Isolation	PlantUML diagrams in exam exercises render in isolation without leaking state between participants
Test Exams	Test Exam Creation & Deletion	Creating and deleting test exams
	Test Exam Management	Managing test exam configuration and settings
	Test Exam Participation	Participating in test exams as a student
	Test Exam Student Exams	Generating and managing individual student exam instances
	Test Exam Test Runs	Creating and managing test runs within a test exam
Exercises	Exercise Import	Importing text, quiz, modeling & programming exercises
File Upload Exercises	Management	Creating and deleting file upload exercises
	Participation	Students can participate in a file upload exercise
	Assessment & Feedback	Assessing submissions, student feedback visibility, and complaint handling
Modeling Exercises	Management	Creating, editing, and deleting modeling exercises
	Visibility Controls	Students can access released/unreleased exercises
	Participation	Students can start and submit models
	Assessment & Complaints	Instructor and tutor assessments, student feedback, and complaint resolution
Programming Exercises	Management	Creating and deleting programming exercises
	Team Management	Forming and managing exercise teams
	Assessment	Assessing programming exercise submissions
	Participation	Submitting code through the code editor and Git (HTTPS & SSH), submissions for Java, C, and Python, team participation
	Static Code Analysis	Configuring SCA grading and handling submissions with SCA errors
Quiz Exercises	Management	Creating quizzes with multiple-choice, short-answer, and drag-and-drop questions
	Deletion & Export	Ensures quizzes can be deleted and exported
	Participation	Tests student participation in hidden, scheduled, and batch-based quizzes
	Assessment	Verifies automatic assessment for multiple-choice and short-answer quizzes
	Drag-and-Drop Mechanics	Ensures correct placement of draggable quiz elements
Text Exercises	Management	Creating and deleting text exercises
	Participation	Ensures students can submit text exercises
	Assessment & Complaints	Tests instructor assessments, feedback visibility, and complaint handling
Lectures	Lecture Management	Creating and deleting lectures, managing existing lectures
Authentication	Logging in	Logging in via UI and programmatically, login failures
	Logging out	Logging out successfully and canceling logout
System Status	System status indicators	Continuous integration & VC server health; Database, Hazelcast, and WebSocket health; Readiness and ping checks