Write the Exercise Code and Tests

An exercise is only as good as its three repositories. The template is what students start from, the solution proves the exercise is solvable, and the tests decide what students are scored on. This page covers filling all three, writing the problem statement that ties tasks to those tests, and writing test cases that hold up against code you did not write.

You can work either locally with Git or in the browser, and the two can be mixed freely.

Update Exercise Code in Repositories

You have two alternatives to update the exercise code:

Alternative 1: Clone and Edit Locally

1. Clone the 3 repositories and adapt the code on your local computer in your preferred IDE
2. To execute tests locally:
   • Copy template (or solution) code into an assignment folder (location depends on language)
   • Execute tests (e.g., using mvn clean test for Java)
3. Commit and push your changes via Git

Special Notes for Haskell

The build file expects the solution repository in the solution subdirectory and allows a template subdirectory for easy local testing.

Convenient checkout script:

#!/bin/sh
# Arguments:
# $1: exercise short name
# $2: (optional) output folder name

if [ -z "$1" ]; then
  echo "No exercise short name supplied."
  exit 1
fi

EXERCISE="$1"
NAME="${2:-$1}"

# Adapt BASE to your repository URL
BASE="ssh://git@artemis.tum.de:7999/$EXERCISE/$EXERCISE"

git clone "$BASE-tests.git" "$NAME" && \
  git clone "$BASE-exercise.git" "$NAME/template" && \
  git clone "$BASE-solution.git" "$NAME/solution" && \
  cp -R "$NAME/template" "$NAME/assignment" && \
  rm -r "$NAME/assignment/.git/"

Special Notes for OCaml

Tests expect to be in a tests folder next to assignment and solution folders.

Convenient checkout script:

#!/bin/sh
# Arguments:
# $1: exercise short name
# $2: (optional) output folder name

PREFIX=  # Set your course prefix

if [ -z "$1" ]; then
  echo "No exercise short name supplied."
  exit 1
fi

EXERCISE="$PREFIX$1"
NAME="${2:-$1}"

BASE="ssh://git@artemis.tum.de:7999/$EXERCISE/$EXERCISE"

git clone "$BASE-tests.git" "$NAME/tests"
git clone "$BASE-exercise.git" "$NAME/template"
git clone "$BASE-solution.git" "$NAME/solution"

# Hardlink assignment interfaces
rm "$NAME/template/src/assignment.mli"
rm "$NAME/tests/assignment/assignment.mli"
rm "$NAME/tests/solution/solution.mli"
ln "$NAME/solution/src/assignment.mli" "$NAME/template/src/assignment.mli"
ln "$NAME/solution/src/assignment.mli" "$NAME/tests/assignment/assignment.mli"
ln "$NAME/solution/src/assignment.mli" "$NAME/tests/solution/solution.mli"

Test script:

#!/bin/sh
dir="$(realpath ./)"

cd .. || exit 1
rm ./assignment
ln -s "$dir" ./assignment
cd tests || exit 1
./run.sh

Alternative 2: Edit in Browser

Open Edit in Editor in Artemis and adapt the code in the online editor.

Editing the template repository

You can switch between different repositories and Submit the code when done.

ℹ️

Information:

The online code editor supports real-time collaborative editing. Multiple instructors or editors can edit the same file simultaneously with automatic conflict resolution and cursor awareness.

Collaborative Editing

Artemis supports real-time collaborative editing for programming exercises. Multiple instructors and editors can work on the same exercise simultaneously, with no additional setup required. Collaboration spans two views:

• Online Code Editor (Edit in Editor) — live synchronization of code files, the file tree, and the problem statement.
• Edit Page (Edit) — synchronization of exercise settings and the problem statement.

To start collaborating, have a second instructor or editor open the same exercise. Both users are connected automatically.

Live Code Editing: When two or more users open the same file in the same repository, their edits are synchronized automatically. This is supported across all repository types: Template, Solution, Tests, and Auxiliary. Each collaborator's cursor and text selection are visible with a colored marker and name label.

When someone pushes a commit to a repository (e.g., via Git CLI, an IDE, or "Submit" button) while the online code editor is open, a new commit alert is displayed. This advises you to save your work by clicking Submit and refresh the page to incorporate the new changes. The alert only appears for the repository that received the new commit.

Live collaborative code editing with cursor awareness

Alert shown when an external commit is detected

File Tree Synchronization: File creation, deletion, and renaming are broadcast to all connected editors immediately. If another user had a deleted or renamed file open, it is handled gracefully.

Problem Statement Synchronization: The markdown problem statement editor supports the same real-time collaboration with cursor awareness. This works both in the online code editor (Edit in Editor) and on the edit page (Edit).

Live collaborative problem statement editing

Exercise Settings Synchronization: On the edit page (Edit), changes to exercise settings (title, points, dates, grading criteria, etc.) are synchronized to other users when saved. Non-conflicting changes are applied automatically. If both instructors modified the same fields, a merge conflict modal lets you choose which values to keep.

Resolving conflicting exercise setting changes

ℹ️

Information: Tips and Limitations:

• Click Submit frequently to ensure your work is committed to the repository.
• Avoid mixing the online editor and external Git tools at the same time — pick one workflow per editing session to reduce the chance of conflicts.
• Use cursor awareness to see where your collaborators are editing.
• Handle commit alerts promptly — save and refresh before continuing to edit.
• Unsaved/unsubmitted content is not preserved on the server. When all users close the page, unsaved content is lost.
• Binary files are not synchronized through the collaborative editor.

Review Earlier Versions of the Exercise

Artemis keeps a version history for a programming exercise, so you can see how it changed over a semester and what a previous run of the course actually looked like. Open the exercise in Course Management and choose Version History.

The timeline lists the versions on the left; selecting one shows the exercise metadata as it was at that point, including the programming-specific settings that are easy to forget you changed:

• the default branch,
• the checkout paths for the assignment, solution and tests repositories,
• the container configuration and the Online IDE image,
• how submissions after the due date are handled,
• the full build plan configuration and build script.

Values that were not set show as Unset. If a whole area is missing from a snapshot — the programming metadata, the build plan configuration or the build script — the view says so explicitly instead of showing an empty section.

Write the Interactive Problem Statement

The problem statement is what students read, and it is interactive: when you bind a task to test cases, Artemis shows each student whether their latest submission passes that task, right inside the description. Getting these bindings right is what turns a wall of text into a checklist students can work through.

Edit it either with Edit on the exercise, or with Edit in Editor to have it side by side with the code. Both editors synchronize in real time when several people work at once — see Collaborative Editing.

The problem statement is Markdown, plus the two Artemis-specific constructs below.

Tasks

A task is a named step bound to one or more test cases:

[task][Implement BubbleSort](testBubbleSort,testBubbleSortHidden)

• The literal [task] marks the construct.
• The second bracket holds the task name shown to students. It may contain anything except square brackets.
• The parentheses hold a comma-separated list of test case names, exactly as they appear in the test repository. Whitespace after a comma is ignored.

Test names may carry parameters or indices, so all of these are valid: testName, testName(), testName(1234, 12), testName(testValue)[1], and Test Name. A task with no tests at all — [task][Read the introduction]() — is allowed and simply renders without a status.

Artemis resolves these names against the test cases reported by the solution build plan, so a task only lights up once the solution build has run at least once. The same names drive task-based grading.

UML Diagrams

Embed a PlantUML diagram between @startuml and @enduml. Diagrams can be interactive too: color an element according to whether a test passes by wrapping it in testsColor:

@startuml
class BubbleSort {
  <color:testsColor(testBubbleSort)>+ performSort(int[])</color>
}
@enduml

The element is drawn in the success color once that test passes, and in the failure color while it does not. Each element takes exactly one test case; you cannot bind several tests to a single UML element.

Check the Bindings

Below the problem statement editor, Artemis continuously analyses the bindings and reports:

• Invalid test cases — a name used in the problem statement that does not exist in the test repository, usually a typo or a renamed test.
• Missing test cases — a test that exists in the test repository but is not referenced by any task, so students never see its result attributed to anything.
• Repeated test cases — a test referenced by more than one task, which makes the task statuses ambiguous.

Problem statement analysis below the editor

The status reads Test cases ok. when none of the three apply. Artemis also warns when an exercise has more than 15 tasks, which is usually a sign the exercise should be split.

ℹ️

Information:

Artemis can draft, refine and review the problem statement for you, and can check it against the three repositories for inconsistencies. See AI-Assisted Exercise Authoring and Consistency Check.

Testing Frameworks by Language

Write test cases in the Test repository using language-specific frameworks:

No.	Language	Package Manager	Build System	Testing Framework
1	Java	Maven / Gradle	Maven / Gradle	JUnit 5 with Ares
2	Python	pip	pip	pytest
3	C	-	Makefile	Python scripts / FACT
4	Haskell	Stack	Stack	Tasty
5	Kotlin	Maven	Maven	JUnit 5 with Ares
6	VHDL	-	Makefile	Python scripts
7	Assembler	-	Makefile	Python scripts
8	Swift	SwiftPM	SwiftPM	XCTest
9	OCaml	opam	Dune	OUnit2
10	Rust	cargo	cargo	cargo test
11	JavaScript	npm	npm	Jest
12	R	built-in	-	testthat
13	C++	-	CMake	Catch2
14	TypeScript	npm	npm	Jest
15	C#	NuGet	dotnet	NUnit
16	Go	built-in	built-in	go testing
17	Bash	-	-	Bats
18	MATLAB	mpminstall	-	matlab.unittest
19	Ruby	Gem	Rake	minitest
20	Dart	pub	built-in	package.test

Check the build plan results:

• Template and solution build plans should not have Build Failed status
• If the build fails, check build errors in the build plan

⚠️

Warning:

Hint: Test cases should only reference code available in the template repository. If this is not possible, try the Sequential Test Runs option.

Testing with Ares

Ares is a JUnit 5 extension for easy and secure Java testing on Artemis.

Main features:

• Security manager to prevent students from crashing tests or cheating
• More robust tests and builds with limits on time, threads, and I/O
• Support for public and hidden Artemis tests with custom due dates
• Utilities for improved feedback (multiline error messages, exception location hints)
• Utilities to test exercises using System.out and System.in

For more information see Ares GitHub

Best Practices for Writing Test Cases

The following sections describe best practices for writing test cases. Examples are specifically for Java (using Ares/JUnit5), but practices can be generalized for other languages.

General Best Practices

Write Meaningful Comments for Tests

Comments should contain:

• What is tested specifically
• Which task from problem statement is addressed
• How many points the test is worth
• Additional necessary information

Keep information consistent with Artemis settings like test case weights.

/**
 * Tests that borrow() in Book successfully sets the available attribute to false
 * Problem Statement Task 2.1
 * Worth 1.5 Points (Weight: 1)
 */
@Test
public void testBorrowInBook() {
    // Test Code
}

Better yet, use comments in display names for manual correction:

@DisplayName("1.5 P | Books can be borrowed successfully")
@Test
public void testBorrowInBook() {
    // Test Code
}

Use Appropriate and Descriptive Names for Test Cases

Test names are used for statistics. Avoid generic names like test1, test2, test3.

@Test
public void testBorrowInBook() {
    // Test Code
}

If tests are in different (nested) classes, add class name to avoid duplicates:

@Test
public void test_LinkedList_add() {
    // Test Code
}

ℹ️

Information:

For Java: If all test methods are in a single class, this is unnecessary (compiler won't allow duplicate methods).

Use Appropriate Timeouts for Test Cases

For regular tests, @StrictTimeout(1) (1 second) is usually sufficient. For shorter timeouts:

@Test
@StrictTimeout(value = 500, unit = TimeUnit.MILLISECONDS)
public void testBorrowInBook() {
    // Test Code
}

Can also be applied to entire test class.

⚠️

Warning:

Remember tests run on CI servers (build agents). Tests execute slower than on local machines.

Avoid Assert Statements

Use conditional fail() calls instead to hide confusing information from students.

❌ Not recommended:

@Test
public void testBorrowInBook() {
    Object book = newInstance("Book", 0, "Some title");
    invokeMethod(book, "borrow");
    assertFalse((Boolean) invokeMethod(book, "isAvailable"),
                "A borrowed book must be unavailable!");
}

Shows: org.opentest4j.AssertionFailedError: A borrowed book must be unavailable! ==> Expected <false> but was <true>

✅ Recommended:

@Test
public void testBorrowInBook() {
    Object book = newInstance("Book", 0, "Some title");
    invokeMethod(book, "borrow");
    if ((Boolean) invokeMethod(book, "isAvailable")) {
        fail("A borrowed book is not available anymore!");
    }
}

Shows: org.opentest4j.AssertionFailedError: A borrowed book is not available anymore!

Write Tests Independent of Student Code

Students can break anything. Use reflective operations instead of direct code references.

❌ Not recommended (causes build errors):

@Test
public void testBorrowInBook() {
    Book book = new Book(0, "Some title");
    book.borrow();
    if (book.isAvailable()) {
        fail("A borrowed book must be unavailable!");
    }
}

✅ Recommended (provides meaningful errors):

@Test
public void testBorrowInBook() {
    Object book = newInstance("Book", 0, "Some title");
    invokeMethod(book, "borrow");
    if ((Boolean) invokeMethod(book, "isAvailable")) {
        fail("A borrowed book must be unavailable!");
    }
}

Error message: The class 'Book' was not found within the submission. Make sure to implement it properly.

Check for Hard-Coded Student Solutions

Students may hardcode values to pass specific tests. Verify solutions fulfill actual requirements, especially in exams.

Avoid Relying on Specific Task Order

Tests should cover one aspect without requiring different parts to be implemented.

Example: Testing translate and runService methods where runService calls translate.

❌ Not recommended (assumes translate is implemented):

@Test
public void testRunServiceInTranslationServer() {
    String result = translationServer.runService("French", "Dog");
    assertEquals("Dog:French", result);
}

✅ Recommended (overrides translate to test runService independently):

@Test
public void testRunServiceInTranslationServer() {
    TranslationServer testServer = new TranslationServer() {
        public String translate(String word, String language) {
            return word + ":" + language;
        }
    };
    String expected = "Dog:French";
    String actual = testServer.runService("French", "Dog");
    if(!expected.equals(actual)) fail("Descriptive fail message");
}

⚠️

Warning:

Handle students making classes/methods final via problem statement or tests to avoid compilation errors.

Catch Possible Student Errors

Handle student mistakes appropriately. For example, null returns can cause NullPointerException.

@Test
public void testBorrowInBook() {
    Object book = newInstance("Book", 0, "Some title");
    Object result = invokeMethod(book, "getTitle");
    if (result == null) {
        fail("getTitle() returned null!");
    }
    // Continue with test
}

Java Best Practices

Use Constant String Attributes for Base Package

Avoid repeating long package identifiers:

private static final String BASE_PACKAGE = "de.tum.in.ase.pse.";

@Test
public void testBorrowInBook() {
    Object book = newInstance(BASE_PACKAGE + "Book", 0, "Some title");
    // Test Code
}

Use JUnit5 and Ares Features

More information: JUnit5 Documentation and Ares GitHub

Useful features:

• Nested Tests to group tests
• @Order to define custom test execution order
• assertDoesNotThrow for exception handling with custom messages
• assertAll to aggregate multiple assertion failures
• Dynamic Tests for special needs
• Custom extensions
• JUnit Platform Test Kit for testing tests

Define Custom Annotations

Combine annotations for better readability:

@Test
@StrictTimeout(10)
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface LongTest {
}

Consider Using jqwik for Property-Based Testing

jqwik allows testing with arbitrary inputs and shrinks errors to excellent counter-examples (usually edge cases).

Eclipse Compiler and Best-Effort Compilation

Use Eclipse Java Compiler for partial, best-effort compilation. Useful for exam exercises and complicated generics.

Compilation errors are transformed into errors thrown where code doesn't compile (method/class level). The body is replaced with throw new Error("Unresolved compilation problems: ...").

⚠️

Warning:

Important: Only method/class bodies should fail to compile, not complete test classes. Anything outside method/nested class bodies must compile, including method signatures, return types, parameter types, and lambdas. Use nested classes for fields/methods with student class types that might not compile.

ℹ️

Information:

The Eclipse Compiler may not support the latest Java version. You can compile student code with the latest Java and test code with the previous version.

Configure it yourself by pointing the Maven compiler plugin at the Eclipse compiler, which is published as org.eclipse.jdt:ecj, via the plugin's compilerId. No Artemis template ships this configuration — the Java test templates use the standard maven-compiler-plugin — so this is a change you make in the test repository's pom.xml.

Common Pitfalls / Problems

• Reflection API limitation: Constant attributes (static final primitives/Strings) are inlined at compile-time, making them impossible to change at runtime
• Long output: Arrays or Strings with long output may be unreadable or truncated after 5000 characters

Java DejaGnu: Blackbox Testing

Classical testing frameworks like JUnit allow writing whitebox tests, which enforce assumptions about code structure (class names, method names, signatures). This requires specifying all structural aspects for tests to run on student submissions. That may be okay or even desired for a beginner course.

For advanced courses, this is a downside: students cannot make their own structural decisions and gain experience in this important programming aspect.

DejaGnu enables blackbox tests for command line interfaces. Tests are written in Expect Script (extension of Tcl). Expect is a Unix utility for automatic interaction with programs exposing text terminal interfaces in a robust way.

Test scripts:

• Start the program as a separate process (possibly several times)
• Interact via textual inputs (standard input)
• Read outputs and make assertions (exact or regex matching)
• Decide next inputs based on output, simulating user interaction

For exercises, only specify:

• Command line interface syntax
• Rough output format guidance

Source code structure is up to students as far as you want.

Source code structure quality assessment can be done manually after submission deadline. The template uses Maven to compile student code, so it can be extended with regular unit tests (e.g., architecture tests for cyclic package dependencies) and report the results for both to the student.

Usage: Consult the official documentation alongside the content the test repository ships with. Everything DejaGnu-related lives under testsuite, laid out like this:

testsuite/
├── config/default.exp              helper functions such as PROGRAM_TEST
├── lib/<package>.exp               exercise-specific procedures
├── testfiles/public/               input files the students' tests may read
├── testfiles/secret/               input files only the hidden tests read
└── <package path>.tests/           the test execution scripts
    ├── public.exp
    ├── secret.exp
    └── advanced.exp

<package path> is the exercise's package name with the dots replaced by slashes, so a package of de.tum.in.ase.pse puts the three scripts in testsuite/de/tum/in/ase/pse.tests/.

Example: PROGRAM_TEST {add x} {} puts "add x" into the program and expects no output.

Helper functions like PROGRAM_TEST are defined in config/default.exp.

Variables in SCREAMING_SNAKE_CASE (e.g., MAIN_CLASS) are replaced with actual values in previous build plan steps. For example, the build plan finds the Java class with main method and replaces MAIN_CLASS.

Best Expect documentation: Exploring Expect book. The Artemis default template contains reusable helper functions in config/default.exp for common I/O use cases.

This exercise type makes it quite easy to reuse existing exercises from the Praktomat autograder system.

Prevent Maven Central Rate Limits (Java and Kotlin)

⚠️

Warning:

Important for Java and Kotlin exercises: Maven Central rate-limits download requests. Because all builds of an Artemis instance originate from the same IP addresses, builds can suddenly fail with HTTP 429 (Too Many Requests) errors — especially during high-load periods such as exams, when many students submit at the same time. We strongly recommend configuring a Maven repository mirror as the first repository in the test repository's build configuration, as described below.

Affected builds fail with errors such as:

Could not GET 'https://repo.maven.apache.org/maven2/de/tum/in/ase/artemis-java-test-sandbox/1.11.3/artemis-java-test-sandbox-1.11.3.pom'.
Received status code 429 from server: Too Many Requests

Only Java and Kotlin exercises are affected, because their builds resolve dependencies via Maven Central. Whether you have to do anything depends on your Artemis instance: if its administrators configured a mirror centrally, every newly created Java and Kotlin exercise already resolves through it and you can skip this section. Otherwise you configure the mirror in the exercise's test repository yourself, as described below - which is also how you add a mirror to exercises that were created before your instance configured one.

When Artemis detects a build that failed due to Maven Central rate limiting, it automatically notifies the instructors of the affected course by email (at most once per day per exercise), including a link to this section and to the online editor of the affected exercise.

For Artemis instances operated by TUM, the Artemis team provides a Reposilite mirror at https://reposilite.aet.cit.tum.de that proxies and caches all required artifacts. Other institutions can operate their own mirror and configure it in the same way.

For Administrators: Configure the Mirror Once for the Whole Instance

Administrators can point every newly created Java and Kotlin test repository at a mirror by setting a single property, so instructors do not have to touch individual exercises:

artemis:
    programming:
        maven-central-mirror-url: https://reposilite.aet.cit.tum.de/releases

Artemis then declares this mirror as the first repository for both dependencies and plugins in the test repository of every Java and Kotlin exercise it creates. Leave the property unset to keep resolving from Maven Central directly. Exercises that already exist are not modified - use the instructions below for those.

Recommended: Add the Mirror to the Test Repository

Gradle exercises: In the build.gradle file of the test repository, add the mirror as the first entry in the repositories block:

repositories {
    maven {
        name "reposiliteRepositoryReleases"
        url "https://reposilite.aet.cit.tum.de/releases"
    }
    mavenCentral()
}

⚠️

Warning:

The repositories block does not cover Gradle plugins. The plugins { id '...' } block in build.gradle (e.g. id 'org.springframework.boot') is not resolved through repositories { ... }. Gradle resolves plugins via the Gradle Plugin Portal, which falls back to Maven Central — bypassing the mirror entirely and still hitting HTTP 429 rate limits.

To route plugin resolution through the mirror as well, add a pluginManagement block at the top of settings.gradle (create the file if it does not exist yet):

pluginManagement {
    repositories {
        maven {
            name "reposiliteRepositoryReleases"
            url "https://reposilite.aet.cit.tum.de/releases"
        }
        gradlePluginPortal()
    }
}

As with dependencies, the mirror must be listed before gradlePluginPortal() so that plugins are resolved from the mirror first.

Maven exercises: In the pom.xml file of the test repository, add the mirror as the first entry in the <repositories> section (create the section if it does not exist yet):

<repositories>
    <repository>
        <id>reposilite-repository-releases</id>
        <name>Reposilite Repository</name>
        <url>https://reposilite.aet.cit.tum.de/releases</url>
    </repository>
</repositories>

Maven resolves plugins through a separate list of repositories, so add the mirror there as well:

<pluginRepositories>
    <pluginRepository>
        <id>reposilite-repository-releases</id>
        <name>Reposilite Repository</name>
        <url>https://reposilite.aet.cit.tum.de/releases</url>
    </pluginRepository>
</pluginRepositories>

⚠️

Warning:

The repository order matters: Repositories are queried in the order they are declared. The mirror must come before Maven Central so that all dependencies are resolved from the mirror and Maven Central is not contacted at all. If the mirror is declared after Maven Central, builds will still hit the rate limit.

Alternative: Configure the Mirror in the Docker Image

Alternatively, the mirror can be baked into the Docker image used for the build (e.g., via a Gradle init script or a Maven settings.xml inside the image). We mention this option for completeness, but recommend the test repository configuration above because it is much more flexible:

• While the mirror has been stable so far, its availability cannot be guaranteed. If it ever becomes unavailable, changing a single entry in the test repository (e.g., via the online code editor) takes seconds and immediately applies to all subsequent builds.
• Switching or rebuilding a Docker image is much slower and must be repeated for every exercise that uses the image.