Configure Grading and Verify the Exercise

Grading decides how test results become points: which tests count, how much, and when students get to see them. Once that is set, verifying the exercise means checking that the template scores zero, the solution scores full marks, and every task is bound to a test that actually exists.

This page also covers what you do once students are working: reviewing common mistakes with feedback analysis, and assessing submissions by hand where the exercise calls for it.

Configure Grading

The grading configuration determines how test cases contribute to the overall score and allows configuration of code quality issue penalties.

General Actions

Grading Configuration Update Information

• Save: Save current grading configuration
• Reset: Reset the current grading configuration of the open tab to the default values. For Test Case Tab, all test cases are set to weight 1, bonus multiplier 1, bonus points 0, and their visibility back to the exercise default — which is Always for a course exercise but After Due Date for an exam exercise, so a reset in exam mode also changes when students see feedback. For the Code Analysis Tab, the default configuration depends on the selected programming language.
• Re-evaluate All: Re-evaluate scores using currently saved settings and existing feedback
• Trigger All: Trigger all build plans to create new results with updated configuration

⚠️

Warning:

Artemis always grades new submissions with the latest configuration, but existing submissions might use outdated configuration. Artemis warns about grading inconsistencies with the Updated grading badge.

Test Case Configuration

Adapt test case contribution to the overall score or set grading based on tasks.

Test Case Grading Configuration

ℹ️

Information:

Artemis registers tasks and test cases from the Test repository using results from the Solution build plan. Test cases only appear after the first Solution build plan execution.

⚠️

Warning:

If your problem statement does not contain tasks, task-based grading is not available. You can still configure grading based on individual test cases.

Left side configuration options:

• Task/Test Name: Task names (bold) are from the problem statement. Test names are from the Test repository
• Weight: Points for a test are proportional to weight (sum of weights as denominator). For tasks, weight is evenly distributed across test cases
• Bonus multiplier: Multiply points for passing a test without affecting other test points. For tasks, multiplier applies to all contained tests
• Bonus points: Add flat point bonus for passing a test. For tasks, bonus points are evenly distributed across test cases
• Visibility: Control feedback visibility:
  • Always: Feedback visible immediately after grading
  • After Due Date: Feedback visible only after due date (or individual due dates)
  • Never: Feedback never visible to students. Not considered in score calculation
• Passed %: Statistics showing percentage of students passing/failing the test

⚠️

Warning:

Bonus points (score > 100%) are only achievable if at least one bonus multiplier > 1 or bonus points are given for a test case.

⚠️

Warning:

For manual assessments, all feedback details are visible to students even if the due date hasn't passed for others. Set an appropriate assessment due date to prevent information leakage.

Examples:

ℹ️

Information:

Example 1: Given an exercise with 3 test cases, maximum points of 10 and 10 achievable bonus points. The highest achievable score is 200%. Test Case (TC) A has weight 2, TC B and TC C have weight 1 (bonus multipliers 1 and bonus points 0 for all test cases). A student that only passes TC A will receive 50% of the maximum points (5 points).

ℹ️

Information:

Example 2: Given the configuration of Example 1 with an additional bonus multiplier of 2 for TC A. Passing TC A accounts for (2 × 2) / (2 + 1 + 1) × 100 = 100% of the maximum points (10). Passing TC B or TC C accounts for 1/4 × 100 = 25% of the maximum points (2.5). If the student passes all test cases they will receive a score of 150%, which amounts to 10 points and 5 bonus points.

ℹ️

Information:

Example 3: Given the configuration of Example 2 with additional bonus points of 5 for TC B. The points achieved for passing TC A and TC C do not change. Passing TC B now accounts for 2.5 points plus 5 bonus points (7.5). If the student passes all test cases they will receive 10 (TC A) + 7.5 (TC B) + 2.5 (TC C) points, which amounts to 10 points and 10 bonus points and a score of 200%.

The right side displays statistics:

• Weight Distribution: Impact of each test on score
• Total Points: Percentage of points awarded per test case across all students

Code Analysis Configuration

Configure visibility and grading of code quality issues by category.

Code Analysis Grading Configuration

ℹ️

Information:

The Code Analysis tab is only available if static code analysis is enabled for the exercise.

Issues are grouped into categories. The following table shows the category mappings for Java, Swift, and C:

Category	Description	Java	Swift	C
Bad Practice	Code that violates recommended and essential coding practices	Spotbugs BAD_PRACTICE Spotbugs I18N PMD Best Practices		GCC BadPractice
Code Style	Code that is confusing and hard to maintain	Spotbugs STYLE Checkstyle blocks Checkstyle coding Checkstyle modifier PMD Code Style	Swiftlint (all rules)
Potential Bugs	Coding mistakes, error-prone code or threading errors	Spotbugs CORRECTNESS Spotbugs MT_CORRECTNESS PMD Error Prone PMD Multithreading		GCC Memory
Duplicated Code	Code clones	PMD CPD
Security	Vulnerable code, unchecked inputs and security flaws	Spotbugs MALICIOUS_CODE Spotbugs SECURITY PMD Security		GCC Security
Performance	Inefficient code	Spotbugs PERFORMANCE PMD Performance
Design	Program structure/architecture and object design	Checkstyle design PMD Design
Code Metrics	Violations of code complexity metrics or size limitations	Checkstyle metrics Checkstyle sizes
Documentation	Code with missing or flawed documentation	Checkstyle javadoc Checkstyle annotation PMD Documentation
Naming & Format	Rules that ensure the readability of the source code (name conventions, imports, indentation, annotations, white spaces)	Checkstyle imports Checkstyle indentation Checkstyle naming Checkstyle whitespace
Miscellaneous	Uncategorized rules	Checkstyle miscellaneous		GCC Misc

ℹ️

Information:

For Swift, only the category Code Style can contain code quality issues currently. All other categories displayed on the grading page are dummies.

ℹ️

Information:

The GCC SCA option for C does not offer categories by default. The issues were categorized during parsing with respect to the rules. For details on the default configuration and active rules, see the GCC - Static Code Analysis Default Configuration section below.

Other languages use categories defined by their static code analysis tool:

• JavaScript, TypeScript, C++, R: Only use the Lint category
• Dart: Dart's analyzer can generate these categories:
  • TODO
  • HINT
  • COMPILE_TIME_ERROR
  • CHECKED_MODE_COMPILE_TIME_ERROR
  • STATIC_WARNING
  • SYNTACTIC_ERROR
  • LINT
• Python: Uses Ruff's tool names
• Ruby: Uses Rubocop's Department names
• Rust: Uses Clippy's lint groups

On the left side of the page, instructors can configure the static code analysis categories:

• Category: The name of the category defined by Artemis
• State:
  • INACTIVE: Code quality issues of an inactive category are not shown to students and do not influence the score calculation
  • FEEDBACK: Code quality issues of a feedback category are shown to students but do not influence the score calculation
  • GRADED: Code quality issues of a graded category are shown to students and deduct points according to the Penalty and Max Penalty configuration
• Penalty: Artemis deducts the selected amount of points for each code quality issue from points achieved by passing test cases
• Max Penalty: Limits the amount of points deducted for code quality issues belonging to this category
• Detected Issues: Visualizes how many students encountered a specific number of issues in this category

Submission Policy Configuration

Submission policies define the effect of a submission on participant progress. A programming exercise can have zero or one submission policy (never more than one). Policies are specified during exercise creation and can be adjusted in the grading configuration later.

ℹ️

Information:

Definition: One submission = one push to the exercise repository by the participant that triggers automatic tests resulting in feedback. Automatic test runs triggered by instructors are not counted as submissions.

Submission Policy Types

Choosing the right policy depends on the exercise and teaching style. Lock repository and submission penalty policies combat trial-and-error solving approaches.

None

No submission policy. Participants can submit as often as they want until the due date.

No Submission Policy

Lock Repository

Participants can submit a fixed number of times. After reaching the limit, the repository is locked and further submissions are prevented.

Lock Repository Policy Configuration

With the example configuration shown above, participants can submit 5 times. After the 5th submission, Artemis locks the repository, preventing further pushes.

⚠️

Warning:

If locking fails and the participant submits again, Artemis attempts to lock again and sets the new result to Not Graded.

ℹ️

Information:

The participant may still work on their solution locally, but cannot submit it to Artemis to receive feedback.

Submission Penalty

Participants can submit as often as they want. For each submission exceeding the limit, the penalty is deducted from the score.

Submission Penalty Policy Configuration

With the example configuration above:

• First 3 submissions: no penalty
• 4th submission: 1.5 points deducted
• 5th submission: 3 points deducted (1.5 × 2 submissions exceeding limit)
• Score cannot be negative

Example: Student achieves 6 out of 12 points on 4th submission. With 1.5 point penalty, final score is 4.5 out of 12.

Students receive feedback explaining the deduction:

Submission Penalty Feedback Example

Submission Penalty Feedback Details

Managing Submission Policies

After generating an exercise, submission policies can be updated on the grading page.

Submission Policy Management on Grading Page

(De)activating Submission Policies

• Active policy: Shows Deactivate button
• Inactive policy: Shows Activate button

When deactivated, Artemis no longer enforces the policy. Locked repositories are unlocked. For submission penalty policies, press Re-evaluate All to apply changes.

Updating Submission Policies

Modify configuration and press Update. The effect of the former policy is removed and the new policy is applied. For submission penalty policies, press Re-evaluate All to update results.

Deleting Submission Policies

Select None as policy type and press Update. Locked repositories are unlocked. For submission penalty policies, press Re-evaluate All to revert effects.

Verify Exercise Configuration

Open the View page of the programming exercise.

Template and Solution Results

Verify:

• Template result: Score 0% with "0 of X passed" (or "0 of X passed, 0 issues" with SCA)
• Solution result: Score 100% with "X of X passed" (or "X of X passed, 0 issues" with SCA)

ℹ️

Information:

If static code analysis finds issues in template/solution, improve the code or disable the problematic rule.

Review Template/Solution Differences

Review differences between template and solution repositories to verify expected student changes.

Template/Solution Comparison in Exercise Management

Review Changes opens the comparison view.

Template/Solution Comparison View

Verify Problem Statement Integration

Click Edit and check the analysis below the problem statement. It should read Test cases ok., meaning every test named in a task exists in the test repository, every test in the repository is referenced by a task, and no test is referenced twice.

If it reports issues instead, hover it to see which test names are affected. See Check the Bindings.

Reset the Exercise

If the exercise is in a state you cannot repair by editing it — a broken template that every student has already cloned, or a build plan that no longer matches the repositories — you can reset it. Open the exercise in Course Management and choose Reset.

The dialog lets you select which operations to perform:

• Delete all student participations, submissions and results. Every student starts again from the current template.
• Recreate the build plans (BASE and SOLUTION) from the latest exercise template, overwriting any changes you made to them. For an exercise with a customized build plan, this resets that customization to the template default instead.

To confirm, you have to type the exercise title.

⚠️

Warning:

A reset cannot be undone, and deleting participations discards every submission and result the students have produced. Review the selection carefully before confirming.

ℹ️

Information:

Resetting is rarely the right tool once students have started working. If only the grading is wrong, Re-evaluate All or Trigger All on the grading page fix the results without discarding anyone's work.

Feedback Analysis

After verifying the exercise configuration, the Feedback Analysis feature helps identify common student mistakes and improve grading efficiency.

Accessing Feedback Analysis:

1. Navigate to Exercise Management
2. Open the programming exercise
3. Go to the grading section
4. Click the Feedback Analysis tab

Feedback Analysis Overview

Key Features:

1. Filtering and Sorting

   Filter by tasks, test cases, error categories, and occurrence frequency. Sort based on count or relevance.

   

   Filtering Options

2. Affected Students Overview

   View students affected by specific feedback and access their repositories for review.

   

   Affected Students View

3. Aggregated Feedback

   • To avoid many single entries in case of testcases using random values and generating similar feedback, Groups similar feedback messages together all feedback to highlight frequently occurring errors.
   • Displays grouped occurrence counts and relative frequencies to help instructors prioritize common issues.

4. Detailed Inspection

   Click feedback entries to view full details in a modal window.

   

   Detailed Feedback View

5. Communication & Collaboration

   Create discussion channels directly from the feedback analysis view for collaborative grading.

   

   Creating a Feedback Discussion Channel

Manual Assessment

You can use the online editor to assess student submissions directly in the browser.

Manual Assessment in Online Editor

The online editor provides features tailored to assessment:

1. File browser: Shows student submission files. Changed files are highlighted in yellow
2. Build output: Shows build process output, useful for build errors
3. Read-only editor: View student code with changed lines highlighted
4. Instructions: Provides structured grading criteria and problem statement, including tasks successfully solved (determined by test cases). Review test cases by clicking the passing test count next to tasks
5. Result: Top right corner shows current submission result. Click to review test cases and feedback

Add feedback directly in source code by hovering over a line and clicking Add Feedback. Alternatively, press "+" key when editor is focused to open feedback widget at cursor line.

Editing Feedback in Online Editor

After clicking Add Feedback:

• Enter feedback comment and score
• Or drag structured assessment criteria from instructions to feedback area
• Click Save to save
• Click Cancel to discard

General Feedback in Online Editor

Add general feedback (not tied to specific file/line) by scrolling to bottom and clicking Add General Feedback.

Save changes with Save (top right). Finalize assessment with Submit Assessment.

ℹ️

Information:

You can save multiple times before submitting. Once submitted, you cannot make changes unless you are an instructor.