Benchmarking Tool
The Artemis Benchmarking Tool is a web application designed for Artemis administrators to test the performance of Artemis under heavy load. It is available on GitHub.
The Benchmarking Tool for Artemis is designed to simulate realistic user interactions with the Artemis platform, particularly focusing on exam conduction where the system's performance and scalability are most critical.
This tool enables system administrators and developers to assess and analyze Artemis's behavior under simulated load conditions, providing valuable insights into response times, system workload, and overall stability. These insights can be used to identify the limits of the hardware infrastructure as well as the software architecture, thereby allowing for targeted optimizations and improvements. The main goal is to ensure that the respective Artemis setup can handle the expected load during exams without any performance issues.
During an exam simulation, the tool participates in the exam on behalf of a configurable number of students. It performs various actions, such as logging in, starting the exam, submitting answers, and git operations. For each action, the tool measures the time it takes to complete the action and records the result. Additionally, the tool can be connected to Prometheus to collect and visualize workload metrics.
For more information on how to set up and use the Benchmarking Tool, please refer to the README.
Looking for how much hardware an installation needs rather than for how to measure it? See Scaling Artemis, which carries the sizing guidance drawn from the runs on this page.
Running a comparable scale test
A benchmark number is only useful if someone else can reproduce it. Record the following alongside any result, so that two runs can be compared and so that a number keeps its meaning after the hardware or the release changes:
| What | Why it matters |
|---|---|
| Artemis version | Performance characteristics change between releases. |
| Deployment topology | Number of Artemis nodes, number of build agents, and whether a load balancer sits in front. |
| Hardware per node | vCPUs, memory, disk type. A number without hardware is not a number. |
| Database | PostgreSQL or MySQL, version, and whether it runs on its own host. |
| Scenario | Exam or course, number of simulated students, exercise types, and the ramp-up profile. |
| Result | Response time percentiles (median, p95, p99) per action, error rate, and the observed limit. |
| Observed bottleneck | What saturated first: CPU, database connections, build agents, network. |
The most informative result is not the largest number the setup survived, but the point at which a percentile started to degrade and which resource caused it.
Benchmark scenarios
A single large number answers almost nothing. What an operator needs to know is where the response time starts to degrade and which resource caused it, so the runs below form a ladder: each one keeps the exam shape fixed and changes only the number of simulated students.
The tool's exam contains four exercises (one programming, one modeling, one quiz, one text). Each simulated student logs in, fetches and starts the exam, submits each exercise, pushes commits for the programming exercise, and submits the exam.
A simulated student is not a real student
Read this before choosing a size, because it decides what the numbers mean. The simulation is a closed-loop maximum-throughput test, not a scaled-down exam:
- Think time is short. A student pauses two to four seconds after each action, and a tenth of a second before each push. A real exam spreads the same work over an hour or more, so a run still compresses the load by a large factor.
- The default push count is high. The form defaults to 8 to 15 commits and pushes per student, and every push is a build. At the sizes below that is tens of thousands of builds.
Together these make a simulated student far more demanding than a real one, so a run of N simulated students corresponds to a considerably larger real cohort. Treat the student count as a load setting, not as a headcount.
Two consequences for how a ladder is run:
- Keep the pushes low, one or two per student, or the build queue rather than Artemis becomes the
thing being measured. Estimate the drain as
students x pushes / concurrent build slots x build timebefore committing to a maintenance window. - Client concurrency is capped at 200 students in flight by default, independent of the tool host's
CPU count, and configurable through
benchmarking.simulation.max-concurrency. Above that many students a larger run adds duration and accumulated state rather than more simultaneous users. A student who is in its think time does not occupy one of those slots. Record the setting with the result, or two runs are not comparable.
The ladder
Start with a smoke test that is not reported: ten students, one push each. It proves the users, the admin or instructor rights, the build agents and the git path work, and it gives the first real measurement of build time, which every duration estimate depends on.
Then hold the exam shape, the push range and the repository access mix fixed, and change only the student count:
| # | Students | Purpose |
|---|---|---|
| S1 | 100 | Baseline. Every later run is compared against these percentiles. |
| S2 | 200 | First check that nothing degrades when the volume doubles. |
| S3 | 500 | Where queueing usually becomes visible. |
| S4 | 1000 | The decision point: read the metrics here before going further. |
Extend to 1500 and then 2000 only if S4 leaves headroom. Decide from S4's numbers rather than planning it in advance: if a percentile is already bending, or the build queue has not drained inside the window, a larger run costs hours and answers nothing.
Hold the repository access mix fixed across the whole ladder. The tool draws it per student from four percentages that must add up to 100 (online IDE, password, participation token, and SSH), so record all four alongside the result. Note that SSH clones bypass any rate limiting a reverse proxy applies to git over HTTP, which makes an all-SSH mix a poor choice for a first run against an unfamiliar deployment.
Two variations at the target size
Run these at whichever of S3 or S4 turned out to be the interesting one. They separate causes that the ladder alone cannot distinguish:
| Variation | Change | What it isolates |
|---|---|---|
| Online editor | Online IDE share set to 100% | Moves programming traffic off git clone and push and onto the REST and WebSocket path, separating version control cost from application cost. |
| No programming exercise | Exam with only modeling, quiz, and text | Removes the build agents and the git server from the picture, so the remaining latency is attributable to Artemis and the database. |
Keep everything else identical between a run and its variation, or the comparison is worthless.
Reading a run
Compare the per-action figures the tool reports (authentication, get and start student exam, submit exercise, submit exam, clone, push) across the ladder rather than the totals. The action that degrades first names the bottleneck:
- Clone and push degrade first: the git server or its storage.
- Submit exercise degrades first: Artemis or the database.
- Authentication degrades first: the identity provider, or connection pool exhaustion.
- Builds per minute plateaus while push stays flat: build agent capacity. That is expected, and is a capacity-planning input rather than a defect.
Enable the tool's Prometheus integration for these runs. Without it a result describes symptoms; with it the same result names the saturated resource.
Publishing a result
Scale claims about Artemis should be traceable to a run described this way. If you have run a
benchmark on your own deployment, please consider contributing the result: it helps other institutions
size their infrastructure, and it turns a claim into evidence. Open a pull request against this page or
write to artemis@xcit.tum.de.
Reference run
Artemis is highly optimized for performance and sustainability. Response time is not a back-office metric here, it is part of the user experience: a student waiting on a page during an exam is a student losing exam time, and an instructor who dreads opening a dashboard stops using it. So the platform is engineered to stay fast under the heaviest load it ever sees, and to do it on modest hardware rather than by throwing servers at the problem.
Exams are where that gets tested for real. Every student arrives in the same minute, opens the same exam, and starts submitting at once. There is no gentle ramp and no second chance.
Here is what that looks like measured.
Four thousand students, and nothing broke
| Students | 4,000 |
| Requests handled | 390,988 |
| Failed requests | 0 |
| Average response time | 0.09 seconds |
| Git operations | all of them, over authenticated HTTPS |
Every one of those 4,000 students logged in, opened the exam, worked on and submitted all four exercise types, pushed code to their own git repository and handed in the exam, pausing only a couple of seconds between actions. A 90-minute exam, compressed into sixteen. Not a single request failed, and the average interaction came back in 88 milliseconds.
On hardware you would not expect
Three Artemis core nodes, three build agents, one database and one file server, plus the load balancer and the supporting services behind them: eleven machines in total. Every one of them a virtual machine with 4 CPUs and about 3.8 GB of memory, each individually weaker than a typical laptop.
This is the sustainability point, and it is a budget point too. Artemis does not need a data centre to run a large exam. Fewer machines, less power, less hardware to buy and replace.
Fast where students feel it
| What a student does | With 4,000 students on the system |
|---|---|
| Open the exam | 0.02 seconds |
| Load their exam | 0.01 seconds |
| Submit an exercise | 0.02 seconds |
| Hand in the exam | 0.01 seconds |
| Download their code repository | 0.24 seconds |
| Push their code | 0.21 seconds |
Programming exams are the hardest case any platform faces, because every student is moving real code in and out of real version control. Every one of these 4,000 students cloned and pushed over authenticated HTTPS, the way a student working in their own IDE does. Even so, a push completed in a fifth of a second.
Where the engineering shows
Three measurements say more about how Artemis is built than any total.
The database has capacity to spare. It answered 3.7 million statements in 303 seconds of total execution time across the run. At the busiest instant 69 of its 120 pooled connections were executing a query; the median across the run was two. Databases are what usually buckle first under exam load, because a careless platform queries the same data over and over on every page. Artemis issues few, well-targeted queries per interaction, and the component most platforms saturate first spent most of the run idle.
Handling student code is nearly free. Storage is the usual suspect when a platform slows during a programming exam, because every clone and every push is real file work on real repositories. With all 4,000 students cloning and pushing, a clone took 0.24 seconds and a push 0.21 seconds. Nothing in that path needed extra hardware to get there.
There is room left over. The three Artemis servers averaged 20% to 27% CPU across the run and the load balancer under 5%, on virtual machines weaker than a laptop. When a course does outgrow its cluster, you add another Artemis server: they are interchangeable, they join automatically, and nothing needs reconfiguring. Build agents scale separately again, so how quickly students get feedback on their code is a decision you make rather than a limit you inherit.
Being straight about what this shows
Two honest notes, because a benchmark that flatters itself is worth nothing.
It gets cheaper per student, not dearer. The same cluster was measured at 3,000 students first, at 0.08 seconds. Adding a third more students cost 11% more response time, not a third more and not the doubling that a system near its limit would show. 4,000 is the largest run reported here, not the largest Artemis handles.
A simulated student is harder work than a real one. Real students read the question, think, and type; these ones pause for a couple of seconds and move on, compressing a 90-minute exam into sixteen minutes. A real cohort of the same size is a considerably lighter load, so the 4,000 here stands for substantially more than 4,000 people in a room. The run below was measured before the tool paused between actions at all, which compressed it further still.
That last point lines up with operational history: Artemis runs courses and exams with thousands of students and has done for years. This run is independent evidence, on small hardware, that the room behind that number is real.
Want to check it on your own infrastructure? The rest of this page explains how, and we would welcome your results.
Measured results
The section above reports the headline run. This one reports both clusters in more detail, plus one single machine, so you can see what the load actually did to each component. The metrics stay high-level: how much hardware, how many students, how long the work took, and what ran out first.
If you are here to size an installation rather than to reproduce a measurement, read Scaling Artemis instead: it draws the sizing conclusions from these runs and links back to the detail below.
Both cluster runs used the same scenario, the same load generator and the same student count, so they are comparable with each other. Every student logs in, opens the exam, submits all four exercise types, pushes code to their own git repository, and hands the exam in, with no pause between actions.
What was tested
| staging1 | staging2 | |
|---|---|---|
| Virtual machines | 11 | 11 |
| CPUs in total | 44 | 44 |
| Memory in total | 41.5 GB | 41.5 GB |
| Artemis core nodes | 3 | 3 |
| Build agent machines | 3 | 3 |
| Concurrent builds | 6 | 9 |
| Database | MySQL, own machine | MySQL, own machine |
| Coordination between servers | Hazelcast | Redis |
| Repository storage | shared network storage | shared network storage |
| Simulated students | 2,000 | 2,000 |
Every machine is a virtual machine with 4 CPUs and about 3.8 GB of memory. The clusters get their capacity from having eleven of them rather than from any machine being large.
What happened
| Action | staging1 | staging2 |
|---|---|---|
| Load the exam | 0.14 s | 0.11 s |
| Open the exam | 0.23 s | 0.29 s |
| Submit an exercise | 0.75 s | 0.31 s |
| Hand in the exam | 0.29 s | 0.23 s |
| Download a code repository | 2.04 s | 0.98 s |
| Push code | 3.11 s | about 2 s |
| Everything else | 0.27 s | 0.13 s |
| Average across all actions | 0.45 s | 0.30 s |
| Requests handled | 72,024 | 70,197 |
| Time to work through the exam | 3 minutes | 2.5 minutes |
The staging2 push figure comes from a repeat run, reported approximately. In the first staging2 run a number of pushes failed server-side and were not retried, which produced a misleading 4.27 s; the repeat completed every push in about two seconds with none failing. The remaining staging2 figures in this table are from the first run, so the 0.30 s average still includes those slow pushes and is, if anything, slightly pessimistic.
Where the load landed
| Machine | staging1 | staging2 |
|---|---|---|
| Busiest Artemis core node | 87% CPU | 77% CPU |
| Other two core nodes | 66%, 60% | 42%, 43% |
| Database | 47% CPU | 47% CPU |
| Repository storage | 11% CPU | 12% CPU |
| Coordination service | not measured separately | 10% CPU |
| Reverse proxy | 11% CPU | 8% CPU |
| Busiest network link | under 5% of capacity | under 5% of capacity |
Reading the results
Both clusters had room left. The busiest core node finished at 87% in one cluster and 77% in the other, and everything behind it was far quieter: the database below half, the storage holding every student repository around a tenth, the network at a few percent. Neither run was pushed until it failed, so both results are lower limits: Artemis handled at least this much, and how much more it would have handled is unknown.
The supporting services are not where capacity goes. This is the most transferable result on the page. Storage, coordination and the proxy never exceeded 12% on either cluster, while the core nodes came close to their limit. Capacity planning should follow that imbalance.
staging2 was faster on almost every action, and the difference is large: submitting an exercise took 0.31 s against 0.75 s, and downloading a repository 0.98 s against 2.04 s. Two things differ between the clusters, so it is not possible to say which of them caused the improvement: staging2 coordinates through Redis rather than Hazelcast, and it was running a newer Artemis version. What can be said is that the coordination layer was never the constraint on either cluster: Redis peaked at 10% CPU and 44 MB of memory while handling 2,500 commands per second, and never had to discard data to stay within its memory limit.
Pushing code looked slower on staging2 at first, and that turned out to be a fault rather than a limit. The initial 4.27 s came from a run in which a number of pushes failed server-side and were not retried, which also explains the lower request count. A repeat run completed every push in about two seconds with none failing. With that corrected, opening the exam is the only action on which staging2 was slower than staging1, by 0.06 s.
How the students authenticated
Both cluster runs above had every student use a repository access token, which is what Artemis hands out through the Code button and what most students use. This matters more than it sounds: an earlier run with every student typing a password instead was markedly slower, because verifying a password is deliberately expensive and it dominated everything else on the core nodes. That comparison is an unverified observation rather than a result from the reference run above — it predates these runs, and the metric, workload and Artemis version behind the roughly 58% figure quoted at the time were not recorded alongside it, so it should not be cited as a benchmark. If you benchmark your own deployment, record the authentication mix alongside the result, or two numbers will not be comparable.
What a single small machine can do
Not every installation is a cluster, so it is worth knowing what the smallest sensible one can do. The measurements below come from one virtual machine with 4 CPUs and 3.8 GB of memory running everything at once: Artemis, the build agent, PostgreSQL and the reverse proxy. Unlike the cluster runs above, the authentication mix of these three runs was not recorded, so read the repository column on its own rather than against the cluster figures: password and token authentication differ enough that the two are not comparable.
| Simulated students | Average response | Submit an exercise | Download a repository |
|---|---|---|---|
| 50 | 0.42 s | 1.05 s | 0.88 s |
| 100 | 0.65 s | 1.78 s | 1.69 s |
| 200 | 1.13 s | 2.69 s | 3.47 s |
Every one of those runs completed with zero failed requests.
A run of 100 simulated students completed on this hardware with no failed requests. At that size the average interaction comes back in two thirds of a second and submitting an exercise takes under two seconds, which a student in an exam would not notice. Read these as configured student counts rather than verified simultaneous users: as noted above, the tool caps client concurrency at ten threads per CPU of the machine running it, so whether a given row was genuinely concurrent depends on the load generator's host rather than on Artemis. Remember also that the simulation is harsher than the real thing: it gives students no thinking time at all, so a real cohort of the same size is a lighter load than this. An institution running a course rather than a faculty can start here.
Beyond that the machine runs out. At 200 students response times roughly double again, and it is memory that runs out rather than processor: the machine is already using swap at 50 students and has exhausted it by 200. Adding memory helps this configuration more than adding cores would, and giving the database a machine of its own helps more than either.
Sizing an installation from these results
The conclusions these measurements support, from how many core nodes and build agents to run to what to check when something is slow and where additional capacity does not help, are collected in Scaling Artemis. That is the page to read before buying hardware; this one is the evidence behind it.