Distributed Data

Artemis runs as a cluster: several core nodes behind a load balancer without sticky sessions, plus build agents that pick up jobs from a shared queue. Everything those nodes share — the build job queue, feature toggles, scheduling messages, websocket broker status, LTI state, Pyris jobs, rate-limit buckets, the @Cacheable caches — lives in a distributed store.

❗

Danger:

Never use Hazelcast or Redis directly. All cross-node state goes through DistributedDataProvider. An ArchUnit rule (DistributedDataProviderArchitectureTest) fails the build if a production class outside the small, explicitly named set of provider adapters depends on com.hazelcast.., org.redisson.. or org.springframework.data.redis...

Why the abstraction exists

Artemis supports two providers and intends to be able to switch between them:

• Hazelcast — embedded in every node, no extra container. This is what production runs today.
• Redis — an external server, selected with artemis.distributed-data.provider: Redis. With it, no Hazelcast instance is created at all.

A third, Local, keeps everything inside one JVM. It exists for single-node development and has no cross-node semantics.

That choice only stays real as long as no application code names a provider. A single hazelcastInstance.getMap(...) in a service pins the whole deployment to Hazelcast, and it does so without failing any test: the other provider simply never sees that state, and nothing reads it back to notice. That is exactly how the abstraction drifted the first time, when 21 services across 8 modules had each grown their own Hazelcast usage, and where the providers disagreed nobody found out until a shared contract test suite was written for them.

The API

DistributedDataProvider hands out the structures, and each has an implementation per provider plus a case in the shared contract suite AbstractDistributedDataTest, which every provider runs:

Method	Use for
getMap(name)	Shared key/value state that never expires. Rejects a per-entry time-to-live, so the expiry requirement stays visible at the call site instead of hiding in provider configuration.
getExpiringMap(name, ttl)	Shared state that has to expire. Entries stored with put(k, v) live for ttl; put(k, v, otherTtl) overrides it per entry.
getQueue(name) / getPriorityQueue(name)	Work queues. A priority queue dequeues by the items' natural ordering.
getSet(name)	A shared set, for example the courses with a plagiarism check in flight.
getTopic(name)	Fire-and-forget broadcast, for state that self-heals on the next heartbeat.
getReliableTopic(name)	Broadcast where losing one message has a lasting effect — the scheduling messages, where a dropped message means an exercise or quiz is never scheduled.
getLock(name)	A cluster-wide mutex. Deliberately AP, not a CP fenced lock: it prevents concurrent work under normal operation, it does not survive a split brain.

putIfAbsent, remove(key, value) and computeIfAbsent on a map are genuine atomics and are the right way to claim a job slot from exactly one node.

⚠️

Warning:

A provider-level map configuration is not a substitute for asking for what you need. A Hazelcast MapConfig time-to-live applies on Hazelcast only, and silently does nothing on Redis. Request the lifetime at the call site with getExpiringMap.

Selecting the provider

artemis.distributed-data.provider takes Hazelcast (default), Redis or Local. As an environment variable that is ARTEMIS_DISTRIBUTEDDATA_PROVIDER. The superseded artemis.continuous-integration.data-store, which was named when the abstraction only carried the Local CI queue, is still honoured as a fallback for one release.

An unsupported value fails at startup rather than leaving the application with no provider at all.

Redis needs no service registry

Eureka exists in Artemis for one purpose: finding the addresses of Hazelcast members so a member or client can join the cluster. Every reader of the registry — HazelcastConfiguration, HazelcastClusterManager, EurekaHazelcastDiscoveryStrategy and the EurekaInstanceHelper they share — is conditional on Hazelcast. Nothing else in Artemis reads it: there are no @LoadBalanced clients, no lb:// URIs and no Spring Cloud Config server.

So on Redis, RedisDiscoveryEnvironmentPostProcessor sets both eureka.client.enabled=false and spring.cloud.discovery.enabled=false before the context is built, and Spring Cloud's own conditions then keep every Eureka and discovery auto-configuration out of it. Both are needed: the first stops the registration, the heartbeat and the registry health indicator, but on its own it leaves Spring Cloud's discovery health indicators behind reporting Discovery Client not initialized, which drags the whole health endpoint down. A Redis node ends up with no Eureka client, no registration, no heartbeat thread and no discovery health indicator at all — verified on a three-node cluster: zero Eureka threads, zero live com.netflix.* objects, zero references to port 8761, and a health endpoint with no discovery entries.

A Redis deployment therefore does not need to run a JHipster registry at all. The local multi-node runner does not start one for --middleware redis, which is what keeps this property honest: if anything started depending on the registry again, that stack would stop coming up.

Caches

Spring @Cacheable resolves against a RoutingCacheManager that splits caches by value shape:

• Blob caches (files, plantUmlPng, plantUmlSvg) are per-node Caffeine caches bounded by bytes, because their values range from kilobytes to tens of megabytes. Pushing those through a shared grid costs a network transfer per read, and on Redis it is worse: Redis is single-threaded, so one multi-MB value stalls every other operation including the CI queue. Coherence comes from BlobCacheEvictionService, which broadcasts evictions, plus a time-to-live so a dropped broadcast self-corrects.
• Everything else is served by DistributedDataCacheManager, which is built on DistributedDataProvider and therefore behaves the same on every provider.

The rest of the caching policy — in particular that Hibernate second-level cache is disabled cluster-wide — is on the Caching page.

Adding a capability

If something you need is genuinely missing from DistributedDataProvider, add it there rather than reaching past it:

1. Add the method to DistributedDataProvider with Javadoc that says what guarantee callers may rely on.
2. Implement it for all three providers (hazelcast, redisson, local).
3. Add a case to AbstractDistributedDataTest, so every provider has to agree. Behaviour that only one provider supports must fail uniformly rather than work on one and silently do nothing on the other.

Only then add the class to the allowlist in DistributedDataProviderArchitectureTest if it truly has to name a provider — each entry there needs a recorded reason, and two guard tests fail if an entry stops matching a class or stops using a provider.

Changing a stored type

Hazelcast discards its data when the cluster stops. Redis and Valkey retain it so that, for example, a queued build survives a deployment. Persistent data also means that a new release can encounter bytes written by an older one. Queue entries, set elements, topic messages, and map keys use positional Kryo encoding without an embedded schema version. An incompatible change can therefore cause a KryoBufferUnderflowException or produce plausible but incorrect values. Issue #12137 is an example.

Redis keys include DistributedDataSchema.VERSION, for example artemis:v1:{buildJobQueue}. A release reads only its own namespace. The version protects production only when every incompatible change is detected, reviewed, and paired with an explicit migration decision.

Compatibility checks

DistributedDataSurfaceTest records the distributed wire surface in src/test/resources/config/distributed-data-surface.txt. It covers the declared root payloads, recursively follows Artemis-owned component types, discovers every concrete PyrisJob, and discovers return values written through distributed @Cacheable and @CachePut operations. The three per-node blob caches are excluded. For records the test captures component names, types, and order; for enums it captures names and order; and it also records effective serialVersionUID values.

Run the focused check with:

./gradlew test --tests de.tum.cit.aet.artemis.core.service.distributed.DistributedDataSurfaceTest

If the surface differs, the test fails and writes the proposed surface to build/distributed-data-surface.actual.txt. Review it before changing the recorded baseline:

diff -u src/test/resources/config/distributed-data-surface.txt \
    build/distributed-data-surface.actual.txt

The test detects a change that might require migration; it cannot decide whether two encodings are compatible. That decision requires review. It also cannot discover an entirely new payload passed directly to a newly added distributed structure. When adding such a use, add its root type to DistributedDataSurfaceTest.DECLARED_ROOTS. Types reachable from an existing root do not need a second entry. PyrisJob implementations and distributed Spring cache values are discovered automatically.

Decision workflow

When the compatibility check reports a change:

1. Identify every distributed map, queue, set, or topic that stores or transports the affected type.
2. Decide whether the previous release can read the new encoding and whether the new release can read existing data. Adding, removing, reordering, or retyping a record component and changing enum order should be treated as incompatible unless compatibility is demonstrated explicitly.
3. If the change is wire-compatible, explain why in the pull request and update the recorded surface.
4. If the change is incompatible, increment DistributedDataSchema.VERSION by exactly one and add an explicit migration from the previous version.
5. For each affected structure, choose whether to preserve or discard it. Preserve data that cannot be reconstructed, such as queued or running builds, unwritten results, and runtime feature toggles. Registrations, presence data, and caches can normally start empty.
6. If preserved data itself has an incompatible representation, write a custom transformation using the previous DTO and codec. The raw-byte mover is only valid when the representation is compatible.
7. Review and copy build/distributed-data-surface.actual.txt to the recorded surface only after the version and migration decision are implemented.
8. Add migration tests for successful conversion, retained data, discarded data, expiry, interruption recovery, and rejection of unsupported versions as applicable.
9. Document the coordinated upgrade and rollback impact in the release notes and the administrator guide.

RedissonDistributedDataMigratorTest exercises the initial migration, restart recovery, version validation, and rejection of a missing adjacent migration. The production migrator also validates the complete adjacent path before it moves any entry. Together with DistributedDataSurfaceTest, this makes an overlooked change visible, but the reviewer still owns the compatibility and preservation decision.

Migration invariants

Every transition owns its carry-over list. A structure omitted from that transition starts empty. This makes discarding data a reviewed code change rather than an emergency production command.

Migration runs under a distributed lock before the provider exposes any structure. It moves entries incrementally to keep peak memory bounded. Queue moves are atomic; map and set moves write the target before removing the source; and the schema version is published only after the step completes. Every step must remain safe to run again after interruption.

The raw-byte mover opens a structure with ByteArrayCodec, avoiding accidental deserialization through the current codec. It does not make incompatible bytes readable. Use it only when the old and new representations are compatible.

The braces in artemis:v1:{buildJobQueue} are a Redis Cluster hash tag. They place every version of one logical structure in the same cluster slot, allowing an atomic server-side queue move. Different logical structures continue to hash independently.

Locks remain unversioned because they contain no serialized payload. Old and new releases must still contend for the same mutex. This does not make a mixed-version deployment safe: administrators must stop every old core and build-agent node before starting a release with a new distributed-data schema.

The first versioned release treats existing unversioned structures as schema version 0 and drains preserved data into the artemis:v1: namespace. It cannot safely delete all other unversioned keys because version 0 covered the entire keyspace. Those obsolete keys are ignored by the new release. A store that reaches a release declaring a later schema continues through every remaining transition in the same startup, each with its own carry-over list, so artemis:v1: can be an intermediate namespace rather than the final one. The administrator guide describes safe upgrade, verification, cleanup, and rollback procedures.

Testing

• AbstractDistributedDataTest is the contract suite; LocalDataTest, HazelcastDistributedDataTest and RedissonDistributedDataTest run it against the three providers.
• The multi-node E2E runners take --middleware hazelcast|redis and run the identical suite on either provider:

./supporting_scripts/e2e/run-e2e-tests-local-multinode-fast.sh --middleware redis
./supporting_scripts/e2e/run-e2e-tests-local-multinode.sh --middleware redis --filter "ClusterFormation"

Hazelcast remains the default, because that is what production runs.