Operating LocalCI on Kubernetes

This page covers the day-to-day operation of an installation that uses the Kubernetes runner and the Helm chart deployment/helm/artemis: which component does what, where each setting goes, how to size and scale the installation, how to update it, and what to monitor. The installation itself is described in Deploying in the build runners guide, and the chart's own README.md and CLUSTER-SETUP.md document the lifecycle commands of the local acceptance cluster.

⚠️

Warning:

The Kubernetes runner and the chart are a proof of concept. They have not been through a production deployment. The chart provides no high availability for PostgreSQL, the broker or the registry, no network policies, no resource quotas and no backups. Autoscaling of the build agents is optional and needs KEDA and Prometheus. This page says where each of these is left to you.

What the chart deploys

Component	Kind	Purpose
Leader core	StatefulSet, one pod	An Artemis core node that also runs the scheduling profile, so that scheduled tasks run once.
Member cores	StatefulSet, artemis.replicaCount - 1 pods	Further core nodes. They serve HTTP and git and join the leader's cluster.
Build agent controllers	StatefulSet in the Artemis namespace, buildAgents.replicaCount pods	Take jobs from the shared queue and create one Kubernetes Job per build. They never run student code.
Build Jobs	Jobs in buildAgents.namespace	One per build. They run the exercise image and a trusted helper container.
PostgreSQL	StatefulSet, optional	The database. Disable it with postgresql.deploy: false to use an external one.
JHipster Registry	Deployment	The service registry through which core nodes find each other.
ActiveMQ	Deployment, optional	The relay for WebSocket messages between core nodes.
Gateway resources	Gateway API objects, optional	HTTP, HTTPS and git over SSH.

The pod name of a build agent controller is its identity: it becomes the short name of the build agent and the ownership label on every Job it creates. This is why the controllers are a StatefulSet and not a Deployment, and why the names are stable across restarts.

Configuring Artemis

Three places hold configuration, depending on what the setting is.

What	Where
Settings that the chart knows	The values file, see the sections artemis, buildAgents, postgresql, registry, broker, gateway and sharedStorage in values.yaml.
Secrets and the installation metadata	The values artemis.config.admin.password, artemis.config.jwtBase64Secret, postgresql.auth.password, registry.password, broker.auth.password and artemis.config.operator.*. The chart does not render without them.
Any other Artemis property of the core nodes	artemis.extraEnv, a list of environment variables, and artemis.extraProfiles for additional Spring profiles.

An Artemis property becomes an environment variable by upper-casing it and removing the dashes and dots, for example artemis.continuous-integration.concurrent-result-processing-size becomes ARTEMIS_CONTINUOUSINTEGRATION_CONCURRENTRESULTPROCESSINGSIZE, which is how the chart's own ConfigMap sets its values. Use it for settings from the Settings at a glance table that apply to core nodes:

artemis:
  extraEnv:
    - name: ARTEMIS_CONTINUOUSINTEGRATION_BUILDTIMEOUTSECONDS_MAX
      value: "300"

artemis.extraEnv applies to the core nodes only. The settings of the build agent controllers are the values under buildAgents, which the chart maps to the properties below artemis.continuous-integration.kubernetes, to the number of concurrent builds, and to the agent's resources. Settings such as the maximum build timeout must be equal on core nodes and build agents, see Which setting goes on which node. The chart has no hook for further agent-only environment variables, so a setting of that kind needs a change to the chart.

Every pod runs with the prod profile and therefore refuses to start with a credential that Artemis publishes as an example. The shared build agent git credential is deliberately not configurable: every node runs LocalCI, and the build agents authenticate with the clone token of the job they run.

Sizing

Build capacity

The number of builds that run at the same time is

buildAgents.replicaCount × buildAgents.concurrentBuildsPerAgent

With the defaults of two controllers and one build each, that is two builds. Each running build is a Pod on the build nodes, and it requests the resources of the exercise or, when the exercise sets none, buildAgents.defaultResources (2 CPUs, 2 GiB of memory and 4 GiB of ephemeral storage by default). The build nodes therefore need room for the maximum number of concurrent builds times these values, in addition to what else runs on them. The controllers themselves are small, see buildAgents.controllerResources (250 millicores and 512 MiB requested). A build whose Pod cannot be scheduled waits for resources, and the wait counts against pod-start-timeout-seconds, not against the build timeout.

Choose the number of controllers from the peak, using the sizing table in Scaling Artemis, and raise concurrentBuildsPerAgent only together with the capacity of the build nodes. Keep build workloads on their own nodes with buildAgents.workloadNodeSelector and workloadTolerations, so that they cannot take the memory of core nodes.

Core nodes

artemis.replicaCount is the total number of core nodes, one leader plus the members. The defaults request 1 CPU and 2.5 GiB of memory with a 4 GB heap, and allow up to 2 CPUs and 5 GiB. Memory limits the core nodes before processor, see Scaling Artemis. More than one core node needs a ReadWriteMany storage class for sharedStorage, because all cores mount the same /opt/artemis/data. The leader is always a single pod. If it is unavailable, no scheduled task runs until it is back, while the members keep serving users.

Scaling

The build agent controllers can be scaled by hand or automatically. The core nodes are always scaled by hand.

Manually

Change the value in the values file and run helm upgrade, so that the chart stays the single source of truth. A replica count that is changed with kubectl scale is not recorded in the release.

• More build capacity: raise buildAgents.replicaCount. A new controller registers with the cluster within seconds and takes jobs at once. Make sure the build nodes have room first.
• Less build capacity: lower buildAgents.replicaCount. The StatefulSet removes the highest ordinal first. A build that runs on a removed controller is put back into the queue and runs again on another agent, at most five times, see Operating build agents. To avoid the repetition, and the extra load described under Automatically, pause the agent on the Build Agents page and wait for its builds to finish before you lower the count.
• More core capacity: raise artemis.replicaCount. A new member waits for the leader to be ready and then joins the cluster.

The values to decide with are the queue and the utilization on the Build Agents page, and the gauges artemis.global.localci.queued, running, agents and maxConcurrentBuilds. The core pods carry the annotations prometheus.io/scrape, prometheus.io/path (/management/prometheus) and prometheus.io/port for a Prometheus that honors them.

Automatically

With buildAgents.autoscaling.enabled: true the chart creates a KEDA ScaledObject for the build agent controllers. It needs KEDA in the cluster and a Prometheus that scrapes the core pods, and the chart installs neither. Two settings make the metrics usable:

1. Scope the query to this installation. A Prometheus often scrapes several Artemis installations, for example production and a test installation. Without a scope, the backlog of one installation would scale the controllers of the other, and the metrics of the other would hide a missing scrape. buildAgents.autoscaling.prometheus.selector holds label matchers that select the metrics of this installation, and the chart refuses to render without it, unless you set your own query. Which labels exist depends on the scrape configuration of your Prometheus, so use labels that identify this installation, for example the namespace of the core pods.
2. Allow Prometheus to scrape Artemis. /management/prometheus only answers requests from the addresses in spring.prometheus.monitoringIp, which is 127.0.0.1 by default, and the chart does not change it. Without the setting, the scrape of the core pods is denied and KEDA has no metrics. Set it for the core nodes through artemis.extraEnv to the source IP addresses of Prometheus, comma-separated and as exact addresses, not as ranges. Use the addresses that the core pods see, which are the address of the Prometheus pod unless your network translates them.

artemis:
  extraEnv:
    - name: SPRING_PROMETHEUS_MONITORINGIP
      value: "10.42.1.15"
buildAgents:
  concurrentBuildsPerAgent: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    prometheus:
      serverAddress: http://prometheus-server.monitoring.svc:80
      selector: namespace="artemis"

• Target. The number of controllers that the queue needs: ceil((queued + running builds) / concurrentBuildsPerAgent), from the cluster-wide gauges artemis.global.localci.queued and running, limited to minReplicas and maxReplicas. autoscaling.prometheus.query replaces the default query, which is built from the selector.
• Replica count. The StatefulSet no longer sets replicas, so buildAgents.replicaCount is ignored and helm upgrade does not reset the count. The upgrade that first enables autoscaling on an existing release is the exception: Helm can remove the field, which makes Kubernetes reset the StatefulSet to one replica until KEDA has reconciled it. The controllers above the first one are removed and their running builds are interrupted. Enable autoscaling while no builds run: pause the build agents, wait until their builds have finished, as in Updating, and set minReplicas to the capacity that you want to keep.
• Speed. Controllers are added quickly (four per minute by default) and removed slowly (one every five minutes, after ten minutes below the target). The scaleUp and scaleDown values change this.
• Missing metric. A missing series is an error and not zero, so a broken scrape leaves the count unchanged instead of scaling down while builds wait.
• Scale-down. The controller with the highest ordinal is removed, whether it is busy or not. Its builds are put back into the queue and run again, at most five times, and are marked failed after that. The Kubernetes Job of the removed controller is not stopped. The other controllers only clean up the Jobs of their own name, so it runs until it finishes or reaches its active deadline, which is the build timeout plus the budgets for the pod start, the result collection and the grace period. A retry can therefore run next to it and use extra build capacity for that time. A controller that starts again under the same name removes the Jobs it no longer tracks. The slow scale-down keeps all of this rare, but it cannot rule it out.
• Paused controllers. A paused controller counts as a replica but takes no builds, and the scaler adds no capacity for it. Pausing is a maintenance action, and a new controller would start unpaused and override it. Builds queued while all controllers are paused can start on a new controller without a resume action: when the queue raises the target above the current replica count, KEDA starts a controller, and it starts unpaused. With a paused controller that stays at the current count, the builds wait until a controller is resumed.
• Capacity. maxReplicas × concurrentBuildsPerAgent builds must fit on the build nodes, otherwise the additional build Pods wait for resources. A cluster node autoscaler can add the nodes when Pods wait, but it does not change the number of controllers.

Updating

⚠️

Warning:

The chart does not set an update strategy, so Kubernetes updates the StatefulSets as a rolling update and the leader, the members and the build agents change independently of each other. That contradicts the rule of Deploying a New Version: no instance may serve traffic against a database schema it was not built for. For a release that contains database migrations, use the staged procedure below.

1. Back up the database and the shared storage, and make sure that my-values.yaml below holds all values of the installation. A value that is missing from it falls back to the chart default in both upgrades, so the leader could, for example, mount a new, empty volume instead of the existing shared storage. If you have no such file, export the current values and add the new image tags:

   helm get values <release> --namespace <namespace> --output yaml > my-values.yaml

2. Pause the build agents on the Build Agents page and wait until their builds have finished.

3. Stop all core nodes and build agents, and wait until they are gone. Otherwise the new leader can migrate the database while old members still serve requests against the old schema. Use the namespace and the release of your installation. List the StatefulSets of the release with kubectl -n <namespace> get statefulsets -l app.kubernetes.io/instance=<release>, which also works when the release uses fullnameOverride. Their names end in -leader, -member (only with more than one core node) and -build-agent. Scale each one that exists, with <name> as listed:

   kubectl -n <namespace> scale statefulset <name>-build-agent --replicas=0
   kubectl -n <namespace> scale statefulset <name>-member --replicas=0
   kubectl -n <namespace> scale statefulset <name>-leader --replicas=0
   kubectl -n <namespace> get pods

   Skip the member command when there is no member StatefulSet. Repeat the last command until no core node and no build agent pod is left. PostgreSQL, the registry and the broker keep running.

   With automatic scaling, KEDA would start the build agents again. Instead of scaling the build agent StatefulSet, pin it to zero through the ScaledObject, which has the same name:

   kubectl -n <namespace> annotate scaledobject <name>-build-agent autoscaling.keda.sh/paused-replicas="0"

4. Update the same release with a single core node and no build agents:

   helm upgrade <release> ./deployment/helm/artemis --namespace <namespace> \
     --values my-values.yaml \
     --set artemis.replicaCount=1 --set buildAgents.replicaCount=0 \
     --set buildAgents.autoscaling.enabled=false \
     --wait --timeout 25m

   The leader starts alone and applies the Liquibase migrations. The command returns when it is ready. The buildAgents.autoscaling.enabled option removes the ScaledObject for this step and is only needed with automatic scaling.

5. Run the same command with the same my-values.yaml, without the --set options. This restores the intended replica counts, and the members and the build agents start with the new version. With automatic scaling the ScaledObject is created again.

6. The new build agents start unpaused and take builds as soon as they are ready, so there is nothing to resume. Check the Build Agents page for the number of agents and their status, and trigger a build.

Take the image tags from the values file and use versioned tags, for the Artemis image and for the helper image. For a release without database migrations, a plain helm upgrade is enough. Rolling back an Artemis version after a migration needs the backup of step 1, see the warning in Deploying a New Version.

With Hazelcast, the chart default, the queue lives in the memory of the cluster. Builds that are queued when all nodes stop are not kept, and the missing-build check puts them back into the queue after the restart.

Monitoring

What	How
Pods and their readiness	kubectl -n artemis get pods -o wide. The probes use /management/health/readiness.
Build Jobs	kubectl -n artemis-builds get jobs,pods shows the builds that are running. The agent deletes a Job and its Pods as soon as it has parsed the result, so a finished build is not kept. buildAgents.jobTtlSeconds (300) is only a fallback for Jobs that were left behind, for example when a controller restarted mid-build. Read the output of a finished build in its stored build log in Artemis.
Build log of a running build	kubectl -n artemis-builds logs job/<name> -c builder
Controllers and runner version	Administration → Build System → Build Agents. The Build Runner row shows Kubernetes and its version.
Queue and capacity	The same page, and the gauges artemis.global.localci.queued, running, agents and maxConcurrentBuilds.
Permissions	The two kubectl auth can-i checks in On an existing cluster.

Failure patterns and their causes are listed under Troubleshooting in the build runners guide.

What is left to the cluster

The chart sets none of the following. Add them according to your requirements.

• Resource quotas and limit ranges for buildAgents.namespace. A quota there caps the concurrent builds independently of the number of controllers, and a build Pod that exceeds it fails to start.
• Network policies. Build Pods run student code. Restrict what they may reach beyond the isolation that the exercise's network setting provides, see Network access.
• Pod Security. The build namespace is labeled privileged through buildAgents.podSecurityLevel, because exercise images run as root and the network isolation container needs NET_ADMIN.
• High availability and backups of PostgreSQL, the broker and the registry. The bundled PostgreSQL is a single instance, and the registry and the broker are single pods. Use managed services where availability matters.
• Backups of the shared storage. It holds the LocalVC repositories, uploads, exports and build logs.
• TLS and ingress. The chart creates Gateway API resources, and a certificate can come from cert-manager through gateway.tls.certManagerClusterIssuer.
• Secret management. The values file holds the secrets in plain text unless you inject them another way.