Rotate Object Storage Credentials

Overview

All bundled S3 consumers in Ilum read their object-storage credentials from a single Kubernetes Secret named ilum-objectstorage-credentials. Rotation is performed by editing that Secret and restarting the consumer Pods. No Helm release upgrade is required.

The Secret exposes six aliased keys so each consumer can reference the credential pair under its native naming convention:

Key	Consumed by
access-key	ilum-core, ilum-jupyter, ilum-history-server, ilum-hive-metastore, MLflow externalS3
secret-key	same as access-key
root-user	Bitnami MinIO chart auth.existingSecret
root-password	same
RUSTFS_ACCESS_KEY	RustFS chart secret.existingSecret
RUSTFS_SECRET_KEY	same

All six keys must hold identical values for the access and secret components, or consumers will diverge on the active provider.

Prerequisites

• kubectl access to the namespace where the release is installed.
• The release was installed with helm_aio 6.7.2-RC2 or later. Earlier versions stored credentials in per-consumer literal values and require a Helm upgrade before rotation works.
• Read-write permission on Secret/ilum-objectstorage-credentials in the release namespace.

Procedure

Step 1. Patch the Secret

Both halves of the credential pair are written in a single kubectl patch so consumers do not see a half-rotated state.

NEW_ACCESS=admin
NEW_SECRET=$(openssl rand -hex 16)

kubectl -n ilum patch secret ilum-objectstorage-credentials --type=strategic -p "{
  \"stringData\": {
    \"access-key\": \"$NEW_ACCESS\",
    \"secret-key\": \"$NEW_SECRET\",
    \"root-user\": \"$NEW_ACCESS\",
    \"root-password\": \"$NEW_SECRET\",
    \"RUSTFS_ACCESS_KEY\": \"$NEW_ACCESS\",
    \"RUSTFS_SECRET_KEY\": \"$NEW_SECRET\"
  }
}"

Step 2. Verify the Secret holds the expected values

kubectl -n ilum get secret ilum-objectstorage-credentials \
  -o jsonpath='{.data.RUSTFS_SECRET_KEY}' | base64 -d

The output must match $NEW_SECRET.

Step 3. Restart the consumer workloads

A pod restart is required because credentials are mounted as environment variables and Kubernetes does not re-inject Secret references into running containers. Every consumer that reads its credentials through secretKeyRef (Ilum core, Hive Metastore, History Server, Jupyter, Trino, Airflow, Kestra, Langfuse, RustFS itself) must be restarted; restarting only RustFS is not sufficient.

kubectl -n ilum rollout restart \
  deploy/ilum-core \
  deploy/ilum-jupyter \
  deploy/ilum-rustfs \
  deploy/ilum-history-server \
  statefulset/ilum-hive-metastore

kubectl rollout restart works for Deployment workloads. StatefulSet workloads with updateStrategy.type: OnDelete (the default for ilum-hive-metastore) ignore rollout restart — explicitly delete each Pod instead so the controller schedules a replacement that picks up the rotated env vars:

kubectl -n ilum delete pod ilum-hive-metastore-0

When optional components are enabled, restart them as well:

kubectl -n ilum rollout restart \
  deploy/ilum-trino-coordinator \
  deploy/ilum-trino-worker \
  deploy/ilum-airflow-api-server \
  deploy/ilum-airflow-scheduler \
  deploy/ilum-airflow-dag-processor \
  deploy/ilum-kestra-standalone \
  deploy/ilum-langfuse-web \
  deploy/ilum-langfuse-worker

Step 4. Wait for the rollouts to complete

kubectl -n ilum rollout status deploy/ilum-core --timeout=10m
kubectl -n ilum rollout status deploy/ilum-rustfs --timeout=5m
kubectl -n ilum rollout status statefulset/ilum-hive-metastore --timeout=5m

Step 5. Verify authentication end-to-end

A simple bucket listing from ilum-jupyter confirms the rotated credentials are being used by the consumer Pods and accepted by the active provider.

kubectl -n ilum exec deploy/ilum-jupyter -- python3 -c "
import os, s3fs
fs = s3fs.S3FileSystem(
    key=os.environ['AWS_ACCESS_KEY_ID'],
    secret=os.environ['AWS_SECRET_ACCESS_KEY'],
    endpoint_url='http://ilum-objectstorage:9000',
)
print(fs.ls('/'))
"

The expected output is the list of default buckets. An AccessDenied response indicates the provider Pod has not yet picked up the new root credentials, or the active provider is MinIO and was bootstrapped with different credentials (refer to the limitations section below).

Limitations

MinIO does not support live root rotation

The MinIO server records its root user during the first start and treats the values in MINIO_ROOT_USER / MINIO_ROOT_PASSWORD as bootstrap-only. Restarting an existing MinIO Pod with new env values does not change the live root identity.

When MinIO is the active provider, the recommended path is to add a new MinIO user with mc admin user add and switch consumers to that user, or to reinstall MinIO on a fresh PVC. RustFS does not have this limitation: its root credentials are read from environment variables on every Pod start.

GitOps deployments

The objectStorage.credentials.preserveExisting flag uses Helm's lookup function to read the live Secret on helm upgrade. Pure helm template pipelines (without a connected cluster) cannot perform this lookup, so credentials must be supplied through values.yaml, sealed secrets, or external-secrets-operator. Set objectStorage.credentials.preserveExisting=false to disable the lookup and force a deterministic render.

References

For the full credential consolidation context refer to the Upgrade Notes.

For the migration procedure between any pair of supported providers refer to Migrate Between Providers.