Skip to main content

Migrate Between Object Storage Providers

Overview​

Ilum supports several S3-compatible object storage backends as the data plane: the bundled RustFS (default since 6.7.2-RC2), the bundled MinIO (opt-in), and any external S3-compatible service such as AWS S3, Wasabi, or Backblaze B2. Switching between them is performed entirely through Helm values. The procedure below is the canonical migration playbook. The most-traveled path, MinIO to RustFS, is documented in detail; the Other migration recipes section at the end covers the reverse path and migrations involving external providers.

A pre-upgrade hook protects existing installs from accidental data loss, so the safe path is to make a deliberate choice rather than to upgrade with no overrides.

See first

How migration works​

Migration in Ilum is parameterized: the same playbook covers any source-and-target pair. Three properties of the chart make this work:

  • Provider-agnostic alias. A stable Service named ilum-objectstorage is provisioned as a label-selector alias that routes to whichever provider is currently active. Bundled consumers target this alias rather than provider-specific Service names. Switching providers does not require consumer reconfiguration.
  • Shared credentials Secret. A single Secret, ilum-objectstorage-credentials, carries the S3 root credentials used by every consumer. The Secret exposes six aliased keys (access-key, secret-key, root-user, root-password, RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY) so each consumer can read the same value under its native env-var name. For rotation, refer to Rotate Object Storage Credentials.
  • Pre-upgrade safety hook. A pre-upgrade Helm hook detects an existing MinIO PersistentVolumeClaim and refuses to proceed when disabling MinIO would orphan the PVC without an acknowledged cutover. The hook is gated behind preUpgradeChecks.enabled (default true).

For the release-by-release record of when each property landed, refer to the Upgrade Notes. For the upstream chart status and known caveats of each provider, refer to the per-provider reference pages linked from the Object Storage Overview.

The cutover model​

The migration playbook is parameterized by three Helm values. Operators running 6.7.2-RC2 or later should prefer the new names; the legacy flag from earlier releases continues to work as a back-compat alias.

Helm valueTypePurpose
objectStorage.activeProviderstringExplicit override. Set to a provider name (rustfs, minio, ...) to pin the alias to that provider. Defaults to auto, which defers to the resolution rules.
objectStorage.previousProviderstringNames the data-bearing side during a cutover when two providers are enabled and activeProvider=auto. Defaults to minio.
objectStorage.cutoverAcknowledgedboolFlips the alias from previousProvider to the other enabled provider once the operator has finished migrating data. Defaults to false.
rustfs.migrationAcknowledgedboolLegacy name from 6.7.2-RC2 onward; honored as an alias for objectStorage.cutoverAcknowledged. Still accepted; deprecated in favour of the new name in a future major release.

Both objectStorage.cutoverAcknowledged=true and the legacy rustfs.migrationAcknowledged=true produce the same result. The examples below use the legacy flag where it matches the existing MinIO to RustFS migration path. Subsequent migrations between other providers should prefer the generalized objectStorage.cutoverAcknowledged name.

Choose a path​

Three operator paths
  • Path A — Keep MinIO. Set two Helm flags. No data movement. No reconfiguration of consumers.
  • Path B — Migrate to RustFS. Run both providers side by side, mirror data, then disable MinIO. Either automated or manual.
  • Path C — Net-new install. Nothing to do. RustFS is the default; the alias Service is created automatically.

Path A: Keep MinIO​

Run a single helm upgrade with two explicit flags. The pre-upgrade hook accepts the upgrade because MinIO remains enabled.

helm upgrade ilum ilum/helm_aio \
  --set minio.enabled=true \
  --set rustfs.enabled=false

The ilum-objectstorage Service alias is rendered with a selector matching MinIO pods, so consumer traffic continues to reach MinIO without any further reconfiguration.

Path B: Migrate to RustFS​

The migration runs both providers concurrently, mirrors bucket contents, and then disables MinIO. Either an automated migration Job or the manual mc mirror procedure can be used.

The bundled migration Job is gated on migration.minioToRustfs.enabled. It refuses to render unless both minio.enabled and rustfs.enabled are true.

Step 1. Run both providers and acknowledge the cutover.

helm upgrade ilum ilum/helm_aio \
  --set minio.enabled=true \
  --set rustfs.enabled=true \
  --set rustfs.migrationAcknowledged=true

Step 2. Dry-run the migration. The Job invokes mc mirror --fake and reports what would be copied without writing.

helm upgrade ilum ilum/helm_aio \
  --set minio.enabled=true \
  --set rustfs.enabled=true \
  --set rustfs.migrationAcknowledged=true \
  --set migration.minioToRustfs.enabled=true \
  --set migration.minioToRustfs.dryRun=true

Step 3. Inspect the Job logs.

kubectl -n ilum logs -l app.kubernetes.io/component=migration --tail=-1

Step 4. Run the migration for real.

helm upgrade ilum ilum/helm_aio \
  --set minio.enabled=true \
  --set rustfs.enabled=true \
  --set rustfs.migrationAcknowledged=true \
  --set migration.minioToRustfs.enabled=true

kubectl -n ilum wait job -l app.kubernetes.io/component=migration \
  --for=condition=complete --timeout=600s

Step 5. Verify (see the next section), then disable MinIO.

helm upgrade ilum ilum/helm_aio \
  --set rustfs.migrationAcknowledged=true

Verify the cutover​

CheckCommandExpected
Object Storage view loads in UInavigate to /object-storageiframe renders the RustFS console
Bucket paritymc diff src/<bucket> dst/<bucket>empty output for every default bucket
Spark writesubmit a job that writes to s3a://ilum-files/probe/object visible in the RustFS pod
Default buckets presentmc ls dst/seven default buckets
MinIO PVC retainedkubectl get pvc -n ilum | grep ilum-minioPVC still exists until manually deleted

Migration-window operational FAQ​

Can consumers continue writing while mc mirror is running?​

Yes. Before objectStorage.cutoverAcknowledged is set to true, the ilum-objectstorage Service alias still targets the source provider, so every consumer continues to write there. Objects written to the source after the first mc mirror pass started may be missed by that pass. The recommended pattern is therefore to run the mirror twice: a first pass during normal operation, then a brief read-only window (quiesce schedulers and Spark jobs, scale long-running consumers to zero) during which a second mc mirror run finalizes the copy.

What happens if the migration Job fails mid-run?​

The bundled migration Job is configured with backoffLimit: 0. A failure exits the Job and leaves whichever buckets had completed in their post-mirror state on the destination. Re-running the Job is safe: mc mirror --preserve is idempotent and resumes from the existing destination state. Delete the failed Job before the next helm upgrade if the Job's name would otherwise conflict:

kubectl -n ilum delete job -l app.kubernetes.io/component=migration

How do I verify the mirror was complete?​

mc diff src/<bucket> dst/<bucket> returns no output when source and destination are byte-identical. Run it for every bucket in objectStorage.defaultBuckets:

for bucket in ilum-files ilum-data ilum-tables ilum-mlflow ilum-kestra ilum-ducklake ilum-langfuse; do
  echo "=== $bucket ==="
  mc diff src/$bucket dst/$bucket
done

Empty output for every bucket means the cutover is safe. Any objects listed by mc diff should be reconciled (typically by running mc mirror again) before acknowledging the cutover.

Rollback​

Helm preserves PVCs across helm upgrade and helm rollback by default, so the original MinIO data is recoverable as long as its PVC has not been manually deleted. Re-enable MinIO and disable RustFS:

helm upgrade ilum ilum/helm_aio \
  --set minio.enabled=true \
  --set rustfs.enabled=false \
  --set rustfs.migrationAcknowledged=true

Consumer traffic returns to MinIO automatically because the ilum-objectstorage Service alias re-targets MinIO pods.

Other migration recipes​

The MinIO to RustFS path above is the canonical example. The same playbook applies to any source-and-target pair. The recipes below cover the three other paths that operators commonly take.

RustFS to MinIO (downgrade or rollback)​

The rollback procedure from the Rollback section above is the supported way to revert from RustFS back to MinIO, provided the original MinIO PersistentVolumeClaim still exists.

When the original MinIO PVC has been deleted, the procedure becomes symmetric to the MinIO to RustFS playbook:

  1. Enable both providers: --set minio.enabled=true --set rustfs.enabled=true --set objectStorage.previousProvider=rustfs.
  2. Run mc mirror from rustfs/<bucket> to minio/<bucket> for every default bucket.
  3. Set objectStorage.cutoverAcknowledged=true to flip the alias to MinIO.
  4. Once MinIO is verified, disable RustFS with --set rustfs.enabled=false.

Migrate to an external S3 provider (AWS S3, Wasabi, Backblaze B2)​

Migrating to a managed external backend has no in-cluster Helm install for the target. Only the alias target and the data copy change.

  1. Provision the external bucket(s) and an IAM credential pair with read and write access. Make sure the bucket names match objectStorage.defaultBuckets.

  2. Update ilum-objectstorage-credentials to carry the external credential pair, or create a new Secret and point objectStorage.credentials.existingSecret at it.

  3. Run mc mirror from the in-cluster source to the external destination using the external provider's S3 endpoint as the destination alias.

  4. Disable the in-cluster provider and point objectStorage.endpoint at the external service:

    helm upgrade ilum ilum/helm_aio \
      --set minio.enabled=false \
      --set rustfs.enabled=false \
      --set objectStorage.endpoint=https://s3.us-east-1.amazonaws.com

For provider-specific endpoint examples, refer to Provider Reference: External S3.

Migrate to a non-bundled provider added via the registry​

When the target is a provider that Ilum does not yet ship as a sub-chart (for example, Garage or SeaweedFS), the migration follows the same shape as the MinIO to RustFS path but the bundled migration Job does not apply: it is hard-coded for MinIO to RustFS. The procedure is:

  1. Provision the target. Stand up the new provider in the release namespace, with pods carrying labels app.kubernetes.io/name: <provider> and app.kubernetes.io/instance: <release>. Follow Add a New Provider for the registry entry and the per-provider Service shape.
  2. Mirror the buckets manually. Use the mc mirror recipe from the Manual mc mirror tab, substituting the destination alias for the new provider's in-cluster Service name.
  3. Flip the alias. Set objectStorage.activeProvider=<provider> to pin the alias to the new backend explicitly. Skip objectStorage.cutoverAcknowledged (it applies only to the auto-mode resolution between two enabled providers).
  4. Verify with mc diff for every default bucket, then disable the source provider once the verification is clean.

Migrate from external S3 back to a bundled provider​

To move from a managed external backend to a bundled provider (for example, repatriating storage to keep everything in-cluster), reverse the steps above:

  1. Enable the target bundled provider (--set rustfs.enabled=true or --set minio.enabled=true). The ilum-objectstorage Service alias renders automatically.
  2. Run mc mirror from the external source to the bundled destination (http://ilum-rustfs-svc:9000 or http://ilum-minio:9000).
  3. Update objectStorage.endpoint to point at the bundled provider's in-cluster Service, or remove the override so the chart resolves the alias automatically.

If the cutover goes wrong​

If the Ilum UI returns 502 Bad Gateway after a cutover, the ilum-objectstorage Service alias most likely lost its endpoints. The fastest recovery is a helm rollback to the previous release revision. For the full diagnosis and recovery procedure, refer to Troubleshoot Object Storage: 502 from the Object Storage view.

Known limitations​

  • RustFS distributed mode is "under testing" upstream. The bundled defaults configure standalone mode with a single PVC.
  • Auto-wiring of Hydra OIDC into RustFS is not yet shipped. Operators that depend on OIDC against the object-storage console should override rustfs.extraEnv directly until the integration stabilizes.
  • MinIO root credentials are recorded by the MinIO server on first install. Rotating the shared Secret after MinIO has been bootstrapped does not change the live MinIO root user. RustFS reads its root credentials from the env on every Pod start, so RustFS supports live rotation.

References​

For the upgrade summary and the Helm chart changelog refer to the Upgrade Notes, which mirrors the in-tree helm/CHANGELOG.md.

For the upstream RustFS chart refer to the Artifact Hub listing.

For the mc client reference refer to the MinIO Client documentation.

Last updated on