Upgrade Notes

Upgrade notes

NOTE TEMPLATE

1. Change

Feature:

Feature description

Values deleted - chart name

Name	Reason
helm.value	Helm value deletion reason

Values added - chart name

Values section description

Name	Description	Value
helm.value	Helm value description	default value

Names changed - chart name

Old Name	New Name
old name	new name

⚠️⚠️⚠️ Warnings

Values changed - chart name

Name	Old value	New Value
helm.value	old value	new value

NEXT RELEASE

RELEASE 6.6.1

1. Ugraded JupyterHub experience

Feature:

Ugraded helm_jupyterhub to bundle Ilum-specific SSH/Git/LDAP bootstrap logic, curated notebooks, and tailored singleuser defaults so helm_aio merely enables the dependency, pins fullnameOverride, and surfaces only the still-relevant overrides. This also keeps the SSH network policy open for port 2222 and ensures the shared ilum-jupyter-ssh-keys secret remains stable, while c.JupyterHub.cleanup_servers = True guarantees the SSH service and user pods stop with the release.

Values added - helm_aio

Name	Description	Value
ilum-jupyterhub.enabled	Enables the curated Ilum JupyterHub chart	false

Values added - helm_jupyterhub

Name	Description	Value
fullnameOverride	Override for the full resource name	ilum-jupyterhub
enabled	Chart enabled flag	false

SSH configuration

Name	Description	Value
ssh.enabled	Enables the bundled SSH operator, service, and shared key workflow	false
ssh.keysSecret	Secret that provides the stable host and authorized keys	ilum-jupyter-ssh-keys
ssh.mode	SSH authentication mode: master (shared authorized_keys from keysSecret) or per-user (individual secrets per user)	master
ssh.perUserSecretNameTemplate	Template for per-user secret names when using per-user mode	ssh-keys-{username}
ssh.perUserAuthorizedKeysKey	Key name in per-user secrets containing authorized_keys	authorized_keys
ssh.service.type	Type of service fronting port 2222	NodePort
ssh.service.port	Port exposed for SSH traffic	2222
ssh.service.targetPort	Target port for SSH traffic	2222
ssh.service.nodePort	NodePort number (empty for auto-assignment)	""
ssh.service.clusterIP	ClusterIP address (empty for auto-assignment)	""
ssh.service.loadBalancerIP	LoadBalancer IP address	""
ssh.service.annotations	Annotations for the SSH service	{}
ssh.service.prefix	Prefix for SSH service resources	ilum-jupyter-ssh
ssh.sshdConfig.customConfig	Custom sshd_config lines	[]
ssh.operatorImage.name	SSH operator image repository	docker.ilum.cloud/ilum-jupyterhub
ssh.operatorImage.tag	SSH operator image tag	ssh-operator-4.3.1
ssh.extraEnv	Extra environment variables for SSH operator	[]

Git configuration

Name	Description	Value
git.existingSecret	Credentials that allow the Git init job to seed the notebooks repository	ilum-git-credentials
git.email	Git email for commits	ilum@ilum
git.repository	Git repository name	jupyter
git.address	Gitea server address	ilum-gitea-http:3000
git.url	Gitea endpoint URL used to seed the ilum-jupyterhub org	http://ilum-gitea-http:3000
git.orgName	Organization managed by the git-init job	ilum-jupyterhub
git.operatorImage.name	Git operator image repository	docker.ilum.cloud/ilum-jupyterhub
git.operatorImage.tag	Git operator image tag	gitea-operator-4.3.1
git.secret.name	Secret containing credentials referenced by the operator	ilum-git-credentials
git.secret.usernameKey	Key for username in the secret	username
git.secret.passwordKey	Key for password in the secret	password

LDAP configuration

Name	Description	Value
ldap.enabled	Keeps the LDAP authenticator wired into Ilum JupyterHub	true
ldap.urls	LDAP server endpoints that front the Ilum directory	["ldap://ilum-openldap:389"]
ldap.base	Search base for Ilum users and groups	"dc=ilum,dc=cloud"
ldap.username	Bind DN used for authentication	"cn=admin,dc=ilum,dc=cloud"
ldap.password	Password for the bind DN	Not@SecurePassw0rd
ldap.adminUsers	LDAP accounts with admin privileges in JupyterHub	["ilumadmin","admin"]
ldap.userSearchBase	Base DN where user entries live	"ou=people,dc=ilum,dc=cloud"
ldap.userSearchFilter	Filter for user lookups	"uid={0}"
ldap.groupSearchBase	Base DN where group entries live	"ou=groups,dc=ilum,dc=cloud"
ldap.groupSearchFilter	Filter that matches members	"(member={0})"
ldap.allowedGroups	Empty list allows all groups unless specified	[]
ldap.userAttribute	User attribute for username	"uid"
ldap.fullnameAttribute	Attribute for user's full name	"cn"
ldap.emailAttribute	Attribute for user's email	"mail"
ldap.groupNameAttribute	Attribute for group name	"cn"
ldap.groupMemberAttribute	Attribute for group membership	"member"
ldap.useSsl	Use SSL for LDAP connection	false
ldap.startTls	Use STARTTLS for LDAP connection	false
ldap.lookupDn	Lookup DN before binding	true

Hub configuration

Name	Description	Value
hub.image.name	Hub image repository	docker.ilum.cloud/ilum-jupyterhub
hub.image.tag	Hub image tag	jupyterhub-4.3.1
hub.contentSecurityPolicy.enabled	Turns the managed CSP header injection on/off	true
hub.contentSecurityPolicy.frameAncestors	Origins allowed to embed JupyterHub in an iframe	["'self'","http://localhost:9777"]
hub.gitInit.enabled	Runs the job that ensures the ilum-jupyterhub organization/repo exist	true

Singleuser runtime defaults

Name	Description	Value
singleuser.startupArgs.iopubDataRateLimit	Raised output bandwidth ceiling for Ilum workloads	1000000000
singleuser.startupArgs.extraArgs	Additional CLI arguments forwarded to the user server	[]
singleuser.nodeSelector	Architecture-agnostic placement (empty by default)	{}
singleuser.tolerations	Allows scheduling on tainted nodes when needed	[]

Image pull credentials

Name	Description	Value
imagePullSecret.create	Create the pull secret in-cluster	false
imagePullSecret.automaticReferenceInjection	Auto-inject the created secret into JupyterHub workloads	true
imagePullSecret.registry	Registry host for the pull secret	""
imagePullSecret.username	Registry username for the pull secret	""
imagePullSecret.password	Registry password for the pull secret	""
imagePullSecret.email	Registry email for the pull secret	""
imagePullSecret.name	Existing secret name to reference instead of the autogenerated pull secret	""
imagePullSecrets	Additional pull secrets injected into all hub-managed pods	[]

Values changed - helm_jupyterhub

Name	Old value	New Value
singleuser.networkPolicy.allowedIngressPorts	[]	[2222]

Instructions

• Keep ilum-jupyter-ssh-keys stable outside Helm so the SSH host fingerprint survives upgrades; rotating the secret requires removing stale entries from user known_hosts.
• Ensure ilum-git-credentials contains valid credentials for a Gitea account with org-level write access—both the SSH operator and Git init job rely on the rendered token.
• To refresh the curated notebooks, update helm_jupyterhub/files/examples (and their config map templates) so the init container can push them into the ilum-jupyterhub repo again.

⚠️⚠️⚠️ Warnings

• Port 2222 is opened via the SSH operator’s shared service; if you switch to per-user authorized keys, keep the service numbering and secrets aligned.

• Cleanup is forced (c.JupyterHub.cleanup_servers = True), so user pods and the SSH service terminate with the Helm release. Manage any long-lived workloads outside this chart.

2. Upgraded Livy compatible API to version 0.8.0

Feature:

Upgraded Livy compatible API to version 0.8.0 with enhanced configuration options for compression, server version control, and TTL-based session cleanup.

Values added - ilum-core

Livy Compression Configuration

Name	Description	Value
livy.compression.enabled	Enable response compression for Livy endpoints	false

Livy Server Configuration

Name	Description	Value
livy.server.version	Livy server version identifier	0.8.0
livy.server.sendServerVersion	Send server version in response headers	false
livy.server.allowCustomClasspath	Allow custom classpath in session creation	false

Livy TTL Session Cleanup Configuration

Name	Description	Value
livy.ttl.checkPeriod	Background sweep period in milliseconds for checking expired sessions	300000
livy.ttl.checkInitialDelay	Initial delay in milliseconds before first TTL background check	60000

Values added - ilum-aio

Same values as ilum-core but under the ilum-core. prefix (e.g., ilum-core.livy.compression.enabled).

⚠️⚠️⚠️ Important Notes

For most users: ✅ No action required. All new configuration options have safe defaults and are backward compatible.

Optional Performance Optimization: If you handle large Livy responses, you may enable compression by setting livy.compression.enabled: true.

Session Management: The new TTL cleanup uses a hybrid approach (lazy + background sweep) to automatically clean up expired sessions. Default settings should work for most deployments.

3. Updated default Spark version and added autopause configuration

Feature:

Updated default Spark version to 3.5.7-delta in kubernetes.defaultCluster.config. Added spark.ilum.autopause: "true" to kubernetes.defaultCluster.config to set the default behavior of the autopause feature.

Values changed - ilum-core

Name	Old value	New Value
kubernetes.defaultCluster.config.spark.kubernetes.container.image	ilum/spark:3.5.6-delta	ilum/spark:3.5.7-delta

Values added - ilum-core

Name	Description	Value
kubernetes.defaultCluster.config.spark.ilum.autopause	Sets the default behavior of autopause feature	"true"

1. HTTP cookie-based access control now disabled by default

Feature:

Fixed stability issues and changed HTTP cookie-based access control to be disabled by default for all external services (Jupyter, Airflow, MLflow, Grafana, etc.).

What Changed?

• External services are now open by default - no cookie requirements
• All users can access services without any special configuration
• System works out-of-the-box

What This Means for You

• ✅ Your services will become more accessible
• ✅ Users can access Jupyter, Airflow, MLflow, etc. without cookie setup
• ✅ No action required for most deployments

If you need to restrict access:

• Use the built-in OAuth2/Hydra authentication (recommended for production)
• Or manually enable cookie-based access control per service (see below)

When to Use Cookie-Based Access Control?

This is an advanced feature for specific use cases:

• ✅ Temporary access restrictions for specific users/sessions
• ✅ Custom access control integrated with your frontend application

To enable for a specific service:

nginx:
  config:
    http_cookie:
      enabled: true
      ilum-jupyter:
        enabled: true

Values changed - ilum-ui

Name	Old value	New Value
nginx.config.http_cookie.enabled	true	false

Values changed - ilum-aio

Name	Old value	New Value
ilum-ui.nginx.config.http_cookie.enabled	true	false

⚠️⚠️⚠️ Important Notes

For most users: ✅ No action required. This change makes services more accessible.

If you customized cookie settings in 6.6.0: You may need to review your configuration. The system now defaults to open access instead of requiring cookies.

RELEASE 6.6.0

1. Upgraded Apache Airflow to 3.1.1

Feature:

Upgraded Apache Airflow from 3.0.5 to 3.1.1 with improved OIDC authentication support using authlib OAuth providers (AUTH_OAUTH) instead of deprecated flask-oidc (AUTH_OIDC). This upgrade includes fixes for OAuth redirect URI patterns and proper volume mounting for OIDC client secrets in init containers.

Values changed - ilum-aio

Name	Old value	New Value
airflow.airflowVersion	3.0.5	3.1.1
airflow.images.airflow.tag	3.0.5	3.1.1
airflow.apiServer.extraInitContainers[0].image	ilum/airflow:3.0.5	ilum/airflow:3.1.1

Values added - ilum-aio

Name	Description	Value
airflow.migrateDatabaseJob.useHelmHooks	Disable Helm hooks for database migration job	false
airflow.apiServer.extraInitContainers[0]	Modified create admin user init container	See below
airflow.apiServer.apiServerConfigConfigMapName	Custom webserver_config.py configmap	ilum-api-server-config

Init Container Volume Mount Configuration

- name: create-admin-user
  image: ilum/airflow:3.1.1
  command: ["/bin/bash", "/scripts/init.sh"]
  volumeMounts:
    - name: ilum-airflow-create-user-secret
      mountPath: /scripts
    - name: config
      mountPath: /opt/airflow/airflow.cfg
      subPath: airflow.cfg
    - name: oauth-secret-volume
      mountPath: /opt/airflow/client-secret
      readOnly: true
    - name: webserver-config-volume
      mountPath: /opt/airflow/webserver_config.py
      subPath: webserver_config.py
      readOnly: true

⚠️⚠️⚠️ Warnings

Configuration value airflow.apiServer.apiServerConfigConfigMapName is preconfigured to use a ConfigMap named ilum-api-server-config. But the name of this configMap must follow pattern <release-name>-api-server-config to be properly mounted as it is Airflow's chart requirement. So if your release name is different from ilum, please change this value accordingly. For example use:

airflow:
  apiServer:
    apiServerConfigConfigMapName: <your-release-name>-api-server-config
    extraVolumes:
      - name: oauth-secret-volume
        secret:
          secretName: ilum-hydra-client-secret
      - name: ilum-airflow-create-user-secret
        secret:
          secretName: ilum-airflow-create-user-secret
      - name: webserver-config-volume
        configMap:
          name: <your-release-name>-api-server-config

2. Enhanced Jupyter startup configuration

Feature:

Added configurable startup arguments for Jupyter notebook server, allowing users to customize base URL, IOPub data rate limits, and pass additional command-line arguments. Also added support for extra environment variables with templating support.

Values added - ilum-jupyter

Startup and environment configuration

Name	Description	Value
token	Jupyter notebook authentication token	""
startupArgs.baseUrl	Jupyter base URL path for reverse proxy configurations	/external/jupyter/
startupArgs.iopubDataRateLimit	IOPub data rate limit in bytes/sec (controls output bandwidth)	1000000000
startupArgs.extraArgs	Additional command-line arguments to pass to Jupyter server	[]
extraEnv	Additional environment variables for Jupyter container as string template	""

⚠️⚠️⚠️ Warnings

• All startup arguments configured via startupArgs.* can be completely overridden by setting the args parameter in values.yaml. When args is set, all default startup arguments are ignored, giving you full control over the Jupyter server startup command.
• The extraEnv parameter accepts a string template (multiline YAML using |). Example usage:

  extraEnv: |
    - name: MY_VAR
      value: value

3. Added Apache NiFi module

Feature:

Added Apache NiFi to ilum-aio as a new module. This will allow users to easily deploy NiFi and use it next to Ilum.

Values added - ilum-aio

Name	Description	Value
nifi.enabled	Flag to enable NiFi deployment in ilum-aio	false
nifi.fullnameOverride	Full name override for NiFi	ilum-nifi
nifi.image.tag	Tag of the source NiFi image	2.5.0
nifi.properties.safetyValve	Additional properties passed to nifi.properties	See values.yaml
nifi.persistence.enabled	Enables PVC for the data directory	true
nifi.persistence.subpath.enabled	Enabled one PVC instead of many	true
nifi.persistence.subpath.size	Size of the data directory	10Gi
nifi.zookeeper.enabled	Enables bundled Zookeeper deployment	false
nifi.registry.enabled	Enables bundled NiFi registry deployment	false
nifi.ca.enabled	Enables bundled CA deployment	false
nifi.openldap.enabled	Enables bundled openLDAP deployment	false

Values added - ilum-ui

Name	Description	Value
runtimeVars.nifiUrl	URL of the NiFi instance	http://ilum-nifi:8443/nifi/
runtimeVars.nifiPath	Proxy path of NiFi	/external/nifi/
nginx.config.nifi.enabled	Enable proxy for NiFi	false
nginx.config.http_cookie.nifi.enabled	Enables cookie mapping for NiFi	true

4. Added integration with Project Nessie metastore

Feature:

Added integration with Project Nessie metastore in Ilum, which allows the use of Nessie as a metastore for Spark jobs.

Values added - ilum-core

Name	Description	Value
metastore.type	Indicates the default metastore	hive
metastore.nessie.address	The address of the default Nessie metastore	http://ilum-nessie:19120/api/v2
metastore.nessie.warehouseDir	The location of the warehouse of the default Nessie metastore	s3a://ilum-data/nessie_catalog
metastore.nessie.s3Endpoint	The S3 API endpoint to use for the default Nessie metastore	http://ilum-minio:9000
metastore.nessie.s3PathStyleAccess	Whether to use path style access for the S3 Nessie connection	true
metastore.nessie.authType	Auth type of the default Nessie metastore	NONE
metastore.nessie.ref	The branch to use for the default Nessie metastore	main
metastore.nessie.cacheEnabled	Enables caching in the default Nessie metastore	false
metastore.nessie.catalog_name	The name of the catalog for the default Nessie metastore	nessie_catalog
metastore.nessie.config	Additional config to add for the Spark job	See values.yaml
metastore.nessie.statusProbe	Status probe for the Nessie metastore, so Ilum-core does not launch too quickly	See values.yaml

Values added - ilum-aio

Name	Description	Value
nessie.enabled	Enables or disables bundled Nessie deployment	false
nessie.fullnameOverride	Full name override for Nessie	ilum-nessie
nessie.versionStoreType	Type of persistent metadata storage	JDBC2
nessie.extraInitContainers	Adds init containers to Nessie (waiting for database)	See values.yaml
nessie.jdbc.jdbcUrl	Url for DB connection	jdbc:postgresql://ilum-postgresql-hl:5432/nessie
nessie.jdbc.secret.name	Secret containing DB credentials	ilum-postgres-credentals
nessie.jdbc.secret.username	Key of username in the secret	username
nessie.jdbc.secret.password	Key of password in the secret	password

Names changed - ilum-core

Old Name	New Name
hiveMetastore.enabled	metastore.enabled
hiveMetastore.*	metastore.hive.*

Values changed - ilum-core

Name	Old value	New Value
kubernetes.defaultCluster.config	See values.yaml	See values.yaml

⚠️⚠️⚠️ Warnings

This is an important change that will need to be addressed if any custom changes to the default configuration were made. Please carefully review the changes and make sure they will not break your deployment.

5. Livy API now fully served by ilum-core (embedded)

Feature:

Livy API is implemented and served directly by ilum-core (embedded). The legacy Livy proxy is deprecated but can still be turned on for backward compatibility.

Values added - ilum-aio

Name	Description	Value
ilum-core.livy.enabled	Enables embedded Livy integration in AIO via ilum-core	true
ilum-core.livy.ilumUI.publicEndpoint	Public endpoint of ilum-ui used for Livy links/integration	http://localhost:9777
ilum-livy-proxy.legacy.enabled	Turns on the legacy Livy proxy resources (ConfigMap/Deployment/etc.)	false

⚠️⚠️⚠️ Warnings

• The compat Service (ilum-livy-proxy → ilum-core) is deprecated and will be removed in a future release.
• The legacy proxy is also deprecated and will be removed after the transition period.
• Mode matrix (AIO):
  • ilum-livy-proxy.enabled=false & ilum-livy-proxy.legacy.enabled=false → nothing created; call ilum-core directly.
  • ilum-livy-proxy.enabled=true & ilum-livy-proxy.legacy.enabled=false → create compat Service pointing to the new ilum-core Livy API.
  • ilum-livy-proxy.enabled=false & ilum-livy-proxy.legacy.enabled=true → deploy legacy proxy.
• When legacy is enabled, update client endpoints to use the legacy Service:
  • ilum-jupyter.livyEndpoint = http://ilum-livy-proxy:8998
  • ilum-zeppelin.livyEndpoint = http://ilum-livy-proxy:8998
  • Airflow connection: AIRFLOW_CONN_ILUM-LIVY-PROXY=livy://ilum-livy-proxy:8998
• Some resources (e.g., ConfigMap/Ingress) are rendered based on ilum-livy-proxy.legacy.enabled only (not on ilum-livy-proxy.enabled).

6. Added cronjob cleaning after uninstalling

Feature:

A pre-delete hook will now clean up kubernetes cronjobs after uninstalling the chart.

Values added - ilum-core

Name	Description	Value
cronjob.cleanup.enabled	Enable cronjob cleanup after uninstalling the chart	true
cronjob.cleanup.image	Image used for the cleanup job	alpine/kubectl:1.34.1

RELEASE 6.5.2

1. SSH mode implementation for helm_jupyter

Feature:

SSH mode in the helm_jupyter chart has been implemented to provide SSH access directly within the main Jupyter container. This allows users to access their Jupyter environment via SSH while maintaining workspace consistency between web and SSH interfaces.

Values added - helm_jupyter

SSH access configuration

Name	Description	Value
ssh.enabled	Enable SSH access in the Jupyter container	true
ssh.keysSecret	Name of the secret containing SSH keys	ilum-jupyter-ssh-keys
ssh.service.type	SSH service type	NodePort
ssh.service.port	SSH service port	2222
ssh.service.nodePort	SSH service node port (when service type is NodePort)	""
ssh.service.clusterIP	SSH service cluster IP	""
ssh.service.loadBalancerIP	SSH service load balancer IP	""
ssh.service.annotations	SSH service annotations	{}
ssh.sshdConfig.customConfig	Custom SSH server configuration	[]

⚠️⚠️⚠️ Warnings

• SSH server runs directly in the main Jupyter container on port 2222 internally
• SSH keys must be provided via a Kubernetes Secret referenced by ssh.keysSecret parameter
• The secret should contain both SSH host keys and authorized_keys for authentication
• SSH access provides direct access to the /home/jovyan/work directory (same as web interface)
• Custom SSH server configuration can be provided via ssh.sshdConfig.customConfig array

2. Address Bitnami’s move to bitnamilegacy + bitnamisecure

Feature:

Bitnami has moved to bitnamilegacy + bitnamisecure for their images after the 18th of August 2025. This change moves used images to the new repositories.

Values added - ilum-aio

Name	Description	Value
global.security.allowInsecureImages	Allows images from outside of bitnami repository in Bitnami's charts	true
kafka.image.repository	Repository for Kafka's image	bitnamilegacy/kafka
minio.image.repository	Repository for Minio's image	bitnamilegacy/minio
mlflow.image.repository	Repository for MlFlow's image	bitnamilegacy/mlflow
postgresql.image.repository	Repository for Postgresql's image	bitnamilegacy/postgresql

Values changed - ilum-aio

Name	Old value	New Value
airflowExtensions.git.image	bitnami/git:2.48.1	bitnamisecure/git@sha256:72ae5bd9715fc81446becc0418011883479c593bac427911aa62ecf27ef96546
postgresExtensions.image	bitnami/postgresql:16	bitnamilegacy/postgresql:16

Values changed - ilum-core

Name	Old value	New Value
kafka.statusProbe.image	bitnami/kafka:3.4.1	bitnamilegacy/kafka:3.4.1

Values changed - ilum-hive-metastore

Name	Old value	New Value
postgresql.image	bitnami/postgresql:16	bitnamilegacy/postgresql:16

Values changed - ilum-jupyter

Name	Old value	New Value
git.init.image	bitnami/git:2.48.1	bitnamisecure/git@sha256:72ae5bd9715fc81446becc0418011883479c593bac427911aa62ecf27ef96546

Values changed - ilum-marquez

Name	Old value	New Value
marquez.db.image	bitnami/postgresql:16	bitnamilegacy/postgresql:16

1. Updated Airflow defaults in ilum-aio

Feature:

Bumped Airflow to 3.0.5 and streamlined default connection/env configuration. Removed legacy cleanup and scheduler overrides in favor of chart defaults and connection-based setup.

Values added - ilum-aio

Name	Description	Value
airflow.enableBuiltInSecretEnvVars.AIRFLOW__CORE__FERNET_KEY	Enable default fernet key generation	false

Values changed - ilum-aio

Name	Old value	New Value
airflow.extraEnv	See values.yaml	See values.yaml
airflow.airflowVersion	3.0.3	3.0.5
airflow.images.airflow.tag	3.0.3	3.0.5
airflow.apiServer.extraInitContainers[0].image	ilum/airflow:3.0.3	ilum/airflow:3.0.5

Values deleted - ilum-aio

Name	Reason
airflow.scheduler.args	Revert to chart default scheduler command as we manage connections via env variables now
airflow.cleanup.enabled	Use executor's instant cleanup
airflow.config.kubernetes_executor.delete_worker_pods	Use chart defaults
airflow.config.kubernetes_executor.delete_worker_pods_on_failure	Use chart defaults

⚠️⚠️⚠️ Warnings

As the default Airflow’s fernet key creation mechanism made it impossible to enable Airflow via values upgrade, the mechanism will get disabled by default. To use it once again, manually create a Kubernetes secret and set required values in the airflow chart.

3. Added configurable HTTP cookie mappings in ilum-ui

Feature:

Added configurable HTTP cookie mappings in the ilum-ui nginx configuration. This allows users to enable or disable cookie-based access control for individual services or turn off the entire cookie mapping section. Each service can be controlled individually while maintaining backward compatibility with all options enabled by default.

Values added - ilum-ui

HTTP cookie mapping configuration

Name	Description	Value
nginx.config.http_cookie.enabled	Global flag to enable HTTP cookie mappings	true
nginx.config.http_cookie.historyServer.enabled	Enable cookie mapping for history server access	true
nginx.config.http_cookie.mlflow.enabled	Enable cookie mapping for MLflow access	true
nginx.config.http_cookie.ilum-jupyter.enabled	Enable cookie mapping for Jupyter notebook access	true
nginx.config.http_cookie.gitea.enabled	Enable cookie mapping for Gitea access	true
nginx.config.http_cookie.n8n.enabled	Enable cookie mapping for n8n access	true
nginx.config.http_cookie.minio.enabled	Enable cookie mapping for MinIO access	true
nginx.config.http_cookie.airflow.enabled	Enable cookie mapping for Airflow access	true
nginx.config.http_cookie.superset.enabled	Enable cookie mapping for Superset access	true
nginx.config.http_cookie.grafana.enabled	Enable cookie mapping for Grafana access	true
nginx.config.http_cookie.kestra.enabled	Enable cookie mapping for Kestra access	true
nginx.config.http_cookie.mageai.enabled	Enable cookie mapping for Mage AI access	true

4. PostgreSQL Max Connections Configuration

Feature:

Added PostgreSQL max_connections configuration to address database connection limits in high-load scenarios.

Values added - ilum-aio

PostgreSQL configuration

Name	Description	Value
postgresql.primary.extendedConfiguration	Extended PostgreSQL configuration to set max_connections parameter	max_connections = 1000

RELEASE 6.4.3

1. Added Mage to ilum-aio

Feature:

Added Mage OSS to ilum-aio as a new module. This will allow users to easily deploy Mage and use it next to Ilum.

Values added - ilum-aio

Name	Description	Value
mageai.enabled	Flag to enable Mage OSS deployment in ilum-aio	false
mageai.fullnameOverride	Overrides the full name of the Mage deployment	ilum-mageai
mageai.image.repository	Repository of the source Mage image	ilum/mageai
mageai.image.tag	Tag of the source Mage image	0.9.76
mageai.rootPath	The root path for the Mage web server	external/mageai
mageai.service.type	Type of Mage's kubernetes service	ClusterIP
mageai.redis.enabled	Enables Redis for the Mage deployment	false
mageai.postgresql.enabled	Enables external Postgres for Mage	true
mageai.postgresql.deploy	Deploys external Postgres for Mage	false
mageai.postgresql.fullnameOverride	The name of the Postgres service	ilum-postgresql-hl
mageai.postgresql.auth.username	Username of the Postgres user	ilum
mageai.postgresql.auth.password	Password of the Postgres user	CHANGEMEPLEASE
mageai.postgresql.auth.database	The database Mage should use	mageai
mageai.persistence.enabled	Enables PVC for the main data directory	true

2. Updated Airflow’s chart and values to support Airflow 3.0

Feature:

Updated Airflow’s chart and values to support Airflow 3.0, which includes changes in the configuration and deployment of Airflow.

Values added - ilum-aio

Name	Description	Value
airflow.airflowVersion	Sets chart compatibility for given Airflow version	3.0.3
airflow.apiServer.*	apiServer replaces webserver settings	See values.yaml
airflow.config.core.auth_manager	Class of auth manager used in Airflow	airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
airflow.config.api.base_url	The base path of the Airflow web app	http://localhost:9777/external/airflow
airflow.config.api.enable_xcom_deserialize_support	Enables XCom deserialization in Airflow API	True
airflow.config.logging.colored_console_log	Enables colored console log in Airflow	True
airflow.config.kubernetes_executor.delete_worker_pods	Enables instant deletion of worker pods	False
airflow.config.kubernetes_executor.delete_worker_pods_on_failure	Enables instant deletion of failed worker pods	False
airflow.cleanup.enabled	Enables periodic deletion of worker pods	True

Values changed - ilum-aio

Name	Old value	New Value
airflow.images.airflow.tag	2.9.3	3.0.3
airflow.executor	LocalKubernetesExecutor	KubernetesExecutor
airflow.extraEnv	See values.yaml	See values.yaml
airflow.webserver.extraInitContainers	See values.yaml	See values.yaml

Values deleted - ilum-aio

Name	Reason
airflow.migrateDatabaseJob.useHelmHooks	Revert to chart default

Values changed - ilum-ui

Name	Old value	New Value
runtimeVars.airflowUrl	http://ilum-airflow-webserver:8080	http://ilum-airflow-api-server:8080

⚠️⚠️⚠️ Warnings

Ilum’s changes, Airflow 3.0 and the new Airflow chart version bring significant changes to the Airflow configuration and deployment. Please review the new values and adjust your configuration accordingly.

If you are a user of Ilum’s OAuth2 provider, this update may require you to manually update some configuration, as Helm is likely to not be able to automatically migrate the values.

3. Added securityContext configuration for ilum charts

Feature:

Added comprehensive securityContext configuration for enhanced security across all ilum Helm charts. This includes both pod-level and container-level security contexts with non-root user execution, capability dropping, and seccomp profiles.

Values added - ilum-core

Name	Description	Value
securityContext.pod.runAsNonRoot	Run container as non-root user	true
securityContext.pod.runAsUser	User ID to run the container	1001
securityContext.pod.runAsGroup	Group ID to run the container	1001
securityContext.pod.fsGroup	File system group ID	1001
securityContext.pod.seccompProfile.type	Seccomp profile type	Unconfined
securityContext.container.allowPrivilegeEscalation	Allow privilege escalation	false
securityContext.container.readOnlyRootFilesystem	Read-only root filesystem	false
securityContext.container.runAsNonRoot	Run container as non-root user	true
securityContext.container.runAsUser	User ID to run the container	1001
securityContext.container.runAsGroup	Group ID to run the container	1001
securityContext.container.capabilities.drop	Capabilities to drop	["ALL"]
securityContext.container.seccompProfile.type	Seccomp profile type	Unconfined

Values added - ilum-ui

Name	Description	Value
securityContext.pod.runAsNonRoot	Run pod as non-root user	true
securityContext.pod.runAsUser	User ID to run the pod	101
securityContext.pod.runAsGroup	Group ID to run the pod	101
securityContext.pod.fsGroup	File system group ID	101
securityContext.pod.seccompProfile.type	Seccomp profile type	Unconfined
securityContext.container.allowPrivilegeEscalation	Allow privilege escalation	false
securityContext.container.readOnlyRootFilesystem	Read-only root filesystem	false
securityContext.container.runAsNonRoot	Run container as non-root user	true
securityContext.container.runAsUser	User ID to run the container	101
securityContext.container.runAsGroup	Group ID to run the container	101
securityContext.container.capabilities.drop	Capabilities to drop	["ALL"]
securityContext.container.seccompProfile.type	Seccomp profile type	Unconfined

Values added - ilum-jupyter

Name	Description	Value
securityContext.container.allowPrivilegeEscalation	Allow privilege escalation	false
securityContext.container.readOnlyRootFilesystem	Read-only root filesystem	false
securityContext.container.runAsNonRoot	Run container as non-root user	true
securityContext.container.runAsUser	User ID to run the container	1000
securityContext.container.runAsGroup	Group ID to run the container	100
securityContext.container.capabilities.drop	Capabilities to drop	["ALL"]
securityContext.container.seccompProfile.type	Seccomp profile type	Unconfined
securityContext.initContainer.allowPrivilegeEscalation	Allow privilege escalation for init container	false
securityContext.initContainer.readOnlyRootFilesystem	Read-only root filesystem for init container	false
securityContext.initContainer.runAsNonRoot	Run init container as non-root user	false
securityContext.initContainer.runAsUser	User ID to run the init container	0
securityContext.initContainer.runAsGroup	Group ID to run the init container	0
securityContext.initContainer.capabilities.drop	Capabilities to drop for init container	["ALL"]
securityContext.initContainer.seccompProfile.type	Seccomp profile type for init container	Unconfined

Values added - ilum-livy-proxy

Name	Description	Value
securityContext.pod.runAsNonRoot	Run pod as non-root user	true
securityContext.pod.runAsUser	User ID to run the pod	1001
securityContext.pod.runAsGroup	Group ID to run the pod	1001
securityContext.pod.fsGroup	File system group ID	1001
securityContext.pod.seccompProfile.type	Seccomp profile type	Unconfined
securityContext.container.allowPrivilegeEscalation	Allow privilege escalation	false
securityContext.container.readOnlyRootFilesystem	Read-only root filesystem	false
securityContext.container.runAsNonRoot	Run container as non-root user	true
securityContext.container.runAsUser	User ID to run the container	1001
securityContext.container.runAsGroup	Group ID to run the container	1001
securityContext.container.seccompProfile.type	Seccomp profile type	Unconfined

Values added - ilum-aio

Added securityContext configuration for postgresExtensions

Name	Description	Value
postgresExtensions.securityContext.pod.runAsNonRoot	Run pod as non-root user	true
postgresExtensions.securityContext.pod.runAsUser	User ID to run the pod	999
postgresExtensions.securityContext.pod.runAsGroup	Group ID to run the pod	999
postgresExtensions.securityContext.pod.fsGroup	File system group ID	999
postgresExtensions.securityContext.pod.seccompProfile.type	Seccomp profile type	Unconfined
postgresExtensions.securityContext.container.allowPrivilegeEscalation	Allow privilege escalation	false
postgresExtensions.securityContext.container.readOnlyRootFilesystem	Read-only root filesystem	false
postgresExtensions.securityContext.container.runAsNonRoot	Run container as non-root user	true
postgresExtensions.securityContext.container.runAsUser	User ID to run the container	999
postgresExtensions.securityContext.container.runAsGroup	Group ID to run the container	999
postgresExtensions.securityContext.container.capabilities.drop	Capabilities to drop	["ALL"]
postgresExtensions.securityContext.container.seccompProfile.type	Seccomp profile type	Unconfined

RELEASE 6.4.2

1. changed openldap chart provider to jp-gouin's

Feature:

Changed openldap chart provider to jp-gouin's, which is more actively maintained and has better support for features like TLS. Because of that, the default configuration of openldap was changed to reflect the new provider's defaults.

Values changed - ilum-core

Name	Old value	New Value
security.ldap.userMapping.enabled	sn	employeeType
security.ldap.userMapping.enabledValue	~	active
security.ldap.password	admin	Not@SecurePassw0rd

Values added - ilum-aio

Added new values for openldap configuration

Name	Description	Value
global.ldapDomain	Domain of the LDAP configuration	ilum.cloud
openldap.replicaCount	Replica count of openLDAP	1
openldap.replication.enabled	Enable HA for openLDAP	false
openldap.ltb-passwd.enabled	Enable ltb-passwd service	false
openldap.phpldapadmin.enabled	Enable PhpLdapAdmin	false

Values changed - ilum-aio

Name	Old value	New Value
openldap.env.LDAP_BACKEND	hdb	mdb
openldap.customLdifFiles	see values.yaml	see values.yaml

Values deleted - ilum-aio

Name	Reason
openldap.env.LDAP_ORGANISATION	Managed by the chart
openldap.env.LDAP_DOMAIN	Managed by the chart
openldap.env.LDAP_TLS	Managed by the chart
openldap.env.LDAP_TLS_ENFORCE	Managed by the chart

RELEASE 6.4.1

1. Adapt hydra to https

Values added - ilum-aio

Name	Description	Value
global.security.hydra.uiDomain	Domain where ilum-ui can be accessed from browser	``
global.security.hydra.uiProtocol	Protocol used to access ilum-ui: http or https	http

Values deleted - ilum-aio

Name	Reason
global.security.hydra.uiUrl	Replaced with uiDomain and uiProtocol

Values added - ilum-core

Name	Description	Value
hydra.cookies.same_site_mode	SameSite value for hydra cookies in set-cookie header	Lax

2. Added openldap to helm chart and ilum-to-ldap synchronization

Values added - ilum-aio

Added openldap configuration values

Name	Description	Value
openldap.enabled	Flag used to enable openldap	false
openldap.adminPassword	Password of admin ldap user	admin
openldap.fullnameOverride	Name of Openldap helm chart resources	ilum-openldap
openldap.persistence.enabled	Flag to enable persistence by openldap	true
openldap.persistence.size	Memory used by openldap for storage	1Gi
openldap.env.LDAP_ORGANIZATION	Organization name of main ldap domain	Ilum
openldap.env.LDAP_DOMAIN	Main domain used in ldap by admin	ilum.cloud
openldap.env.LDAP_BACKEND	Type of ldap backend	hdb
openldap.env.LDAP_TLS	Flag used to enable TLS in ldap	false
openldap.env.LDAP_TLS_ENFORCE	Flag used to enforce TLS in ldap	false
openldap.env.LDAP_REMOVE_CONFIG_AFTER_SETUP	Flag used to update config	true
openldap.customLdifFiles.schemas.ldif	File with custom schem applied at the startup	<initial-schema.ldif>

Values added - ilum-core

Added configurations for synchronization of ldap with ilum

Name	Description	Value
security.ldap.ilumToLdapSync	Flag used to enable ilum to ldap sync	false
security.ldap.userMapping.oc	OC values used during insertion of ilum users into ldap	<default-oc-list>
security.ldap.groupMapping.oc	OC values used during insertion of ilum groups into ldap	<default-oc-list>

Values changed - ilum-core

Name	Old value	New Value
security.ldap.urls	[]	[ "ldap://ilum-openldap:389" ]
security.ldap.base	""	dc=ilum,dc=cloud
security.ldap.username	""	cn=admin,dc=ilum,dc=cloud
security.ldap.password	""	admin
security.ldap.adminUsers	[ "admin" ]	[ "admin", "ilumadmin" ]
security.ldap.userMapping.base	""	ou=people
security.ldap.userMapping.fullname	""	cn
security.ldap.userMapping.description	""	description
security.ldap.userMapping.email	""	mail
security.ldap.userMapping.enabled	""	sn
security.ldap.userMapping.base	""	ou=groups
security.ldap.userMapping.description	""	description

3. Restricted RBAC Mode for ilum-core service

Feature:

Introduced a new rbac.restricted.enabled flag in the ilum-core chart. When set to true, this option applies a more restrictive set of RBAC permissions for the service account. This enhances security by adhering to the principle of least privilege and is recommended for production or security-sensitive environments.

Values added - ilum-core

Added a flag to enable a more restrictive RBAC configuration.

Name	Description	Value
rbac.restricted.enabled	If true, applies a more restrictive, non-cluster-wide set of RBAC permissions for Spark applications.	false

4. Added enabled flag to Trino in ilum-sql

Feature:

Added enabled flag to Trino in ilum-sql, which allows users to disable Trino if they do not need it. This also will help ilum-ui with the configuration of Trino.

Values added - ilum-sql

Name	Description	Value
config.trino.enabled	Flag to enable Trino in ilum-sql	false

⚠️⚠️⚠️ Warnings

Trino was enabled by default, so if you wish to enable it after the version upgrade, you need to set ilum-sql.config.trino.enabled to true in your helm values.

RELEASE 6.4.0

1. Addition of OAuth Provider and its integration with Services

Feature:

Added Hydra deployment to helm chart and fields to configure it

Values added - ilum-core

Name	Description	Value
global.security.hydra.enabled	Flag to enable hydra	false
global.security.hydra.uiUrl	Ilum UI url required to configure OpenID connect	``
global.security.hydra.clientId	Client Id of OpenID client created in hydra	ilum-client
global.security.hydra.cliendSecret	Client Secret of OpenId Client created in hydra	secret
hydra.dsn	DSN for database used by hydra	postgres://ilum:CHANGEMEPLEASE@ilum-postgresql:5432/hydra?sslmode=disable
hydra.secretsSystem	Secret used by hydra to securily store data	CHANGEMEPLEASE
hydra.recreateClient	Boolean flag for OpenId client recreation during hydra startup	true
hydra.resources.requests	Memory and CPU limits and requests used by hydra deployment	null
hydra.imagePullPolicy	Hydra container image pull policy	IfNotPresent
hydra.service.domain	Domain used by hydra service	ilum-hydra
hydra.service.publicPort	Port that exposes public api of hydra	4444
hydra.service.adminPort	Port that exposes admin api of hydra	4445
hydra.service.type	Hydra service type	ClusterIP
hydra.service.publicNodePort	Hydra service node port assigned to public port	``
hydra.service.publicNodePort	Hydra service node port assigned to admin port	``
hydra.service.clusterIP	Hydra service cluster IP	``
hydra.service.loadBalancerIP	Hydra service load balancer IP	``
hydra.service.annotations	Annotations used by hydra service	{}
hydra.separateDeployment	Flag to launch hydra in a separate deployment or in ilum-core	true

Values added - ilum-ui

Name	Description	Value
runtimeVars.hydraUrl	Url of Hydra Public API	http://ilum-hydra:4444
runtimeVars.hydraPath	ilum-ui proxy-path to hydra public api	/external/hydra

Feature

Added helm values to specify how roles and groups from ilum-core are going to be mapped to microservices of ilum

Values added - ilum-core

Name	Description	Value
hydra.rewriteMapping	Boolean flag for recreation of ilum-to-services roles config after ilum-core restart	true
hydra.mapping.minioMinAccessRole	Default role assigned to ilum users with access to minio	readonly
hydra.mapping.airflowMinAccessRole	Default role assigned to ilum users with access to airflow	Viewer
hydra.mapping.supersetMinAccessRole	Default role assigned to ilum users with access to superset	Gamma
hydra.mapping.grafanaMinAccessRole	Default role assigned to ilum users with access to grafana	Viewer
hydra.mapping.giteaMinAccessRole	Default role assigned to ilum users with access to gitea	``
hydra.mapping.groupsToMinio	Map of ilum groups to a list of minio policies	null
hydra.mapping.groupsToSuperset	Map of ilum groups to a list of superset roles	null
hydra.mapping.groupsToAirflow	Map of ilum groups to a list of airflow roles	null
hydra.mapping.groupsToGrafana	Map of ilum groups to a list of grafana roles	null
hydra.mapping.groupsToGitea	Map of ilum groups to a list of gitea roles	null
hydra.mapping.groupsToMinio[*].ilumObj	Name of ilum group to be mapped	``
hydra.mapping.groupsToMinio[*].serviceObjs	List of minio policies that the ilum group is mapped to	``
hydra.mapping.groupsToSuperset[*].ilumObj	Name of ilum group to be mapped	``
hydra.mapping.groupsToSuperset[*].serviceObjs	List of superset roles that the ilum group is mapped to	``
hydra.mapping.groupsToAirflow[*].ilumObj	Name of ilum group to be mapped	``
hydra.mapping.groupsToAirflow[*].serviceObjs	List of airflow roles that the ilum group is mapped to	``
hydra.mapping.groupsToGrafana[*].ilumObj	Name of ilum group to be mapped	``
hydra.mapping.groupsToGrafana[*].serviceObjs	List of grafana roles that the ilum group is mapped to	``
hydra.mapping.groupsToGitea[*].ilumObj	Name of ilum group to be mapped	``
hydra.mapping.groupsToGitea[*].serviceObjs	List of gitea roles that the ilum group is mapped to	``
hydra.mapping.rolesToGitea	Map of ilum roles to a list of gitea roles	null
hydra.mapping.rolesToMinio[0].ilumObj	Name of ilum role to be mapped	ADMIN
hydra.mapping.rolesToMinio[0].serviceObjs	List of minio policies that the ilum role is mapped to	[ consoleAdmin ]
hydra.mapping.rolesToMinio[1].ilumObj	Name of ilum role to be mapped	DATA_ENGINEER
hydra.mapping.rolesToMinio[1].serviceObjs	List of minio policies that the ilum role is mapped to	[ readonly, writeonly, diagnostics ]
hydra.mapping.rolesToSuperset[0].ilumObj	Name of ilum role to be mapped	ADMIN
hydra.mapping.rolesToSuperset[0].serviceObjs	List of superset roles that the ilum role is mapped to	[ Admin ]
hydra.mapping.rolesToSuperset[1].ilumObj	Name of ilum role to be mapped	DATA_ENGINEER
hydra.mapping.rolesToSuperset[1].serviceObjs	List of superset roles that the ilum role is mapped to	[ Alpha ]
hydra.mapping.rolesToAirflow[0].ilumObj	Name of ilum role to be mapped	ADMIN
hydra.mapping.rolesToAirflow[0].serviceObjs	List of airflow roles that the ilum role is mapped to	[ Admin ]
hydra.mapping.rolesToAirflow[1].ilumObj	Name of ilum role to be mapped	DATA_ENGINEER
hydra.mapping.rolesToAirflow[1].serviceObjs	List of airflow roles that the ilum role is mapped to	[ User ]
hydra.mapping.rolesToGrafana[0].ilumObj	Name of ilum role to be mapped	ADMIN
hydra.mapping.rolesToGrafana[0].serviceObjs	List of grafana roles that the ilum role is mapped to	[ Admin ]
hydra.mapping.rolesToGrafana[1].ilumObj	Name of ilum role to be mapped	DATA_ENGINEER
hydra.mapping.rolesToGrafana[1].serviceObjs	List of grafana roles that the ilum role is mapped to	[ Editor ]
hydra.mapping.rolesToMinio[*].ilumObj	Name of ilum role to be mapped	``
hydra.mapping.rolesToMinio[*].serviceObjs	List of minio policies that the ilum role is mapped to	``
hydra.mapping.rolesToSuperset[*].ilumObj	Name of ilum role to be mapped	``
hydra.mapping.rolesToSuperset[*].serviceObjs	List of superset roles that the ilum role is mapped to	``
hydra.mapping.rolesToAirflow[*].ilumObj	Name of ilum role to be mapped	``
hydra.mapping.rolesToAirflow[*].serviceObjs	List of airflow roles that the ilum role is mapped to	``
hydra.mapping.rolesToGrafana[*].ilumObj	Name of ilum role to be mapped	``
hydra.mapping.rolesToGrafana[*].serviceObjs	List of grafana roles that the ilum role is mapped to	``
hydra.mapping.rolesToGitea[*].ilumObj	Name of ilum role to be mapped	``
hydra.mapping.rolesToGitea[*].serviceObjs	List of gitea roles that the ilum role is mapped to	``

Feature

Integrated minio with Hydra OIDC in values.yaml by adding new environment variables with oidc client data taken from global.security.hydra

Values added - minio

| minio.extraEnvVars | | |

Name	Old value	New Value
minio.extraEnvVars	...	...

Feature

Integrated airflow with Hydra OIDC in values.yaml

Values added - airflow

Name	Description	Value
airflow.webserver.extraVolumes[0].name	Name of additional airflow volume with oidc config	oauth-secret-volume
airflow.webserver.extraVolumes[0].secret.secretName	Airflow secret with hydra oidc client config	ilum-hydra-client-secret
airflow.webserver.extraVolumeMounts[0].name	Name of volume-mount of secret with oidc config	oauth-secret-volume
airflow.webserver.extraVolumeMounts[0].mountPath	Path for volume-mount of secret with oidc config	/opt/airflow/client-secret
airflow.webserver.extraVolumeMounts[0].readOnly	readonly flag of volume mount with oidc config	true

Feature

Integrated grafana with Hydra OIDC in values.yaml

Values added - grafana

Name	Description	Value
grafana.grafana.ini.auth.generic_oauth.enabled	Flag to enable oauth in grafana, taken from global.security.hydra.enabled by default	false
grafana.grafana.ini.auth.generic_oauth.name	Name of oauth client	Ilum
grafana.grafana.ini.auth.generic_oauth.allow_sign_up	Flag to enable user creation when signing in with oauth	true
grafana.grafana.ini.auth.generic_oauth.client_id	Id of oauth client, taken from global.security.hydra.clientId by default	ilum-client
grafana.grafana.ini.auth.generic_oauth.client_secret	Secret of oauth client, taken from global.security.hydra.clientSecret by default	secret
grafana.grafana.ini.auth.generic_oauth.scopes	Scopes requested from oauth	openid profile email offline_access
grafana.grafana.ini.auth.generic_oauth.auth_url	Url used to initiate oauth authentication, uses global.security.hydra.uiUrl as base by default	/external/hydra/oauth2/auth
grafana.grafana.ini.auth.generic_oauth.token_url	Url used for tokens exchange in oauth workflow, uses global.security.hydra.uiUrl as base by default	/external/hydra/oauth2/token
grafana.grafana.ini.auth.generic_oauth.api_url	Url used to access user info, uses global.security.hydra.uiUrl as base by default	/external/hydra/userinfo
grafana.grafana.ini.auth.generic_oauth.login_attribute_path	Id token claim used to distinguish different users	userId
grafana.grafana.ini.auth.generic_oauth.email_attribute_name	Id token claim with email	email
grafana.grafana.ini.auth.generic_oauth.role_attribute_path	Expression used to map roles from id_token to grafana	...
grafana.grafana.ini.auth.generic_oauth.role_attribute_strict	Flag to require the role during sign up	true

Feature

Integrated superset with Hydra OIDC in values.yaml

Values added - superset

Name	Description	Value
superset.configOverrides.ilum_oauth_security	Code added to superset config in order to enable oidc	...
superset.extraVolumes[0].name	Name of extra volume with a secret for oidc connection	oauth-secret-volume
superset.extraVolumes[0].secret.secretName	Name of secret used in extra volume for oidc connection	ilum-hydra-client-secret
superset.extraVolumes[1].name	Name of extra volume with superset plugin used to enable oidc	oauth-plugin-volume
superset.extraVolumes[1].secret.secretName	Secret with superset plugin used to enable oidc	ilum-superset-oidc-plugin-secret
superset.extraVolumeMounts[0].name	Name of volume mount with oidc secret data	oauth-secret-volume
superset.extraVolumeMounts[0].mountPath	Path of volume mount with oidc secret data	/app/pythonpath/oauth
superset.extraVolumeMounts[1].name	Name of volume mount with a superset plugin as a python file	oauth-plugin-volume
superset.extraVolumeMounts[1].mountPath	Path of volume mound with a superset plugin	/app/pythonpath/security

2. Addition of examples for ilum-core and superset

Feature:

Added examples for Ilum modules. New Ilum users can use these examples to quickly understand how to use Ilum modules like ilum-sql, superset and others.

Values added - ilum-core

Name	Description	Value
examples.job	Enables creating single job example	true
examples.schedule	Enables creating schedule example	true
examples.sqlNotebook	Enables creating sql notebook example	true
examples.sqlQuery	Enables creating sql query example	true
examples.database	Enables creating database example	true

Values added - ilum-aio

Name	Description	Value
ilum-core.examples.job	Enables creating single job example	true
ilum-core.examples.schedule	Enables creating schedule example	true
ilum-core.examples.sqlNotebook	Enables creating sql notebook example	true
ilum-core.examples.sqlQuery	Enables creating sql query example	true
ilum-core.examples.database	Enables creating database example	true
superset.extraEnv.IMPORT_DASHBOARD	Enables creating superset dashboard example	true
superset.extraVolumes[2].name	Volume name for dashboard import	example-dashboard
superset.extraVolumes[2].configMap.name	ConfigMap that contains base64-encoded dashboard	ilum-superset-example-dashboard
superset.extraVolumeMounts[2].name	Mount name for example dashboard config	example-dashboard
superset.extraVolumeMounts[2].mountPath	Path in container to mount the dashboard config	/config
superset.extraVolumeMounts[2].readOnly	Mount config as read-only	true
superset.init.initscript	Custom init script to conditionally import dashboard	See script in ilum-aio values file

3. Internal users upgrade credentials flag

Values added - ilum-core

Name	Description	Value
ilum-core.security.internal.upgradeCredentials	Enables overriding user password with helm configuration	false

4. Tighter integration of Marquez with Ilum

Feature:

Enhanced Marquez integration with ilum-core, which means no direct communication between ilum-frontend and Marquez is needed anymore. This way, having a customized Marquez build is not necessary anymore.

Values changed - ilum-core

Name	Old value	New Value
job.openLineage.transport.endpoint	/external/lineage/api/v1/lineage	/api/v1/lineage

5. Changed superset load example variable name

Values deleted - ilum-aio

Name	Reason
superset.extraEnv.IMPORT_DASHBOARD	Changed to other variable name

Values added - ilum-aio

Name	Description	Value
superset.init.loadExamples	Enables creating superset dashboard example	true

RELEASE 6.3.2

1. Addition of frequently used values to ilum-sql

Feature:

Added frequently used values to ilum-sql chart, so that their configuration is easier.

Values added - ilum-sql

Name	Description	Value
config.kyuubi.logLevel	Base log-level of the log4j framework	INFO
config.kyuubi.idleEngineTimeout	Auto-shutdown time of Kyuubi engines	30M
config.kyuubi.idleSessionTimeout	Auto-shutdown time of Kyuubi sessions	30M
config.kyuubi.engineAliveProbe	Whether to create a probe, which will check engine liveness	true
config.kyuubi.cleanupTerminatedSparkDriverPods	Determines which of the terminated Spark engine pods will get deleted. Available: NONE, COMPLETED, ALL	ALL

Values changed - ilum-aio

Name	Old value	New Value
ilum-sql.config.kyuubi.defaults	see values.yaml	~

⚠️⚠️⚠️ Warnings

Because these values were already present in the ilum-aio chart, the change will not be noticeable for users who have changed the value of ilum-sql.config.kyuubi.defaults.

2. Support of Trino in ilum-sql

Feature:

Introduced support for Trino as an SQL engine in ilum-sql chart.

Values added - ilum-aio

Name	Description	Value
trino.enabled	Enables built-in Trino	false
trino.nameOverride	Sets the name override	ilum-trino
trino.coordinatorNameOverride	Sets the name override for the coordinator	ilum-trino-coordinator
trino.workerNameOverride	Sets the name override for the worker nodes	ilum-trino-worker
trino.server.workers	Sets the number of workers	1
trino.catalogs.ilum-delta	Configures the 'ilum-delta' catalog	See values.yaml
ilum-sql.config.trino.catalog	The catalog of choice for Trino	ilum-delta

Values added - ilum-sql

Name	Description	Value
config.trino.url	Url pointing to Trino coordinator	http://ilum-trino:8080
config.trino.catalog	The catalog of choice for Trino	system
config.trino.defaults	Additional settings of Trino engines. All properties must be prefixed with trino.	~

3. Enhanced Oauth2

Feature

Added users, groups and roles mapping during authentication from OAuth2 Autherization server to Ilum Core. Added ability to assign Admin role to oauth2 users.

Values added - ilum-core

Name	Description	Value
security.oauth2.mapping.id	JWT claim that should be mapped to user`s id	""
security.oauth2.mapping.name	JWT claim that should be mapped to user`s name	sub
security.oauth2.mapping.email	JWT claim that should be mapped to user`s email	email
security.oauth2.mapping.fullname	JWT claim that should be mapped to user`s fullname	fullname
security.oauth2.mapping.description	JWT claim that should be mapped to user`s description	""
security.oauth2.mapping.department	JWT claim that should be mapped to user`s department	""
security.oauth2.mapping.groups	JWT claim with a list of groups that user is a part of represented by strings	"groups"
security.oauth2.mapping.roles	JWT claim with a list of roles that user uses represented by strings	"roles"
security.oauth2.mapping.enabled	JWT claim that should be mapped to user`s state	""
security.oauth2.mapping.enabledTrue	Value of JWT claim with the name of mapping.enabled that stands for ENABLED	""
security.oauth2.mapping.singleGroup	JWT claim that contains a string with name of group that the user is part of	""
security.oauth2.mapping.singleRole	JWT claim that contains a string with name of role that the user has	""

4. Addition of kubernetes s3 region to ilum-core and ilum-aio

Feature:

Added the ability to set the S3 region in ilum-core.

Values added - ilum-core

Name	Description	Value
kubernetes.s3.region	default kubernetes cluster S3 storage region to store spark resources	us-east-1

Values added - ilum-aio

Name	Description	Value
ilum-core.kubernetes.s3.region	default kubernetes cluster S3 storage region to store spark resources	us-east-1
ilum-core.kubernetes.defaultCluster.config.spark.hadoop.fs.s3a.bucket.ilum-data.region	default kubernetes cluster S3 storage region to store spark resources	us-east-1

5. Protocol in superset

Feature:

Added the ability to set the protocol in superset.

Values added - superset

Name	Description	Value
protocol	superset protocol	http

6. Addition of hive metastore status probe to ilum-core helm chart

Feature:

Added possibility to enable status probe for hive metastore.

Values added - ilum-core

Name	Description	Value
hiveMetastore.statusProbe.enabled	Hive metastore status probe enabled flag	false
hiveMetastore.statusProbe.image	Hive metastore status probe image	curlimages/curl:8.5.0
hiveMetastore.statusProbe.host	Hive metastore status probe host	ilum-hive-metastore
hiveMetastore.statusProbe.port	Hive metastore status probe port	9083

RELEASE 6.3.1

1. Added extra buckets to storage config

Feature

Added ability to include extra buckets to ilum cluster spark storage configuration.

Values added - ilum-core

Name	Description	Value
kubernetes.s3.extraBuckets	ilum-core default kubernetes cluster S3 storage extra buckets to include	[]
kubernetes.gcs.extraBuckets	ilum-core default kubernetes cluster GCS storage extra buckets to include	[]
kubernetes.wasbs.extraContainers	ilum-core default kubernetes cluster WASBS storage extra containers to include	[]
kubernetes.hdfs.extraCatalogs	ilum-core default kubernetes cluster HDFS storage extra catalogs to include	[]

2. Enhanced LDAP

Added the ability to map users, groups, and roles — along with their properties and relationships — from an LDAP server to Ilum Core based on mapping configurations in Helm. Enabled the option to assign Admin role to LDAP users.

Values added - ilum-core

Name	Description	Value
security.ldap.userMapping.base	LDAP base of user entries	""
security.ldap.userMapping.filter	LDAP filter used for users search	"uid={0}"
security.ldap.userMapping.username	Name of LDAP attribute that should be mapped to user`s username	uid
security.ldap.userMapping.password	Name of LDAP attribute that should be mapped to user`s password	userPassword
security.ldap.userMapping.description	Name of LDAP attribute that should be mapped to user`s description	""
security.ldap.userMapping.fullname	Name of LDAP attribute that should be mapped to user`s fullname	""
security.ldap.userMapping.department	Name of LDAP attribute that should be mapped to user`s department	""
security.ldap.userMapping.email	Name of LDAP attribute that should be mapped to user`s email	""
security.ldap.userMapping.enabled	Name of LDAP attribute that should be mapped to user`s state	""
security.ldap.userMapping.enabledValue	Value of attribute with the name of userMapping.enabled that stands for ENABLED	""
security.ldap.groupMapping.base	LDAP base for group entries	""
security.ldap.groupMapping.filter	LDAP filter used for groups search	(member={0})
security.ldap.groupMapping.name	Name of LDAP attribute that should be mapped to group`s name	cn
security.ldap.groupMapping.description	Name of LDAP attribute that should be mapped to group`s description	""
security.ldap.groupMapping.memberAttribute	Name of LDAP attribute that lists users having the group	uid
security.ldap.groupMapping.roles	LDAP attribute that lists the roles that the group includes	""
security.ldap.groupMapping.roleFilterAttribute	LDAP attribute of roles that represents a role in groupMapping.roles attribute	""
security.ldap.groupMapping.enabled	Name of LDAP attribute that should be mapped to group`s state	""
security.ldap.groupMapping.enabledTrue	Value of attribute from groupMapping.enabled that stands for ENABLED	""
security.ldap.roleMapping.base	LDAP base for role entries	""
security.ldap.roleMapping.filter	LDAP filter used for roles search	""
security.ldap.roleMapping.memberAttribute	Name of LDAP attribute that lists users having the role	""
security.ldap.roleMapping.name	Name of LDAP attribute that should be mapped to role`s name	""
security.ldap.roleMapping.description	Name of LDAP attribute that should be mapped to role`s description	""
security.ldap.roleMapping.enabled	Name of LDAP attribute that should be mapped to role`s state	""
security.ldap.roleMapping.enabledTrue	Value of attribute from roleMapping.enabled that stands for ENABLED	""

Values deleted - ilum-core

Name	Reason
security.ldap.userSearch	Replaced with security.ldap.userMapping
security.ldap.groupSearch	Replaced with security.ldap.groupMapping

Names changed - ilum-core

Old Name	New Name
security.internal.users[*].password	security.internal.users[*].initialPassword

3. sparkmagic default config

Values changed - ilum-jupyter configuration parameters change because of the introduction of the spark session form.

Name	Old value	New Value
sparkmagic.config.sessionConfigs.conf	'{ "pyRequirements": "pandas", "cluster": "default", "autoPause": "false", "spark.example.config": "You can change the default configuration in ilum-jupyter-config k8s configmap" }'	'{}'

Values deleted - ilum-jupyter

Name	Reason
sparkmagic.config.sessionConfigs.executorCores	Not needed anymore because of the new spark session form

Values deleted - ilum-jupyter

Name	Reason
sparkmagic.config.sessionConfigs.driverMemory	Not needed anymore because of the new spark session form

RELEASE 6.3.0

1. Changed image tag version of kyuubi

Values changed - sparkmagic configuration parameters

Name	Old value	New Value
sparkmagic.config.sessionConfigs.conf	'{ "pyRequirements": "pandas", "spark.example.config": "You can change the default configuration in ilum-jupyter-config k8s configmap" }'	'{ "pyRequirements": "pandas", "cluster": "default", "autoPause": "false", "spark.example.config": "You can change the default configuration in ilum-jupyter-config k8s configmap" }'

2. Added property to set kafka address for ilum-core

Feature

Added ability to set kafka address for ilum-core pod, separate from global kafka address configuration for both spark jobs and ilum-core pod set via kafka.address property.

Values added - ilum-core

Name	Description	Value
kafka.ilum.address	ilum-core kafka address only for ilum-core pod, overrides kafka.address	not defined

3. Changed ilum job healthcheck tolerance time

Values changed - ilum job healthcheck configuration parameters

Name	Old value	New Value
job.healthcheck.tolerance	120	3600

4. Introducing embedded git repo

Feature

Added Gitea as a module providing build in git server for ilum platform.

Values added - gitea

Name	Description	Value
gitea.enabled	Enable or disable Gitea deployment	true
gitea.image.rootless	Run Gitea in rootless mode	false
gitea.gitea.config.database.DB_TYPE	Database type used for Gitea	postgres
gitea.gitea.config.database.HOST	Database host and port for Gitea	ilum-postgresql-hl:5432
gitea.gitea.config.database.NAME	Database name for Gitea	gitea
gitea.gitea.config.database.USER	Database username for Gitea	ilum
gitea.gitea.config.database.PASSWD	Database password for Gitea (Change required)	CHANGEMEPLEASE
gitea.gitea.admin.existingSecret	Gitea secret to store init credentials	ilum-git-credentials
gitea.gitea.admin.email	Gitea admin email	ilum@ilum
gitea.gitea.admin.passwordMode	Password mode for admin account	initialOnlyNoReset
gitea.gitea.additionalConfigFromEnvs[0].name	Enable push-create user	GITEA__REPOSITORY__ENABLE_PUSH_CREATE_USER
gitea.gitea.additionalConfigFromEnvs[0].value	Value for enabling push-create user	true
gitea.gitea.additionalConfigFromEnvs[1].name	Enable push-create organization	GITEA__REPOSITORY__ENABLE_PUSH_CREATE_ORG
gitea.gitea.additionalConfigFromEnvs[1].value	Value for enabling push-create organization	true
gitea.gitea.additionalConfigFromEnvs[2].name	Default repository branch	GITEA__REPOSITORY__DEFAULT_BRANCH
gitea.gitea.additionalConfigFromEnvs[2].value	Value for default repository branch	master
gitea.gitea.additionalConfigFromEnvs[3].name	Root URL of the Gitea server	GITEA__SERVER__ROOT_URL
gitea.gitea.additionalConfigFromEnvs[3].value	Value for Gitea server root URL	http://git.example.com/external/gitea/
gitea.gitea.additionalConfigFromEnvs[4].name	Static URL prefix	GITEA__SERVER__STATIC_URL_PREFIX
gitea.gitea.additionalConfigFromEnvs[4].value	Value for static URL prefix	/external/gitea/
gitea.redis-cluster.enabled	Enable or disable Redis cluster	false
gitea.redis.enabled	Enable or disable Redis	false
gitea.postgresql.enabled	Enable or disable standalone PostgreSQL	false
gitea.postgresql-ha.enabled	Enable or disable PostgreSQL HA	false

Values added - ilum-jupyter

Name	Description	Value
ilum-jupyter.git.enabled	Enable or disable Git integration	false
ilum-jupyter.git.username	Git username for authentication	ilum
ilum-jupyter.git.password	Git password for authentication	ilum
ilum-jupyter.git.email	Git email address	ilum@ilum
ilum-jupyter.git.repository	Git repository name	jupyter
ilum-jupyter.git.address	Git server address	ilum-gitea-http:3000
ilum-jupyter.git.init.image	Git initialization image	bitnami/git:2.48.1

Values added - ilum-airflow

Name	Description	Value
airflow.dags.gitSync.enabled	Enable or disable Git synchronization for DAGs	true
airflow.dags.gitSync.repo	Git repository URL for DAGs	http://ilum-gitea-http:3000/ilum/airflow.git
airflow.dags.gitSync.branch	Git branch to sync from	master
airflow.dags.gitSync.ref	Git reference to sync	HEAD
airflow.dags.gitSync.depth	Git clone depth	1
airflow.dags.gitSync.maxFailures	Maximum allowed synchronization failures	0
airflow.dags.gitSync.subPath	Subpath within the repository to sync	""
airflow.dags.gitSync.credentialsSecret	Secret used for Git authentication	ilum-git-credentials

5. Ilum SQL configuration naming changes

Change the naming of Ilum Sql Configuration to better reflect the current usage of Kyuubi

Names changed - ilum-core

Old Name	New Name
kyuubi.*	sql.*

Names changed - ilum-aio

Old Name	New Name
ilum-kyuubi.*	ilum-sql.*

⚠️⚠️⚠️ Warnings

Due to the changes in naming and in the inner workings of the SQL engine launching, and restrictions on what can be done via a helm upgrade, it is required to manually delete the old stateful set (e.g. kubectl delete sts ilum-sql) before upgrading to this version. This will ensure that during the update, a new stateful set is created with the correct configuration. The breaking changes are related to the labels and volume mounts that are used by the ilum-sql stateful set.

6. Add configurations for Ilum Submit for Spark Sql engines

Ilum Submit enhances the process of launching Spark SQL engines via both the Ilum Web Application and the JDBC endpoint by automatically applying the configurations of the selected cluster. This improvement eliminates the need to manually provide Kyuubi's Spark configuration to Ilum Core.

Valued deleted - ilum-core

Name	Reason
sql.sparkConfig	Unnecessary after the change

Values added - ilum-kyuubi

Name	Description	Value
ilumSubmit.enabled	Flag to enable ilum submit service	false
ilumSubmit.ilum.host	Host of Ilum REST service	ilum-core
ilumSubmit.ilum.port	Port of Ilum REST service	9888

Values added - ilum-aio

Name	Description	Value
ilum-sql.ilumSubmit.enabled	Flag to enable SQL engine creation through Ilum	true

⚠️⚠️⚠️ Warnings

Since Kyuubi's Spark config is not needed in Ilum Core anymore, the default spark config should be supplied directly to ilum-sql.config.spark.defaults instead of the global value.

7. Moving security related values in ilum-core from config map to secret

Feature

Security‑related configuration (including internal user credentials, LDAP, OAuth2, JWT, and authorities settings) has been moved from the config map to a dedicated Kubernetes Secret. This improves the security of sensitive data by isolating it from non‑sensitive configuration.

Values added - ilum-core

Name	Description	Value
security.secret.name	Name of the secret that holds security‑related configuration. Use this to override the default secret name.	ilum-security

8. Changed ilum-ui service type

Because of the problems with kubectl port-forward we are exposing a NodePort by default.

Values changed - ilum-ui healthcheck configuration parameters

Name	Old value	New Value
service.type	ClusterIP	NodePort
service.nodePort	``	31777

RELEASE 6.2.1

1. Change the value of Kyuubi's url

Feature

Change the value of Kyuubi's url in ilum-core. The default value should work now out of the box.

Values changed - ilum-core

Name	Old value	New Value
kyuubi.host	ilum-sql-rest	ilum-sql-headless

RELEASE 6.2.1-RC1

1. Spark job's memory settings configuration in ilum-core

Feature

Added spark job's memory settings configuration in ilum-core. When default cluster in ilum-core is being created, it will have memory settings parameters set to those values.

Values added - ilum-core

Name	Description	Value
job.memorysettings.executors	spark jobs executor count	2
job.memorysettings.executorMemory	spark jobs executor memory allocation	1g
job.memorysettings.driverMemory	spark jobs driver memory allocation	1g
job.memorysettings.executorCores	spark jobs executor core count	1
job.memorysettings.driverCores	spark jobs driver core count	1
job.memorysettings.dynamicAllocationEnabled	spark jobs dynamic allocation enabled flag	false
job.memorysettings.minExecutors	spark jobs minimum number of executors	0
job.memorysettings.initialExecutors	spark jobs initial number of executors	0
job.memorysettings.maxExecutors	spark jobs maximum number of executors	20

2. Spark history server retention parameters addition

Feature

Added spark history server retention parameters to ilum-core. These parameters allow the user to configure the retention of spark history server logs.

Values added - ilum-core

Name	Description	Value
historyServer.parameters.spark.history.fs.cleaner.enabled	history server cleaner enabled flag	true
historyServer.parameters.spark.history.fs.cleaner.interval	history server cleaner interval	1d
historyServer.parameters.spark.history.fs.cleaner.maxAge	history server logs max age	7d

3. Split Kyuubi's url into host and port

Feature

Split Kyuubi's url into host and port in ilum-core. This change was necessary for us to be able to create custom engines.

Values added - ilum-core

Name	Description	Value
kyuubi.host	Kyuubi host	ilum-sql-rest
kyuubi.port	Kyuubi port	10099

Values deleted - ilum-core

Name	Reason
kyuubi.url	Unnecessary after the change

4. Extra entries in helm_ui nginx server config map

Feature

Added enabled flags for history server, minio, ilum-jupyter, airflow, mlflow and lineage to ilum-ui. These flags allow the user to enable or disable the access to these services through ilum-ui. These values will be used in nginx server config map.

Values added - ilum-ui chart

Name	Description	Value
nginx.config.ilum-jupyter.enabled	ilum-ui nginx config ilum-jupyter enabled flag	false
nginx.config.airflow.enabled	ilum-ui nginx config airflow enabled flag	false
nginx.config.minio.enabled	ilum-ui nginx config minio enabled flag	false
nginx.config.historyServer.enabled	ilum-ui nginx config historyServer enabled flag	false
nginx.config.mlflow.enabled	ilum-ui nginx config mlflow enabled flag	false
nginx.config.lineage.enabled	ilum-ui nginx config lineage enabled flag	false

5. Superset in ilum-aio chart

Feature

Superset in ilum AIO chart. Superset is fast, lightweight, intuitive, and loaded with options that make it easy for users of all skill sets to explore and visualize their data, from simple line charts to highly detailed geospatial charts. Superset is one of modules integrated with Ilum platform.

Values added - ilum-ui

log aggregation config

Name	Description	Value
runtimeVars.supersetUrl	superset service url	http://ilum-superset:8088/
nginx.config.superset.enabled	ilum-ui nginx config superset enabled flag	false

4. Ilum default kubernetes cluster config from helm values

Feature

From now on, the default ilum cluster parameters will be set based on the helm values.

Values added - ilum-core chart

Name	Description	Value
kubernetes.defaultCluster.config	ilum-core default kubernetes cluster configuration	config:   spark.driver.extraJavaOptions: "-Divy.cache.dir=/tmp -Divy.home=/tmp"   spark.kubernetes.container.image: "ilum/spark:3.5.2-delta"   spark.databricks.delta.catalog.update.enabled: "true"

RELEASE 6.2.0

1. Changed image tag version of kyuubi

Values changed - ilum-kyuubi chart

Name	Old value	New Value
image.tag	1.9.2-spark	1.10.0-spark

2. Changed kyuubi spark configuration in ilum-kyuubi chart

Added spark.driver.memory=2g in global.kyuubi.sparkConfig

RELEASE 6.2.0-RC2

1. Minio status probe addition

Feature

Added status probe in ilum-core that checks whether minio storage is ready

Values added - ilum-core

Name	Description	Value
minio.statusProbe.enabled	minio status probe enabled flag	true
minio.statusProbe.image	minio status probe image	curlimages/curl:8.5.0
minio.statusProbe.baseUrl	minio base url	"http://ilum-minio:9000"

2. Kyuubi configuration in ilum-core

Feature

Added Kyuubi configuration in ilum-core helm chart. Kyuubi will allow the user to execute SQL queries on many different data sources using ILUM UI.

Values added - ilum-core

Name	Description	Value
kyuubi.enabled	Kyuubi enabled flag	true
kyuubi.url	Url of Kyuubi's rest service	http://ilum-sql-rest:10099

⚠️⚠️⚠️ Warnings

In order to properly manage SQL engines, we need to pass Kyuubi's spark configuration to ilum-core. This is done by configuring Kyuubi's spark in global.kyuubi.sparkConfig and allows the user to write one configuration which can be passed to both Kyuubi and ilum-core.

3. MongoDb uri configuration in ilum-core

Feature

Change the way mongoDb uri is passed to ilum-core. Now it is passed as a single string, which enables the user to provide more granular configuration such as authSource.

Values added - ilum-core

Name	Description	Value
mongo.uri	MongoDb connection string	mongodb://mongo:27017/ilum-default?replicaSet=rs0

Values deleted - ilum-core

Name	Reason
mongo.instances	Unnecessary after the change
mongo.replicaSetName	Unnecessary after the change

⚠️⚠️⚠️ Warnings

The mongo.uri, if set incorrectly, will cause the application to not work properly. Make sure to provide the correct connection string.

Previously the format was: mongodb://{ mongo.instances }/ilum-{ release_namespace }?replicaSet={ mongo.replicaSetName } By default in the ilum-aio chart these values were:

• mongo.instances - ilum-mongodb-0.ilum-mongodb-headless:27017,ilum-mongodb-1.ilum-mongodb-headless:27017
• mongo.replicaSetName - rs0
• release_namespace - default

4. Autopausing configuration in ilum-core

Feature

Added autopausing in ilum-core, which periodically checks if any groups are idle for the specified time and pauses the group. Each group has to have autopausing exclicitly turned on for this to take place.

Values added - ilum-core

Name	Description	Value
job.autoPause.enabled	Feature flag to enable auto pausing	true
job.autoPause.period	Interval in seconds to check the idleness groups	180
job.autoPause.idleTime	Time in seconds that the group needs to be idle to be auto paused	3600

5. Graphite exporter in ilum-aio chart

Feature

Graphite exporter in ilum AIO chart and Graphite configuration in ilum-core chart. Graphite exporter is a Prometheus exporter for metrics exported in the Graphite plaintext protocol.

Values added - graphite-exporter

Newly added whole chart, check its values on the chart's page

6. Graphite configuration in ilum-core

Feature

Added Graphite configuration in ilum-core helm chart. Graphite will allow Spark jobs to send their metrics to graphite sink, which will be scraped by Prometheus.

Values added - ilum-core

Name	Description	Value
job.graphite.enabled	Graphite enabled flag	false
job.graphite.host	Graphite host	ilum-graphite-graphite-tcp
job.graphite.port	Graphite port	9109
job.graphite.period	Interval between sending job metrics	10
job.graphite.units	Time unit	seconds

RELEASE 6.1.4

1. Jupyter default sparkmagic configuration change

Feature

Changed method of passing spark default configs to jupyter notebook, now it is passed as json string

Values added - ilum-jupyter

sparkmagic configuration parameters

Name	Description	Value
sparkmagic.config.sessionConfigs.conf	sparkmagic session spark configuration	'{ "pyRequirements": "pandas", "spark.jars.packages": "io.delta:delta-core_2.12:2.4.0", "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension", "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog"}'
sparkmagic.config.sessionConfigsDefaults.conf	sparkmagic session defaults spark configuration	'{ "pyRequirements": "pandas", "spark.jars.packages": "io.delta:delta-core_2.12:2.4.0", "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension", "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog"}'

2. Kyuubi in ilum-aio chart

Feature

Kyuubi in ilum AIO chart. Kyuubi is a distributed multi-tenant gateway providing SQL query services for data warehouses and lakehouses. It provides both JDBC and ODBC interfaces, and a REST API for clients to interact with.

Values added - ilum-kyuubi

Newly added whole chart, check its values on the chart's page

RELEASE 6.1.3

1. Jupyter configuration and persistent storage

Feature

Added extended configuration of jupyter notebook helm chart through helm values. Moreover added persitent storage to jupyter pod. All data saved in work directory will now be available after jupyter restart/update.

Values added - ilum-jupyter

pvc parameters

Name	Description	Value
pvc.annotations	persistent volume claim annotations	{}
pvc.selector	persistent volume claim selector	{}
pvc.accessModes	persistent volume claim accessModes	ReadWriteOnce
pvc.storage	persistent volume claim storage requests	4Gi
pvc.storageClassName	persistent volume claim storageClassName	``

sparkmagic configuration parameters

Name	Description	Value
sparkmagic.config.kernelPythonCredentials.username	sparkmagic python kernel username	""
sparkmagic.config.kernelPythonCredentials.password	sparkmagic python kernel password	""
sparkmagic.config.kernelPythonCredentials.auth	sparkmagic python kernel auth mode	"None"
sparkmagic.config.kernelScalaCredentials.username	sparkmagic python kernel username	""
sparkmagic.config.kernelScalaCredentials.password	sparkmagic scala kernel password	""
sparkmagic.config.kernelScalaCredentials.auth	sparkmagic scala kernel auth mode	"None"
sparkmagic.config.kernelRCredentials.username	sparkmagic r kernel username	""
sparkmagic.config.kernelRCredentials.password	sparkmagic r kernel password	""
sparkmagic.config.waitForIdleTimeoutSeconds	sparkmagic timeout waiting for idle state	15
sparkmagic.config.livySessionStartupTimeoutSeconds	sparkmagic timeout waiting for the session to start	300
sparkmagic.config.ignoreSslErrors	sparkmagic ignore ssl errors flag	false
sparkmagic.config.sessionConfigs.conf	sparkmagic session spark configuration	[pyRequirements: pandas, spark.jars.packages: io.delta:delta-core_2.12:2.4.0, spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension,spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog]
sparkmagic.config.sessionConfigs.driverMemory	sparkmagic session driver memory	1000M
sparkmagic.config.sessionConfigs.executorCores	sparkmagic session executor cores	2
sparkmagic.config.sessionConfigsDefaults.conf	sparkmagic session defaults spark configuration	[pyRequirements: pandas, spark.jars.packages: io.delta:delta-core_2.12:2.4.0, spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension,spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog]
sparkmagic.config.sessionConfigsDefaults.driverMemory	sparkmagic session defaults driver memory	1000M
sparkmagic.config.sessionConfigsDefaults.executorCores	sparkmagic session defaults executor cores	2
sparkmagic.config.useAutoViz	sparkmagic use auto viz flag	true
sparkmagic.config.coerceDataframe	sparkmagic coerce dataframe flag	true
sparkmagic.config.maxResultsSql	sparkmagic max sql result	2500
sparkmagic.config.pysparkDataframeEncoding	sparkmagic pyspark dataframe encoding	utf-8
sparkmagic.config.heartbeatRefreshSeconds	sparkmagic heartbeat refresh seconds	30
sparkmagic.config.livyServerHeartbeatTimeoutSeconds	sparkmagic livy server heartbeat timeout seconds	0
sparkmagic.config.heartbeatRetrySeconds	sparkmagic heartbeat retry seconds	10
sparkmagic.config.serverExtensionDefaultKernelName	sparkmagic server extension default kernel name	pysparkkernel
sparkmagic.config.retryPolicy	sparkmagic retry policy	configurable
sparkmagic.config.retrySecondsToSleepList	sparkmagic retry seconds to sleep list	[0.2, 0.5, 1, 3, 5]
sparkmagic.config.configurableRetryPolicyMaxRetries	sparkmagic retry policy max retries	8

RELEASE 6.1.2

1. Hive metastore in ilum-aio chart

Feature

Hive metastore in ilum AIO chart. HMS is a central repository of metadata for Hive tables and partitions in a relational database, and provides clients (including Hive, Impala and Spark) access to this information using the metastore service API. With hive metastore enabled in ilum AIO helm stack spark jobs run by ilum can be configured to autmatically access it.

Values added - ilum-hive-metastore

Newly added whole chart, check its values on chart page

Values added - ilum-core

Name	Description	Value
hiveMetastore.enabled	passing hive metastore config to ilum spark jobs flag	false
hiveMetastore.address	hive metastore address	thrift://ilum-hive-metastore:9083
hiveMetastore.warehouseDir	hive metastore warehouse directory	s3a://ilum-data/

2. Postgres extensions added

Feature

Few of ilum AIO subchars use postgresql, to make it easier to manage deployment of them we have added postgres extension resource to create postgresql databases for ilum sucharts.

Values added - ilum-aio

postgresql extensions parameters

Name	Description	Value
postgresExtensions.enabled	postgres extensions enabled flag	true
postgresExtensions.image	image to run extensions in	bitnami/postgresql:16
postgresExtensions.pullPolicy	image pull policy	IfNotPresent
postgresExtensions.imagePullSecrets	image pull secrets	[]
postgresExtensions.host	postgresql database host	ilum-postgresql-0.ilum-postgresql-hl
postgresExtensions.port	postgresql database port	5432
postgresExtensions.databasesToCreate	comma separated list of databases to create	marquez,airflow,metastore
postgresExtensions.auth.username	postgresql account username	ilum
postgresExtensions.auth.password	postgresql account password	CHANGEMEPLEASE
postgresExtensions.nodeSelector	postgresql extensions pods node selector	{}
postgresExtensions.tolerations	postgresql extensions pods tolerations	[]

3. Loki and promtail in ilum-aio chart

Feature

Loki and promtail in ilum AIO chart. Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. Promtail is an agent which ships the contents of local logs to a Grafana Loki instance. Ilum will now use loki to aggregate logs from spark job pods to be able to clean cluster resources after jobs are done. Loki and promtail are preconfigured to scrap logs only from spark pods run by ilum in order to fetch job logs after their finish.

Values added - ilum-core

log aggregation config

Name	Description	Value
global.logAggregation.enabled	ilum log aggregation flag, if enabled Ilum will fetch logs of finished kubernetes spark pods from loki	false
global.logAggregation.loki.url	loki gateway address to access logs	http://ilum-loki-gateway

Values added - ilum-aio

log aggregation - loki config

Name	Description	Value
loki.nameOverride	subchart name override	ilum-loki
loki.monitoring.selfMonitoring.enabled	self monitoring enabled flag	false
loki.monitoring.selfMonitoring.grafanaAgent.installOperator	self monitoring grafana agent operator install flag	false
loki.monitoring.selfMonitoring.lokiCanary.enabled	self monitoring canary enabled flag	false
loki.test.enabled	tests enabled flag	false
loki.loki.auth_enabled	authentication enabled flag	false
loki.loki.storage.bucketNames.chunks	storage chunks bucket	ilum-files
loki.loki.storage.bucketNames.ruler	storage ruler bucket	ilum-files
loki.loki.storage.bucketNames.admin	storage admin bucket	ilum-files
loki.loki.storage.type	storage type	s3
loki.loki.s3.endpoint	s3 storage endpoint	http://ilum-minio:9000
loki.loki.s3.region	s3 storage endpoint	us-east-1
loki.loki.s3.secretAccessKey	s3 storage secret access key	minioadmin
loki.loki.s3.accessKeyId	s3 storage access key id	minioadmin
loki.loki.s3.s3ForcePathStyle	s3 storage path style access flag	true
loki.loki.s3.insecure	s3 storage insecure flag	true
loki.loki.compactor.retention_enabled	logs retention enabled flag	true
loki.loki.compactor.deletion_mode	deletion mode	filter-and-delete
loki.loki.compactor.shared_store	shared store	s3
loki.loki.limits_config.allow_deletes	allow logs deletion flag	true

log aggregation - loki config

Name	Description	Value
promtail.config.clients[0].url	first client url	http://ilum-loki-write:3100/loki/api/v1/push
promtail.snippets.pipelineStages[0].match.selector	pipeline stage to drop non ilum logs selector	{ilum_logAggregation!="true"}
promtail.snippets.pipelineStages[0].match.action	pipeline stage to drop non ilum logs action	drop
promtail.snippets.pipelineStages[0].match.drop_counter_reason	pipeline stage to drop non ilum logs drop_counter_reason	non_ilum_log
promtail.snippets.extraRelabelConfigs[0].action	relabel config to keep ilum pod labels action	labelmap
promtail.snippets.extraRelabelConfigs[0].regex	relabel config to keep ilum pod labels regex	__meta_kubernetes_pod_label_ilum(.*)
promtail.snippets.extraRelabelConfigs[0].replacement	relabel config to keep ilum pod labels replacement	ilum${1}
promtail.snippets.extraRelabelConfigs[1].action	relabel config to keep spark pod labels action	labelmap
promtail.snippets.extraRelabelConfigs[1].regex	relabel config to keep spark pod labels regex	__meta_kubernetes_pod_label_spark(.*)
promtail.snippets.extraRelabelConfigs[1].replacement	relabel config to keep spark pod labels replacement	spark${1}

RELEASE 6.1.1

1. Added health checks for ilum interactive jobs

Feature

To prevent situations with unexpected crushes of ilum groups we added healthchecks to make sure they work as they should.

Values added - ilum-core

ilum-job parameters

Name	Description	Value
job.healthcheck.enabled	spark interactive jobs healthcheck enabled flag	true
job.healthcheck.interval	spark interactive jobs healthcheck interval in seconds	300
job.healthcheck.tolerance	spark interactive jobs healthcheck response time tolerance in seconds	120

2. Parameterized replica scale for ilum scalable services

Feature

The configuration of the number of replicas for ilum scalable services was extracted to helm values.

Values added - ilum-core

ilum-core common parameters

Name	Description	Value
replicaCount	number of ilum-core replicas	1

Values added - ilum-ui

ilum-ui common parameters

Name	Description	Value
replicaCount	number of ilum-ui replicas	1

RELEASE 6.1.0

1. Deleted unneeded parameters from ilum cluster wasbs storage

Feature

WASBS storage containers no longer needs to have sas token porvided in helm values as it turned out to be unnecessary

Values deleted - ilum-core

wasbs storage parameters

Name	Reason
kubernetes.wasbs.sparkContainer.name	Moved to kubernetes.wasbs.sparkContainer value
kubernetes.wasbs.sparkContainer.sasToken	Turned out to be unnecessary
kubernetes.wasbs.dataContainer.name	Moved to kubernetes.wasbs.dataContainer value
kubernetes.wasbs.dataContainer.sasToken	Turned out to be unnecessary

Values added - ilum-core

wasbs storage parameters

Name	Description	Value
kubernetes.wasbs.sparkContainer	default kubernetes cluster WASBS storage container name to store spark resources	ilum-files
kubernetes.wasbs.dataContainer	default kubernetes cluster WASBS storage container name to store ilum tables	ilum-tables

2. Added init containers to check service availability

Feature

To make Ilum deployment more gracefully, from now on Ilum containers have containers waiting for the availability of the services they depend on.

Values added - ilum-core

Name	Description	Value
mongo.statusProbe.enabled	mongo status probe enabled flag	true
mongo.statusProbe.image	init container that waits for mongodb to be available image	mongo:7.0.5
kafka.statusProbe.enabled	kafka status probe enabled flag	true
kafka.statusProbe.image	init container that waits for kafka to be available image	bitnami/kafka:3.4.1
historyServer.statusProbe.enabled	ilum history server ilum-core status probe enabled flag	true
historyServer.statusProbe.image	ilum history server init container that waits for ilum-core to be available image	curlimages/curl:8.5.0

Values added - ilum-livy-proxy

Name	Description	Value
statusProbe.enabled	ilum-core status probe enabled flag	true
statusProbe.image	init container that waits for ilum-core to be available image	curlimages/curl:8.5.0

Values added - ilum-ui

Name	Description	Value
statusProbe.enabled	ilum-core status probe enabled flag	true
statusProbe.image	init container that waits for ilum-core to be available image	curlimages/curl:8.5.0

3. Parameterized kafka producers in ilum-core chart

Feature

In kafka communication mode ilum interactive jobs responses to interactive job instances using kafka producers. With newly added helm values kafka producer can be adapted to match user needs.

Values added - ilum-core

kafka parameters

Name	Description	Value
kafka.maxPollRecords	kafka max.poll.records parameter for ilum jobs kafka consumer, it determines how much requests ilum-job kafka consumer will fetch with each poll	500
kafka.maxPollInterval	kafka max.poll.interval.ms parameter for ilum jobs kafka consumer, it determines the maximum delay between invocations of poll, which in ilum-job context means time limit for processing requests fetched in poll	60000

RELEASE 6.1.0-RC1

1. added support for service annotations

Feature

Ilum helm charts services annotations may now be configured through helm values

Values added - ilum-core

service parameters

Name	Description	Value
service.annotations	service annotations	{}
grpc.service.annotations	grpc service annotations	{}
historyServer.service.annotations	history server service annotations	{}

Values added - ilum-jupyter

service parameters

Name	Description	Value
service.annotations	service annotations	{}

Values added - ilum-livy-proxy

service parameters

Name	Description	Value
service.annotations	service annotations	{}

Values added - ilum-ui

service parameters

Name	Description	Value
service.annotations	service annotations	{}

Values added - ilum-zeppelin

service parameters

Name	Description	Value
service.annotations	service annotations	{}

2. Pulled out security oauth2 parameters to global values

Feature

Ilum security oauth2 configuration is now being set through global values

Values added - ilum-aio

security parameters

Name	Description	Value
global.security.oauth2.clientId	oauth2 client ID	``
global.security.oauth2.issuerUri	oauth2 URI that can either be an OpenID Connect discovery endpoint or an OAuth 2.0 Authorization Server Metadata endpoint defined by RFC 8414	``
global.security.oauth2.audiences	oauth2 audiences	``
global.security.oauth2.clientSecret	oauth2 client secret	``

Values deleted - ilum-core

security parameters

Name	Reason	Value
security.oauth2.clientId	oauth2 security parameters are now configured through global values	``
security.oauth2.issuerUri	oauth2 security parameters are now configured through global values	``

3. Runtime environment variables for frontend

Feature

Configuration for frontend environment variables throuhg helm ui values.

Values added - ilum-ui

runtime variables

Name	Description	Value
runtimeVars.defaultConfigMap.enabled	default config map for frontend runtime environment variables	true
runtimeVars.debug	debug logging flag	false
runtimeVars.backenUrl	ilum-core backend url	http://ilum-core:9888
runtimeVars.historyServerUrl	url of history server ui	http://ilum-history-server:9666
runtimeVars.jupyterUrl	url of jupyter ui	http://ilum-jupyter:8888
runtimeVars.airflowUrl	url of airflow ui	http://ilum-webserver:8080
runtimeVars.minioUrl	url of minio ui	http://ilum-minio:9001
runtimeVars.mlflowUrl	url of mlflow ui	http://mlflow:5000
runtimeVars.historyServerPath	ilum-ui proxy path to history server ui	/external/history-server/
runtimeVars.jupyterPath	ilum-ui proxy path to jupyter ui	/external/jupyter/lab/tree/work/IlumIntro.ipynb
runtimeVars.airflowPath	ilum-ui proxy path to airflow ui	/external/airflow/
runtimeVars.dataPath	ilum-ui proxy path to minio ui	/external/minio/
runtimeVars.mlflowPath	ilum-ui proxy path to mlflow ui	/external/mlflow/

Values deleted - ilum-ui

Name	Reason
debug	moved to runtimeVars section
backenUrl	moved to runtimeVars section
historyServerUrl	moved to runtimeVars section
jupyterUrl	moved to runtimeVars section
airflowUrl	moved to runtimeVars section

4. Kube-prometheus-stack in ilum-aio chart

Feature

Kube prometheus stack in ilum AIO chart. Preconfigured to automatically work wiht ilum deployment in order to collect metrics of ilum pods and spark jobs run by ilum. Ilum provides prometheus service monitors to autoamtically scrape metrics from spark driver pods run by ilum and ilum backend services. Additionally ilum_aio chart provides built-in grafana dashboards that can be found in Ilum folder.

Values added - ilum-aio

kube-prometheus-stack variables - for extended configuration check kube-prometheus stack helm chart

Name	Description	Value
kube-prometheus-stack.enabled	kube-prometheus-stack enabled flag	false
kube-prometheus-stack.releaseLabel	kube-prometheus-stack flag to watch resource only from ilum_aio release	true
kube-prometheus-stack.kubeStateMetrics.enabled	kube-prometheus-stack Component scraping kube state metrics enabled flag	false
kube-prometheus-stack.nodeExporter.enabled	kube-prometheus-stack node exporter daemon set deployment flag	false
kube-prometheus-stack.alertmanager.enabled	kube-prometheus-stack alert manager flag	false
kube-prometheus-stack.grafana.sidecar.dashboards.folderAnnotation	kube-prometheus-stack, If specified, the sidecar will look for annotation with this name to create folder and put graph here	grafana_folder
kube-prometheus-stack.grafana.sidecar.dashboards.provider.foldersFromFilesStructure	kube-prometheus-stack, allow Grafana to replicate dashboard structure from filesystem	true

Values added - ilum-core

Name	Description	Value
job.prometheus.enabled	prometheus enabled flag, If true spark jobs run by Ilum will share metrics in prometheus format	true

5. Marquez OpenLineage in ilum-aio chart

Feature

Marquez OpenLineage in ilum AIO chart. Marquez enables consuming, storing, and visualizing OpenLineage metadata from across an organization, serving use cases including data governance, data quality monitoring, and performance analytics. With marquez enabled in ilum AIO helm stack spark job run by Ilum will share lineage information with marquez backend. Marquez web interface visualize data lienage information collected from spark jobs and it is accesible through ilum UI as iframe.

Values added - ilum-aio

Name	Description	Value
global.lineage.enabled	marquez enabled flag	false

Values added - ilum-core

Name	Description	Value
job.openLineage.transport.type	marquez communication type	http
job.openLineage.transport.serverUrl	marquez backend url	http://ilum-marquez:9555/
job.openLineage.transport.endpoint	marquez backend endpoint	/external/lineage/api/v1/lineage

Values added - ilum-marquez

Newly added whole chart, check its values on chart page

Values added - ilum-ui

Name	Description	Value
runtimeVars.lineageUrl	url to provide marquez openlineage UI iframe	http://ilum-marquez-web:9444
runtimeVars.lineagePath	ilum-ui proxy path to marquez openlineage UI	/external/lineage/

RELEASE 6.0.3

1. Parameterized kafka producers max.request.size parameter in ilum-core chart

Feature

In kafka communication mode ilum interactive jobs responses to interactive job instances using kafka producers. With newly added helm value max.request.size kafka producer parameter can be adapted to match responses size needs.

Values added - ilum-core

kafka parameters

Name	Description	Value
kafka.requestSize	kafka max.request.size parameter for ilum jobs kafka producers	20000000

RELEASE 6.0.2

1. Support for hdfs, gcs and azure blob storage in ilum-core chart

Feature

Ilum cluster no longer has to be attached to s3 storage, from now default cluster can be configured to use hdfs, gcs or azure blob as storage as well. It can be achieved using newly added values in ilum-core helm chart.

Values deleted - ilum-core

Name	Reason
kubernetes.s3.bucket	From now on two separated buckets must be set with new values: kubernetes.s3.sparkBucket, kubernetes.s3.dataBucket

Values added - ilum-core

kubernetes storage parameters

Name	Description	Value
kubernetes.upgradeClusterOnStartup	default kubernetes cluster upgrade from values in config map flag	false
kubernetes.storage.type	default kubernetes cluster storage type, available options: s3, gcs, wasbs, hdfs	s3

s3 kubernetes storage parameters

Name	Description	Value
kubernetes.s3.host	default kubernetes cluster S3 storage host to store spark resources	s3
kubernetes.s3.port	default kubernetes cluster S3 storage port to store spark resources	7000
kubernetes.s3.sparkBucket	default kubernetes cluster S3 storage bucket to store spark resources	ilum-files
kubernetes.s3.dataBucket	default kubernetes cluster S3 storage bucket to store ilum tables	ilum-tables
kubernetes.s3.accessKey	default kubernetes cluster S3 storage access key to store spark resources	""
kubernetes.s3.secretKey	default kubernetes cluster S3 storage secret key to store spark resources	""

gcs kubernetes storage parameters

Name	Description	Value
kubernetes.gcs.clientEmail	default kubernetes cluster GCS storage client email	""
kubernetes.gcs.sparkBucket	default kubernetes cluster GCS storage bucket to store spark resources	"ilum-files"
kubernetes.gcs.dataBucket	default kubernetes cluster GCS storage bucket to store ilum tables	"ilum-tables"
kubernetes.gcs.privateKey	default kubernetes cluster GCS storage private key to store spark resources	""
kubernetes.gcs.privateKeyId	default kubernetes cluster GCS storage private key id to store spark resources	""

wasbs kubernetes storage parameters

Name	Description	Value
kubernetes.wasbs.accountName	default kubernetes cluster WASBS storage account name	""
kubernetes.wasbs.accessKey	default kubernetes cluster WASBS storage access key to store spark resources	""
kubernetes.wasbs.sparkContainer.name	default kubernetes cluster WASBS storage container name to store spark resources	"ilum-files"
kubernetes.wasbs.sparkContainer.sasToken	default kubernetes cluster WASBS storage container sas token to store spark resources	""
kubernetes.wasbs.dataContainer.name	default kubernetes cluster WASBS storage container name to store ilum tables	"ilum-tables"
kubernetes.wasbs.dataContainer.sasToken	default kubernetes cluster WASBS storage container sas token to store ilum tables	""

hdfs kubernetes storage parameters

Name	Description	Value
kubernetes.hdfs.hadoopUsername	default kubernetes cluster HDFS storage hadoop username	""
kubernetes.hdfs.config	default kubernetes cluster HDFS storage dict of config files with name as key and base64 encoded content as value	""
kubernetes.hdfs.sparkCatalog	default kubernetes cluster HDFS storage catalog to store spark resources	"ilum-files"
kubernetes.hdfs.dataCatalog	default kubernetes cluster HDFS storage catalog to store ilum-tables	"ilum-tables"
kubernetes.hdfs.keyTab	default kubernetes cluster HDFS storage keytab file base64 encoded content	""
kubernetes.hdfs.principal	default kubernetes cluster HDFS storage principal name	""
kubernetes.hdfs.krb5	default kubernetes cluster HDFS storage krb5 file base64 encoded content	""
kubernetes.hdfs.trustStore	default kubernetes cluster HDFS storage trustStore file base64 encoded content	""
kubernetes.hdfs.logDirectory	default kubernetes cluster HDFS storage directory absolute path to store eventLog for history server	""

Important! Make sure S3/GCS buckets or WASBS containers are already created and reachable!

2. Added spark history server to ilum-core helm chart

Feature

Spark history server can be deployed from now on along with ilum-core. History server config is being passed to every spark job ilum runs. History server UI can now be accesesed by ilum UI. If enabled it will use default kubernetes cluster storage configured with kubernetes.[STORAGE_TYPE].[PARAMETER] values as eventLog storage.

Values added - ilum-core

history server parameters

Name	Description	Value
historyServer.enabled	spark history server flag	true
historyServer.image	spark history server image	ilum/spark-launcher:spark-3.5.3
historyServer.address	spark history server address	http://ilum-history-server:9666
historyServer.pullPolicy	spark history server image pull policy	IfNotPresent
historyServer.imagePullSecrets	spark history server image pull secrets	[]
historyServer.parameters	spark history server custom spark parameters	[]
historyServer.resources	spark history server pod resources	limits: memory: "500Mi" requests: memory: "300Mi"
historyServer.service.type	spark history server service type	ClusterIP
historyServer.service.port	spark history server service port	9666
historyServer.service.nodePort	spark history server service nodePort	""
historyServer.service.clusterIP	spark history server service clusterIP	""
historyServer.service.loadBalancerIP	spark history server service loadbalancerIP	""
historyServer.ingress.enabled	spark history server ingress flag	false
historyServer.ingress.version	spark history server ingress version	"v1"
historyServer.ingress.className	spark history server ingress className	""
historyServer.ingress.host	spark history server ingress host	"host"
historyServer.ingress.path	spark history server ingress path	"/(.*)"
historyServer.ingress.pathType	spark history server ingress pathType	Prefix
historyServer.ingress.annotations	spark history server annotations	nginx.ingress.kubernetes.io/rewrite-target: /$1 nginx.ingress.kubernetes.io/proxy-body-size: "600m" nginx.org/client-max-body-size: "600m"

Warnings

1. Make sure HDFS logDirectory (helm value kubernetes.hdfs.logDirectory) is absolute path of configured sparkCatalog with /ilum/logs suffix! Eg for kubernetes.hdfs.sparkCatalog=spark-catalog put hdfs://name-node/user/username/spark-catalog/ilum/logs

3. Job retention in ilum-core chart

Feature

Ilum jobs will be deleted after the configured retention period expires

Values added - ilum-core

job retention parameters

Name	Description	Value
job.retain.hours	spark jobs retention hours limit	168