Production Deployment Guide

This comprehensive guide provides detailed instructions for deploying Ilum in production environments with enhanced security, namespace separation, and multiple configuration options to meet diverse operational requirements.

Overview
Kubernetes Prerequisites
Architecture
High Availability Configuration
Failure Domains & Resilience
Dynamic Resource Scaling
Namespace Separation Strategy
Security Configuration
Dependency Deployment
Pre-configured Stack Options
Helm Values Configuration
Installation Instructions
Post-Installation Configuration
Troubleshooting

Overview

For production environments, it's strongly recommended to deploy critical dependencies in separate namespaces to achieve:

Enhanced Security: Namespace-level isolation and RBAC policies
Resource Management: Independent resource quotas and limits
Operational Excellence: Simplified maintenance and upgrades
Compliance: Meeting organizational separation requirements
Scalability: Independent scaling of components

Critical Components for Namespace Separation

The following components should be deployed in separate namespaces for production:

PostgreSQL: Primary metadata store for ilum-core and shared dependency for Marquez, Hive Metastore, Airflow, Superset, MLflow, and others
MongoDB: Legacy metadata store, still supported for existing deployments
Apache Kafka: Message broker and communication layer (required for ilum-core HA)
MinIO: Object storage for Spark applications and data

Kubernetes Prerequisites

Ilum has been extensively tested across all leading Kubernetes environments, ensuring compatibility with a variety of deployment scenarios:

Supported Platforms

Lightweight Distributions: k3s, Rancher, MicroK8s
Bare-metal Clusters: Self-managed Kubernetes installations
Managed Services:
- Google Kubernetes Engine (GKE)
- Amazon Elastic Kubernetes Service (EKS)
- Azure Kubernetes Service (AKS)
- DigitalOcean Kubernetes
- Red Hat OpenShift

Minimum Suggested Requirements

Component	Requirement
Kubernetes Version	1.20+
CPU	8 cores minimum, 16+ recommended
Memory	16GB minimum, 32GB+ recommended

Air-gapped (Offline) Environments

For air-gapped installations, refer to our comprehensive Air-gapped Installation Guide.

Testing vs Production

Minikube is used throughout our documentation for demonstration purposes but is not suitable for production due to limitations in scalability, resource management, and high availability.

Architecture

Components and modules

The Ilum platform consists of three core services and a curated set of optional modules:

ilum-core: Main backend service. Hosts the public REST API, job and cluster orchestration, multi-engine SQL execution, security, and lineage capture.
ilum-ui: React-based web frontend. Hosts the SQL Editor, Table Explorer, Lineage view, Workloads management, and the Modules registry.
ilum-api: Module-management microservice. Drives Helm-based install, upgrade, and disable of optional Ilum modules at runtime via cluster-scoped RBAC. Future releases will extend ilum-api with Model Context Protocol (MCP) capabilities and open APIs for third-party extension.

Optional modules include execution engines (Trino, Apache Flink), additional catalogs (Project Nessie, Unity Catalog, DuckLake), notebooks (JupyterHub, Zeppelin), orchestration (Airflow, Kestra, Mage, n8n, NiFi), BI tools (Superset, Streamlit), AI and ML stacks (MLflow, LangFuse), and observability components (Kube Prometheus stack, Loki, Promtail).

PostgreSQL

PostgreSQL is the primary metadata store for ilum-core (accessed via R2DBC with jOOQ-generated SQL DSL). It is also used by shared dependencies including Marquez, Hive Metastore, Airflow, Superset, MLflow, Hydra, Gitea, n8n, and Kestra to store their respective metadata.

PostgreSQL is enabled in Ilum by default.

If you want to control whether PostgreSQL is enabled or not, you can use the helm value postgresql.enabled. For example, to disable it, you can add --set postgresql.enabled=false to your installation command.

MongoDB

MongoDB is supported as a legacy metadata store for ilum-core. New deployments should use PostgreSQL; existing MongoDB-backed deployments remain fully supported. Ilum ships migration tooling (M001 through M009 scripts) for moving from MongoDB to PostgreSQL.

Ilum automatically creates all necessary databases and collections during the startup process when MongoDB is in use.

Apache Kafka

Apache Kafka serves as Ilum's communication layer, facilitating interaction between Ilum-Core and Spark jobs, as well as between different Ilum-Core instances when scaled. It is critical to ensure Apache Kafka brokers are accessible by both Ilum-Core and Spark jobs, especially when Spark jobs are launched on a different Kubernetes cluster.

Ilum utilizes Kafka to carry out communication using several topics, all created during Ilum's startup. Therefore, users don't need to manage these topics manually.

MinIO

Ilum uses MinIO as the storage layer for Spark application components. All files (including jars, configurations, data files) needed for the operation of Spark components (driver, executors) are stored and made available for download via MinIO.

MinIO implements the S3 interface, which also enables it to store input/output data.

Ilum-Livy-proxy

Ilum-Livy proxy is our implementation of Livy Api, that integrates spark code with Ilum Groups in services such as Jupyter, Zeppelin, Airflow

Ilum Livy-proxy is enabled in Ilum by default.

In case you want to add or remove Ilum-Livy-proxy, you can use ilum-livy-proxy.enabled helm value to manage it. For example: --set ilum-livy-proxy.enabled=false to disable it.

Jupyter

Jupyter is a Notebook - sophisticated development environment which allows you to have code, charts, explanations and more in one executable document.

Jupyter is enabled in Ilum by default.

However, in case you want to control whether it is enabled or not, you can use helm value ilum-jupyter.enabled. For example, you can add --set ilum-jupyter.enabled=false to your installation command to disable it.

Be aware, that Jupyter makes use of Ilum-Livy-proxy to integrate with Ilum Groups. Therefore, you should enable it as well: --set ilum-livy-proxy.enabled=true

If you want to access the Jupyter UI, you can do it by:

using Ilum UI: go to Modules > Jupyter
configuring an ingress
using the port-forward command kubectl port-forward svc/ilum-jupyter 8888:8888

Apache Zeppelin

Zeppelin is a Notebook - sophisticated development environment which allows you to have code, charts, explanations and more in one executable document.

Please be aware, that Zeppelin notebook is not bundled in ilum package by default. If you want to run this service, add --set ilum-zeppelin.enabled=true to your installation command.

Be aware, that Zeppelin makes use of Ilum-Livy-proxy to integrate with Ilum Groups. Therefore, you should enable it as well: --set ilum-livy-proxy.enabled=true

If you want to access the Zeppelin UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-zeppelin 8080:8080

Hive Metastore

Hive Metastore is a metadata storage used to store your Spark catalogs (Spark tables, databases, views, and more) in a database instead of runtime memory. You can view these schemas later on the Table Explorer page.

info

Hive Metastore is not enabled in Ilum by default.

To enable the Hive Metastore bundled instance, set the following values in your Helm installation command:

ilum-core:
  metastore:
    enabled: true
    type: hive

ilum-hive-metastore:
  enabled: true

note

Hive Metastore uses PostgreSQL database to store metadata. You can read about Postgres in Ilum below.

Project Nessie

Nessie is a transactional catalog for your data. It was inspired by Git and is designed to support a wide range of data-lake tooling. It works best with Apache Iceberg tables.

To learn more about Nessie, visit the Nessie documentation page.

info

Project Nessie is not enabled in Ilum by default. To enable it, set the following values in your Helm installation command:

ilum-core:
  metastore:
    enabled: true
    type: nessie

nessie:
  enabled: true

Ilum SQL (Kyuubi Gateway)

note

Ilum SQL is not enabled in Ilum by default.

Ilum SQL is the multi-engine SQL gateway built on Apache Kyuubi. It exposes a single JDBC and REST entry point for queries that route to Apache Spark, Trino, DuckDB, or Apache Flink based on the engine selected for each query, or chosen by the automatic engine router.

To enable Ilum SQL, add --set ilum-sql.enabled=true to enable the SQL execution host and --set ilum-core.sql.enabled=true to enable the SQL features inside Ilum itself.

Read more about the SQL Editor on the SQL Editor page.

Trino

Note: Trino is not enabled in Ilum by default. To enable it, add --set trino.enabled=true to deploy a built-in Trino distribution.

Trino is a distributed SQL query engine well suited to interactive analytics on medium-to-large datasets and for federated queries across multiple data sources. It complements Spark for workloads that benefit from low-latency response times.

Once enabled, Trino is reachable through the Ilum SQL gateway and selectable from the Engine Selector in the SQL Editor. Read more on the SQL Editor page.

DuckDB and DuckLake

DuckDB is enabled by default and provides single-node SQL execution for small-to-medium data, ad-hoc exploration, and DuckLake-managed tables. It runs in-process with ilum-core and is selectable from the Engine Selector in the SQL Editor.

DuckLake is the DuckDB-native catalog, enabled by default, with table data stored in MinIO (or any configured S3-compatible backend). No additional Helm values are required to use DuckDB or DuckLake.

Apache Flink

Apache Flink support is available for Enterprise deployments as a Beta feature. Flink is exposed through the Kyuubi SQL gateway for low-latency stream processing. Contact Ilum for enablement details.

n8n

Note: n8n is not enabled in Ilum by default. To enable it, add --set ilum-n8n.enabled=true to enable a built-in n8n distribution.

n8n is a fair-code workflow automation platform with native AI capabilities.

Apache Airflow

Apache Airflow is a powerful platform for orchestrating and managing data workflows. To read more about Airflow in Ilum, visit the Airflow documentation page.

note

Airflow is not enabled in the Ilum package by default. To deploy Airflow, add --set airflow.enabled=true to your installation command.

Once enabled, Airflow will appear in the Ilum UI under the Modules section.

Apache Livy

Airflow can leverage Ilum’s Livy proxy to easily create jobs within Ilum. For more details, see the Livy proxy section above.

Marquez

Marquez is an open-source metadata management tool that focuses on capturing, aggregating, and visualizing the lineage of data assets within an organization’s data ecosystem. It tracks how datasets are produced and consumed by different jobs and provides a central view of these dependencies.

Please be aware that Marquez is not bundled in Ilum package by default. If you want to run this service, add --set global.lineage.enabled=true to your installation command.

Take into account that Marquez makes use of PostgreSQL database to store the metadata. You can read about it below.

Additionally, if you wish to use Marquez’s web client instead of Ilum’s UI, enable the default web client with --set ilum-marquez.web.enabled=true and set up one of the access methods:

use the port-forward command kubectl port-forward svc/ilum-marquez-web 9444:9444
configure an ingress

Read more about Marquez and Ilum Lineage here

Kestra

Kestra is an open-source data orchestration platform designed for orchestrating and automating data pipelines and business workflows. You can read about it here.

Kestra is not enabled in Ilum by default. To enable it, add --set kestra.enabled=true to your installation command.

Kestra uses PostgreSQL database to store the data about the jobs and tasks and Minio for general file storage

Mage

Mage is an open-source data engineering platform that simplifies the process of building, deploying, and maintaining data pipelines. It provides a user-friendly interface for creating data workflows, integrating with various data sources, and managing data transformations.

To read about Mage next to Ilum, visit the documentation page.

info

Mage is not enabled in Ilum by default. To enable it, add --set mageai.enabled=true to your installation command.

note

Ilum deploys Mage OSS, which is the open-source version of Mage, and does not include the commercial features available in Mage Pro (Cloud).

NiFi

Apache NiFi is a software project for building data processing pipelines. It provides a user interface for creating, managing, and deploying data processing pipelines.

To read about NiFi next to Ilum, visit the documentation page.

info

NiFi is not enabled in Ilum by default. To enable it, add --set nifi.enabled=true to your installation command.

Streamlit

Streamlit is a library for creating beautiful, performant, and scalable data apps in Python. It is used to build custom data apps accessible from the Ilum UI.

To read about Streamlit next to Ilum, visit the documentation page.

info

Streamlit is not enabled in Ilum by default. To enable it, add --set streamlit.enabled=true to your installation command.

Additionally, can provide a docker image with your streamlit application running in it. To see how to do it, visit our documentation page.

Kube Prometheus Stack

Kube Prometheus Stack includes Prometheus, Grafana and other tools for monitoring your data infrastructure

Please be aware, that Kube Prometheus Stack is not bundled in ilum package by default. If you want to run this service, add --set kube-prometheus-stack.enabled=true to your installation command.

If you are upgrading an existing Ilum Helm chart that previously did not have the Kube Prometheus Stack enabled, you must first install the required Prometheus Custom Resource Definitions (CRDs) before proceeding with the upgrade. To do this, run the following commands:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusagents.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_scrapeconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

If you want to access the Prometheus UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/prometheus-operated 9090:9090

If you want to access the Grafana UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-grafana 8080:80

Loki and Promtail

Loki is used to gather and manage logs of your data infrastructure. Promtail is used as an agent that pushes logs into Loki

Please be aware, that Loki is not enabled in Ilum by default. If you want to run this service, add --set global.logAggregation.loki.enabled=true to your installation command.

Promtail also is not enabled in Ilum by default. To enable it add --set global.logAggreagtion.promtail.enabled=true to your installation command

If you want to access Loki and run Loki Queries, you can configure an ingress or use the port-forward command kubectl port-forward svc/ilum-loki-read 3100:3100 for read queries and kubectl port-forward svc/ilum-loki-write 3100:3100 for write queries. You can also use service ilum-loki-gateway to link grafana to loki

Production Architecture Overview

Recommendations on which optionally deployed components to place in which namespace, whether in the one belonging to the Ilum release or in separate, dedicated ones.

┌─────────────────────────┐     ┌────────────────────────────┐
│   ILUM                  │     │   Dependencies             │
│   components            │     │   separated                │
│   namespace             │     │   namespaces               │
├─────────────────────────┤     ├────────────────────────────┤
│ • Ilum Core **          │     │ • PostgreSQL **            │
│ • Ilum UI **            │     │ • MongoDB (legacy)         │
│ • Ilum API **           │     │ • Kafka **                 │
│ • Ilum Livy-Proxy *     │     │ • MinIO *                  │
│ • Ilum Jupyter *        │     │ • Kube-Prometheus-Stack *  │
│ • Ilum Zeppelin         │     │ • Loki & Promtail *        │
│ • Ilum Hive Metastore * │     └────────────────────────────┘
│ • Ilum Kyuubi *         │
│ • Trino                 │
│ • Marquez *             │
│ • Unity Catalog         │
│ • Nessie                │
│ • Airflow               │
│ • MLflow                │
│ • LangFuse              │
│ • Superset              │
│ • Streamlit             │
│ • Gitea *               │
│ • N8N                   │
│ • Kestra                │
│ • Mage                  │
│ • OpenLDAP              │
└─────────────────────────┘
**  Mandatory components for production release
*   Recommended components for production release

Network Communication

Inter-namespace Communication: Configured via Kubernetes Services and NetworkPolicies
Service Discovery: DNS-based service resolution
Security: TLS encryption for all inter-service communication

High Availability Configuration

For production deployments, critical services should run with multiple replicas to eliminate single points of failure. The following table shows recommended configurations:

Component	Minimum Replicas	Recommended	Helm Value	Notes
ilum-core	2	3	`ilum-core.replicaCount: 3`	Stateless; requires Kafka communication mode for HA
ilum-api	2	2	`ilum-api.replicaCount: 2`	Stateless module-management microservice; cluster-scoped RBAC
PostgreSQL	2	3	`postgresql.replicaCount: 3`	Primary metadata store; primary + standby with streaming replication
MongoDB (legacy)	3	3	`mongodb.replicaCount: 3`	Only for legacy deployments; replica set with automatic failover
Apache Kafka	3	3	`kafka.replicaCount: 3`	Required for HA communication; minimum 3 for quorum
MinIO	4	4	`minio.replicas: 4`	Erasure coding requires minimum 4 nodes
Hive Metastore	2	2	`ilum-hive-metastore.replicaCount: 2`	Stateless; connects to shared PostgreSQL backend
Kyuubi (ilum-sql)	2	2	`ilum-sql.replicaCount: 2`	SQL gateway; stateless

Example Helm values for a fully HA deployment:

ilum-core:
  replicaCount: 3

mongodb:
  replicaCount: 3

kafka:
  replicaCount: 3

minio:
  replicas: 4

postgresql:
  replicaCount: 3

warning

High Availability for ilum-core requires Apache Kafka as the communication type. The default gRPC communication does not support HA. Set ilum-core.communication.type: kafka to enable HA mode.

Failure Domains & Resilience

Ilum's architecture is designed to eliminate single points of failure through several mechanisms:

Stateless Core Design

The ilum-core component is fully stateless. It persists no local data and can recover its entire state from PostgreSQL (or MongoDB on legacy deployments) and Kafka after a crash or restart. This means any ilum-core replica can handle any request, enabling seamless horizontal scaling and zero-downtime rolling updates.

The ilum-api module-management microservice is also stateless. It can be scaled horizontally and tolerates pod restarts without losing in-flight Helm operations, thanks to a long termination grace period.

Pod Anti-Affinity

Configure Kubernetes pod anti-affinity rules to ensure replicas of critical services are spread across different nodes and availability zones:

ilum-core:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values:
                    - ilum-core
            topologyKey: kubernetes.io/hostname

Namespace-Level Isolation

Deploying dependencies (MongoDB, Kafka, MinIO, PostgreSQL) in separate namespaces provides blast radius containment. A failure in one component's namespace does not cascade to others. See Namespace Separation Strategy for details.

Leader Election

For components that require coordination (such as the Spark History Server or scheduled maintenance tasks), Kubernetes-native leader election ensures that exactly one instance acts as the primary while others stand by for automatic failover.

Dynamic Resource Scaling

Ilum supports multiple layers of dynamic scaling to match compute resources with workload demands.

Spark Dynamic Allocation

Enable Spark's built-in dynamic allocation to automatically adjust the number of executors based on workload:

# Cluster-level Spark defaults in ilum
spark.dynamicAllocation.enabled: "true"
spark.dynamicAllocation.minExecutors: "1"
spark.dynamicAllocation.maxExecutors: "20"
spark.dynamicAllocation.executorIdleTimeout: "60s"
spark.dynamicAllocation.schedulerBacklogTimeout: "5s"

With dynamic allocation, idle executors are released back to the cluster, freeing resources for other jobs. New executors are added automatically when tasks queue up.

Kubernetes Horizontal Pod Autoscaler (HPA)

For always-on services like Trino coordinators or ilum-core, use Kubernetes HPA to scale based on CPU or memory utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ilum-core-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ilum-core
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Cluster Autoscaler Integration

Ilum works seamlessly with Kubernetes cluster autoscalers (available on GKE, EKS, AKS, and via Cluster API). When Spark jobs request more executor pods than the current cluster can accommodate, the cluster autoscaler provisions additional nodes automatically. When pods are released, nodes are drained and removed.

tip

Combine Spark dynamic allocation with cluster autoscaling for full elasticity: Spark scales executors based on workload, and the cluster scales nodes based on pending pods.

Auto-Pause on Idle

Ilum's interactive services (SQL Editor sessions, Jupyter notebooks) can be configured to automatically pause after a period of inactivity, releasing compute resources:

ilum-core:
  sql:
    duckdb:
      idleTimeout: 1h
  job:
    retain:
      hours: 24

Namespace Separation Strategy

Recommended Namespace Structure

# Core Ilum Components placed in ilum namespace
ilum:          # Main application services (ilum-core, ilum-ui, ilum-api)

# Critical Dependencies (Separate Namespaces)
ilum-postgresql:    # PostgreSQL databases (primary metadata store)
ilum-mongodb:       # MongoDB cluster (legacy metadata store)
ilum-kafka:         # Kafka brokers (required for ilum-core HA)
ilum-minio:         # Object storage cluster
ilum-kps:           # Kube-prometheus-stack monitoring resources
ilum-logs:          # Loki & Promtail log aggregation resources

Namespace Creation

Create all required namespaces before deployment:

⚠️ Warning: Create namespaces only for components you plan to deploy

# Ilum namespace
kubectl create namespace ilum

# Dependency namespaces
kubectl create namespace ilum-mongodb
kubectl create namespace ilum-kafka
kubectl create namespace ilum-minio
kubectl create namespace ilum-postgresql
kubectl create namespace ilum-kps
kubectl create namespace ilum-logs

Security keys

This application uses JSON Web Tokens (JWT) for authentication purposes. By default, the application employs an RSA key pair, which is randomly generated at runtime, to sign these tokens.

In its standard configuration, the application creates a fresh RSA key pair each time it starts. This approach simplifies local development and testing by automatically handling the key generation process. However, it must be emphasized that this approach is not suitable for a production environment.

The primary issue with using randomly generated keys in a production environment is the lack of persistence. Each time the application restarts, it generates a new RSA key pair, invalidating all previously issued tokens. This could lead to an abrupt and unanticipated logout for all users, disrupting user experience and potentially leading to data loss.

Generate private key

For a production environment, a stable and secure key pair should be manually generated and used consistently. This ensures that tokens remain valid across multiple application restarts, thus providing a consistent user experience.

You can generate an RSA key pair manually using tools like OpenSSL. A common command to generate a 2048-bit RSA private key is as follows:

openssl genpkey -algorithm RSA \
    -pkeyopt rsa_keygen_bits:2048 \
    -pkeyopt rsa_keygen_pubexp:65537 | \
  openssl pkcs8 -topk8 -nocrypt -outform pem > private-key.p8

The contents of the private key should look like the following:

In order to use private key as the setting ilum-core.security.jwt.privateKey, remove header and footer from the key.

Generate public key

To generate the corresponding public key, use:

openssl pkey -pubout -inform pem -outform pem -in private-key.p8 -out public-key.spki

The contents of the public key should look like the following:

In order to use public key as the setting ilum-core.security.jwt.publicKey, remove header and footer from the key.

Authentication Methods

Ilum as Identity Provider

Ilum can be deployed as an Identity Provider. With this feature, you can manage users exclusively within Ilum and authenticate them across other microservices such as Airflow, Superset, Grafana, Gitea, and Minio.

To enable Ilum's Identity Provider, add the following flags: --set global.security.hydra.enabled=true and --set global.security.hydra.uiUrl=<your-ilum-ui-domain>.

To learn more about Identity Provider configuration, visit this page

OpenLDAP

OpenLDAP is the open-source LDAP server. It is not enabled in Ilum by default. To enable it add --set openldap.enabled=true to your installation command.

If you want to access OpenLDAP you can use port-forward command kubectl port-forward svc/ilum-openldap 1389:389.

Ilum-Core is preconfigured to connect to OpenLDAP server deployed by Ilum. However you must switch security type to LDAP by adding --set ilum-core.security.type=ldap to your installation command.

To learn more about OpenLDAP and LDAP in Ilum, visit the LDAP documentation page here.

Dependency Deployment

Note: The components described below are required or recommended dependencies for Ilum. You can choose to deploy them using the provided instructions or integrate with your own existing managed services.

MongoDB

Deploy MongoDB in a dedicated namespace with high availability

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create MongoDB Values File

# mongodb-values.yaml
image:
  registry: docker.io
  repository: ilum/mongodb
  tag: 6.0.5
fullnameOverride: "ilum-mongodb"
architecture: "replicaset"
auth:
  enabled: false
volumePermissions:
  enabled: true

3. Deploy MongoDB, Ilum by default uses chart in ^13.x version but should also work with newer versions

helm install mongodb bitnami/mongodb \
  --namespace ilum-mongodb \
  --version 13.18.5 \
  --values mongodb-values.yaml

For more information about mongo deployment refer to mongodb chart details

4. Ilum AIO chart configuration values that need to be changed

# Disable bundled mongo
mongodb:
  enabled: false

ilum-core:
  mongo:
    uri: "mongodb://ilum-mongodb-0.ilum-mongodb-headless.ilum-mongodb:27017,ilum-mongodb-1.ilum-mongodb-headless.ilum-mongodb:27017/ilum?replicaSet=rs0"
         # or use your mongo instance uri

Apache Kafka

Deploy Kafka cluster in a dedicated namespace

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create Kafka Values File

# kafka-values.yaml
extraConfig: |
  auto.create.topics.enable=false
  message.max.bytes=20000000
fullnameOverride: "ilum-kafka"
listeners:
  client:
    protocol: PLAINTEXT

3. Deploy Kafka, Ilum by default uses chart in ^25.x version but should also work with newer versions

helm install kafka bitnami/kafka \
  --namespace ilum-kafka \
  --version 25.3.5 \
  --values kafka-values.yaml

For more information about kafka deployment refer to kafka chart details

4. Ilum AIO chart configuration values that need to be changed

ilum-core:
  communication:
    type: kafka
  kafka:
    address: "ilum-kafka-controller-0.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-1.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-2.ilum-kafka-controller-headless.ilum-kafka:9092"
    # or use your kafka instance address

MinIO Deployment

Deploy MinIO cluster for object storage:

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create MinIO Values File

# minio-values.yaml
extraEnvVars: |
  - name: MINIO_BROWSER_REDIRECT_URL
    value: "http://ilum-minio.ilum-minio:9001/external/minio/"
 # Environment variables for hydra
  - name: MINIO_IDENTITY_OPENID_CONFIG_URL
    value: "ILUM_UI_PROTOCOL://ILUM_UI_DOMAIN/external/hydra/.well-known/openid-configuration"
  - name: MINIO_IDENTITY_OPENID_CLIENT_ID
    value: "HYDRA_CLIENT_ID"
  - name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
    value: "HYDRA_CLIENT_SECRET"
  - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
    value: "ILUM_UI_PROTOCOL://ILUM_UI_DOMAIN/external/minio/oauth_callback"
  - name: MINIO_IDENTITY_OPENID_SCOPES
    value: "openid,profile,email"
  - name:  MINIO_IDENTITY_OPENID_CLAIM_NAME
    value: "minio_policies"
 # end of hydra related environment variables
fullnameOverride: "ilum-minio"
defaultBuckets: "ilum-files, ilum-data, ilum-tables, ilum-mlflow, ilum-kestra"
auth:
  rootUser: "minioadmin"
  rootPassword: "minioadmin"
persistence:
  size: 16Gi

3. Deploy MinIO, Ilum by default uses chart in ^15.x version but should also work with newer versions

helm install minio bitnami/minio \
  --namespace ilum-minio \
  --version 15.0.7 \
  --values minio-values.yaml

For more information about minio deployment refer to minio chart details

4. Ilum AIO chart configuration values that need to be changed

# Disable bundled minio
minio:
  enabled: false

ilum-core:
  minio:
    statusProbe:
      baseUrl: "http://ilum-minio.ilum-minio:9000"
  kubernetes:
    storage:
      type: s3
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

# If hive metastore is deployed with minio as the storage
ilum-hive-metastore:
  storage:
    metastore:
      warehouse: "s3a://ilum-data/"
    type: "s3"
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

# If trino is deployed with minio as the storage
trino:
  catalogs:
    ilum-delta: |
      connector.name=delta_lake
      delta.metastore.store-table-metadata=true
      delta.register-table-procedure.enabled=true
      hive.metastore.uri=thrift://ilum-hive-metastore:9083
      fs.native-s3.enabled=true
      s3.endpoint=http://ilum-minio.ilum-minio:9000
      s3.region=us-east-1
      s3.path-style-access=true
      s3.aws-access-key=minioadmin
      s3.aws-secret-key=minioadmin

# If Loki is deployed with s3 as the storage
loki:
  loki:
    auth_enabled: false
    storage:
      bucketNames:
        chunks: ilum-files
        ruler: ilum-files
        admin: ilum-files
      type: s3
      s3:
        endpoint: http://ilum-minio.ilum-minio:9000
        region: us-east-1
        secretAccessKey: minioadmin
        accessKeyId: minioadmin
        s3ForcePathStyle: true
        insecure: true

# If minio is deployed
minioExtensions:
  host: http://ilum-minio.ilum-minio:9000

# If airflow is deployed with minio as the logs storage
airflow:
  extraEnv: |
    - name: AIRFLOW__API__AUTH_BACKENDS
      value: "airflow.api.auth.backend.default"
    - name: MINIO_USERNAME
      value: "minioadmin"
    - name: MINIO_PASSWORD
      value: "minioadmin"
    - name: MINIO_ENDPOINT
      value: "http://ilum-minio.ilum-minio:9000"

# If mlflow is deployed
mlflow:
  externalS3:
    host: "ilum-minio.ilum-minio"

# If kestra is deployed with minio as the storage
kestra:
  configuration:
    kestra:
      storage:
        type: minio
        minio:
          endpoint: ilum-minio.ilum-minio
          port: 9000
          secure: false
          accessKey: minioadmin
          secretKey: minioadmin
          region: "default"
          bucket: ilum-kestra

PostgreSQL Deployment

Deploy PostgreSQL for metadata storage:

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create PostgreSQL Values File

# postgresql-values.yaml
fullnameOverride: ilum-postgresql
auth:
  postgresPassword: "CHANGEMEPLEASE"
  username: ilum
  password: "CHANGEMEPLEASE"

3. Deploy PostgreSQL, Ilum by default uses chart in ^13.x version but should also work with newer versions

helm install postgresql bitnami/postgresql \
  --namespace ilum-postgresql \
  --version 13.4.4 \
  --values postgresql-values.yaml

For more information about postgresql deployment refer to postgresql chart details

4. Ilum AIO chart configuration values that need to be changed

# Disable bundled PostgreSQL
postgresql:
  enabled: false

#Postgres extensions for databases auto-creation
postgresExtensions:
  host: "ilum-postgresql-hl.ilum-postgresql"
  port: 5432
  databasesToCreate: marquez,airflow,metastore,mlflow,mlflow_auth,superset,gitea,n8n,hydra,kestra
  auth:
    username: ilum
    password: "CHANGEMEPLEASE"

# Configure hydra database if hydra is enabled in ilum-core deployment
ilum-core:
  hydra:
    dns: postgres://ilum:CHANGEMEPLEASE@ilum-postgresql.ilum-postgresql:5432/hydra?sslmode=disable

# If ilum-hive-metastore is deployed
ilum-hive-metastore:
  postgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    database: metastore
    auth:
      username: ilum
      password: "CHANGEMEPLEASE"

# If lineage is enabled
ilum-marquez:
  marquez:
    db:
      image: "bitnami/postgresql:16"
      host: "ilum-postgresql-0.ilum-postgresql-hl.ilum-postgresql"
      port: 5432
      name: marquez
      user: ilum
      password: "CHANGEMEPLEASE"

# If gitea is deployed
gitea:
  initPreScript: "while ! nc -z ilum-postgresql-hl.ilum-postgresql 5432 2>/dev/null; do echo 'Waiting for Postgres...'; sleep 5; done; echo 'Postgres is ready!'"
  gitea:
    config:
      server:
        ROOT_URL: http://gitea.example.com/external/gitea
      database:
        DB_TYPE: postgres
        HOST: "ilum-postgresql-hl.ilum-postgresql:5432"
        NAME: gitea
        USER: ilum
        PASSWD: "CHANGEMEPLEASE"

# If airflow is deployed with postgres as metadata connection
airflow:
  data:
    metadataConnection:
      user: postgres
      pass: "CHANGEMEPLEASE"
      protocol: postgresql
      host: "ilum-postgresql-hl.ilum-postgresql"
      port: 5432
      db: airflow
      sslmode: disable

# If superset is deployed with postgres as the storage
superset:
  supersetNode:
    connections:
      db_host: "ilum-postgresql-hl.ilum-postgresql"
      db_port: "5432"
      db_user: ilum
      db_pass: "CHANGEMEPLEASE"
      db_name: superset

# If mlflow is deployed with postgres as the external database
mlflow:
  externalDatabase:
    dialectDriver: "postgresql"
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    user: ilum
    database: mlflow
    authDatabase: mlflow_auth
    existingSecret: "ilum-postgresql"
    existingSecretPasswordKey: "postgres-password"

# If n8n is deployed
n8n:
  db:
    type: postgresdb
  externalPostgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    username: "ilum"
    password: "CHANGEMEPLEASE"
    database: "n8n"

# If kestra is deployed
kestra:
  configuration:
    datasources:
      postgres:
        url: "jdbc:postgresql://ilum-postgresql-hl.ilum-postgresql:5432/kestra"
        driverClassName: "org.postgresql.Driver"
        username: ilum
        password: "CHANGEMEPLEASE"
  initContainers:
    - name: wait-for-postgres
      image: "bitnami/postgresql:16"
      command:
        - /bin/bash
      args:
        - -ec
        - |
          until pg_isready -h ilum-postgresql-hl.ilum-postgresql -p 5432 -U ilum -d kestra; do
            echo "Waiting for PostgreSQL server to be ready..."
            sleep 2
          done
          echo "Database is ready!"

Kube-Prometheus-Stack Deployment (If you plan to use it)

Deploy Kube-Prometheus-Stack for monitoring purposes:

1. Add Prometheus Community Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2. Create Kube-Prometheus-Stack Values File

# kps-values.yaml
nameOverride: "ilum-kps"
kubeStateMetrics:
  enabled: false
nodeExporter:
  enabled: false
alertmanager:
  enabled: false
grafana:
  nameOverride: "ilum-grafana"
  grafana.ini:
    server:
      root_url: "%(protocol)s://%(domain)s:%(http_port)s/external/grafana"
      serve_from_sub_path: "true"
    auth.generic_oauth:
      enabled: false
      name: Ilum
      allow_sign_up: true
      login_attribute_path: userId
      email_attribute_name: email
      role_attribute_path: contains(grafana_roles, 'Admin') && 'Admin' || contains(grafana_roles, 'Editor') && 'Editor' || contains(grafana_roles, 'Viewer') && 'Viewer'
      role_attribute_strict: true
  adminPassword: admin
  sidecar:
    dashboards:
      folderAnnotation: "grafana_folder"
      provider:
        foldersFromFilesStructure: "true"

3. Deploy kube-prometheus-stack, Ilum by default uses chart in ^54.x version but should also work with newer versions

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace ilum-kps \
  --version 54.2.2 \
  --values kps-values.yaml

For more information about kube-prometheus-stack deployment refer to kps chart details

4. Define prometheus PodMonitors and ServiceMonitors

You can define your own Pod and ServiceMonitors for prometheus. Example PodMonitors defined in Ilum AIO chart you can use

Ilum Pods Monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ilum-pods-monitor
  namespace: ilum-kps
  labels:
    ilum.prometheus: "true"
spec:
  selector:
    matchLabels:
      ilum.prometheus: "true"
  podMetricsEndpoints:
    - interval: 5s
      port: http
      path: /actuator/prometheus
  namespaceSelector:
    matchNames:
      - ilum

Spark Driver Pods Monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ilum-driver-pods-monitor
  namespace: ilum-kps
  labels:
    ilum.resourceType: driver
spec:
  selector:
    matchLabels:
      ilum.resourceType: driver
  podMetricsEndpoints:
    - interval: 5s
      port: spark-ui
      path: /metrics/executors/prometheus/
    - interval: 5s
      port: spark-ui
      path: /metrics/prometheus/
  namespaceSelector:
    any: true

Loki & Promtail Deployment (If you plan to use it)

Deploy Loki & Promtail for logs aggregation:

1. Add Grafana Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

2. Create Loki and Promtail Values File

# loki-values.yaml
nameOverride: ilum-loki
monitoring:
  selfMonitoring:
    enabled: false
    grafanaAgent:
      installOperator: false
    lokiCanary:
      enabled: false
test:
  enabled: false
loki:
  auth_enabled: false
  storage:
    bucketNames:
      chunks: ilum-files
      ruler: ilum-files
      admin: ilum-files
    type: s3
    s3:
      endpoint: http://ilum-minio.ilum-minio:9000
      region: us-east-1
      secretAccessKey: minioadmin
      accessKeyId: minioadmin
      s3ForcePathStyle: true
      insecure: true
  compactor:
    retention_enabled: true
    deletion_mode: filter-and-delete
    shared_store: s3
  limits_config:
    allow_deletes: true

# promtail-values.yaml
config:
  clients:
    - url: http://ilum-loki-write:3100/loki/api/v1/push
  snippets:
    pipelineStages:
      - match:
          selector: '{ilum_logAggregation!="true"}'
          action: drop
          drop_counter_reason: non_ilum_log
    extraRelabelConfigs:
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_ilum(.*)"
        replacement: "ilum${1}"
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_spark(.*)"
        replacement: "spark${1}"

    scrapeConfigs: |
      - job_name: kubernetes-pods
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
            namespaces:
              names:
              - ilum
        relabel_configs:
          - source_labels:
              - __meta_kubernetes_pod_controller_name
            regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
            action: replace
            target_label: __tmp_controller_name
          - source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
              - __meta_kubernetes_pod_label_app
              - __tmp_controller_name
              - __meta_kubernetes_pod_name
            regex: ^;*([^;]+)(;.*)?$
            action: replace
            target_label: app
          - source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_instance
              - __meta_kubernetes_pod_label_instance
            regex: ^;*([^;]+)(;.*)?$
            action: replace
            target_label: instance
          - source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_component
              - __meta_kubernetes_pod_label_component
            regex: ^;*([^;]+)(;.*)?$
            action: replace
            target_label: component
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - replacement: kubernetes-pods
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}
          {{- with .Values.config.snippets.extraRelabelConfigs }}
          {{- toYaml . | nindent 4 }}
          {{- end }}

3. Deploy Loki, Ilum by default uses chart in 3.5.0 version but should also work with newer versions

helm install loki grafana/loki \
  --namespace ilum-logs \
  --version 3.5.0 \
  --values loki-values.yaml

For more information about loki deployment refer to loki chart details

4. Deploy Promtail, Ilum by default uses chart in ^6.15.x version but should also work with newer versions

helm install promtail grafana/promtail \
  --namespace ilum-logs \
  --version 6.15.5 \
  --values promtail-values.yaml

For more information about promtail deployment refer to promtail chart details

5. Ilum AIO chart configuration values that need to be changed

global:
  logAggregation:
    enabled: true
    loki:
      enabled: false
      url: http://ilum-loki-gateway.ilum-logs
    promtail:
      enabled: false

Pre-configured Stack Examples

Note: In those examples security configuration has been ignored, so those deployments stick to Ilum internal security mechanism, if you want to know how to use other security methods refer to LDAP or OAUTH2

Option 1: Minimal Production Stack

Suitable for deployments with basic requirements. Extra enabled modules:

ilum-sql
ilum-hive-metastore
ilum-marquez (lineage)

Minimal needed values file taking into account the components deployment described above. For more detailed configuration you can examine dedicated charts, for example ilum-core

# minimal-production-values.yaml
global:
  lineage:
    enabled: true

ilum-core:
  security:
    jwt:
      privateKey: Key generated as described in Security-keys section
      publicKey: Key generated as described in Security-keys section
  sql:
    enabled: true
  metastore:
    enabled: true
    type: hive
  mongo:
    uri: "mongodb://ilum-mongodb-0.ilum-mongodb-headless.ilum-mongodb:27017,ilum-mongodb-1.ilum-mongodb-headless.ilum-mongodb:27017/ilum?replicaSet=rs0"
  communication:
    type: kafka
  kafka:
    address: "ilum-kafka-controller-0.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-1.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-2.ilum-kafka-controller-headless.ilum-kafka:9092"
  minio:
    statusProbe:
      baseUrl: "http://ilum-minio.ilum-minio:9000"
  kubernetes:
    storage:
      type: s3
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

gitea:
  initPreScript: "while ! nc -z ilum-postgresql-hl.ilum-postgresql 5432 2>/dev/null; do echo 'Waiting for Postgres...'; sleep 5; done; echo 'Postgres is ready!'"
  gitea:
    config:
      server:
        ROOT_URL: http://gitea.example.com/external/gitea
      database:
        DB_TYPE: postgres
        HOST: "ilum-postgresql-hl.ilum-postgresql:5432"
        NAME: gitea
        USER: ilum
        PASSWD: "CHANGEMEPLEASE"

ilum-marquez:
  marquez:
    db:
      host: "ilum-postgresql-0.ilum-postgresql-hl.ilum-postgresql"

# Disable bundled dependencies
mongodb:
  enabled: false
kafka:
  enabled: false
minio:
  enabled: false
postgresql:
  enabled: false

# Above-default modules
ilum-sql:
  enabled: true

ilum-hive-metastore:
  enabled: true
  storage:
    metastore:
      warehouse: "s3a://ilum-data/"
    type: "s3"
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"
  postgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    database: metastore
    auth:
      username: ilum
      password: "CHANGEMEPLEASE"

# Ilum add-ons
postgresExtensions:
  host: "ilum-postgresql-hl.ilum-postgresql"
  port: 5432
  databasesToCreate: marquez,airflow,metastore,mlflow,mlflow_auth,superset,gitea,n8n,hydra,kestra
  auth:
    username: ilum
    password: "CHANGEMEPLEASE"

minioExtensions:
  host: http://ilum-minio.ilum-minio:9000

Option 2: Enterprise Production Stack

Comprehensive setup with monitoring, logging aggregation, and more optional modules. Extra enabled modules:

ilum-sql
ilum-hive-metastore
ilum-marquez (lineage)
superset
mlflow
kube-prometheus-stack
Loki & Promtail (log aggregation)

Minimal needed values file taking into account the components deployment described above. For more detailed configuration you can examine dedicated charts, for example ilum-core

# enterprise-production-values.yaml
global:
  lineage:
    enabled: true
  logAggregation:
    enabled: true
    loki:
      enabled: false
      url: http://ilum-loki-gateway.ilum-logs
    promtail:
      enabled: false

ilum-core:
  security:
    jwt:
      privateKey: Key generated as described in Security-keys section
      publicKey: Key generated as described in Security-keys section
  sql:
    enabled: true
  metastore:
    enabled: true
    type: hive
  mongo:
    uri: "mongodb://ilum-mongodb-0.ilum-mongodb-headless.ilum-mongodb:27017,ilum-mongodb-1.ilum-mongodb-headless.ilum-mongodb:27017/ilum?replicaSet=rs0"
  communication:
    type: kafka
  kafka:
    address: "ilum-kafka-controller-0.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-1.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-2.ilum-kafka-controller-headless.ilum-kafka:9092"
  minio:
    statusProbe:
      baseUrl: "http://ilum-minio.ilum-minio:9000"
  kubernetes:
    storage:
      type: s3
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

gitea:
  initPreScript: "while ! nc -z ilum-postgresql-hl.ilum-postgresql 5432 2>/dev/null; do echo 'Waiting for Postgres...'; sleep 5; done; echo 'Postgres is ready!'"
  gitea:
    config:
      server:
        ROOT_URL: http://gitea.example.com/external/gitea
      database:
        DB_TYPE: postgres
        HOST: "ilum-postgresql-hl.ilum-postgresql:5432"
        NAME: gitea
        USER: ilum
        PASSWD: "CHANGEMEPLEASE"

ilum-marquez:
  marquez:
    db:
      host: "ilum-postgresql-0.ilum-postgresql-hl.ilum-postgresql"

# Disable bundled dependencies
mongodb:
  enabled: false
kafka:
  enabled: false
minio:
  enabled: false
postgresql:
  enabled: false

# Above-default modules
ilum-sql:
  enabled: true

ilum-hive-metastore:
  enabled: true
  storage:
    metastore:
      warehouse: "s3a://ilum-data/"
    type: "s3"
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"
  postgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    database: metastore
    auth:
      username: ilum
      password: "CHANGEMEPLEASE"

# Ilum add-ons
postgresExtensions:
  host: "ilum-postgresql-hl.ilum-postgresql"
  port: 5432
  databasesToCreate: marquez,metastore,mlflow,mlflow_auth,superset,gitea
  auth:
    username: ilum
    password: "CHANGEMEPLEASE"

minioExtensions:
  host: http://ilum-minio.ilum-minio:9000

superset:
  enabled: true
  supersetNode:
    connections:
      db_host: "ilum-postgresql-hl.ilum-postgresql"
      db_port: "5432"
      db_user: ilum
      db_pass: "CHANGEMEPLEASE"
      db_name: superset

mlflow:
  enabled: true
  externalS3:
    host: "ilum-minio.ilum-minio"
  externalDatabase:
    dialectDriver: "postgresql"
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    user: ilum
    database: mlflow
    authDatabase: mlflow_auth
    existingSecret: "ilum-postgresql"
    existingSecretPasswordKey: "postgres-password"

Helm Values Configuration

Complete Production Values Template

Reference the official Helm chart values at: https://artifacthub.io/packages/helm/ilum/ilum?modal=values

Ilum AIO Installation

Step 1: Verify Dependencies

# Check MongoDB
kubectl get pods -n ilum-mongodb
kubectl logs -f mongodb-0 -n ilum-mongodb

# Check Kafka
kubectl get pods -n ilum-kafka
kubectl logs -f kafka-0 -n ilum-kafka

# Check MinIO
kubectl get pods -n ilum-minio
kubectl logs -f minio-0 -n ilum-minio

# Check PostgreSQL
kubectl get pods -n ilum-postgresql
kubectl logs -f postgresql-primary-0 -n ilum-postgresql

Step 2: Deploy Ilum

Ilum in defragmented deployment needs ilum-minio secret

kubectl -n ilum create secret generic ilum-minio \
  --from-literal=root-user=minioadmin \
  --from-literal=root-password=minioadmin

ilum-postgresql secret

kubectl -n ilum create secret generic ilum-postgresql \
--from-literal=password=CHANGEMEPLEASE \
--from-literal=postgres-password=CHANGEMEPLEASE

Install Ilum

helm install ilum ilum/ilum \
  --namespace ilum \
  --values production-values.yaml

# Monitor deployment
kubectl get pods -n ilum -w

Post-Installation Configuration

1. Verify Installation

# Check all pods
kubectl get pods --all-namespaces | grep ilum

# Check services
kubectl get services --all-namespaces | grep ilum

# Check ingress (if enabled)
kubectl get ingress --all-namespaces

2. Access Ilum UI

# Port forward (for testing)
kubectl port-forward svc/ilum-ui 9777:9777 -n ilum

# Or access via ingress
http://ilum.company.com

# Or access via NodePort
kubectl get svc
http://K8S_NODE_ADDRESS:ILUM_UI_NODEPORT

Troubleshooting

Image Pulling Errors

During the installation of Ilum on your cluster, Helm will pull Docker images, which may be as large as 10 GB, depending on the additional modules you enable. Consequently, with a slow internet connection, you might encounter Image Pull Timeout errors if the image download time exceeds the configured timeout. To resolve this issue, you can:

Pull Docker image manually by running:

minikube ssh docker pull image
# for example
minikube ssh docker pull ilum/core-6.1.3

Change the image pull timeout in your kubernetes configurations like this:

minikube start --extra-config=kubelet.runtime-request-timeout=5m

or like this:

minikube start --extra-config=kubelet.image-pull-progress-deadline=5m

Default Passwords / Credentials

Ilum comes with predefined credentials for various modules to simplify initial setup and testing. However, for production deployments, it is critical to change these default credentials to ensure security and prevent unauthorized access.

Default Credentials

Application	Default Username	Default Password
Ilum UI	admin	admin
MinIO Console	minioadmin	minioadmin
Airflow Web UI	admin	admin
Superset UI	admin	admin
Gitea UI	ilum	ilum
Grafana	admin	admin

Database Credentials (For Internal Use)

Database	Default Username	Default Password
PostgreSQL	postgres	CHANGEMEPLEASE
Marquez	postgres	CHANGEMEPLEASE
Hive Metastore	postgres	CHANGEMEPLEASE

Table of Contents​

Overview​

Critical Components for Namespace Separation​

Kubernetes Prerequisites​

Supported Platforms​

Minimum Suggested Requirements​

Air-gapped (Offline) Environments​

Testing vs Production​

Architecture​

Components and modules​

PostgreSQL​

MongoDB​

Apache Kafka​

MinIO​

Ilum-Livy-proxy​

Jupyter​

Apache Zeppelin​

Hive Metastore​

Project Nessie​

Ilum SQL (Kyuubi Gateway)​

Trino​

DuckDB and DuckLake​

Apache Flink​

n8n​

Apache Airflow​

Marquez​

Kestra​

Mage​

NiFi​

Streamlit​

Kube Prometheus Stack​

Loki and Promtail​

Production Architecture Overview​

Network Communication​

High Availability Configuration​

Failure Domains & Resilience​

Stateless Core Design​

Pod Anti-Affinity​

Namespace-Level Isolation​

Leader Election​

Dynamic Resource Scaling​

Spark Dynamic Allocation​

Kubernetes Horizontal Pod Autoscaler (HPA)​

Cluster Autoscaler Integration​

Auto-Pause on Idle​

Namespace Separation Strategy​

Recommended Namespace Structure​

Namespace Creation​

Security keys​

Generate private key​

Generate public key​

Authentication Methods​

Ilum as Identity Provider​

OpenLDAP​

Dependency Deployment​

MongoDB​

1. Add Bitnami Repository​

2. Create MongoDB Values File​

3. Deploy MongoDB, Ilum by default uses chart in ^13.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

Apache Kafka​

1. Add Bitnami Repository​

2. Create Kafka Values File​

3. Deploy Kafka, Ilum by default uses chart in ^25.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

MinIO Deployment​

1. Add Bitnami Repository​

2. Create MinIO Values File​

3. Deploy MinIO, Ilum by default uses chart in ^15.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

PostgreSQL Deployment​

1. Add Bitnami Repository​

2. Create PostgreSQL Values File​

3. Deploy PostgreSQL, Ilum by default uses chart in ^13.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

Kube-Prometheus-Stack Deployment (If you plan to use it)​

1. Add Prometheus Community Repository​

2. Create Kube-Prometheus-Stack Values File​

3. Deploy kube-prometheus-stack, Ilum by default uses chart in ^54.x version but should also work with newer versions​

4. Define prometheus PodMonitors and ServiceMonitors​

Table of Contents

Overview

Critical Components for Namespace Separation

Kubernetes Prerequisites

Supported Platforms

Minimum Suggested Requirements

Air-gapped (Offline) Environments

Testing vs Production

Architecture

Components and modules

PostgreSQL

MongoDB

Apache Kafka

MinIO

Ilum-Livy-proxy

Jupyter

Apache Zeppelin

Hive Metastore

Project Nessie

Ilum SQL (Kyuubi Gateway)

Trino

DuckDB and DuckLake

Apache Flink

n8n

Apache Airflow

Marquez

Kestra

Mage

NiFi

Streamlit

Kube Prometheus Stack

Loki and Promtail

Production Architecture Overview

Network Communication

High Availability Configuration

Failure Domains & Resilience

Stateless Core Design

Pod Anti-Affinity

Namespace-Level Isolation

Leader Election

Dynamic Resource Scaling

Spark Dynamic Allocation

Kubernetes Horizontal Pod Autoscaler (HPA)

Cluster Autoscaler Integration

Auto-Pause on Idle

Namespace Separation Strategy

Recommended Namespace Structure

Namespace Creation

Security keys

Generate private key

Generate public key

Authentication Methods

Ilum as Identity Provider

OpenLDAP

Dependency Deployment

MongoDB

1. Add Bitnami Repository

2. Create MongoDB Values File

3. Deploy MongoDB, Ilum by default uses chart in ^13.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

Apache Kafka

1. Add Bitnami Repository

2. Create Kafka Values File

3. Deploy Kafka, Ilum by default uses chart in ^25.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

MinIO Deployment

1. Add Bitnami Repository

2. Create MinIO Values File

3. Deploy MinIO, Ilum by default uses chart in ^15.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

PostgreSQL Deployment

1. Add Bitnami Repository

2. Create PostgreSQL Values File

3. Deploy PostgreSQL, Ilum by default uses chart in ^13.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

Kube-Prometheus-Stack Deployment (If you plan to use it)

1. Add Prometheus Community Repository

2. Create Kube-Prometheus-Stack Values File

3. Deploy kube-prometheus-stack, Ilum by default uses chart in ^54.x version but should also work with newer versions

4. Define prometheus PodMonitors and ServiceMonitors