Skip to main content

Production

For production environments, it's recommended to deploy all dependencies in separate namespaces.

Prerequisites

The table below provides necessary prerequisites and related instructions.

PrerequisiteInstruction
MongoDBRefer to https://bitnami.com/stack/mongodb/helm
KafkaRefer to https://bitnami.com/stack/kafka/helm
ObjectStorageRefer to https://min.io/docs/minio/kubernetes/upstream/operations/installation.html

ilum-core

helm install ilum-core --create-namespace -n ilum --set mongo.instances=<mongo uri> --set kafka.address=<kafka broker address> --set s3a.host=<s3 host> --set s3a.port=<s3 port> ilum/ilum-core

ilum-ui

helm install ilum-ui --create-namespace -n ilum ilum/ilum-ui

MongoDB

Ilum employs MongoDB as its storage layer, preserving all data required between restarts within the MongoDB database. Ilum automatically creates all necessary databases and collections during the startup process.

Apache Kafka

Apache Kafka serves as Ilum's communication layer, facilitating interaction between Ilum-Core and Spark jobs, as well as between different Ilum-Core instances when scaled. It is critical to ensure Apache Kafka brokers are accessible by both Ilum-Core and Spark jobs, especially when Spark jobs are launched on a different Kubernetes cluster.

Ilum utilizes Kafka to carry out communication using several topics, all created during Ilum's startup. Therefore, users don't need to manage these topics manually.

MinIO

Ilum uses MinIO as the storage layer for Spark application components. All files (including jars, configurations, data files) needed for the operation of Spark components (driver, executors) are stored and made available for download via MinIO.

MinIO implements the S3 interface, which also enables it to store input/output data.

Starter Kits

Ilum provides several starter kits to facilitate the integration and deployment of various services.

ilum-livy-proxy

To include the Ilum-Livy-Proxy service in your Ilum installation, you need to specify this during the installation process. Add the --set ilum-livy-proxy.enabled=true flag to your installation command.

Jupyter

Jupyter notebook is another service that can be included in your Ilum package. To enable it, include the --set ilum-jupyter.enabled=true flag in your installation command.

Apache Zeppelin

Similar to the other services, the Apache Zeppelin notebook is not bundled in the default Ilum package. To include it, use the --set ilum-zeppelin.enabled=true flag during your Ilum installation.