Production
For production environments, it's recommended to deploy all dependencies in separate namespaces.
Prerequisites
The table below provides necessary prerequisites and related instructions.
| Prerequisite | Instruction |
|---|---|
| MongoDB | Refer to https://bitnami.com/stack/mongodb/helm |
| Kafka | Refer to https://bitnami.com/stack/kafka/helm |
| ObjectStorage | Refer to https://min.io/docs/minio/kubernetes/upstream/operations/installation.html |
ilum-core
helm install ilum-core --create-namespace -n ilum --set mongo.instances=<mongo uri> --set kafka.address=<kafka broker address> --set s3a.host=<s3 host> --set s3a.port=<s3 port> ilum/ilum-core
ilum-ui
helm install ilum-ui --create-namespace -n ilum ilum/ilum-ui
MongoDB
Ilum employs MongoDB as its storage layer, preserving all data required between restarts within the MongoDB database. Ilum automatically creates all necessary databases and collections during the startup process.
Apache Kafka
Apache Kafka serves as Ilum's communication layer, facilitating interaction between Ilum-Core and Spark jobs, as well as between different Ilum-Core instances when scaled. It is critical to ensure Apache Kafka brokers are accessible by both Ilum-Core and Spark jobs, especially when Spark jobs are launched on a different Kubernetes cluster.
Ilum utilizes Kafka to carry out communication using several topics, all created during Ilum's startup. Therefore, users don't need to manage these topics manually.
MinIO
Ilum uses MinIO as the storage layer for Spark application components. All files (including jars, configurations, data files) needed for the operation of Spark components (driver, executors) are stored and made available for download via MinIO.
MinIO implements the S3 interface, which also enables it to store input/output data.
Starter Kits
Ilum provides several starter kits to facilitate the integration and deployment of various services.
ilum-livy-proxy
To include the Ilum-Livy-Proxy service in your Ilum installation, you need to specify this during the installation process. Add the --set ilum-livy-proxy.enabled=true flag to your installation command.
Jupyter
Jupyter notebook is another service that can be included in your Ilum package. To enable it, include the --set ilum-jupyter.enabled=true flag in your installation command.
Apache Zeppelin
Similar to the other services, the Apache Zeppelin notebook is not bundled in the default Ilum package. To include it, use the --set ilum-zeppelin.enabled=true flag during your Ilum installation.