For production environments, it's recommended to deploy all dependencies in separate namespaces.
The table below provides necessary prerequisites and related instructions.
|MongoDB||Refer to |
|Kafka||Refer to |
|ObjectStorage||Refer to |
helm install ilum-core --create-namespace -n ilum --set mongo.instances=<mongo uri> --set kafka.address=<kafka broker address> --set s3a.host=<s3 host> --set s3a.port=<s3 port> ilum/ilum-core
helm install ilum-ui --create-namespace -n ilum ilum/ilum-ui
Ilum employs MongoDB as its storage layer, preserving all data required between restarts within the MongoDB database. Ilum automatically creates all necessary databases and collections during the startup process.
Apache Kafka serves as Ilum's communication layer, facilitating interaction between Ilum-Core and Spark jobs, as well as between different Ilum-Core instances when scaled. It is critical to ensure Apache Kafka brokers are accessible by both Ilum-Core and Spark jobs, especially when Spark jobs are launched on a different Kubernetes cluster.
Ilum utilizes Kafka to carry out communication using several topics, all created during Ilum's startup. Therefore, users don't need to manage these topics manually.
Ilum uses MinIO as the storage layer for Spark application components. All files (including jars, configurations, data files) needed for the operation of Spark components (driver, executors) are stored and made available for download via MinIO.
MinIO implements the S3 interface, which also enables it to store input/output data.
Ilum provides several starter kits to facilitate the integration and deployment of various services.
To include the Ilum-Livy-Proxy service in your Ilum installation, you need to specify this during the installation process. Add the
--set ilum-livy-proxy.enabled=true flag to your installation command.
Jupyter notebook is another service that can be included in your Ilum package. To enable it, include the
--set ilum-jupyter.enabled=true flag in your installation command.
Similar to the other services, the Apache Zeppelin notebook is not bundled in the default Ilum package. To include it, use the
--set ilum-zeppelin.enabled=true flag during your Ilum installation.