Skip to main content

All in one

All In One helm chart which contains all dependencies in a single package.

TL;DR​

$ helm repo add ilum https://charts.ilum.cloud
$ helm install ilum ilum/ilum
$ kubectl port-forward svc/ilum-ui 9777:9777

Installing the Chart​

To install the chart with the release name ilum:

$ helm install --dependency-update ilum ilum/ilum

The command deploys ilum on the Kubernetes cluster in the default configuration.

Uninstalling the Chart​

To uninstall/delete the ilum deployment:

$ helm delete ilum

The command removes all the Kubernetes components associated with the chart and deletes the release.

Parameters​

All the default parameters to ilum components are provided in values.yaml file.

They are configured with some default values for ilum to be operational just out of the box.

To check all supported parameters and their default values visit README.md files in ilum-core and ilum-ui charts.

Example deployments​

Using kafka as a communication type​

helm install --dependency-update --set kafka.enabled=true --set ilum-core.communication.type=kafka ilum ilum/ilum

Using gRPC as a communication type (default)​

helm install --dependency-update ilum ilum/ilum

Using gRPC as a communication type and gRPC port exposed​

e.g. for Yarn to be able to reach gRPC service

helm install --dependency-update --set kafka.enabled=false --set ilum-core.communication.type=grpc --set ilum-core.grpc.host=<k8s-exposed-host> --set ilum-core.grpc.nodePort=<k8s-exposed-port> --set ilum-core.grpc.type=NodePort ilum ilum/ilum

ilum-livy-proxy​

Please be aware, that ilum-livy-proxy is not bundled in ilum package by default. If you want to run this service, add --set ilum-livy-proxy.enabled=true to your installation command.

Jupyter​

Please be aware, that Jupyter notebook is not bundled in ilum package by default. If you want to run this service, add --set ilum-jupyter.enabled=true to your installation command.

If you want to access the Jupyter UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-jupyter 8888:8888

Apache Zeppelin​

Please be aware, that Zeppelin notebook is not bundled in ilum package by default. If you want to run this service, add --set ilum-zeppelin.enabled=true to your installation command.

If you want to access the Zeppelin UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-zeppelin 8080:8080

Apache Airflow​

Apache Airflow is a powerful platform for orchestrating and managing data workflows. To read more about Airflow in Ilum, visit the Airflow documentation page.

note

Airflow is not enabled in the Ilum package by default. To deploy Airflow, add --set airflow.enabled=true to your installation command.

Once enabled, Airflow will appear in the Ilum UI under the Modules section.

info

Airflow can leverage Ilum’s Livy proxy to easily create jobs within Ilum. For more details, see the Livy proxy section above.

Airflow is preconfigured to use the default port-forward method of connection. This means that even if you access Ilum via a different URL than localhost:9777, Airflow will still try to redirect you to the default URL. To avoid this, you can configure the Airflow base URL in the Helm values:

  airflow:
    config:
      api:
        base_url: "http://<your-address>:<your-port>/external/airflow"

Marquez​

Please be aware, that Marquez is not bundled in ilum package by default. If you want to run this service, add --set global.lineage.enabled=true to your installation command.

If you want to access the Marquez UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-marquez-web 9444:9444

PostgreSQL​

Please be aware, that PostgreSQL is not bundled in ilum package by default. If you want to run this service, add --set postgresql.enabled=true to your installation command.

Kube Prometheus Stack​

Please be aware, that Kube Prometheus Stack is not bundled in ilum package by default. If you want to run this service, add --set kube-prometheus-stack.enabled=true to your installation command.

If you are upgrading an existing Ilum Helm chart that previously did not have the Kube Prometheus Stack enabled, you must first install the required Prometheus Custom Resource Definitions (CRDs) before proceeding with the upgrade. To do this, run the following commands:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusagents.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_scrapeconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

If you want to access the Prometheus UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/prometheus-operated 9090:9090

If you want to access the Grafana UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-grafana 8080:80

Last updated on