Skip to main content

Ilum Core

Ilum Core helm chart config.

TL;DR

$ helm repo add ilum https://charts.ilum.cloud
$ helm install ilum-core ilum/ilum-core

Installing the Chart

To install the chart with the release name ilum-core:

$ helm install ilum-core ilum/ilum-core

The command deploys ilum-core on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Uninstalling the Chart

To uninstall/delete the ilum-core deployment:

$ helm delete ilum-core

The command removes all the Kubernetes components associated with the chart and deletes the release.

Parameters

Common parameters

NameDescriptionValue
nameOverrideString to partially override ilum-core.fullname template (will maintain the release name)""
replicaCountNumber of ilum-core replicas1
fullnameOverrideString to fully override ilum-core.fullname template""
nodeSelectorilum-core node selection constraint
tolerationsilum-core pods tolerations[]
affinityilum-core node affinity

ilum-core deployment parameters

NameDescriptionValue
imageilum-core imageilum/core:6.2.0
pullPolicyilum-core image pull policyIfNotPresent
imagePullSecretsilum-core image pull secrets[]

ilum-core communication parameters

NameDescriptionValue
communication.typeilum-core communication type with spark jobs, available options: grpc, kafkagrpc

ilum-core service parameters

NameDescriptionValue
service.typeilum-core service typeClusterIP
service.portilum-core service port9888
service.nodePortilum-core service node port - required when type is LoadBalancer or NodePort""
service.clusterIPilum-core service cluster IP - required when type is ClusterIP""
service.loadBalancerIPilum-core service load balancer IP - required when type is LoadBalancer""
service.annotationsilum-core service annotations{}

ilum-core ingress parameters

NameDescriptionValue
ingress.enabledilum-core ingress enabled flagfalse
ingress.versionilum-core ingress versionv1
ingress.classNameilum-core ingress class name""
ingress.hostilum-core ingress hosthost
ingress.tlsilum-core ingress tls configuration[]
ingress.tls[x].secretNameilum-core ingress secret name to apply for a single tls configuration entry
ingress.tls[x].hostsilum-core ingress hosts list to apply for a single tls configuration entry
ingress.annotationsilum-core ingress annotations in yaml formatnginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/proxy-body-size: "600m"
nginx.org/client-max-body-size: "600m"
ingress.pathilum-core ingress path/(.*)
ingress.pathTypeilum-core ingress pathTypePrefix

ilum-core livenessProbe/readinessProbe parameters

NameDescriptionValue
readinessProbeilum-core readinessProbe configuration
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/dev/reactive/health
port: http
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
livenessProbeilum-core livenessProbe configuration
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/dev/reactive/health
port: http
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1

ilum-core mongo parameters

NameDescriptionValue
mongo.instancesilum-core mongo instances to connect tomongo:27017
mongo.replicaSetNameilum-core mongo replica set namers0
mongo.statusProbe.enanbledilum-core mongo healthcheck flagtrue
mongo.statusProbe.imageilum-core mongo healthcheck imageilum/mongodb:6.0.5

ilum-core kafka parameters

NameDescriptionValue
kafka.addressilum-core kafka address to connect tokafka:9092
kafka.requestSizeilum-core kafka max.request.size parameter for ilum jobs kafka producers20000000
kafka.maxPollRecordsilum-core kafka max.poll.records parameter for ilum jobs kafka consumers500
kafka.maxPollIntervalilum-core kafka max.poll.interval.ms parameter for ilum jobs kafka consumers60000
kafka.statusProbe.enanbledilum-core kafka healthcheck flagtrue
kafka.statusProbe.imageilum-core kafka healthcheck imagebitnami/kafka:3.4.1

ilum-core grpc service parameters

NameDescriptionValue
grpc.service.typeilum-core grpc service typeClusterIP
grpc.service.portilum-core grpc service port9999
grpc.service.nodePortilum-core grpc service node port - required when type is LoadBalancer or NodePort""
grpc.service.clusterIPilum-core grpc service cluster IP - required when type is ClusterIP""
grpc.service.loadBalancerIPilum-core grpc service load balancer IP - required when type is LoadBalancer""
grpc.service.annotationsilum-core grpc service annotations{}

ilum-core grpc parameters for spark job

NameDescriptionValue
grpc.job.hostilum-core grpc host for spark job to connect toilum-grpc
grpc.job.portilum-core grpc port for spark job to connect to9999

ilum-core kubernetes cluster initializer parameters

NameDescriptionValue
kubernetes.initClusterOnStartupilum-core default kubernetes cluster initialization flagtrue
kubernetes.upgradeClusterOnStartupilum-core default kubernetes cluster upgrade from values in config map flagfalse
kubernetes.api.urlilum-core default kubernetes cluster api urlhttps://kubernetes.default.svc
kubernetes.container.imageilum-core default kubernetes cluster container imageilum/spark:3.5.2-delta
kubernetes.sparkNamespaceilum-core default kubernetes cluster namespace to store spark resources{{ .Release.Namespace }}
kubernetes.storage.typeilum-core default kubernetes cluster storage type, available options: s3, gcs, wasbs, hdfss3

s3 kubernetes storage parameters

NameDescriptionValue
kubernetes.s3.hostilum-core default kubernetes cluster S3 storage host to store spark resourcess3
kubernetes.s3.portilum-core default kubernetes cluster S3 storage port to store spark resources7000
kubernetes.s3.sparkBucketilum-core default kubernetes cluster S3 storage bucket to store spark resourcesilum-files
kubernetes.s3.dataBucketilum-core default kubernetes cluster S3 storage bucket to store ilum tablesilum-tables
kubernetes.s3.accessKeyilum-core default kubernetes cluster S3 storage access key to store spark resources""
kubernetes.s3.secretKeyilum-core default kubernetes cluster S3 storage secret key to store spark resources""

gcs kubernetes storage parameters

NameDescriptionValue
kubernetes.gcs.clientEmaililum-core default kubernetes cluster GCS storage client email""
kubernetes.gcs.sparkBucketilum-core default kubernetes cluster GCS storage bucket to store spark resources"ilum-files"
kubernetes.gcs.dataBucketilum-core default kubernetes cluster GCS storage bucket to store ilum tables"ilum-tables"
kubernetes.gcs.privateKeyilum-core default kubernetes cluster GCS storage private key to store spark resources""
kubernetes.gcs.privateKeyIdilum-core default kubernetes cluster GCS storage private key id to store spark resources""

wasbs kubernetes storage parameters

NameDescriptionValue
kubernetes.wasbs.accountNameilum-core default kubernetes cluster WASBS storage account name""
kubernetes.wasbs.accessKeyilum-core default kubernetes cluster WASBS storage access key to store spark resources""
kubernetes.wasbs.sparkContainerilum-core default kubernetes cluster WASBS storage container name to store spark resources"ilum-files"
kubernetes.wasbs.dataContainerilum-core default kubernetes cluster WASBS storage container name to store ilum tables"ilum-tables"

hdfs kubernetes storage parameters

NameDescriptionValue
kubernetes.hdfs.hadoopUsernameilum-core default kubernetes cluster HDFS storage hadoop username""
kubernetes.hdfs.configilum-core default kubernetes cluster HDFS storage dict of config files with name as key and base64 encoded content as value""
kubernetes.hdfs.sparkCatalogilum-core default kubernetes cluster HDFS storage catalog to store spark resources"ilum-files"
kubernetes.hdfs.dataCatalogilum-core default kubernetes cluster HDFS storage catalog to store ilum-tables"ilum-tables"
kubernetes.hdfs.keyTabilum-core default kubernetes cluster HDFS storage keytab file base64 encoded content""
kubernetes.hdfs.principalilum-core default kubernetes cluster HDFS storage principal name""
kubernetes.hdfs.krb5ilum-core default kubernetes cluster HDFS storage krb5 file base64 encoded content""
kubernetes.hdfs.trustStoreilum-core default kubernetes cluster HDFS storage trustStore file base64 encoded content""
kubernetes.hdfs.logDirectoryilum-core default kubernetes cluster HDFS storage directory path to store eventLog for history server""

Important! Make sure S3/GCS buckets or WASBS containers are already created and reachable! Important! Make sure HDFS logDirectory is absolute path of configured sparkCatalog with /ilum/logs suffix! Eg hdfs://name-node/user/username/spark-catalog/ilum/logs

ilum-core security parameters

NameDescriptionValue
security.typeilum-core authentication type, available options: internal, ldap, oauth2internal
security.jwt.issuerUrlilum-core frontend URI used in the jwt iss claimhttps://ilum.cloud
security.jwt.timeToLiveilum-core jwt time to live in specified time units8h
security.jwt.publicKeyilum-core base64 encoded string containing the X.509 RSA 2048 bit public key""
security.jwt.privateKeyilum-core base64 encoded string containing the PKCS8 RSA 2048 bit private key""
security.authoritiesilum-core authorities mapping rules configuration. Used when authorization is ldap or oauth2. Allows to translate external auth provider groups and scopes to ilum roles
authorities:
roles:
prefix: ROLE_
claim-name: groups
scopes:
prefix: SCOPE__
claim-name: scp
security.authorities.roles.prefixilum-core authentication role prefixROLE_
security.authorities.roles.claimNameilum-core external authentication provider ID of the jwt claim which contains list of roles/groupsgroups
security.authorities.roles.mappingsilum-core role mapping definitions in form of a map external_role: ilum_role{}
security.authorities.scopes.prefixilum-core authentication scope prefixSCOPE_
security.authorities.scopes.claimNameilum-core external authentication provider ID of the jwt claim which contains list of scopesscp
security.authorities.scopes.mappingsilum-core scope mapping definitions in form of a map external_scope: ilum_scope{}

Internal config-map based authentication parameters

NameDescriptionValue
security.internal.usersilum-core internal users configuration
users:
- username: "admin"
password: "admin"
roles:
- "ADMIN"
security.internal.users[].usernameilum-core user username""
security.internal.users[].passwordilum-core user plain password""
security.internal.users[].rolesilum-core user roles, available options: ADMIN, USER, VIEWER[]

LDAP based authentication parameters

NameDescriptionValue
security.ldap.urlsilum-core LDAP URLs of the server[]
security.ldap.baseilum-core LDAP base suffix from which all operations should originate""
security.ldap.usernameilum-core LDAP login username of the server""
security.ldap.passwordilum-core LDAP login password of the server""
security.ldap.passwordEncoderilum-core LDAP password encoder. LDAP server authenticates users (bind operations) if empty, available options:
adaptive - password encoder that delegates to another encoder based upon a prefixed identifier
bcrypt
md5
sha256
""
security.ldap.userSearch.baseilum-core LDAP base DN from which the search for an user should be performed""
security.ldap.userSearch.filterilum-core LDAP pattern to be used for the user search. 0 is the usernameuid={0}
security.ldap.userSearch.passwordAttrilum-core LDAP ID of the attribute which contains the password of a useruserPassword
security.ldap.groupSearch.baseilum-core LDAP base DN from which the search for group membership should be performed""
security.ldap.groupSearch.filterilum-core LDAP pattern to be used for the group search. 0 is the user's DN(member={0})
security.ldap.groupSearch.roleAttrilum-core LDAP ID of the attribute which contains the role name for a groupcn

OAuth2 based authentication parameters

NameDescriptionValue
security.oauth2.clientIdilum-core oauth2 Client ID""
security.oauth2.clientSecretilum-core oauth2 Client SECRET""
security.oauth2.issuerUriilum-core oauth2 URI that can either be an OpenID Connect discovery endpoint or an OAuth 2.0 Authorization Server Metadata endpoint defined by RFC 8414.""

ilum-core license parameters

NameDescriptionValue
license.privateKeyilum license key""

ilum-core external spark submit parameters

NameDescriptionValue
externalSparkSubmit.enabledilum-core external spark-submit flagfalse
externalSparkSubmit.imageilum-core external spark-submit base imagesilum/spark-launcher:spark-3.5.1
externalSparkSubmit.resourcesilum-core external spark-submit pod kubernetes resources<pre>limits:<br/> memory: "500Mi"<br/>requests:<br/> memory: "300Mi"</pre>

ilum-core spark history server parameters

NameDescriptionValue
historyServer.enabledilum-core spark history server flagtrue
historyServer.imageilum-core spark history server imageilum/spark-launcher:spark-3.5.1
historyServer.addressilum-core spark history server addresshttp://ilum-history-server:9666
historyServer.pullPolicyilum-core spark history server image pull policyIfNotPresent
historyServer.imagePullSecretsilum-core spark history server image pull secrets[]
historyServer.parametersilum-core spark history server custom spark parameters
spark.history.fs.cleaner.enabled: true
spark.history.fs.cleaner.interval: 1d
spark.history.fs.cleaner.maxAge: 7d
historyServer.resourcesilum-core spark history server pod resources
limits:
memory: "500Mi"
requests:
memory: "300Mi"
historyServer.service.typeilum-core spark history server service typeClusterIP
historyServer.service.portilum-core spark history server service port9666
historyServer.service.nodePortilum-core spark history server service nodePort""
historyServer.service.clusterIPilum-core spark history server service clusterIP""
historyServer.service.loadBalancerIPilum-core spark history server service loadbalancerIP""
historyServer.service.annotationsilum-core history server service annotations{}
historyServer.ingress.enabledilum-core spark history server ingress flagfalse
historyServer.ingress.versionilum-core spark history server ingress version"v1"
historyServer.ingress.classNameilum-core spark history server ingress className""
historyServer.ingress.hostilum-core spark history server ingress host"host"
historyServer.ingress.pathilum-core spark history server ingress path"/(.*)"
historyServer.ingress.pathTypeilum-core spark history server ingress pathTypePrefix
historyServer.ingress.annotationsilum-core spark history server annotationsnginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/proxy-body-size: "600m"
nginx.org/client-max-body-size: "600m"
historyServer.statusProbe.enabledilum-core spark history server ilum-core healthcheck flagtrue
historyServer.statusProbe.imageilum-core spark history server ilum-core healthcheck imagecurlimages/curl:8.5.0
historyServer.nodeSelectorilum-core spark history server nodeSelector
historyServer.tolerationsilum-core spark history server tolerations[]
historyServer.affinityilum-core spark history server affinity

ilum-core job parameters

NameDescriptionValue
job.retain.hoursilum-core spark jobs retention hours limit168
job.prometheus.enabledilum-core spark jobs prometheus enabled flagtrue
job.healthcheck.enabledilum-core spark jobs healthcheck enabled flagtrue
job.healthcheck.intervalilum-core spark jobs healthcheck interval in seconds300
job.healthcheck.toleranceilum-core spark jobs healthcheck response time tolerance in seconds120
job.memorysettings.executorsilum-core spark jobs executor count2
job.memorysettings.executorMemoryilum-core spark jobs executor memory allocation1g
job.memorysettings.driverMemoryilum-core spark jobs driver memory allocation1g
job.memorysettings.executorCoresilum-core spark jobs executor core count1
job.memorysettings.driverCoresilum-core spark jobs driver core count1
job.memorysettings.dynamicAllocationEnabledilum-core spark jobs dynamic allocation enabled flagfalse
job.memorysettings.minExecutorsilum-core spark jobs minimum number of executors0
job.memorysettings.initialExecutorsilum-core spark jobs initial number of executors0
job.memorysettings.maxExecutorsilum-core spark jobs maximum number of executors20

ilum-core default metastore parameters

NameDescriptionValue
hiveMetastore.enabledilum-core default hive metastore enabled flagfalse
hiveMetastore.addressilum-core default hive metastore addressthrift://ilum-hive-metastore:9083
hiveMetastore.versionilum-core default hive metastore version3.1.3

ilum-core auto pausing settings

NameDescriptionValue
job.autoPause.enabledilum-core group auto pausing flagtrue
job.autoPause.idleTimeilum-core auto pausing minimum group idle time in seconds3600
job.autoPause.periodilum-core time between checks of group idleness in seconds180

Generating RSA Private and Public Key

In order to create 2048-bit RSA keys in an unencrypted Base64 PEM PKCS#8 format for authentication configuration, openssl was used.

Generate private key

openssl genpkey -algorithm RSA \
-pkeyopt rsa_keygen_bits:2048 \
-pkeyopt rsa_keygen_pubexp:65537 | \
openssl pkcs8 -topk8 -nocrypt -outform pem > private-key.p8

The contents of the private key should look like the following:

-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsRnE83rm6BJya
nTyzVqX0SG+D4zBjkyWsOmGG+CoDdgQ6Z8AaocmnjP1SbRykQsQSMf6SeW+fdpH+
ccmzuHe7pZIa2o2Mg8xbk/UszJDaPztwoQbUt/2gHi/rZP8cIVkquzhnN/yxrMls
...
-----END PRIVATE KEY-----

In order to use private key as the setting security.jwt.privateKey, remove header and footer from the key.

Extract public key

openssl pkey -pubout -inform pem -outform pem -in private-key.p8 -out public-key.spki

The contents of the public key should look like the following:

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArEZxPN65ugScmp08s1al
9Ehvg+MwY5MlrDphhvgqA3YEOmfAGqHJp4z9Um0cpELEEjH+knlvn3aR/nHJs7h3
u6WSGtqNjIPMW5P1LMyQ2j87cKEG1Lf9oB4v62T/HCFZKrs4Zzf8sazJbMN3E/mJ
...
-----END PUBLIC KEY-----

In order to use public key as the setting security.jwt.publicKey, remove header and footer from the key.