Skip to main content

CA Management

Ilum’s truststore management offers a flexible and robust framework for handling Certificate Authority (CA) certificates. This capability is essential for integrating with services that rely on self-signed or internal CAs, a common practice in both development and enterprise environments.

Whether you are securing a local development environment or connecting to internal services such as GitLab, LDAP, or private container registries, Ilum’s truststore can be configured to recognize and trust all your TLS-enabled services.

This guide details the available operational modes, offers practical configuration examples, and provides troubleshooting guidance to help you manage your CA certificates effectively.

To ensure a secure and reliable setup, we recommend adhering to the following best practices:

  1. Use Strong Passwords: Safeguard your truststore and keystore files with strong, unique passwords. This is the first line of defense against unauthorized access to your sensitive certificate information.
  2. Rotate Certificates Regularly: Monitor certificate expiration dates and establish a rotation policy. Proactively rotating certificates prevents service outages caused by expired credentials.
  3. Enforce Minimal Permissions: Apply the principle of least privilege. Use Kubernetes RBAC to grant read-only access to the secrets containing your certificates, minimizing the risk of unauthorized modification or exposure.
  4. Maintain Audit Trails: Enable and review audit trails. Regularly inspect the logs from the generate-trust-store init container to monitor certificate import activities and detect any unusual behavior.
  5. Validate Before Deployment: Test configurations thoroughly. Before deploying to a production environment, always validate your certificate chains and truststore settings in a staging or pre-production environment to catch issues early.

TrustStore Modes​

Ilum supports four distinct truststore modes to accommodate a wide range of certificate management scenarios. Each mode is designed to address specific operational requirements, from simple setups to complex enterprise environments. You can select the mode that best fits your needs by configuring the trustStore.mode property.

The basic truststore configuration is defined as follows:

ilum-core:
  security:
    trustStore:
      enabled: true # Enable or disable the truststore functionality
      mode: "single"  # Supported modes: "single", "multiple", "prebuilt", "directory"
      secretName: "tls-secret" # The Kubernetes secret containing CA certificates
      # password: "CHANGEMEPLEASE" <- Optional, change in production
      # image: "ilum/java:17.0.13_11-jre" <- Default value

Single Certificate Mode​

In single mode, Ilum imports a single CA certificate into the JVM truststore. This is the default and most straightforward mode, making it ideal for simple use cases where you only need to trust one internal or self-signed CA. It is also backward-compatible with older Ilum configurations.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "single"
      secretName: "ca-certificates"
      # password: "CHANGEMEPLEASE" <- Optional, change in production
      single:
        caSecretFileName: "ca.crt"

You can create the required Kubernetes secret with your CA certificate using the following command:

kubectl create secret generic ca-certificates \
  --from-file=ca.crt=/path/to/your/ca.crt
note

In legacy configurations (prior to version 6.4.1), the caSecretFileName property was located directly under trustStore. The current, more structured configuration nests it under the single block to better support multiple modes.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "single"
      secretName: "ca-certificates"
      # password: "CHANGEMEPLEASE" <- Optional, change in production
      # caSecretFileName: "ca.crt" # <- Legacy location
      single:
        caSecretFileName: "ca.crt"

Multiple Certificates Mode​

multiple mode extends the functionality of single mode by allowing you to import several CA certificates from a single Kubernetes secret. This is useful when Ilum needs to communicate with multiple services, each with its own CA. This mode offers two approaches for specifying the certificates: auto-discovery and explicit definition.

Auto-Discovery

With auto-discovery enabled, the init container automatically finds and imports all files with .crt, .pem, or .cer extensions from the specified secret. This is the simplest way to manage multiple certificates.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "multiple"
      secretName: "ca-certificates"
      # password: "CHANGEMEPLEASE" <- Optional, change in production
      multiple:
        autoDiscoverCerts: true

You can create a secret containing multiple CA certificates as shown below:

kubectl create secret generic ca-certificates \
  --from-file=gitlab-ca.crt=/path/to/gitlab-ca.crt \
  --from-file=ldap-ca.crt=/path/to/ldap-ca.crt \
  --from-file=jenkins-ca.pem=/path/to/jenkins-ca.pem

Explicit Definition

For more granular control, you can disable auto-discovery and explicitly list the certificate files to import. This approach also allows you to define a unique alias for each certificate, which is useful for identification and troubleshooting.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "multiple"
      secretName: "ca-certificates"
      # password: "CHANGEMEPLEASE" <- Optional, change in production
      multiple:
        autoDiscoverCerts: false
        certificateFiles:
          - filename: "gitlab-ca.crt"
            alias: "company-gitlab-root-ca" # A unique alias for the GitLab CA
          - filename: "ldap-ca.crt"
            alias: "company-ldap-ca"      # A unique alias for the LDAP CA
          - filename: "artifactory-ca.pem"
            alias: "company-artifactory-ca" # A unique alias for the Artifactory CA

Pre-built TrustStore Mode​

The prebuilt mode is tailored for organizations that maintain a centralized, pre-existing Java KeyStore (JKS) file. This is common in enterprises where a dedicated security team manages a corporate truststore. When this mode is enabled, Ilum will use the provided JKS file directly as its truststore, replacing the default JVM truststore entirely.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "prebuilt"
      secretName: "corporate-truststore"
      # password: "CHANGEMEPLEASE" <- Optional, change in production
      prebuilt:
        truststoreFileName: "corporate-truststore.jks"

Create the Kubernetes secret containing your JKS file:

kubectl create secret generic corporate-truststore \
  --from-file=corporate-truststore.jks=/path/to/corporate-truststore.jks

Directory Scanning Mode​

For the most complex scenarios, directory mode provides maximum flexibility. It allows Ilum to aggregate and import certificates from multiple sources, including different Kubernetes secrets and ConfigMaps. This mode is particularly well-suited for multi-tenant environments or when different teams manage their own sets of certificates.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "directory"
      secretName: "primary-ca-secret"
      password: "your-secure-password"
      directory:
        sources:
          - name: "internal-cas"
            secretName: "internal-ca-certificates"
            mountPath: "/internal-cas"
          - name: "external-cas"
            configMapName: "external-ca-certificates"
            mountPath: "/external-cas"
          - name: "partner-cas"
            secretName: "partner-ca-certificates"
            mountPath: "/partner-cas"
#       patterns: <- Optional, specify file patterns to include
#         - "*.crt"
#         - "*.pem"
#         - "*.cer"

The sources array can reference Kubernetes secret or configMap resources, which are then mounted into the init container.

# Internal certificates (sensitive)
kubectl create secret generic internal-ca-certificates \
  --from-file=gitlab.crt --from-file=jenkins.crt

# External certificates (public)
kubectl create configmap external-ca-certificates \
  --from-file=public-ca-1.pem --from-file=public-ca-2.pem

# Partner certificates
kubectl create secret generic partner-ca-certificates \
  --from-file=partner-a.crt --from-file=partner-b.crt

Real-World Examples​

To illustrate how these modes work in practice, here are some examples of how to configure the truststore for common enterprise scenarios.

Integrating with GitLab and LDAP​

In a typical enterprise setup, you might need to connect to an internal GitLab instance and an LDAP server for authentication, each secured by its own internal CA. multiple mode with explicit definition is a good choice here.

ilum-core:
  security:
    type: "ldap"
    ldap:
      urls:
        - "ldaps://ldap.company.com:636"
      base: "dc=company,dc=com"
      # ... other LDAP configuration
    trustStore:
      enabled: true
      mode: "multiple"
      secretName: "company-ca-certificates"
      password: "secure-truststore-password"
      multiple:
        certificateFiles:
          - filename: "gitlab-ca.crt"
            alias: "gitlab-ca"
          - filename: "ldap-ca.crt"
            alias: "ldap-ca"
    keyStore:
      enabled: true
      secretName: "company-ca-certificates"
      secretFileName: "client-keystore.p12"
      password: "secure-keystore-password"

Enterprise Multi-Source Configuration​

In a large organization, different teams might manage their own sets of certificates. The directory scanning mode is perfect for this scenario, allowing you to aggregate CAs from various sources without centralizing them in one secret.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "directory"
      secretName: "root-ca-certificates"
      password: "enterprise-truststore-password"
      directory:
        sources:
          # CAs managed by the central security team
          - name: "corporate-root-cas"
            secretName: "corporate-root-cas"
            mountPath: "/corporate-cas"
          
          # Certificates for development and testing environments
          - name: "dev-team-cas"
            secretName: "dev-team-certificates"
            mountPath: "/dev-cas"
          
          # Public certificates from external partners
          - name: "partner-cas"
            configMapName: "partner-certificates"
            mountPath: "/partner-cas"

Migration Guides​

Migrating from Legacy Configuration​

The structured truststore modes were introduced in version 6.4.1. If you are using an older configuration, you can migrate it to the equivalent single mode as follows.

Legacy Configuration:

ilum-core:
  security:
    trustStore:
      enabled: true
      secretName: "ca-certificates"
      caSecretFileName: "ca.crt"

New single Mode Configuration:

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "single"
      secretName: "ca-certificates"
      single:
        caSecretFileName: "ca.crt"

Migrating from Single to Multiple Certificates​

If your needs evolve, and you must add more CAs to an existing setup, you can easily migrate from single to multiple mode.

Current single Mode Configuration:

ilum-core:
  security:
    trustStore:
      enabled: true
      secretName: "tls-secret"
      caSecretFileName: "ca.crt"
      password: super-secret-password

Step 1: Update the Secret with Additional Certificates

First, update your Kubernetes secret to include the new CA certificate. The following command reads the existing secret, adds the new file, and applies the changes atomically.

kubectl create secret generic tls-secret \
  --from-file=ca.crt=/path/to/original-ca.crt \
  --from-file=additional-ca.crt=/path/to/additional-ca.crt \
  --dry-run=client -o yaml | kubectl apply -f -

Step 2: Update the Ilum Configuration

Next, modify your values.yaml to switch to multiple mode. Using auto-discovery is the quickest way to include all certificates from the updated secret.

ilum-core:
  security:
    trustStore:
      enabled: true
      mode: "multiple"
      secretName: "tls-secret"
      password: super-secret-password
      multiple:
        autoDiscoverCerts: true

Troubleshooting​

When issues arise, the first place to look is the logs of the generate-trust-store init container. These logs provide detailed information about the certificate import process.

First, find the ilum-core pod name, then run the following command:

kubectl logs <pod-name> -c generate-trust-store

A successful run will produce logs similar to this, showing the imported certificates and their aliases:

Starting truststore generation in mode: multiple
Auto-discovering certificates in /tls/
Importing certificate: /tls/gitlab-ca.crt with alias: gitlab-ca
Successfully imported /tls/gitlab-ca.crt as gitlab-ca
Importing certificate: /tls/ldap-ca.crt with alias: ldap-ca
Successfully imported /tls/ldap-ca.crt as ldap-ca
Truststore generation completed successfully

Final truststore contents:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

gitlab-ca, Jan 6, 2025, trustedCertEntry,
Certificate fingerprint (SHA-256): AB:CD:EF:...
ldap-ca, Jan 6, 2025, trustedCertEntry,
Certificate fingerprint (SHA-256): 12:34:56:...

Validating a Certificate Chain​

You can use openssl to diagnose certificate issues before applying them to your cluster.

# Test a TLS connection to a server and verify its certificate against your CA file.
openssl s_client -connect ldap.company.com:636 -CAfile /path/to/ldap-ca.crt

# Decode and inspect the details of a certificate.
openssl x509 -in /path/to/certificate.crt -text -noout

# Quickly check the validity period of a certificate.
openssl x509 -in /path/to/certificate.crt -dates -noout

Common Issues​

  1. Certificate Format Errors:

    Warning: Invalid certificate format in /tls/bad-cert.crt, skipping...

    Solution:

    Ensure your certificate files are in a valid, base64-encoded PEM format, including the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- headers and footers. You can verify the format with openssl x509 -in certificate.crt -text -noout.

  2. Duplicate Alias Warnings:

    Warning: Failed to import /tls/cert2.crt, possibly duplicate alias gitlab-ca

    Solution:

    When using auto-discovery, certificate aliases are derived from filenames. Ensure all certificate filenames within the secret are unique. For full control, use the explicit certificateFiles configuration to assign unique aliases manually.

  3. Pre-built TrustStore Not Found:

    Error: Pre-built truststore file not found: /tls/missing-truststore.jks

    Solution:

    Double-check that the truststoreFileName in your configuration exactly matches the filename within the Kubernetes secret. Verify the secret’s contents with kubectl get secret <secret-name> -o yaml.

  4. SSL Handshake Failures in ilum-core:

    The ilum-core pod logs may show Java SSL exceptions like this:

    javax.net.ssl.SSLHandshakeException: PKIX path building failed

    Solution: This error indicates that the JVM could not validate the server’s certificate chain using the CAs in its truststore.

    • Confirm that the correct CA certificate for the service you are connecting to has been added to the truststore.
    • Verify that the certificate has not expired using openssl.
    • Ensure the truststore is being generated and mounted correctly by checking the init container logs for success messages.