Skip to main content

API Reference

The Ilum API forms the backbone of Ilum's functionality, providing a programmatic interface that allows users to interact with their Apache Spark clusters and jobs. This RESTful API is designed to be robust, flexible, and intuitive, enabling developers to seamlessly integrate Ilum's capabilities into their existing applications and workflows.

Ilum's OpenAPI specification can be found here

Importance​

  • Real-Time Interactivity: Perhaps one of the most powerful features of the Ilum API is its ability to handle real-time interactive Spark jobs. This feature allows applications to send data to be processed immediately, without waiting for Spark context initialization. This kind of real-time interactivity can be invaluable for applications that need instant insights from their data.

  • Ease of Integration: The Ilum API allows developers to access and control Spark clusters from within their own applications, eliminating the need for manual intervention or standalone tools. This means that Ilum's capabilities can be leveraged wherever they're needed, whether that's in a custom data analysis application, a machine learning pipeline, or an enterprise data management system.

  • Automation and Control: The Ilum API enables users to automate tasks such as job submission, monitoring, and cluster management. This allows for streamlined operations and less room for error, enhancing efficiency and reliability. Users can also control several Spark clusters from one place, regardless of whether they're deployed in the cloud or on-premises, and whether they use Yarn or Kubernetes as their cluster manager.

  • Scalability and Flexibility: The Ilum API was designed with scalability in mind, allowing it to handle the management of Spark clusters ranging from just a single node to hundreds of nodes. Furthermore, its RESTful design means it's based on standard HTTP methods, making it accessible from any platform or language that can make HTTP requests.

In summary, the Ilum API is a crucial part of Ilum's offering, enabling high levels of automation, integration, and real-time interactivity for managing and monitoring Apache Spark clusters. It's a powerful tool for any organization that wants to leverage the power of Spark in a flexible, scalable, and efficient manner.

Interacting with Ilum API​

This guide provides examples of how to interact with the Ilum API. The examples will use curl command line tool, but the same principles apply regardless of your chosen HTTP client.

Expose Ilum-Core​

Firstly, expose the ilum-core service on port 9888 using the following command:

$ kubectl port-forward svc/ilum-core 9888:9888

This allows for local interaction with the Ilum service.

Get Available Clusters​

To retrieve a list of available clusters, execute the following GET request:

$ curl "http://localhost:9888/api/v1/cluster"

The first cluster is created during the initial deployment and is preconfigured to work with the Kubernetes instance used by Ilum.

Create a Spark Job Group​

To begin interacting with Ilum, you need to create a scalable Spark job group by uploading a JAR file containing your Spark application. Use the following POST request to create a group:

$ curl -X POST 'http://localhost:9888/api/v1/group' \
    --form 'jars=@"/path/to/local/file/ilum-job-example.jar"' \
    --form 'scale="1"' \
    --form 'clusterName="default"' \
    --form 'name="example-group"'

Upon success, this request will return a Group ID. You can retrieve this ID later by running:

$ curl http://localhost:9888/api/v1/group

Run a Job and Get a Result​

To execute a job within the group and retrieve the result, use the following POST request. Make sure to replace GROUP_ID with the actual ID of your group:

$ curl -X POST 'http://localhost:9888/api/v1/group/GROUP_ID/job/execute' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "type": "interactive_job_execute",
        "jobClass": "cloud.ilum.job.example.ExampleJob",
        "jobConfig": {
            "parameter": "Welcome to ILUM"
        }
    }'

This request will start the job in the specified group and return the job's result.

OpenAPI 3.0​

OpenAPI 3.0 is a widely adopted standard for designing, building, and documenting RESTful APIs. Using OpenAPI 3.0 for the Ilum API brings several significant benefits:

  • Standardization: OpenAPI 3.0 provides a clear and standardized structure for designing and documenting APIs. This promotes interoperability and reduces the learning curve for developers who are already familiar with the OpenAPI standard.

  • Interactive Documentation: OpenAPI 3.0 supports the generation of interactive API documentation. This not only helps developers understand how to use the API, but also enables them to try out API calls directly from the documentation, improving usability and aiding in testing.

  • Code Generation: There are numerous tools available that can generate client SDKs in various languages from an OpenAPI specification. This can significantly speed up development and ensure the generated code is in line with the API's design.

  • Validation and Testing: OpenAPI 3.0 specifications can be used to automatically validate API requests and responses, helping to catch errors and inconsistencies early in the development process. They can also be used to generate test cases, further aiding in quality assurance.

By using OpenAPI 3.0, the Ilum API not only gains these benefits, but also aligns itself with a wide ecosystem of tools and best practices, thereby enhancing its usability, reliability, and maintainability.