Ilum API (1.5.3)

Download OpenAPI specification:Download

Ilum API

Group

Returns a page of existing groups

query Parameters

groupStates	Array of strings (groupState) Items Enum: "ACTIVE" "ARCHIVED" The list of group states to apply for filtering
groupKinds	Array of strings (groupKind) Items Enum: "JOB" "CODE" The list of group kinds to apply for filtering
jobStates	Array of strings (jobState) Default: "SUBMITTED&jobStates=STARTING&jobStates=READY&jobStates=BUSY" Items Enum: "SUBMITTED" "STARTING" "READY" "BUSY" "FAILED" "FINISHED" "CANCELED" "UNKNOWN" The list of job states to apply for filtering
clusterNames	Array of strings The list of cluster names to apply for filtering
tags	string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2
search	string The word to apply for filtering
page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order

Responses

Response samples

200

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"id": "string",
"name": "string",
"description": "string",
"kind": "string",
"tags": "string",
"language": "string",
"clusterName": "string",
"state": "string",
"scale": 0,
"jobConfig": { },
"pyRequirements": "string",
"packages": "string",
"createdTime": 0,
"archivedTime": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"autoPause": true,
"jobs": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}
]
}

Creates a group of interactive Spark jobs

Request Body schema: multipart/form-data
required

name required	string
description	string
kind	string Default: "JOB" Enum: "JOB" "CODE"
tags	string Default: "key=value;key2=value2" A semicolon separated list of tags. Parameter key and value should be separated by an equal sign.
language	string Default: "SCALA" Enum: "SCALA" "PYTHON"
clusterName required	string Default: "default"
jars	Array of strings <binary> [ items <binary > ]
files	Array of strings <binary> [ items <binary > ]
pyFiles	Array of strings <binary> [ items <binary > ]
jobConfig	string Default: "spark.key=value;spark.key2=value2" A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign.
pyRequirements	string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign.
packages	string Comma-separated list of Maven coordinates of jars to include on the driver and executor classpaths. The coordinates should be groupId:artifactId:version.
scale	string Default: 1 Integer count of Spark jobs to be launched within this group
executors	string number of executors to run
driverCores	string Number of cores to use for the driver process.
executorCores	string The number of cores to use on each executor.
driverMemory	string Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
executorMemory	string Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
dynamicAllocation	string Default: "false" Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload.
initialExecutors	string Initial number of executors to run if dynamic allocation is enabled.
minExecutors	string Upper bound for the number of executors if dynamic allocation is enabled.
maxExecutors	string Lower bound for the number of executors if dynamic allocation is enabled.
autoPause	string Default: "false" Boolean flag to enable auto-pause feature

Responses

Response samples

200
402
404
409

Content type

application/json

{"groupId": "string"
}

Returns a group found by its id

path Parameters

groupId

required

string

The unique identifier of a group

Responses

Response samples

200
404

Content type

application/json

{"id": "string",
"name": "string",
"description": "string",
"kind": "string",
"tags": "string",
"language": "string",
"clusterName": "string",
"state": "string",
"scale": 0,
"jobConfig": { },
"pyRequirements": "string",
"packages": "string",
"createdTime": 0,
"archivedTime": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"autoPause": true,
"jobs": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}

Deletes a group by its id

path Parameters

groupId

required

string

The unique identifier of a group

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Updates a group by its id

path Parameters

groupId

required

string

The unique identifier of a group

Request Body schema: multipart/form-data
required

description	string
tags	string A semicolon separated list of tags. Parameter key and value should be separated by an equal sign.
jobConfig	string A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign.
pyRequirements	string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign.
packages	string Comma-separated list of Maven coordinates of jars to include on the driver and executor classpaths. The coordinates should be groupId:artifactId:version.
scale	string Integer count of Spark jobs to be launched within this group
executors	string number of executors to run
driverCores	string Number of cores to use for the driver process.
executorCores	string The number of cores to use on each executor.
driverMemory	string Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
executorMemory	string Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
dynamicAllocation	string Default: "false" Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload.
initialExecutors	string Initial number of executors to run if dynamic allocation is enabled.
minExecutors	string Upper bound for the number of executors if dynamic allocation is enabled.
maxExecutors	string Lower bound for the number of executors if dynamic allocation is enabled.
autoPause	string Default: "false" Boolean flag to enable auto-pause feature

Responses

Response samples

402
404
409

Content type

application/json

{"message": "string"
}

Executes an interactive Spark job within chosen group

Executes an interactive Spark job within chosen group and waits for job instance to be completed. Its result is returned as a response.

path Parameters

groupId

required

string

The unique identifier of a group

Request Body schema: application/json
required

type required	string Default: "interactive_job_execute" interactive_job_execute
jobClass required	string
required	object

Responses

Request samples

Payload

Content type

application/json

Example

interactive_job_execute

{"type": "interactive_job_execute",
"jobClass": "string",
"jobConfig": { }
}

Response samples

200
402
404
409

Content type

application/json

{"jobInstanceId": "string",
"jobId": "string",
"groupId": "string",
"startTime": 0,
"endTime": 0,
"jobClass": "string",
"jobConfig": { },
"result": "string",
"error": {"message": "string",
"stackTrace": "string"
}
}

Returns a page of existing group job instance results

path Parameters

groupId

required

string

The unique identifier of a group

query Parameters

search	string The word to apply for filtering
page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order

Responses

Response samples

200

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"jobInstanceId": "string",
"jobId": "string",
"groupId": "string",
"startTime": 0,
"endTime": 0,
"jobClass": "string",
"jobConfig": { },
"result": "string",
"error": {"message": "string",
"stackTrace": "string"
}
}
]
}

Submits an interactive Spark job within chosen group

Submits an interactive Spark job within chosen group. It does not wait for job to be completed. Instead it returns created job instance id. Its result can be fetched by a job instance result update subscription.

path Parameters

groupId

required

string

The unique identifier of a group

Request Body schema: application/json
required

type required	string Default: "interactive_job_execute" interactive_job_execute
jobClass required	string
required	object

Responses

Request samples

Payload

Content type

application/json

Example

interactive_job_execute

{"type": "interactive_job_execute",
"jobClass": "string",
"jobConfig": { }
}

Response samples

200
402
404
409

Content type

application/json

{"jobInstanceId": "string"
}

Changes the count of interactive Spark jobs that chosen group uses

path Parameters

groupId required	string The unique identifier of a group
scale required	integer Number of instances to scale interactive Spark job to within group

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Restarts a chosen group

path Parameters

groupId

required

string

The unique identifier of a group

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Pauses a chosen group

path Parameters

groupId

required

string

The unique identifier of a group

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Resumes a chosen group

path Parameters

groupId

required

string

The unique identifier of a group

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Stops a chosen group and puts it in ARCHIVED state

path Parameters

groupId

required

string

The unique identifier of a group

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Returns a stream of interactive Spark job result changes

query Parameters

jobIds	Array of strings The list of job ids to apply for filtering
groupIds	Array of strings The list of group ids to apply for filtering
clusterNames	Array of strings The list of cluster names to apply for filtering
groupNames	Array of strings The list of group names to apply for filtering
tags	string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2

Responses

Returns a group found by its name

path Parameters

groupName

required

string

The unique name of a group

Responses

Response samples

200
404

Content type

application/json

{"id": "string",
"name": "string",
"description": "string",
"kind": "string",
"tags": "string",
"language": "string",
"clusterName": "string",
"state": "string",
"scale": 0,
"jobConfig": { },
"pyRequirements": "string",
"packages": "string",
"createdTime": 0,
"archivedTime": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"autoPause": true,
"jobs": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}

Schedule

Returns a page of existing schedules

query Parameters

scheduleStates	Array of strings (scheduleState) Default: "ENABLED" Items Enum: "ENABLED" "DISABLED" "PAUSED" The list of schedule states to apply for filtering
scheduleCrons	Array of strings The list of schedule crons to apply for filtering
jobStates	Array of strings (jobState) Default: "SUBMITTED&jobStates=STARTING&jobStates=READY&jobStates=BUSY" Items Enum: "SUBMITTED" "STARTING" "READY" "BUSY" "FAILED" "FINISHED" "CANCELED" "UNKNOWN" The list of job states to apply for filtering
clusterNames	Array of strings The list of cluster names to apply for filtering
tags	string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2
search	string The word to apply for filtering
page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order

Responses

Response samples

200

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"id": "string",
"name": "string",
"description": "string",
"cron": "string",
"tags": "string",
"language": "string",
"clusterName": "string",
"jobConfig": { },
"pyRequirements": "string",
"packages": "string",
"state": "string",
"jobClass": "string",
"args": ["string"
],
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"createdTime": 0,
"disabledTime": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"jobs": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}
]
}

Creates a schedule of single Spark jobs

Request Body schema: multipart/form-data
required

name required	string
description	string
cron required	string
tags	string Default: "key=value;key2=value2" A semicolon separated list of tags. Parameter key and value should be separated by an equal sign.
language	string Default: "SCALA" Enum: "SCALA" "PYTHON"
clusterName required	string Default: "default"
jars	Array of strings <binary> [ items <binary > ]
files	Array of strings <binary> [ items <binary > ]
pyFiles	Array of strings <binary> [ items <binary > ]
jobConfig	string Default: "spark.key=value;spark.key2=value2" A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign.
jobClass required	string
args	string list of spark arguments to be applied for the Spark job.
pyRequirements	string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign.
packages	string Comma-separated list of Maven coordinates of jars to include on the driver and executor classpaths. The coordinates should be groupId:artifactId:version.
startTime	string The time when the schedule will start in epoch milliseconds (e.g. 1727733600000, which means 01.10.2024). If not provided, the schedule will start immediately.
endTime	string The time when the schedule will be suspended in epoch milliseconds (e.g. 1727820000000, which means 02.10.2024). If not provided, the schedule will create jobs until schedule disabling.
maxRetries	string Default: 0 Max number of retries to be performed if the job fails
executors	string number of executors to run
driverCores	string Number of cores to use for the driver process.
executorCores	string The number of cores to use on each executor.
driverMemory	string Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
executorMemory	string Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
dynamicAllocation	string Default: "false" Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload.
initialExecutors	string Initial number of executors to run if dynamic allocation is enabled.
minExecutors	string Upper bound for the number of executors if dynamic allocation is enabled.
maxExecutors	string Lower bound for the number of executors if dynamic allocation is enabled.

Responses

Response samples

200
402
404
409

Content type

application/json

{"scheduleId": "string"
}

Returns a schedule found by its id

path Parameters

scheduleId

required

string

The unique identifier of a schedule

Responses

Response samples

200
404

Content type

application/json

{"id": "string",
"name": "string",
"description": "string",
"cron": "string",
"tags": "string",
"language": "string",
"clusterName": "string",
"jobConfig": { },
"pyRequirements": "string",
"packages": "string",
"state": "string",
"jobClass": "string",
"args": ["string"
],
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"createdTime": 0,
"disabledTime": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"jobs": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}

Updates a schedule by its id

path Parameters

scheduleId

required

string

The unique identifier of a schedule

Request Body schema: multipart/form-data
required

description	string
tags	string A semicolon separated list of tags. Parameter key and value should be separated by an equal sign.
jobConfig	string A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign.
cron	string Cron expression to be applied for the schedule
args	string A semicolon separated list of spark arguments to be applied for the Spark job.
jobClass	string
pyRequirements	string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign.
packages	string Comma-separated list of Maven coordinates of jars to include on the driver and executor classpaths. The coordinates should be groupId:artifactId:version.
startTime	string The time when the schedule will start in epoch milliseconds (e.g. 1727733600000, which means 01.10.2024). If not provided, the schedule will start immediately.
endTime	string The time when the schedule will be suspended in epoch milliseconds (e.g. 1727820000000, which means 02.10.2024). If not provided, the schedule will create jobs until schedule disabling.
maxRetries	string Max number of retries to be performed if the job fails
executors	string number of executors to run
driverCores	string Number of cores to use for the driver process.
executorCores	string The number of cores to use on each executor.
driverMemory	string Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
executorMemory	string Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
dynamicAllocation	string Default: "false" Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload.
initialExecutors	string Initial number of executors to run if dynamic allocation is enabled.
minExecutors	string Upper bound for the number of executors if dynamic allocation is enabled.
maxExecutors	string Lower bound for the number of executors if dynamic allocation is enabled.

Responses

Response samples

402
404
409

Content type

application/json

{"message": "string"
}

Deletes a schedule by its id

path Parameters

scheduleId

required

string

The unique identifier of a schedule

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Pauses a chosen schedule

path Parameters

scheduleId

required

string

The unique identifier of a schedule

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Resumes a chosen schedule

path Parameters

scheduleId

required

string

The unique identifier of a schedule

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Stops a chosen schedule and puts it in DISABLED state

path Parameters

scheduleId

required

string

The unique identifier of a schedule

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Returns a schedule found by its name

path Parameters

scheduleName

required

string

The unique name of a schedule

Responses

Response samples

200
404

Content type

application/json

{"id": "string",
"name": "string",
"description": "string",
"cron": "string",
"tags": "string",
"language": "string",
"clusterName": "string",
"jobConfig": { },
"pyRequirements": "string",
"packages": "string",
"state": "string",
"jobClass": "string",
"args": ["string"
],
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"createdTime": 0,
"disabledTime": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"jobs": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}

Job

Returns a page of existing jobs

query Parameters

jobTypes	Array of strings (jobType) Items Enum: "SINGLE" "INTERACTIVE" "SHELL" The list of job types to apply for filtering
jobStates	Array of strings (jobState) Default: "SUBMITTED&jobStates=STARTING&jobStates=READY&jobStates=BUSY" Items Enum: "SUBMITTED" "STARTING" "READY" "BUSY" "FAILED" "FINISHED" "CANCELED" "UNKNOWN" The list of job states to apply for filtering
clusterNames	Array of strings The list of cluster names to apply for filtering
tags	string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2
search	string The word to apply for filtering
page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order

Responses

Response samples

200

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}
]
}

Returns a job found by its id

path Parameters

jobId

required

string

The unique identifier of a job

Responses

Response samples

200
404

Content type

application/json

{"jobId": "string",
"groupId": "string",
"scheduleId": "string",
"jobName": "string",
"jobType": "string",
"tags": "string",
"language": "string",
"jobClass": "string",
"clusterName": "string",
"jobConfig": "string",
"args": ["string"
],
"pyRequirements": "string",
"packages": "string",
"appId": "string",
"state": "string",
"error": "string",
"submitTime": 0,
"startTime": 0,
"endTime": 0,
"maxRetries": 0,
"attempt": 0,
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
}
}

Deletes a job found by its id

path Parameters

jobId

required

string

The unique identifier of a job

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Stops a chosen job and puts it in CANCELED state

path Parameters

jobId

required

string

The unique identifier of a job

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Returns logs for all job resources

path Parameters

jobId

required

string

The unique identifier of a job

query Parameters

logSize	integer <int64> Default: 100 Max number of log lines to return. The limit in the API is set to 25000 lines, you can change it by setting ilum.api.job.logs.maxLines parameter in the configuration file of ilum-core
from	integer <int64> Default: 0 Offset from start of log

Responses

Response samples

200
404

Content type

application/json

[{"name": "string",
"type": "string",
"role": "string",
"state": "string",
"from": 0,
"logSize": 0,
"fileName": "string",
"log": ["string"
]
}
]

Submits a single Spark job

Request Body schema: multipart/form-data
required

name required	string
tags	string Default: "key=value;key2=value2" A semicolon separated list of tags. Parameter key and value should be separated by an equal sign.
language	string Default: "SCALA" Enum: "SCALA" "PYTHON"
clusterName required	string Default: "default"
jobClass	string
jars	Array of strings <binary> [ items <binary > ]
files	Array of strings <binary> [ items <binary > ]
pyFiles	Array of strings <binary> [ items <binary > ]
jobConfig	string Default: "spark.key=value;spark.key2=value2" A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign.
pyRequirements	string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign.
packages	string Comma-separated list of Maven coordinates of jars to include on the driver and executor classpaths. The coordinates should be groupId:artifactId:version.
args	string A semicolon separated list of spark arguments to be applied for the Spark job.
maxRetries	string Default: 0 Max number of retries to be performed if the job fails
executors	string number of executors to run
driverCores	string Number of cores to use for the driver process.
executorCores	string The number of cores to use on each executor.
driverMemory	string Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
executorMemory	string Amount of memory to use per executor process, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
dynamicAllocation	string Default: "false" Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload.
initialExecutors	string Initial number of executors to run if dynamic allocation is enabled.
minExecutors	string Upper bound for the number of executors if dynamic allocation is enabled.
maxExecutors	string Lower bound for the number of executors if dynamic allocation is enabled.

Responses

Response samples

200
402
404
409

Content type

application/json

{"jobId": "string"
}

Returns a stream of Spark job changes

query Parameters

jobIds	Array of strings The list of job ids to apply for filtering
groupIds	Array of strings The list of group ids to apply for filtering
clusterNames	Array of strings The list of cluster names to apply for filtering
tags	string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2

Responses

Cluster

Creates a cluster definition

Request Body schema: application/json
required

name required	string
description	string
	object
sparkStorage	string Default: "s3"
metastoreName	string
	object (memorySettingsCreation)
defaultApplicationConfig	string
type required	string Default: "k8s" k8s
url required	string
caCert	string
clientCert	string
clientKey	string
clientKeyAlgorithm	string
clientKeyPassphrase	string
username	string
password	string
	object (clusterResourceQuota)
	object (clusterLimitRange)

Responses

Request samples

Payload

Content type

application/json

Example

k8s

{"name": "string",
"description": "string",
"storages": {"property1": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
},
"property2": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
}
},
"sparkStorage": "s3",
"metastoreName": "string",
"memorySettings": {"executors": "string",
"driverCores": "string",
"executorCores": "string",
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": "false",
"initialExecutors": "string",
"minExecutors": "string",
"maxExecutors": "string"
},
"defaultApplicationConfig": "string",
"type": "k8s",
"url": "string",
"caCert": "string",
"clientCert": "string",
"clientKey": "string",
"clientKeyAlgorithm": "string",
"clientKeyPassphrase": "string",
"username": "string",
"password": "string",
"resourceQuota": {"limitsPod": "string",
"limitsCpu": "string",
"limitsMemory": "string",
"requestsCpu": "string",
"requestsMemory": "string"
},
"limitRange": {"cpuLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
},
"memoryLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
}
}
}

Response samples

200
402
409

Content type

application/json

{"clusterId": "string"
}

Returns a page of existing clusters

query Parameters

search	string The word to apply for filtering
page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order

Responses

Response samples

200

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"id": "string",
"name": "string",
"description": "string",
"sparkMaster": "string",
"sparkDeployMode": "string",
"sparkStorage": "string",
"metastoreName": "string",
"storages": {"property1": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
},
"property2": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
}
},
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"defaultApplicationConfig": "string",
"type": "k8s",
"url": "string",
"caCert": "string",
"clientCert": "string",
"clientKey": "string",
"clientKeyAlgorithm": "string",
"clientKeyPassphrase": "string",
"username": "string",
"password": "string",
"resourceQuota": {"limitsPod": "string",
"limitsCpu": "string",
"limitsMemory": "string",
"requestsCpu": "string",
"requestsMemory": "string"
},
"limitRange": {"cpuLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
},
"memoryLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
}
}
}
]
}

Returns a cluster found by its id

path Parameters

clusterId

required

string

The unique identifier of a cluster

Responses

Response samples

200
404

Content type

application/json

Example

k8s

{"id": "string",
"name": "string",
"description": "string",
"sparkMaster": "string",
"sparkDeployMode": "string",
"sparkStorage": "string",
"metastoreName": "string",
"storages": {"property1": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
},
"property2": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
}
},
"memorySettings": {"executors": 0,
"driverCores": 0,
"executorCores": 0,
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": true,
"initialExecutors": 0,
"minExecutors": 0,
"maxExecutors": 0
},
"defaultApplicationConfig": "string",
"type": "k8s",
"url": "string",
"caCert": "string",
"clientCert": "string",
"clientKey": "string",
"clientKeyAlgorithm": "string",
"clientKeyPassphrase": "string",
"username": "string",
"password": "string",
"resourceQuota": {"limitsPod": "string",
"limitsCpu": "string",
"limitsMemory": "string",
"requestsCpu": "string",
"requestsMemory": "string"
},
"limitRange": {"cpuLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
},
"memoryLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
}
}
}

Updates a cluster by its id

path Parameters

clusterId

required

string

The unique identifier of a cluster

Request Body schema: application/json
required

name required	string
description	string
	object
sparkStorage	string Default: "s3"
metastoreName	string
	object (memorySettingsCreation)
defaultApplicationConfig	string
type required	string Default: "k8s" k8s
url required	string
caCert	string
clientCert	string
clientKey	string
clientKeyAlgorithm	string
clientKeyPassphrase	string
username	string
password	string
	object (clusterResourceQuota)
	object (clusterLimitRange)

Responses

Request samples

Payload

Content type

application/json

Example

k8s

{"name": "string",
"description": "string",
"storages": {"property1": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
},
"property2": {"sparkBucket": "string",
"dataBucket": "string",
"extraBuckets": ["string"
],
"type": "s3",
"endpoint": "string",
"region": "us-east-1",
"accessKey": "string",
"secretKey": "string"
}
},
"sparkStorage": "s3",
"metastoreName": "string",
"memorySettings": {"executors": "string",
"driverCores": "string",
"executorCores": "string",
"driverMemory": "string",
"executorMemory": "string",
"dynamicAllocation": "false",
"initialExecutors": "string",
"minExecutors": "string",
"maxExecutors": "string"
},
"defaultApplicationConfig": "string",
"type": "k8s",
"url": "string",
"caCert": "string",
"clientCert": "string",
"clientKey": "string",
"clientKeyAlgorithm": "string",
"clientKeyPassphrase": "string",
"username": "string",
"password": "string",
"resourceQuota": {"limitsPod": "string",
"limitsCpu": "string",
"limitsMemory": "string",
"requestsCpu": "string",
"requestsMemory": "string"
},
"limitRange": {"cpuLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
},
"memoryLimitRange": {"defaultLimit": "string",
"defaultRequest": "string",
"minRequest": "string",
"maxLimit": "string",
"maxLimitRequestRatio": "string"
}
}
}

Response samples

402
404
409

Content type

application/json

{"message": "string"
}

Deletes a cluster by its id

path Parameters

clusterId

required

string

The unique identifier of a cluster

query Parameters

force

boolean

Default: false

Force delete flag

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Returns names of existing clusters

Responses

Response samples

200

Content type

application/json

{"names": ["string"
]
}

Creates a namespace for a cluster

Request Body schema: application/json
required

name

required

string

Responses

Request samples

Payload

Content type

application/json

{"name": "string"
}

Response samples

402
409

Content type

application/json

{"message": "string"
}

License

Returns current license status

Responses

Response samples

200

Content type

application/json

{"status": "GREEN",
"expire": "string"
}

Attaches license key

Request Body schema: application/json
required

key

required

string

Responses

Request samples

Payload

Content type

application/json

{"key": "string"
}

Response samples

409

Content type

application/json

{"message": "string"
}

SQL

Returns information about SQL engines

path Parameters

clusterId

required

string

The unique identifier of a cluster

Responses

Response samples

200
409

Content type

application/json

[{"id": "string",
"type": "SPARK_SQL",
"state": "RUNNING"
}
]

Creates an SQL engine

path Parameters

clusterId

required

string

The unique identifier of a cluster

Request Body schema: application/json

type	string (sqlEngineType) Default: "SPARK_SQL" Enum: "SPARK_SQL" "TRINO"
async	boolean Default: false If true, the request will return immediately, meaning that the engine may not be started yet
force	boolean Default: false If true, when there is a starting engine, it will be stopped and a new one will be started

Responses

Request samples

Payload

Content type

application/json

{"type": "SPARK_SQL",
"async": false,
"force": false
}

Response samples

200
402
409

Content type

application/json

{"id": "string"
}

Shuts down an SQL engine by its id

path Parameters

sqlEngineId

required

string

The unique identifier of a SQL engine

Responses

Response samples

402
404
409

Content type

application/json

{"message": "string"
}

Executes an SQL statement

Request Body schema: application/json
required

statement required	string
clusterId	string
maxRows	integer <int64> Default: 1000
engine	string (sqlEngineType) Default: "SPARK_SQL" Enum: "SPARK_SQL" "TRINO"

Responses

Request samples

Payload

Content type

application/json

{"statement": "string",
"clusterId": "string",
"maxRows": 1000,
"engine": "SPARK_SQL"
}

Response samples

200
402
409

Content type

application/json

{"info": {"state": "string",
"creationTime": 0,
"startTime": 0,
"completionTime": 0,
"exception": "string"
},
"columns": [{"name": "string",
"type": "string",
"index": 0,
"precision": 0,
"scale": 0,
"comment": "string"
}
],
"rows": [[{ }
]
],
"logs": ["string"
]
}

Returns paginated list of saved SQL statements

query Parameters

page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order
search	string The word to apply for filtering

Responses

Response samples

200
409

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"id": "string",
"name": "string",
"statement": "string",
"engine": "SPARK_SQL",
"maxRows": 0
}
]
}

Saves an SQL statement

Request Body schema: application/json
required

name required	string
statement required	string
engine	string (sqlEngineType) Default: "SPARK_SQL" Enum: "SPARK_SQL" "TRINO"
maxRows	integer <int64>

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"statement": "string",
"engine": "SPARK_SQL",
"maxRows": 0
}

Response samples

200
402
409

Content type

application/json

{"id": "string"
}

Returns a saved SQL statement found by its id

path Parameters

sqlSavedId

required

string

The unique identifier of a saved SQL statement

Responses

Response samples

200
404

Content type

application/json

{"name": "string",
"statement": "string",
"engine": "SPARK_SQL",
"maxRows": 0
}

Updates a saved SQL statement found by its id

path Parameters

sqlSavedId

required

string

The unique identifier of a saved SQL statement

Request Body schema: application/json
required

name required	string
statement required	string
engine	string (sqlEngineType) Default: "SPARK_SQL" Enum: "SPARK_SQL" "TRINO"
maxRows	integer <int64>

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"statement": "string",
"engine": "SPARK_SQL",
"maxRows": 0
}

Response samples

402
404

Content type

application/json

{"message": "string"
}

Deletes a saved SQL statement found by its id

path Parameters

sqlSavedId

required

string

The unique identifier of a saved SQL statement

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Returns a page of SQL notebooks

query Parameters

page	integer Default: 0 The number of page to be returned
size	integer Default: 10 The size of page to be returned
sort	Array of strings Example: sort=name,asc The sort rules to be applied for define paging order
search	string The word to apply for filtering

Responses

Response samples

200
409

Content type

application/json

{"page": {"number": 0,
"size": 0,
"offset": 0,
"paged": true
},
"sort": ["string"
],
"totalElements": 0,
"totalPages": 0,
"first": true,
"last": true,
"empty": true,
"content": [{"id": "string",
"name": "string",
"statements": [{"id": "string",
"language": "SQL",
"code": "string"
}
]
}
]
}

Creates an SQL notebook

Request Body schema: application/json
required

name	string
	Array of objects (sqlNotebookCell)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"statements": [{"id": "string",
"language": "SQL",
"code": "string"
}
]
}

Response samples

200
402
409

Content type

application/json

{"id": "string"
}

Returns an SQL notebook found by its id

path Parameters

sqlNotebookId

required

string

The unique identifier of an SQL notebook

Responses

Response samples

200
404

Content type

application/json

{"id": "string",
"name": "string",
"statements": [{"id": "string",
"language": "SQL",
"code": "string"
}
]
}

Updates an SQL notebook found by its id

path Parameters

sqlNotebookId

required

string

The unique identifier of an SQL notebook

Request Body schema: application/json
required

name	string
	Array of objects (sqlNotebookCell)

Responses

Request samples

Payload

Content type

application/json

{"name": "string",
"statements": [{"id": "string",
"language": "SQL",
"code": "string"
}
]
}

Response samples

402
404
409

Content type

application/json

{"message": "string"
}

Deletes an SQL notebook found by its id

path Parameters

sqlNotebookId

required

string

The unique identifier of an SQL notebook

Responses

Response samples

402
404

Content type

application/json

{"message": "string"
}

Ilum API (1.5.3)

Group

Returns a page of existing groups

query Parameters

Responses

Response samples

Creates a group of interactive Spark jobs

Request Body schema: multipart/form-datarequired

Responses

Response samples

Returns a group found by its id

path Parameters

Responses

Response samples

Deletes a group by its id

path Parameters

Responses

Response samples

Updates a group by its id

path Parameters

Request Body schema: multipart/form-datarequired

Responses

Response samples

Executes an interactive Spark job within chosen group

path Parameters

Request Body schema: application/jsonrequired

Responses

Request samples

Response samples

Returns a page of existing group job instance results

path Parameters

query Parameters

Responses

Response samples

Submits an interactive Spark job within chosen group

path Parameters

Request Body schema: application/jsonrequired

Responses

Request samples

Response samples

Changes the count of interactive Spark jobs that chosen group uses

path Parameters

Responses

Response samples

Restarts a chosen group

path Parameters

Responses

Response samples

Pauses a chosen group

path Parameters

Responses

Response samples

Resumes a chosen group

path Parameters

Responses

Response samples

Stops a chosen group and puts it in ARCHIVED state

path Parameters

Responses

Response samples

Returns a stream of interactive Spark job result changes

query Parameters

Responses

Returns a group found by its name

path Parameters

Responses

Response samples

Schedule

Returns a page of existing schedules

query Parameters

Responses

Response samples

Creates a schedule of single Spark jobs

Request Body schema: multipart/form-datarequired

Responses

Response samples

Returns a schedule found by its id

path Parameters

Responses

Response samples

Request Body schema: multipart/form-data
required

Request Body schema: multipart/form-data
required

Request Body schema: application/json
required

Request Body schema: application/json
required

Request Body schema: multipart/form-data
required

Request Body schema: multipart/form-data
required

Request Body schema: multipart/form-data
required

Request Body schema: application/json
required