Returns a page of existing groups
query Parameters
| groupStates | Array of strings (groupState) Items Enum: "ACTIVE" "ARCHIVED" The list of group states to apply for filtering |
| groupKinds | Array of strings (groupKind) Items Enum: "JOB" "CODE" The list of group kinds to apply for filtering |
| jobStates | Array of strings (jobState) Items Enum: "SUBMITTED" "STARTING" "READY" "BUSY" "FAILED" "FINISHED" "UNKNOWN" The list of job states to apply for filtering |
| clusterNames | Array of strings The list of cluster names to apply for filtering |
| tags | string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2 |
| search | string The word to apply for filtering |
| page | integer Default: 0 The number of page to be returned |
| size | integer Default: 10 The size of page to be returned |
| sort | Array of strings Example: sort=name,asc The sort rules to be applied for define paging order |
Responses
Response samples
- 200
{- "page": {
- "number": 0,
- "size": 0,
- "offset": 0,
- "paged": true
}, - "sort": [
- "string"
], - "totalElements": 0,
- "totalPages": 0,
- "first": true,
- "last": true,
- "empty": true,
- "content": [
- {
- "id": "string",
- "name": "string",
- "description": "string",
- "kind": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "state": "string",
- "scale": 0,
- "jobConfig": { },
- "createdTime": 0,
- "archivedTime": 0,
- "jobs": [
- {
- "jobId": "string",
- "groupId": "string",
- "jobName": "string",
- "jobType": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "appId": "string",
- "state": "string",
- "error": "string",
- "submitTime": 0,
- "startTime": 0,
- "endTime": 0
}
]
}
]
}Creates a group of interactive Spark jobs
Request Body schema: multipart/form-datarequired
| name required | string |
| description | string |
| kind | string Default: "JOB" Enum: "JOB" "CODE" |
| tags | string Default: "key=value;key2=value2" A semicolon separated list of tags. Parameter key and value should be separated by an equal sign. |
| language | string Default: "SCALA" Enum: "SCALA" "PYTHON" |
| clusterName required | string Default: "default" |
| jars | Array of strings <binary> [ items <binary > ] |
| files | Array of strings <binary> [ items <binary > ] |
| pyFiles | Array of strings <binary> [ items <binary > ] |
| jobConfig | string Default: "spark.key=value;spark.key2=value2" A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign. |
| pyRequirements | string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign. |
| scale | string Default: 1 Integer count of Spark jobs to be launched within this group |
Responses
Response samples
- 200
- 402
- 404
- 409
{- "groupId": "string"
}Returns a group found by its id
path Parameters
| groupId required | string The unique identifier of a group |
Responses
Response samples
- 200
- 404
{- "id": "string",
- "name": "string",
- "description": "string",
- "kind": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "state": "string",
- "scale": 0,
- "jobConfig": { },
- "createdTime": 0,
- "archivedTime": 0,
- "jobs": [
- {
- "jobId": "string",
- "groupId": "string",
- "jobName": "string",
- "jobType": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "appId": "string",
- "state": "string",
- "error": "string",
- "submitTime": 0,
- "startTime": 0,
- "endTime": 0
}
]
}Executes an interactive Spark job within chosen group
Executes an interactive Spark job within chosen group and waits for job instance to be completed. Its result is returned as a response.
path Parameters
| groupId required | string The unique identifier of a group |
Request Body schema: application/jsonrequired
| type required | string Default: "interactive_job_execute" |
| jobClass required | string |
required | object |
Responses
Request samples
- Payload
{- "type": "interactive_job_execute",
- "jobClass": "string",
- "jobConfig": { }
}Response samples
- 200
- 402
- 404
- 409
{- "jobInstanceId": "string",
- "jobId": "string",
- "groupId": "string",
- "startTime": 0,
- "endTime": 0,
- "jobClass": "string",
- "jobConfig": { },
- "result": "string",
- "error": {
- "message": "string",
- "stackTrace": "string"
}
}Returns a page of existing group job instance results
path Parameters
| groupId required | string The unique identifier of a group |
query Parameters
| search | string The word to apply for filtering |
| page | integer Default: 0 The number of page to be returned |
| size | integer Default: 10 The size of page to be returned |
| sort | Array of strings Example: sort=name,asc The sort rules to be applied for define paging order |
Responses
Response samples
- 200
{- "page": {
- "number": 0,
- "size": 0,
- "offset": 0,
- "paged": true
}, - "sort": [
- "string"
], - "totalElements": 0,
- "totalPages": 0,
- "first": true,
- "last": true,
- "empty": true,
- "content": [
- {
- "jobInstanceId": "string",
- "jobId": "string",
- "groupId": "string",
- "startTime": 0,
- "endTime": 0,
- "jobClass": "string",
- "jobConfig": { },
- "result": "string",
- "error": {
- "message": "string",
- "stackTrace": "string"
}
}
]
}Submits an interactive Spark job within chosen group
Submits an interactive Spark job within chosen group. It does not wait for job to be completed. Instead it returns created job instance id. Its result can be fetched by a job instance result update subscription.
path Parameters
| groupId required | string The unique identifier of a group |
Request Body schema: application/jsonrequired
| type required | string Default: "interactive_job_execute" |
| jobClass required | string |
required | object |
Responses
Request samples
- Payload
{- "type": "interactive_job_execute",
- "jobClass": "string",
- "jobConfig": { }
}Response samples
- 200
- 402
- 404
- 409
{- "jobInstanceId": "string"
}Changes the count of interactive Spark jobs that chosen group uses
path Parameters
| groupId required | string The unique identifier of a group |
| scale required | integer Number of instances to scale interactive Spark job to within group |
Responses
Response samples
- 402
- 404
{- "message": "string"
}Returns a stream of interactive Spark job result changes
query Parameters
| jobIds | Array of strings The list of job ids to apply for filtering |
| groupIds | Array of strings The list of group ids to apply for filtering |
| clusterNames | Array of strings The list of cluster names to apply for filtering |
| groupNames | Array of strings The list of group names to apply for filtering |
| tags | string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2 |
Responses
Returns a group found by its name
path Parameters
| groupName required | string The unique name of a group |
Responses
Response samples
- 200
- 404
{- "id": "string",
- "name": "string",
- "description": "string",
- "kind": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "state": "string",
- "scale": 0,
- "jobConfig": { },
- "createdTime": 0,
- "archivedTime": 0,
- "jobs": [
- {
- "jobId": "string",
- "groupId": "string",
- "jobName": "string",
- "jobType": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "appId": "string",
- "state": "string",
- "error": "string",
- "submitTime": 0,
- "startTime": 0,
- "endTime": 0
}
]
}Returns a page of existing jobs
query Parameters
| jobTypes | Array of strings (jobType) Items Enum: "SINGLE" "INTERACTIVE" "SHELL" The list of job types to apply for filtering |
| jobStates | Array of strings (jobState) Items Enum: "SUBMITTED" "STARTING" "READY" "BUSY" "FAILED" "FINISHED" "UNKNOWN" The list of job states to apply for filtering |
| clusterNames | Array of strings The list of cluster names to apply for filtering |
| tags | string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2 |
| search | string The word to apply for filtering |
| page | integer Default: 0 The number of page to be returned |
| size | integer Default: 10 The size of page to be returned |
| sort | Array of strings Example: sort=name,asc The sort rules to be applied for define paging order |
Responses
Response samples
- 200
{- "page": {
- "number": 0,
- "size": 0,
- "offset": 0,
- "paged": true
}, - "sort": [
- "string"
], - "totalElements": 0,
- "totalPages": 0,
- "first": true,
- "last": true,
- "empty": true,
- "content": [
- {
- "jobId": "string",
- "groupId": "string",
- "jobName": "string",
- "jobType": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "appId": "string",
- "state": "string",
- "error": "string",
- "submitTime": 0,
- "startTime": 0,
- "endTime": 0
}
]
}Returns a job found by its id
path Parameters
| jobId required | string The unique identifier of a job |
Responses
Response samples
- 200
- 404
{- "jobId": "string",
- "groupId": "string",
- "jobName": "string",
- "jobType": "string",
- "tags": "string",
- "language": "string",
- "clusterName": "string",
- "appId": "string",
- "state": "string",
- "error": "string",
- "submitTime": 0,
- "startTime": 0,
- "endTime": 0
}Returns logs for all job resources
path Parameters
| jobId required | string The unique identifier of a job |
query Parameters
| logSize | integer <int64> Default: 100 Max number of log lines to return. The limit in the API is set to 25000 lines, you can change it by setting ilum.api.job.logs.maxLines parameter in the configuration file of ilum-core |
| from | integer <int64> Default: 0 Offset from start of log |
Responses
Response samples
- 200
- 404
[- {
- "name": "string",
- "type": "string",
- "role": "string",
- "state": "string",
- "from": 0,
- "logSize": 0,
- "fileName": "string",
- "log": [
- "string"
]
}
]Submits a single Spark job
Request Body schema: multipart/form-datarequired
| name required | string |
| tags | string Default: "key=value;key2=value2" A semicolon separated list of tags. Parameter key and value should be separated by an equal sign. |
| language | string Default: "SCALA" Enum: "SCALA" "PYTHON" |
| clusterName required | string Default: "default" |
| jobClass | string |
| jars | Array of strings <binary> [ items <binary > ] |
| files | Array of strings <binary> [ items <binary > ] |
| pyFiles | Array of strings <binary> [ items <binary > ] |
| jobConfig | string Default: "spark.key=value;spark.key2=value2" A semicolon separated list of spark parameters to be applied for the Spark job. Parameter key and value should be separated by an equal sign. |
| pyRequirements | string A semicolon separated list of python requirements. Package name and version (if given) should be separated by a double equal sign. |
| args | string A semicolon separated list of spark arguments to be applied for the Spark job. |
Responses
Response samples
- 200
- 402
- 404
- 409
{- "jobId": "string"
}Returns a stream of Spark job changes
query Parameters
| jobIds | Array of strings The list of job ids to apply for filtering |
| groupIds | Array of strings The list of group ids to apply for filtering |
| clusterNames | Array of strings The list of cluster names to apply for filtering |
| tags | string A semicolon separated list of tags to apply for filtering. Pattern: key1=value1;key2=value2 |
Responses
Creates a cluster definition
Request Body schema: application/jsonrequired
| id | string |
| name required | string |
| description | string |
object | |
| sparkStorage | string Default: "s3" |
| defaultApplicationConfig | string |
| type required | string Default: "k8s" |
| url required | string |
| caCert | string |
| clientCert | string |
| clientKey | string |
| clientKeyAlgorithm | string |
| clientKeyPassphrase | string |
| username | string |
| password | string |
Responses
Request samples
- Payload
{- "id": "string",
- "name": "string",
- "description": "string",
- "storages": {
- "property1": {
- "sparkBucket": "string",
- "dataBucket": "string",
- "type": "s3",
- "endpoint": "string",
- "accessKey": "string",
- "secretKey": "string"
}, - "property2": {
- "sparkBucket": "string",
- "dataBucket": "string",
- "type": "s3",
- "endpoint": "string",
- "accessKey": "string",
- "secretKey": "string"
}
}, - "sparkStorage": "s3",
- "defaultApplicationConfig": "string",
- "type": "k8s",
- "url": "string",
- "caCert": "string",
- "clientCert": "string",
- "clientKey": "string",
- "clientKeyAlgorithm": "string",
- "clientKeyPassphrase": "string",
- "username": "string",
- "password": "string"
}Response samples
- 200
- 402
- 409
{- "clusterId": "string"
}Returns a page of existing clusters
query Parameters
| search | string The word to apply for filtering |
| page | integer Default: 0 The number of page to be returned |
| size | integer Default: 10 The size of page to be returned |
| sort | Array of strings Example: sort=name,asc The sort rules to be applied for define paging order |
Responses
Response samples
- 200
{- "page": {
- "number": 0,
- "size": 0,
- "offset": 0,
- "paged": true
}, - "sort": [
- "string"
], - "totalElements": 0,
- "totalPages": 0,
- "first": true,
- "last": true,
- "empty": true,
- "content": [
- {
- "id": "string",
- "name": "string",
- "description": "string",
- "sparkMaster": "string",
- "sparkDeployMode": "string",
- "sparkStorage": "string",
- "storages": {
- "property1": {
- "sparkBucket": "string",
- "dataBucket": "string",
- "type": "s3",
- "endpoint": "string",
- "accessKey": "string",
- "secretKey": "string"
}, - "property2": {
- "sparkBucket": "string",
- "dataBucket": "string",
- "type": "s3",
- "endpoint": "string",
- "accessKey": "string",
- "secretKey": "string"
}
}, - "defaultApplicationConfig": "string",
- "type": "k8s",
- "url": "string",
- "caCert": "string",
- "clientCert": "string",
- "clientKey": "string",
- "clientKeyAlgorithm": "string",
- "clientKeyPassphrase": "string",
- "username": "string",
- "password": "string"
}
]
}Returns a cluster found by its id
path Parameters
| clusterId required | string The unique identifier of a cluster |
Responses
Response samples
- 200
- 404
{- "id": "string",
- "name": "string",
- "description": "string",
- "sparkMaster": "string",
- "sparkDeployMode": "string",
- "sparkStorage": "string",
- "storages": {
- "property1": {
- "sparkBucket": "string",
- "dataBucket": "string",
- "type": "s3",
- "endpoint": "string",
- "accessKey": "string",
- "secretKey": "string"
}, - "property2": {
- "sparkBucket": "string",
- "dataBucket": "string",
- "type": "s3",
- "endpoint": "string",
- "accessKey": "string",
- "secretKey": "string"
}
}, - "defaultApplicationConfig": "string",
- "type": "k8s",
- "url": "string",
- "caCert": "string",
- "clientCert": "string",
- "clientKey": "string",
- "clientKeyAlgorithm": "string",
- "clientKeyPassphrase": "string",
- "username": "string",
- "password": "string"
}