Scheduling Apache Spark Jobs

Scheduling in Ilum allows you to automate the execution of Apache Spark jobs on Kubernetes clusters at specified intervals using CRON expressions. This is essential for setting up reliable ETL pipelines, regular data analysis, or maintenance tasks that need to run without manual intervention.

Example JAR Source

You can use the jar file with spark examples from one of these links:

Spark 4 (default)

Spark 4 / Scala 2.13: spark-examples_2.13-4.1.1.jar

Spark 3

Spark 3 / Scala 2.12: spark-examples_2.12-3.5.7.jar

Step-by-Step Guide: Scheduling a Spark Job

1. Navigate to Schedules: Access the Schedules section in your Ilum dashboard.

2. Create New Schedule: Click the New Schedule + button to start setting up your automated job.

3. Fill Out Schedule Details:

   • General Tab:

     • Name: Enter ScheduledMiniReadWriteTest
     • Cluster: Select your target cluster
     • Class: Enter org.apache.spark.examples.MiniReadWriteTest
     • Language: Select Scala

   • Timing Tab:

     • CRON Expression: Select the Custom tab.
     • Custom expression: Enter @daily
     Timing

     This configuration will trigger the job to run once every day at midnight. You can adjust this to any valid CRON expression (e.g., 0 */12 * * * for every 12 hours).

   • Configuration Tab:

     • Arguments: Enter /opt/spark/examples/src/main/resources/kv1.txt

   • Resources Tab:

     • Jars: Upload the JAR file from the link above.

   • Memory Tab:

     • Leave all settings at their default values for this example.

4. Submit and Monitor:

   • Click Submit to create the schedule.
   • You can see your new schedule in the list.
   • When the scheduled time arrives, a new job instance will be launched automatically. You can view these instances in the Jobs section.

Schedule Configuration Reference

Below is a detailed breakdown of all available settings, organized by tab as they appear in the UI.

General

Parameter	Description
Name	A unique identifier for the schedule.
Cluster	The target cluster where the scheduled jobs will be executed.
Class	The fully qualified class name of the application (e.g., org.apache.spark.examples.SparkPi) or the filename for Python scripts.
Language	The programming language used for the job (Scala or Python).
Description	An optional description to explain the purpose of this schedule.
Max Retries	The maximum number of times Ilum will attempt to restart the job if it fails.

Timing

Parameter	Description
Start Time	(Optional) The specific date and time when the schedule should become active. If left blank, it starts immediately.
End Time	(Optional) The specific date and time when the schedule should stop triggering new jobs.
CRON Expression	Defines the frequency of the job execution. You can use the visual builders (Minutes, Hourly, Daily, Weekly, Monthly) or select Custom to enter a standard Unix-style CRON expression (e.g., 0 12 * * * for noon daily).

Configuration

Parameter	Description
Parameters	Key-value pairs for configuring Spark properties (e.g., spark.executor.memory).
Arguments	Command-line arguments passed to the main method of your application.
Tags	Custom labels to categorize and filter your scheduled jobs.

Resources

Parameter	Description
Jars	Additional JAR files to be included in the classpath.
Files	Auxiliary files to be placed in the working directory of each executor.
PyFiles	Python dependencies (.zip, .egg, .py) for Python jobs.
Requirements	Additional Python packages to install.
Spark Packages	Maven coordinates for Spark JAR packages to be downloaded.

Memory

Parameter	Description
Executors	The number of executor instances to allocate.
Driver Cores	The number of CPU cores assigned to the driver.
Executor Cores	The number of CPU cores assigned to each executor.
Driver Memory	The amount of RAM allocated to the driver.
Executor Memory	The amount of RAM allocated to each executor.
Dynamic Allocation	Enables automatic scaling of executors based on workload.

Frequently Asked Questions

Details

Can I schedule PySpark jobs using Ilum?

Yes, Ilum fully supports scheduling for both Scala/Java (JARs) and Python (PySpark) jobs. Simply select "Python" as the language in the General tab and provide your script.

Details

How does the retry mechanism work?

If a scheduled job fails, Ilum can automatically attempt to restart it based on the "Max Retries" configuration. This ensures transient issues don't break your pipelines.

Details

What CRON formats are supported?

Ilum supports standard Unix-style CRON expressions (e.g., 0 12 * * *) as well as predefined macros like @daily, @hourly, etc.