Skip to main content

How to run interactive spark job

An interactive Spark job is a dynamic and responsive way to execute Spark tasks that allows for real-time manipulation and querying of data directly within a live Spark session. Unlike a traditional Spark job, which is typically batch-oriented and executed as a single, predefined workflow, an interactive job offers a more flexible and exploratory environment.

Interactive Guide​

Guide in Full Screen

Here's a step-by-step guide to setting up a simple Spark job using Ilum. This guide will walk you through configuring, executing, and monitoring a basic job named MiniReadWriteTest within the Ilum platform.

Step-by-Step Guide to Running Interactive Spark Job in Ilum​

  1. Access the Group Section: Navigate to the 'Group' section on the Ilum dashboard.

  2. Create a New Group: Click on the "New Group" button to start setting up your interactive job environment.

  3. Group Details:

    • Group Name: Assign a unique name to your group for easy identification.
    • Cluster Selection: Select the cluster on which you want to start the group.
    • Replicas: Specify the number of replicas if multiple users need access to the interactive group simultaneously.

  4. Resource Setup:

    • Navigate to the resources tab.
    • Upload Job Code: Upload the executable jar file containing your job code. For this example, use the special job designed to count the Pi number interactively, similar to SparkPi but fully interactive.

  5. Activate the Group: Submit the group to activate it.

  6. Execute the Job:

    • Provide the full class name of your interactive job.
    • Enter any necessary job parameters. Customize these according to your needs.

  7. Initial Run:

    • Execute the job. Note that the first run might take longer due to session initialization.
    • Execute the job again to see how the initialization speeds up subsequent runs.

  8. Parameter Adjustment:

    • Modify the parameters as needed and re-execute the job multiple times to observe different outcomes.
    • Increase parameter values to test different computational loads.

  9. Monitoring and Adjustments:

    • Navigate to the group details to review all requests sent to the group.
    • Examine the parameters and results of each specific request.
    • Monitor the execution timeline and the memory usage of each executor in the Executors section.
    • Check the logs for detailed execution information.

  10. Conclusion: Congratulations on successfully setting up and running your interactive job in Ilum! For further information or support, reach out to us at [email protected].

Benefits of interactive spark jobs​

Here's how interactive Spark jobs differ from regular Spark jobs:

  1. Real-time Interaction:

    • Interactive Jobs: Facilitate a direct, real-time interaction with data. Users can dynamically execute queries or commands through Ilum's intuitive UI or programmatically via the API, receiving immediate results. This capability is perfect for exploratory data analysis and iterative development.
    • Regular Jobs: Execute pre-defined scripts in batch mode, designed to run without interaction until completion. Ideal for scheduled data transformations and large-scale processing tasks.

  2. Session Persistence:

    • Interactive Jobs: Sessions remain active across multiple queries, preserving the state and data in memory. This persistence speeds up repeated data operations by avoiding the constant overhead of reloading data or reinitializing environments.
    • Regular Jobs: Each job execution is isolated, starting and ending its lifecycle without retaining any intermediate state between runs.

  3. Use Case Suitability:

    • Interactive Jobs: Especially beneficial in production environments where quick, repetitive job executions are necessary. They can handle simultaneous requests from multiple users efficiently, thanks to Ilum’s built-in load balancing which distributes the workload across available resources.
    • Regular Jobs: Suited for predictable, repetitive tasks where the workflow and data processing requirements are well-defined and stable over time.

  4. Resource Utilization:

    • Interactive Jobs: Continuously consume resources due to their persistent nature, which could lead to higher resource utilization. However, Ilum mitigates this by allowing interactive sessions to be paused after a period of inactivity, effectively freeing up resources. This feature ensures that the sessions are resource-efficient, reactivating quickly upon receiving new user commands via the UI or API.
    • Regular Jobs: Typically more resource-efficient in scenarios where jobs are infrequent or do not require immediate repetition, as resources are only engaged during the job execution period.

This nuanced approach of interactive Spark jobs in Ilum supports not just the execution of data tasks but does so with an optimal balance of speed, interactivity, and resource management, catering to a broad spectrum of data processing scenarios.