Skip to main content

Exporting Notebooks to Pipelines · JPE

The JupyterLab Pipeline Exporter (JPE) is a JupyterLab 4 extension that promotes an interactive Sparkmagic notebook into a production artifact - an Airflow DAG, or a native Ilum Spark job in single, service, or cron mode - without leaving the notebook UI. The notebook is parsed in place, Jupyter-only constructs are stripped, and the resulting code is shipped to the configured backend.

JPE ships pre-installed in the helm_jupyter sub-chart. On a standard Ilum deployment, every backend URL is auto-wired from in-cluster Service DNS; no manual configuration is required.

Deployment Targets​

JPE offers two export targets, each with its own execution modes. Both target cards are enabled when their backend responds at panel load; the backend is probed and reflected as a status badge.

TargetModesMechanismBacking API
Ilum Spark jobsingle · service · cronNative Ilum. The selected mode determines how the notebook is packaged and submitted to ilum-core.POST /api/v1/job/submit, POST /api/v1/group, POST /api/v1/schedule
Airflow DAGper-cell · batchRenders a DAG Python file via Jinja2 using the Livy operator, pushes it to Gitea, and lets git-sync deliver it to the Airflow dag-processor. Optional auto-trigger after visibility.Gitea Contents API + Airflow REST API (/api/v2/dags)

Three further targets - SDP (Spark Declarative Pipelines), dbt project, and DuckDB via Quack - are visible in the panel as coming soon and are not yet selectable.

Independently of the target, any notebook can be downloaded as a Spark-runnable .py file for inspection or manual spark-submit; this download is always available, including fully offline.

Execution Modes​

Ilum Spark job​

The Ilum Spark job target submits to ilum-core in one of three modes, chosen with the mode toggle on the target card:

  • single - Wraps the notebook as a standalone pyFile and submits it as a one-shot job on an ephemeral Spark application that is stopped automatically when the job completes. (POST /api/v1/job/submit)
  • service - Wraps the notebook as an IlumJob subclass and registers a long-running Ilum service with a warm Spark driver, then runs the notebook against it on demand. (POST /api/v1/group, executed via POST /api/v1/group/<id>/job/execute)
  • cron - Registers the service together with a cron expression. Each fire is a single-shot Spark job triggered by ilum-core's internal scheduler - no Kubernetes CronJob is created. (POST /api/v1/schedule)

Airflow DAG​

The Airflow DAG target renders in one of two shapes:

  • per_cell - Default. Each code cell becomes one PythonOperator task chained sequentially; all tasks share a single Livy session opened by a leading livy_session task and closed by a trailing livy_cleanup task. Fine-grained DAG with individually retriable cells. More Livy round-trips than batch.
  • batch - A single PythonOperator task (run_notebook) runs the whole notebook inside one Livy session as one Spark batch job. Minimizes session startup overhead at the cost of cell-level retriability.

Cell Tags​

Two aspects of the generated Airflow DAG are controlled per cell through JupyterLab cell tags. Tags are read from the notebook metadata; they are inert during interactive execution and only consumed at export time.

TagApplies toEffect
task:<name>per_cell task idReplaces the generated id (cell_3, or c2_c3 for merged cells) with a stable, readable task_id.
retries:<n>Per-task retry countOverrides the DAG-level default retry count for that task.

Naming per-cell tasks​

In per_cell mode each task defaults to a generated id. Add a task:<name> cell tag (for example task:load_raw_sales) to assign a stable task_id. The name must be a valid Python identifier; invalid tags are ignored and fall back to the generated id. The active panel surfaces a Name your tasks hint while per_cell is selected, and the export preview shows a tagged-cell counter.

Per-task retry count​

Generated DAGs carry a DAG-level default of three retries with exponential backoff. To override the retry count for an individual task, add a retries:<n> cell tag, where <n> is a non-negative integer (for example retries:5). A flaky ingestion step can be given more retries while a deterministic transform is set to retries:0 to fail fast.

The value is applied per task:

  • In per_cell mode the tag sets retries=<n> on that cell's PythonOperator, overriding the DAG default for that task only.
  • In batch mode there is a single run_notebook task; the first valid retries:<n> tag among the exported cells applies to it.

A tag whose value is not a non-negative integer (for example retries:-1, retries:abc, retries:2.5) is ignored, and the task keeps the DAG-level default. Surrounding whitespace is tolerated (retries: 3). When multiple retries: tags are present on merged cells, the first valid one wins.

# Cell tags: task:load_raw_sales, retries:5
raw = spark.read.parquet("s3a://landing/sales/")
raw.write.saveAsTable("bronze.sales")

The cell above renders as:

load_raw_sales = PythonOperator(
    task_id="load_raw_sales",
    python_callable=_submit_statement,
    op_kwargs={"livy_conn_id": "ilum-livy-proxy", "code": "..."},
    retries=5,
)

The shipped example notebooks (Pipeline_Exporter_Showcase, Pipeline_Exporter_Ilum_Service_Showcase, and the docker-compose quickstarts) demonstrate both tags, assigning higher retry counts to I/O-bound steps and retries:0 to parameter cells.

Operating Modes​

Each backend is probed when the panel loads (≤1.5 s) and reflected as a status badge on the target card.

  • Standalone mode - No remote backend responds. Both target cards are dashed-bordered with a tooltip, and only the .py download is offered. Typical for a pip install jupyterlab-pipeline-exporter setup whose backend URLs have not been pointed at an Ilum or Airflow instance yet.
  • Connected mode - Some backends are reachable. Cards light up independently.
  • Ilum bundled mode - All backends auto-discovered: ilum-core:9888, ilum-airflow-api-server:8080, ilum-gitea-http:3000. DAG auto-trigger mints HS512 tokens locally from AIRFLOW_JWT_SECRET (the FAB+OAuth /auth/token endpoint is incompatible with Airflow 3.x).

Notebook Sanitization​

A Sparkmagic notebook contains constructs that crash an Airflow worker - no IPython display backend, no %magic resolver, no shell. JPE strips these before generating any artifact and reports every stripped line with its cell index and line number.

Removed by default:

  • Line magics - %manage_spark, %matplotlib, %load_ext
  • Shell escapes - !pip install <pkg>
  • Display calls - print(...), display(...) with no terminal sink in batch execution
  • Secret-looking literals - AWS keys, JWT payloads, common password patterns. The request is rejected with HTTP 400 unless allow_secrets: true is set.

Stripped lines appear in the rejected list of every response and in the preview panel. Set keep_rejections: true to retain a magic or display call (for example a print that writes to a log aggregator).

tip

Use Preview before committing. It returns the fully rendered DAG or pyFile together with the rejection report - without pushing to Gitea or Ilum.

Cell-kind filtering and the exported-cell count​

The enabledCellKinds setting decides which cell kinds (spark, pyspark, sql, scala, plain) are exported. The JupyterLab panel defaults to ["spark", "pyspark"], so a Sparkmagic notebook exports only its %%spark / %%pyspark cells; plain Python and %%sql cells are dropped and listed in rejected as filtered-cell:<kind>. A parameter cell is always retained regardless of the filter.

A cell that passes the kind filter but sanitizes to an empty body - for example a setup cell containing only line magics such as %load_ext sparkmagic.magics and %manage_spark - produces no task. It is reported in rejected as emptied-cell and excluded from the count.

Because of this, the cells value returned by an export is the number of cells that actually became tasks (or, for batch and pyFile targets, statements) in the generated artifact - not the raw count of code cells in the notebook. Skipped cells, kind-filtered cells, and emptied setup cells do not contribute to it. If every exportable cell is filtered out, the request is rejected with HTTP 422 so an empty pipeline is never produced.

Pipeline Parameters​

JPE recognizes parameter cells through two conventions:

  • A cell tagged parameters (the papermill convention)
  • A cell whose first non-blank line is # @pipeline-params (kebab) or # @pipeline_params (snake). The marker is a plain Python comment, so the cell executes interactively in any IPython kernel without raising UsageError: Cell magic … not found

Variables assigned in such a cell are emitted as config.get('<name>', <default>) lookups in the generated code, so the same notebook runs unchanged interactively (literal default) and in production (values injected by Airflow or the Ilum service request).

# Tagged: parameters
output_table = "gold.daily_kpi"
run_date = "2026-05-21"
threshold = 0.95

In the generated artifact the same variables resolve to:

output_table = config.get("output_table", "gold.daily_kpi")
run_date = config.get("run_date", "2026-05-21")
threshold = config.get("threshold", 0.95)

History and Re-Execution​

Every submission is recorded in the History tab with the export specification, generated code, and rejection report. The same record is persisted to ~/.jupyter/jpe-audit.jsonl for audit.

For Ilum service submissions JPE also keeps a re-execution payload - the service can be re-invoked from the History tab without re-uploading the pyFile or recreating the service.

Configuration​

All settings are in Settings → Plugin Settings → Pipeline Exporter. Defaults target in-cluster Ilum services; everything is overridable.

SettingDefaultDescription
ilumApiUrlhttp://ilum-core:9888/api/v1Ilum-core REST API used by single, service, and cron targets.
airflowApiUrlhttp://ilum-airflow-api-server:8080Airflow REST API used for DAG visibility checks and auto-trigger.
gitApiUrlhttp://ilum-gitea-http:3000/api/v1Git provider API used to push generated DAGs.
gitProvidergiteaGit provider type - currently Gitea in the bundled deployment.
defaultClusterIddefaultIlum cluster name pre-selected in the dialog.
defaultLivyConnIdilum-livy-proxyAirflow connection ID used by generated operators.
defaultModeper_cellAirflow export shape pre-selected in the dialog (per_cell or batch).
sparkImages6 presetsSpark image dropdown - Cluster default (empty, inherits the cluster's image) plus Spark 4.1.2 with Delta, Sedona, Iceberg, or Trino, and Spark 3.5.8 with Nessie + Sedona.
enabledCellKinds["spark", "pyspark"]Cell kinds (spark, pyspark, sql, scala, plain) included in the export. Cells outside this list are dropped and reported.

Override precedence (highest wins):

  1. Request payload fields - per submission.
  2. Pod environment variables (ILUM_API_URL, AIRFLOW_API_URL, GITEA_API_URL, GITEA_TOKEN, …).
  3. Plugin settings (JupyterLab Settings editor).
note

On a standard Ilum deployment the Helm chart wires every URL, secret, and JWT into the Jupyter pod from helm_jupyter/values.yaml. The Settings editor is only needed for non-bundled deployments or per-user overrides.

  • airflowIntegration.enabled (default true) gates the AIRFLOW_JWT_SECRET / AIRFLOW_API_URL env block. Set it false on deployments without Airflow so the disabled state is explicit at template render time rather than a silently empty token at runtime.
  • git.existingSecret alone gates the GITEA_USERNAME / GITEA_PASSWORD env vars JPE uses to push DAGs. The init-container that seeds the work dir into Gitea is gated separately by git.initialCommit.enabled, so a slim deploy without the gitea sub-chart can keep JPE's Gitea credentials without forcing the init loop to run. git.enabled is retained as a deprecated alias for git.initialCommit.enabled.

Quick Start​

  1. Open the notebook in JupyterLab.
  2. Click the Pipeline Exporter icon in the left sidebar (rank 102, below the file browser).
  3. Pick the Ilum Spark job target card. Select service mode.
  4. Enter a service name (for example daily-kpi-svc) and an optional parameter list.
  5. Click Create Ilum Service. A green banner confirms that the service has been created.
  6. Open the History tab. The new service entry exposes a Run action.
  7. Observe the run under Workloads → Services in the Ilum UI.
# Generated wrapper (excerpt)
class NotebookService(IlumJob):
    def run(self, spark, config):
        output_table = config.get("output_table", "gold.daily_kpi")
        # ... notebook body, sanitized

Generated Artifacts​

Target · modeArtifactLocation
Airflow DAG<dag_id>.pyGitea repository (<owner>/<repo>/<branch>:<subdir>/) - picked up by git-sync and parsed by dag-processor.
Ilum Spark job · singlepipeline_exp_<name>/main.pyUploaded to Ilum as part of the job submission; visible at /workloads/details/job/<id> in the Ilum UI.
Ilum Spark job · serviceNotebookService class in module pipeline_exp_<name>Registered as the Ilum service's main class FQN; invoked via POST /api/v1/group/<id>/job/execute.
Ilum Spark job · cronService + schedule entrySame as service plus a row in /schedules; each fire is a single-shot job triggered by ilum-core.
.py download (any target)Wrapped .py fileBrowser download - useful for inspection, manual spark-submit, or offline development.
warning

Service and cron modes wrap the notebook as an IlumJob subclass, whose base class comes from the ilum-python-job package (from ilum.api import IlumJob). The Spark image used to run the service must provide this package - the Ilum Spark runtime images ship it. If a custom image is used in service or cron mode, install ilum-python-job into it; otherwise the import fails at driver startup. The single mode and the Airflow DAG target do not require it.

Last updated on