Diagnose a failure and hand off the dataset

Inspect a code error in run history, restore the pipeline, and prepare collection access and dataset metadata.

9 min

In this final lesson, you deliberately make the pipeline fail once. You inspect its failed state and error details, restore the original code, and verify that the full pipeline succeeds again.

You then prepare the collection access and metadata that an analyst needs to find and understand the result dataset.

Keep a copy of the working code

This exercise temporarily changes the normalize_price_krw code asset. Copy the working code from Lesson 04 somewhere safe. You must restore it before completing this lesson.

The pipeline should still have this shape:

src_postgres_orders → normalize_price_krw → mart_orders_daily

Run information that is available

The current pipeline screen exposes run results in these places:

Run history in the top bar — small bars summarize recent run states.
The code node on the canvas — the node reflects running, successful, and failed states.
Inspector → History — for the selected code node, this shows each run's state, start time, duration, error description, and stack trace.

This is not an inline row preview for every node. After a run finishes, open the input or output dataset's Data view to inspect actual values.

Introduce a recoverable error

Create a column-name error that is easy to restore.

Edit the normalize_price_krw code asset.
Change pl.col("price_krw_str") to the nonexistent pl.col("price_krw_missing").
Save the code and the pipeline changes.
Select Run now.
Wait for the run to finish with a failed state.

Restore the code as soon as you have observed the failure, before the next scheduled run. On a real production pipeline, notify your team first so another user does not start it during the exercise.

Find the cause in History

Select the failed normalize_price_krw node on the canvas.
Open the inspector's History tab.
Select the newest run and expand its step state.
Find price_krw_missing in the error description or stack trace.
If needed, use the copy button beside the step to copy its status and error details.

Also confirm that the newest item in the top-bar run history is marked as failed. When the reported column matches the code you just changed, you have identified the cause.

Restore and run the full pipeline

Change pl.col("price_krw_missing") back to pl.col("price_krw_str").
Save the code, then select Save if the pipeline has unsaved changes.
Select Run now to execute the full pipeline again.
Confirm the new successful run in both the top-bar run history and the code node's History tab.
Open mart_orders_daily and confirm that its data can be queried again.

The current screen does not offer “restart from the failed node.” Selecting Run now after a fix executes the full pipeline. Because the effect on existing data depends on the output dataset's write settings, review those settings before rerunning a production pipeline.

Make the dataset understandable

A dataset name alone does not tell an analyst its grain or freshness. In the mart_orders_daily detail view, update the metadata that the current screen actually exposes.

Alias — a recognizable display name, such as Daily order amounts.
Description — grain, coverage, and refresh time, such as Order-date and item grain · KRW · Refreshed daily as of 03:10 KST.
Tags — business terms used in search, such as orders, daily, and finance.

Owner information comes from creation and permission relationships; it is not a free-form dataset metadata field. If users need a support contact, add the team name or contact procedure to the description.

Hand off through collection permissions

Use the permissions screen of the collection that contains the dataset.

Open the collection that contains mart_orders_daily.
Open the collection's Permissions screen.
Select the analyst user or group.
Grant Reader when they only need to view the data.
Verify with the analyst account that the collection and dataset are visible.

Use Writer only for collaborators who must change content, and Owner only for people who must manage the collection, including permissions. Reader is the usual starting role for an analyst who consumes the result in dashboards.

Self-check

Did the deliberate failure appear on both the code node and the top-bar run history?
Did History show the incorrect column name and stack trace?
Did you restore the original code and confirm a successful full-pipeline run?
Did you add an accurate alias, description, and tags to the result dataset?
Did you grant the analyst user or group the minimum collection role they need?

Path complete and next steps

You have completed the Data Engineer Path. You can now ingest data, transform it through a pipeline, schedule runs, inspect failures, and hand the result to an analyst.

Continue with one of these learning experiences:

Retail Inventory Intelligence — Practise the engineer-to-analyst flow in one scenario.
Quick scenario import (advanced, 10 min) — Learn how to bring a dhub2-examples scenario into your environment.
Analyst Path — See how a handed-off dataset is used in search, widgets, and dashboards.

Confirm that every lesson is marked complete, then choose your next course from Continue learning on the home page.