Your first pipeline — three nodes and one run

Connect an input dataset, a built-in transform, and an output dataset, then run the batch pipeline once.

8 min

A pipeline connects datasets, code assets, and built-in transforms from one collection into an executable flow.

In this lesson, you rename the source column ord_dt_str to order_date and write the rows to a table dataset. In the current editor, you connect the collection's input dataset → Quick Add's Rename/Cast transform → collection output dataset.

Prerequisite

Use the src_postgres_orders dataset from the previous lesson. If you skipped the connector exercise, use another tabular dataset in the same collection.

Create the output table first

Before opening the pipeline editor, create mart_orders_clean as a real dataset in the collection. A table created through Quick Add can currently leave a built-in transform pointing to an unresolved dataset reference when the pipeline is saved.

Open the collection that contains src_postgres_orders.
Select Add Item → Dataset → Table.
Enter mart_orders_clean as the name.
Define order_id as Text, order_date as Text, and amount as Integer.
Select Create and confirm that mart_orders_clean appears in the collection.

Open the editor and select a collection

Select Pipelines in the left sidebar.
Select Create in the upper-right.
On the new pipeline screen, first select the collection where the pipeline will be saved.

You cannot add or run pipeline items until a collection is selected. A pipeline can use only datasets and code from that selected collection, so choose the collection that contains your input dataset.

The Component Library on the left has a Quick Add section for new items and a Collections section for existing items. The canvas is in the center, and selecting a node opens its configuration and I/O in the inspector on the right.

Add the input dataset

In the Component Library's Collections section, expand the selected collection.
Drag src_postgres_orders onto the canvas.

This node references the dataset already stored in the collection; it does not create another copy.

Add a built-in transform

Drag Quick Add → Transform → Rename/Cast onto the canvas.
Drag from the input dataset's right handle to the transform's left handle.
Select the transform and open the inspector's Config tab.
Choose the connected dataset under Input port.
Add a column mapping with ord_dt_str under Column and order_date under Alias. Keep the data type unchanged.
Confirm that validation reports the node as valid, then select Preview.

The current built-in transforms are Select/Filter, Join, Union, Aggregate, and Rename/Cast. This pipeline uses Rename/Cast.

Add and connect the output dataset

In the Component Library's Collections section, expand the selected collection.
Drag the existing mart_orders_clean dataset onto the canvas. Do not create the output through Quick Add for this exercise.
Connect the transform's right handle to mart_orders_clean.

Your canvas should now show:

src_postgres_orders → Rename/Cast → mart_orders_clean

Save and run once

Select Save in the upper-right.
Enter mart_orders_clean_pipeline and keep the pipeline type as Batch.
After it is saved, select Run now.

Selecting Run now on an unsaved pipeline opens a save-first prompt. Saving and running separately the first time makes it easier to tell a save error from an execution error.

Follow the node states on the canvas and the run history bar at the bottom. When the run finishes, open mart_orders_clean in the collection and verify:

order_date appears instead of ord_dt_str;
order_id and amount match the source; and
all three practice rows are present.

Self-check

Did you select the pipeline collection before adding items?
Did you create mart_orders_clean in the collection before opening the editor?
Are the existing input dataset, Rename/Cast transform, and existing output dataset connected in order?
Did you save the pipeline before selecting Run now?
Does the output dataset show the expected schema and rows?

Next lesson

Next, you write logic that is awkward to express with built-in transforms as a reusable Python or SQL code asset and connect it to a pipeline.