Data Engineer Path

About this Path

This Path teaches D.Hub portal from a data engineer's point of view. If the Analyst Path is about building result screens on top of data that already exists, the Engineer Path is about how that already-existing data got there — pulled in from external systems, transformed, scheduled to run on its own every day, and caught when something fails. Six lessons walk you through the full cycle. Together, about 50 minutes.

Each lesson is sized at 5–10 minutes. Do them in one sitting, or spread one or two a day.

The six lessons connect external data to an analyst handoff as one operating flow.

flowchart LR
    accTitle: Data Engineer Path end-to-end flow
    accDescr: Connect an external system, ingest and transform data through pipelines and Code, save a result dataset, schedule and monitor runs, and hand the dataset to an analyst.
    external[(External system)] --> connector[Connector and ingestion]
    connector --> pipeline[Pipeline]
    pipeline --> code[Code transformation]
    code --> dataset[(Result dataset)]
    schedule[Schedule] --> pipeline
    pipeline --> monitor[Run and error monitoring]
    dataset --> handoff[Analyst handoff]

Terms to know before you start

• Connector: Connection settings and behavior for reaching an external database or service.
• Pipeline: An executable flow that reads inputs, transforms data, and writes outputs.
• Input dataset: An existing dataset that a pipeline reads.
• Output dataset: A dataset that receives transformed rows. Inputs and outputs are the same dataset resource type; their connection position determines their role in a pipeline.
• Transform node: A processing step that selects, renames, casts, joins, or aggregates input data.
• Code node: A pipeline step written in Python or SQL for transformations not covered by standard processing.

Prerequisites

• Portal access at the Editor level or higher. You'll need permission to register connectors and run pipelines.
• Either finishing the Analyst Path, or familiarity with the basics of collections and datasets.
• One of the following practice data sources:
  • A read-only account on an internal production database, or
  • The provided src_postgres_orders.csv (you can still complete the pipeline exercise without connector registration permission)

Prior experience with dbt, Airflow, or Snowflake helps map things faster but isn't required. Lesson 01 lays out the correspondence between those tools and portal surfaces in a single table.

What you'll be able to do

• Describe the four surfaces engineers usually live in inside portal (Connectors, Pipelines, Codes, Datasets), along with the inputs, outputs, and boundaries of each.
• Register one connector against an external system and land production data as a portal dataset.
• Connect and run an input dataset → transform node → output dataset flow in the Workflow editor.
• Write Python or SQL in a code node for transformations the standard nodes can't cover.
• Make the same pipeline run automatically on a fixed schedule and route failure alerts.
• Narrow down a failed run using logs and node states, and decide between a full re-run and a partial re-run.
• Hand the datasets you produced over to an analyst cleanly by tidying up permissions.

What comes after this Path

If you want to go deeper inside the engineer flow, these are the natural next steps.

• Retail Inventory Intelligence — Every surface in this Path (connectors, pipelines, code nodes, scheduling) is wired together inside a real domain scenario. You walk through one full cycle with an analyst alongside you. About 90 minutes.
• Quick scenario import (advanced, 10 min) — Load one scenario from dhub2-examples into your environment with a single command. It uses the same tools as step 1 of the workshop.
• Analyst Path — See how your datasets show up in the analyst's collection tree. This acts as a second pass over the handoff step just before lesson 06.

The checkboxes next to each lesson record progress automatically. Pick one up and start.