Data engineering as a product: how I work with data at a large bank

I'm Gabriel Henrique, a Data Engineer working on Data Architecture at a large bank, in the Wealth Management segment. Before that, I spent three years owning the pipelines that feed a commercial CRM tied to a business operation generating over R$ 500M in revenue, and I was the Data Owner of the internal data catalog.

The idea that changed my work the most along the way was treating every dataset as a product, with an owner, users, a contract and a quality bar. This article explains how that works in practice.

What changes when data has an owner

A pipeline without an owner breaks silently. Someone notices three days later, when the number in the meeting doesn't add up. A data product has one person who answers for it and knows who depends on it.

On the data catalog, I was the lead developer and the Data Owner. That meant answering product, marketing and sales teams about what each dataset means, where it comes from and when it refreshes. With 3,000+ monthly users, the documentation has to be right, because every mistake turns into dozens of wrong decisions.

The five pillars I use

1. An explicit contract

Every consumer needs to know what they will get: schema, business rule, delivery time. Today, in Wealth Management, I maintain the rules and data contracts that govern how curated audiences land in Salesforce. When the Lakehouse and the CRM disagree, the contract shows which one is right.

2. Reproducible, auditable logic

The segmentation rules for Wealth clients translate commercial strategy into filters. I write them as versioned code instead of loose queries, so anyone can run them again and get the same result. In banking, auditability is part of the requirement.

3. Quality and monitoring

I added quality controls and monitoring to the pipelines to catch failures before users do. The questions I automate are simple: did the data arrive on time, is the volume within the expected range, and are the key columns filled in.

4. Performance as a requirement

I refactored legacy SQL queries and ELT processes with Apache Spark on Databricks and cut processing time by more than 80%. Data that arrives late loses value, so I treat runtime as part of the delivery.

5. A bridge to the business

A big part of my job is conversation: gathering requirements with business teams, aligning deadlines and explaining in meetings what engineering will deliver. A technically perfect pipeline that answers the wrong question is useless.

SAS to Databricks: the product being built

My current focus is modernizing legacy SAS routines into Databricks. The work goes beyond translating code: I design canonical and derived data layers consumed by both the legacy CRM and Salesforce.

• Canonical layer: one trusted version of each business entity.
• Derived layers: views ready for each consumer, built from the canonical layer.
• CI/CD promotion: nothing reaches production without going through the deployment flow and internal standards.

I wrote more about this stack in the article on the DP-750 Azure Databricks certification.

Automation is a product too

I built a strategic web crawler and more than ten data governance automations with Python, Databricks, Azure and Apache Airflow. Each one follows the same rule as the pipelines: someone owns it, it is monitored and its result is measured. Orchestrating these routines is the subject of the article on DAG authoring in Airflow 3.

Where to start on your team

1. List your five most used datasets and give each one an owner.
2. Write a one-page contract for each: schema, rule and delivery time.
3. Add three automated checks: lateness, volume and nulls.
4. Measure runtime and attack the slowest pipeline first.

To see the projects where I applied this, the portfolio of Gabriel Henrique, Data Engineer has everything, and the experience section covers my path as a data engineer. I'm Gabriel Henrique, and I write here about data engineering, governance and the certifications I use every day.