Parquet to Delta in Microsoft Fabric: How to automate the conversion for your Supermetrics data

Supermetrics writes Parquet to Fabric on a schedule, so the conversion to Delta needs to run on a schedule too (usually Copy activity with Upsert, with the REST API and Spark notebook covering narrower cases).

Converting a Parquet file to a Delta table in Microsoft Fabric is well documented, and it works fine if you only need to do it once. But a Supermetrics transfer to Microsoft Fabric doesn’t just run once. It runs on a schedule, and every run drops new Parquet files into your Lakehouse. If you want a Delta table that stays current, you need the conversion to run continuously too, not just the first time you check your data.

This article covers the three ways to automate Parquet-to-Delta conversion in Fabric, when each one makes sense, and how to handle the deduplication that comes with Supermetrics’ refresh windows.

There’s no single best approach here. One option covers almost every use case with the least ongoing maintenance, while the other two exist for specific situations you might encounter.

Key takeaways

• Supermetrics writes data to Microsoft Fabric as Parquet files in your Lakehouse’s Files section on a schedule.
• A one-time manual conversion is useful for checking that your first files landed correctly, but it doesn’t scale once your transfers keep landing.
• Fabric’s Data Pipeline Copy activity is the default recommendation for most teams. It converts Parquet to Delta directly and supports Append, Overwrite, and Upsert with key columns.
• Use the Load Table API if you already run an external orchestrator, like Airflow or Azure Functions, and want the conversion embedded there instead.
• Use a Spark notebook only if you need explicit control over schema or column types. It’s the most flexible option and the most overhead to maintain.
• Because you can set a refresh window in Supermetrics to resend a number of days on every run, deduplication matters. Upsert with key columns is how you handle it.

Why does a Parquet file need to become a Delta table in Fabric?

Because Fabric's two most common query paths, the Lakehouse SQL analytics endpoint and Power BI Direct Lake, only read Delta tables. A raw Parquet file in the unmanaged Files section isn't queryable until you convert it.

Right-clicking a Parquet file in the Files section shows every available action on it. Load to Tables is the only path forward, there's no option to query the file directly.

Once that file is loaded into a table, it's immediately queryable through the SQL analytics endpoint using standard T-SQL.

Two other differences also matter once your data is live and updating:

1. Parquet files are immutable. Updating or deleting a row means rewriting the whole file. Delta tables support MERGE, UPDATE, and DELETE directly, so partial changes don’t require a full rewrite.
2. Parquet doesn’t enforce schema. A new column or a changed data type can break a downstream job without warning. Delta enforces schema on write and supports controlled schema evolution, so changes like adding a column don’t break existing readers.

What are your options for automating Parquet-to-Delta conversion?

You have three options: a Data Pipeline Copy activity, the Load Table API, and a Spark notebook. Each is suited to a different situation.

A transfer using the Supermetrics Fabric destination runs on a schedule, pushing Parquet files into your Lakehouse. During a trial, teams start by checking how the data lands: file structure, scheduling, monitoring. That first check is usually done with a manual, one-time conversion, just to look at the file.

The real question comes after those first few days: how do you keep converting new files without doing this by hand every time? There are three viable paths, and they serve different situations rather than competing head-to-head.

Why use the Data Pipeline Copy activity as your default?

It's the simplest option and it's built into Fabric's own automation flow, so most teams don't need to maintain anything external. A Copy activity reads Parquet files from your Lakehouse Files section and writes them directly to a Delta table. You can trigger it on a fixed schedule, or set up a Data Activator rule to run it automatically when a new file arrives.

How do you handle deduplication when your refresh window resends data?

Use the Upsert table action with key columns. This is the Copy activity’s equivalent of Delta’s MERGE, and it’s what keeps repeated data from becoming duplicate data.

Here’s why this matters specifically for Supermetrics data: refresh windows are set to resend a number of days on every run, often matching the maximum attribution window for a paid channel or accounting for data still settling at the source. That’s deliberate. It also means the same day’s data can land more than once across different transfer runs.

The Copy activity gives you three table actions when writing to a Delta destination:

1. Append: adds new rows. On its own, this creates duplicates for any day your refresh window resends.
2. Overwrite: replaces the whole table’s data and schema with the new values.
3. Upsert: inserts new rows and updates existing ones, based on key columns you choose.

For a recurring transfer with an overlapping window, Upsert with key columns is what keeps the table accurate without duplicate rows.

One thing worth checking before you turn it on: Upsert only works as well as the key columns you pick. If the key columns don’t fully capture what makes a row unique, for example missing a date or a campaign-level dimension, rows that should stay separate can get collapsed into one. It’s worth confirming your key columns against the full grain of your destination table before relying on Upsert in production. Also worth knowing: partition columns and key columns can’t overlap, so if you’re partitioning the table, factor that into which columns you choose for the dedup key.

When should you use the Load Table API instead?

If you already run an external orchestrator, like Airflow or Azure Functions, and don’t want a second automation living inside Fabric, the Load Table API is the better fit. It’s the programmatic version of the manual conversion: you call the Load Table API after each file drop and poll until it completes. Not every team wants to maintain an external process, so this isn’t the default, but it’s the right call if that process already exists.

When do you actually need a Spark notebook?

Only when you need explicit control over schema or column types that automatic inference doesn’t give you. Both the Copy activity and the REST API infer schema automatically, which covers most cases. A Spark notebook gives you full programmatic control instead, at the cost of owning that schema logic yourself going forward rather than letting Fabric handle it.

One watch-out worth flagging: external Delta tables created through Spark code aren’t visible to the Lakehouse SQL analytics endpoint by default. If querying through the SQL analytics endpoint or Power BI Direct Lake mode is the whole reason you’re converting to Delta in the first place, you’ll need to add a shortcut in the Tables section to make that external table visible. Otherwise you’ve done the conversion but the table still won’t show up where you need it.

Conclusion

A manual, one-time Parquet-to-Delta conversion is a fine way to check that your first files landed correctly. However, it’s not built to keep up once your Supermetrics transfer is running on a schedule. For most teams, the Data Pipeline Copy activity is the right default, and pairing it with Upsert and key columns handles the deduplication that comes with a recurring refresh window. If you already have an external orchestrator, the Load Table API fits into that process directly. And if you need precise schema control, a Spark notebook gives you that, with more to maintain in exchange.

If you’re still evaluating Fabric as a destination, explore the Supermetrics Fabric destination to see how the transfer itself is configured.

FAQs