Quantcast
Channel: Service topics
Viewing all articles
Browse latest Browse all 61866

New column added from a Dataflow is not appearing when adding a Destination Source

$
0
0

EDIT: I seem to be facing the same issue as this person: https://community.fabric.microsoft.com/t5/Dataflow/Overwrite-a-table-with-different-schema-in-Dataflow-Gen2/m-p/3752099

 

Hello, I am trying to create a Gen2 Dataflow where I can use Power Query to do some data transformations and then load back into the Lakehouse by overwrite.

I'm new to Fabric and trying to set up our data infrastructure on here.

The current goal (or what I am trying to achieve and what makes sense in my head) is:

  1. A data pipeline that copies data from an on-prem SQL Server instance into a Lakehouse on Fabric.

    1. I got this to work

  2. Then, invoking a Dataflow Gen2 that adds a column to the table.

    1. I am having issues with this. In the screenshot below, I created the new column.

    2. Then when I choose the destination, the column is not appearing.

  3. Unless I am not understanding how Dataflows work, I am wanting this to overwrite the table in the Lakehouse with these transformations.

Copy data from on-prem SQL Server > Lakehouse > perform transformations with Power Query > overwrite same table in Lakehouse with transformations.

 

christianunr_0-1727191323387.png

I can see the new column being added through the Applied Steps.

 

christianunr_1-1727191329861.png

But in the destination settings, it is not appearing.

 

christianunr_2-1727191375558.png

Here is the data pipeline I have so far.


Viewing all articles
Browse latest Browse all 61866

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>