EDIT: I seem to be facing the same issue as this person: https://community.fabric.microsoft.com/t5/Dataflow/Overwrite-a-table-with-different-schema-in-Dataflow-Gen2/m-p/3752099
Hello, I am trying to create a Gen2 Dataflow where I can use Power Query to do some data transformations and then load back into the Lakehouse by overwrite.
I'm new to Fabric and trying to set up our data infrastructure on here.
The current goal (or what I am trying to achieve and what makes sense in my head) is:
A data pipeline that copies data from an on-prem SQL Server instance into a Lakehouse on Fabric.
I got this to work
Then, invoking a Dataflow Gen2 that adds a column to the table.
I am having issues with this. In the screenshot below, I created the new column.
Then when I choose the destination, the column is not appearing.
Unless I am not understanding how Dataflows work, I am wanting this to overwrite the table in the Lakehouse with these transformations.
Copy data from on-prem SQL Server > Lakehouse > perform transformations with Power Query > overwrite same table in Lakehouse with transformations.
I can see the new column being added through the Applied Steps.
But in the destination settings, it is not appearing.
Here is the data pipeline I have so far.