I have a report I am trying to put together that is taking a source of data from SQL and transposing it. The problem I am having is that the dataflow is expecting a certain number of columns to be present, but this report will vary it might be 300 columns one month and 200 the next and 500 the month after that.
let
Source = Sql.Database("redacted", "redacted", [Query = "redacted"]),
#"Sorted rows" = Table.Sort(Source, {{"ValueColumn", Order.Ascending}}),
#"Filtered Rows" = Table.SelectRows(#"Sorted rows", each [Record_Type] <> "Org Owned"),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows", {"EMPLOYEE", "ADDDATE"}),
#"Transposed Table" = Table.Transpose(#"Removed Columns"),
#"Filtered Rows2" = Table.SelectRows(#"Transposed Table", each [Column1] <> 1),
#"Capitalized Each Word" = Table.TransformColumns(#"Filtered Rows2",{{"Column4", Text.Proper, typetext}}),
#"Added Conditional Column" = Table.AddColumn(#"Capitalized Each Word", "Custom", eachif [Column4] = "Cif_Key"then"System Information About Owner"elseif [Column4] = "Orig_Person"then"System Information About Account"elsenull),
#"Added Conditional Column1" = Table.AddColumn(#"Added Conditional Column", "Custom.1", eachif [Column4] = "~ZZZ~"then""elsenull),
#"Appended query" = Table.Combine({#"Added Conditional Column1", Query2}),
#"Duplicated Column" = Table.DuplicateColumn(#"Appended query", "Custom.1", "Custom - Copy"),
#"Added Index" = Table.AddIndexColumn(#"Duplicated Column", "Index", 0, 1, Int64.Type),
#"Filled Down" = Table.FillDown(#"Added Index", {"Custom - Copy"}),
#"Reordered columns" = Table.ReorderColumns(#"Filled Down", {"Custom", "Custom.1", "Column1", "Column2"}),
#"Removed columns 1" = Table.RemoveColumns(#"Reordered columns", {"Column3"}),
LSTHeaders = Table.ColumnNames(#"Removed columns 1"),
HowMany = List.Count(LSTHeaders),
Transformation = Table.TransformColumnTypes(#"Removed columns 1", Table.ToRows(Table.FromColumns({LSTHeaders, List.Repeat({typetext}, HowMany )}))),
#"Removed errors" = Table.RemoveRowsWithErrors(Transformation)
in
#"Removed errors"
For testing purposes, I intentionally inflated the number of records for the configuration of the dataflow, then when I remove the padded data I get an error about a column name not being in the dataset.
Error: Expression.Error: The column 'Column2819' of the table wasn't found. <ccon>Column2819</ccon>.
the mscipt doesn't explicity call out that column, is it being stored in-memory via the dynamic column type transformation? If I don't have that step power bi adds a transformation to all the columns shown.
