Is there any way to just manipulate a string in Azure Data Factory?
Is there any way to just manipulate a string in Azure Data Factory?
Something as simple as I have a block of memory with a tab-separated file and I want to move it to the storage table ... but do a few
And I want to say, convert tabs to comma, concatenate columns from 4 to last column
source to share
You can do this with Custom Activity , but all you need to remember about the Azure Data Factory is that it is mostly for orchestration (not transform). This means that he does not have much opportunity to do the conversion.
Another way to do this is to use Azure Data Lake Analytics (ADLA), which is probably going to be my preference as I use it a lot at the moment. If you don't have an ADLA account yet, though it might seem a little overhead.
I took a sample file and converted it as per your requirement using U-SQL using this example file :
DECLARE @inputFilepath string = "input/input67.tsv";
DECLARE @outputFilepath string = "output/output67.csv";
@input =
EXTRACT rowId int,
col1 int,
col2 int,
col3 int,
col4 int
FROM @inputFilepath
USING Extractors.Tsv(skipFirstNRows : 1);
// Concat the four columns
@output =
SELECT rowId,
string.Concat(col1.ToString(), col2.ToString(), col3.ToString(), col4.ToString()) AS col5
FROM @input;
// Export as csv
OUTPUT @output
TO @outputFilepath
USING Outputters.Csv(quoting:false);
source to share