CSV Reference in Azure Stream Analytics
I have a stream parsing application where events are JSON encoded and look like
{customerID: 45, state:"S2" , timestamp:"2017-06-06 14:19:21.77"}
{customerID: 74, state:"S2" , timestamp:"2017-06-06 14:19:26.61"}
{customerID: 79, state:"S2" , timestamp:"2017-06-06 14:19:28.50"}
{customerID: 10, state:"D" , timestamp:"2017-06-06 14:19:31.79"}
{customerID: 70, state:"S2" , timestamp:"2017-06-06 14:19:31.93"}
{customerID: 37, state:"S2" , timestamp:"2017-06-06 14:19:32.17"}
{customerID: 41, state:"D" , timestamp:"2017-06-06 14:19:33.48"}
I have reference data for clients in a CSV file that looks like this:
"CUST_ID", "Age", "Rich"
1, "50", "Y"
2, "22", "N"
I am uploading datafiles in the above mentioned formats and check the following request
select A.[CUSTOMERID], A.[state], B.[AGE], B.[GENDER_CODE] from clickstream A timestamp by A.[TIMESTAMP] left join refdata B on A.[CUSTOMERID]=B.[CUST_ID]
I am getting the following error message without any details:
Now, the same request works fine as long as the data data is represented as JSON. Is there a working example where I can look at the CSV reference data?
source to share
I had the same problem with flow analytics.
Just FYI, converting the CSV file to a .json file (using an online tool for example) and changing the serialization format of the original data to JSON solved the problem. I can now test the request by sending a json file through the Azure portal request tool. It looks like the implementation of the CSV flow analytics is a bug as I am sure my CSV was formatted correctly.
source to share
I have reference data for clients in a CSV file
When we create the input , we need to specify the event serialization format , which allows Stream Analytics to know which serialization format (JSON, CSV, or Avro) we are using for the incoming data streams. Please check the event serialization format of your refdata input and make sure it is set to CSV.
source to share
Test button with reference data didn't work for me at all. I had to use Azure Stream Analytics Tools for Visual Studio for testing on-premises.
Regarding an error in the path pattern. I used YYYY-MM-DD
and was used by default for thread analysis YYYY/MM/DD
. As the work was done, I saw an exclamation mark next to the entrance to the flow analytics dashboard.
To debug the error, you must click on the benchmark data in the Stream Analytics overview and you will see warnings on your client.
Mine warning:
Initializing input without a valid reference data blob for UTC time 11/14/2017 12:22:32 PM, example path: 'https://blabla.blob.core.windows.net/cohorts/2017/11/14/12-22/result.csv'
source to share