Using Azure Factory Data to Get Data from REST API
Can I use Azure Data Factory to fetch data from a REST API and insert it into an Azure database table?
The data factory now has an http connector , which allows you to perform a GET or POST (with a body) on an http endpoint.
For example:
{
"name": "HttpLinkedService",
"properties":
{
"type": "Http",
"typeProperties":
{
"authenticationType": "Anonymous",
"url" : "https://en.wikipedia.org/wiki/"
}
}
}
I did this using Custom.Net Activities. I had to fetch data from the Salesforce API. I have an entry on how to do this here: http://eatcodelive.com/2016/02/06/accessing-azure-data-lake-store-from-an-azure-data-factory-custom-net- activity / uses data storage, but you can store data anywhere.
Also see an example here from Microsoft: https://azure.microsoft.com/en-us/documentation/articles/data-factory-use-custom-activities/
Hope it helps.
This can be achieved with Data Factory. Factory data is useful if you want to run batches on a schedule and have one location for monitoring and control. Our GitHub registry has sample code for an HTTP downloader to blob here https://github.com/Azure/Azure-DataFactory . The blob can then be consumed for other processing, like putting it on your desk. To do this, you will create a pipeline containing your custom loader and then a blob operation as an input dataset and a table as an output dataset.
https://github.com/Azure/Azure-DataFactory/tree/master/Samples/HttpDataDownloaderSample
Hi this is a custom data activity that downloads a file on an HTTP request. You can get what you want by making small changes. Hope it helps.
Is there a way to create paginated API calls using the response header where we get the total number of pages in the source system?
There is now REST support as a data source in Azure Data Factory, including pagination
Short answer: Yes :)
Long answer (mostly a tutorial): https://azure.microsoft.com/en-us/documentation/articles/data-factory-get-started/