Adding a storage container to Azure with a different account?
We are trying to share data across multiple Outlook accounts. Let's say the data is stored in a container that is owned by data@outlook.com and I want to read it as datasc1@outlook.com , my friend wants to read from datasc2@outlook.com.
I have a generic account account name, container name (which is a public container), but when I try to read data using Hive with the command below:
CREATE EXTERNAL TABLE deneme (t1 string, t2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION 'wasb://container@storageaccount.blob.core.windows.net/OUR_DATA.txt';
OR I will also try below command
CREATE EXTERNAL TABLE deneme (t1 string, t2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION 'wasb://container@storageaccount.blob.core.windows.net/OUR_DATA.txt?sig=ACCESS_KEY_OF_CONTAINER';
I am getting the error below:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.hadoop.fs.azure.AzureException Uploads to to public accounts using anonymous access is prohibited.)
We tried some methods, we created a "Public Blob" container type that didn't work. We added our accounts to the default account store and that didn't work either. I tried loading data using PIG, it seemed like it worked, but when I dump the PIG also failed.
It seems strange to me, when I run the code below on the Hadoop command line, it works fine:
hadoop fs -lsr wasb://container@storageaccount.blob.core.windows.net/
:
lsr: DEPRECATED: Please use 'ls -R' instead.
-rwxrwxrwx 1 145391417 2015-05-18 10:58 wasb://container@storageaccount.blob.core.windows.net/OUR_DATA.txt
-rwxrwxrwx 1 25634418 2015-05-18 10:44 wasb://container@storageaccount.blob.core.windows.net/OUR_OTHER_DATA.txt
To summarize our problem, we are reading data from another Azure account with our Azure accounts using HDInsight (Hive / PIG / Hadoop).
Thanks in advance.
source to share
Does this work if you just point to a folder instead of a specific file? Hive expects folder paths to be in place, not specific files.
CREATE EXTERNAL TABLE deneme (t1 string, t2 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
LOCATION 'wasb://container@storageaccount.blob.core.windows.net/';
I was able to create a similar external table for a container configured as "Public Container".
If you don't want to use an open container, you can include the storage key in the config variable directly in the Hive script, for example:
set fs.azure.account.key.storageaccount.blob.core.windows.net=ACCESS_KEY_OF_CONTAINER;
Or, you can configure the cluster by configuring the time with storage account access permissions using the Additional Stores section of the Custom Create Wizard, or using the Add-AzureHDInsightStorage cmdlet to change the cluster configuration before creating the cluster.
This article has a bunch of related information on interactions between HDInsight and Azure Blob storage: http://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/
source to share
I solved this problem by adding below mentioned properties to core-site.xml
<property>
<name>fs.azure.account.key.<storage account name>.blob.core.windows.net</name>
<value><account key value></value>
</property>
Now go to the Vault account with the azure portal and change the access type to container "(by default it is" private ")
source to share