Apache Drill: table not found on bucket s3
I am a beginner with Apache Drill.
The scenario is as follows:
I have an S3 bucket where I put my CSV file named test.csv. I am installing Apache Drill with instructions from the official site.
I followed this tutorial: https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/ to create an S3 plugin.
I run Drill using the correct "workspace" (with: use my-s3;), but when I try to select records from the test.cav file, an error occurs:
Table 's3./test.csv' was not found.
Can anyone help me? Thank!
source to share
Use your workspace name (if you are using it) and back ticks in the USE command like this:
USE `my-s3`.`<workspace-name>`;
SHOW files; //should list test.csv file
SELECT * FROM `test.csv`;
Request a CSV on the local filesystem using the dfs storage plugin config to rule out issues like the header causing the problem. This page might help if you haven't seen it.
The storage plugin mentioned in the comment above:
{
"type": "file",
"enabled": true,
"connection": "s3n://<accesskey>:<secret>@catpaws",
"workspaces": {},
"formats": {
"psv": {
"type": "text",
"extensions": [
"tbl"
],
"delimiter": "|"
},
"csv": {
"type": "text",
"extensions": [
"csv"
],
"delimiter": ","
},
"tsv": {
"type": "text",
"extensions": [
"tsv"
],
"delimiter": "\t"
},
"parquet": {
"type": "parquet"
},
"json": {
"type": "json"
}
}
}
Perhaps it doesn't matter. This is an excerpt from the Amazon S3 Help and has a lot of information:
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>ID</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>SECRET</value>
</property>
source to share