How do I execute adhoc SQL queries on flat files generated by Hive?

We parse the log data with Hive and store the aggregation results in daily partitioned text fields on S3 (let's call them "rough" aggregations).

These aggregation results are pretty small (no more than a few MB per day) and we have a Javascript toolbar that loads and renders some aspects of this data (call them "fine-grained" clusters).

We are now doing "fine-grained" aggregations with Javascript code. I just want to use SQL queries here, for simplicity. I wonder what best practices exist for this kind of problem?

A) We could generate all the "fine-grained" aggregations in Hive. However, working with these small datasets is time consuming in Hive.

B) We could introduce a "quick access layer" between S3 and Javascript that can run SQL queries. What request mechanism would you recommend?

+3
javascript sql hadoop hive hiveql


source to share


No one has answered this question yet

Check out similar questions:

4829
How do I include a JavaScript file in another JavaScript file?
3491
How can I UPDATE from SELECT in SQL Server?
2817
How can I upload files asynchronously?
2776
How can I prevent SQL injection in PHP?
2701
How do I get the values โ€‹โ€‹of a query string in JavaScript?
1762
How to import SQL file using command line in MySQL?
1658
How to return only date from SQL Server DateTime datatype
1587
Insert multiple rows into one SQL query?
1373
How do I execute IF ... THEN in SQL SELECT?
1299
How to query MongoDB with "how"?



All Articles
Loading...
X
Show
Funny
Dev
Pics