Clever Geek Handbook

AppEngine & BigQuery - Where would you put your statistics / monitoring data?

I have an AppEngine app that processes files from Cloud Storage and inserts them into BigQuery.

Because now and also in the future, I would like to know about the health of the application ... I would like to store statistics data in a cloud storage or a Cloud SQL instance.

I have two questions that I would like to ask:

  • Cloud Datastore vs Cloud SQL - which would you use and why? What disadvantages have you experienced so far?

  • Are you using task

    or direct call to insert data

    as well as why? - Could you add a task and then add some users to the data or do a direct paste [regardless of the solution chosen above]. What disadvantages have you experienced so far?

Thank.

+3


source to share


1 answer


  • Cloud SQL is better if you want to execute JOINs or SUMs later, Cloud Datastore will scale more if you have a lot of data to store. Also, in the datastore, if you want to update the statistics object transactionally, you will need to penalize or you will be limited to 5 updates per second.

  • If the data to insert is small (one row to insert into BQ, or one object in the data store), you can do so by calling directly, but you must admit that the call may fail. If you want to retry if it fails, or if the data to insert is large and it will take time, it is best to run it asynchronously in a task. Note that you have to be careful with the y tasks because they can be run more than once.



+3


source






More articles:


All Articles