Cloud Functions for Firebase BigQuery sync error

We are working on a cloud feature that allows us to sync our bigquery and firebase database. The function runs when a place is created / updated / deleted.

Based on the trigger action (create / update / delete) we add the big_query_active property to tell if the object exists or not. The same goes for the date.

Our current problem is that calling a large query sometimes returns an error. This means the data is no longer syncing. How can this be prevented?

'use strict';

// Default imports.

const functions = require('firebase-functions');
const bigQuery = require('@google-cloud/bigquery');

// If you want to change the nodes to listen to REMEMBER TO change the constants below.
// The 'id' field is AUTOMATICALLY added to the values, so you CANNOT add it.

const ROOT_NODE = 'places';
const VALUES = [
    'country_id',
    'category_id',
    'name',
    'active',
    'archived'
];

// This function listens to the supplied root node, but on child added/removed/changed.
// When an object is inserted/deleted/updated the appropriate action will be taken.

exports.children = functions.database.ref(ROOT_NODE + '/{id}').onWrite(event => {
    const query = bigQuery();
    const dataset = query.dataset('stampwallet');
    const table = dataset.table(ROOT_NODE);

    if (!event.data.exists() && !event.data.previous.exists()) {
        return;
    }

    const item = event.data.exists() ? event.data.val() : event.data.previous.val();

    const data = {};

    data['id'] = event.params.id;

    for (let index = 0; index < VALUES.length; index++) {
        const key = VALUES[index];
        data[key] = item[key] !== undefined ? item[key] : null;
    }

    data['big_query_date'] = new Date().getTime() / 1000;
    data['big_query_active'] = event.data.exists();

    return table.insert(data).then(() => {
        return true;
    }).catch((error) => {
        if (error.name === 'PartialFailureError') {
            console.log('A PartialFailureError happened while uploading to BigQuery...');
        } else {
            console.log(JSON.stringify(error));
            console.log('Random error happened while uploading to BigQuery...');
        }
    });
});

      

This is the error we (sometimes) get

{"code":"ECONNRESET","errno":"ECONNRESET","syscall":"read"}

      

How can you prevent data from not syncing? Or is there a way to try again so that it always succeeds?

+3


source to share





All Articles