Clever Geek Handbook

Using Google Cloud Storage with rsync

I am new to Google Cloud. We've historically used AWS for online backups - essentially our on-premise servers ran rsync against an EC2 instance on AWS and it all worked fine. Now I am trying to move from AWS to Google and of course the setup is completely different. With gsutil rsync it looked like I didn't need to spin up the compute engine at all, I could just push it right into the gs: // aws_mnt bucket

By installing the SDK on our AWS instance, I was able to use all of our backups in the gs: // aws_mnt bucket very easily with gsutil cp -n

But in the future, I want to run a cron job on a local server, which understandably uses rsync and not cp.

I have two problems:

  • Despite reading the relevant documentation ( here ) I am so stupid that I can’t figure out how to permanently allow the local server so I don’t need to do gcloud auth login and get the code from the browser on every session, nor does the cron work that doesn’t work.

  • When I try to use gsutil rsync from my local server to a gs: // aws_mnt bucket that was pre-populated with AWS, I get an error:

gsutil rsync / mnt / archive / backups gs: // aws_mnt / kahless State sync state ... Skip the cloud object subdirectory object gs: // aws_mnt / kahless / Start sync

There is some discussion about this error on github and I have drawn a detailed conclusion from

gsutil -D -m rsync / mnt / archive / backups gs: // aws_mnt / kahless

But since this is a brand new SDK install, I can't imagine the stream hasn't been processed already, so I must be doing something wrong?

Rus

+3


source to share


1 answer


In response to your questions:



  • Once you have configured credentials with gcloud auth, the "gcloud auth login" command will force them to select until you log in with different credentials ... and that state will persist and require you to log in browser again until / until you revoke those credentials. Note. If you plan on running commands from a standalone script (e.g. via cron), try using the credentials of a service account. For more information see https://developers.google.com/cloud/sdk/gcloud/#gcloud.auth

  • This "skips ..." message is not an error — it simply informs you that gsutil is skipping the attempt to load a placeholder object, since such objects are not needed in (and will interfere with) directories on the local filesystem. I'll update the post in the next version of gsutil to make this clearer. So you saw that the second run of gsutil rsync found nothing to do after comparing the source and destination, and completed fine.

+1


source






More articles:


All Articles