Using standalone 'gsutil' from within GKE
I'm trying to use a standalone tool gsutil
from a container running on a GKE cluster, but I can't seem to get it to work. I believe the cluster has the appropriate permissions (see below). However, working
./gsutil ls gs://my-bucket/
gives
ServiceException: 401 Anonymous users does not have storage.objects.list access to bucket my-bucket.
Did I miss something? I don't have a file .boto
, as I believe it is not necessary, or is it? This is a list of scopes that have a cluster and a node pool:
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.full_control
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/pubsub
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/trace.append
Short answer:
Yes, you need some kind of boto file.
Long answer:
Generally, for GCE instances, you don't need a file ~/.boto
because the file is /etc/boto.cfg
already there - the Boto library that GSUtil uses knows by default. In Debian images, it contains the following lines:
# This file is automatically created at boot time by the /usr/lib/python
# 2.7/dist-packages/google_compute_engine/boto/boto_config.pyc script.
# Do not edit this file directly. If you need to add items to this file,
# create or edit /etc/boto.cfg.template instead and then re-run the
# script.
[GSUtil]
default_project_id = <PROJECT NUMBER HERE>
default_api_version = 2
[GoogleCompute]
service_account = default
[Plugin]
plugin_directory = /usr/lib/python2.7/dist-packages/google_compute_engine/boto
If you want to reproduce this behavior in your GKE container, you will need to have the python package installed google-compute-engine
and also have a boto file that tells gsutil to download that plugin from where it was installed, as seen above. In GCE (and I accept GKE too, although I haven't tested it), this plugin allows the VM to talk to its MDS server to get the credentials for a specified service account.
You can use gsutil inside a docker container in GKE with a service account or with your own credentials.
Service account
1) Add the file service-account.json
to your project.
2) Add to the project a .boto
file pointing to the file service-account.json
:
[Credentials]
gs_service_key_file = /path/to/service-account.json
3) In your Dockerfile, set an environment variable BOTO_CONFIG
to point to this file .boto
:
ENV BOTO_CONFIG=/path/to/.boto
Own credentials
1) Locally, run gcloud auth login. The file .boto
will be created at ~ / .config / gcloud / legacy_credentials / your@account.com /.boto with the following structure:
[OAuth2]
client_id = <id>.apps.googleusercontent.com
client_secret = <secret>
[Credentials]
gs_oauth2_refresh_token = <token>
2) Copy this .boto
file to your project
3) In your Dockerfile, set an environment variable BOTO_CONFIG
to point to this file .boto
:
ENV BOTO_CONFIG=/path/to/.boto
I have installed standalone gsutil in a docker container using pip install gsutil