Clever Geek Handbook

How do I get a custom health check path in the GCE L7 load balancer serving Kubernetes Ingress?

I am trying to deploy a grafana instance inside Kubernetes (Server 1.6.4) in GCE. I am using the following manifests:

Deployment ( full version ):

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: grafana
    spec:
      initContainers:
        …                                
      containers:
        - name: grafana
          image: grafana/grafana
          readinessProbe:
            httpGet:
              path: /login
              port: 3000
          …

Service

apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  selector:
    name: grafana
  ports:
    - protocol: TCP
      port: 3000
  type: NodePort

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana
spec:
  tls:
    - secretName: grafana.example.com
  backend:
    serviceName: grafana
    servicePort: 3000

It turns out grafana serves 302 on /

, but by default the GCE ingress healthcheck expects 200 on /

( source ). As you can see, the Deployment has a customabilityProbe line (line 22).

As soon as I send these resources to the cube-apiserver everything is created without error. Specifically, Ingress gets a public ip4 address, but the healtcheck is configured by default /

instead of the custom one configured inreadinessProbe

. Because of this, I get 502 if I have curl

Ingress public ip4 address.

This issue was addressed by manually changing the probe path to /login

in the GCE console.

+3


source to share


2 answers


Quoting from here :

GLBC requires you to define a port (in your case 3000) in the Pod specification.



The solution is to declare the port used for health check in ports

, besides adding a custom one readinessProbe

:

containers:
  - name: grafana
    readinessProbe:
      httpGet:
        path: /login
        port: 3000
    ports:
      - name: grafana
        containerPort: 3000
    …
+6


source


Configuring health checks

With GLBC add-on

It's not entirely clear from your question, but if you are using GCE Load-Balancer Controller (GLBC) Cluster Addon

, you can customize the health check path .

Currently, all service servers must meet any of the following requirements in order to pass HTTP (S) health checks sent to it from the GCE loadbalancer:

  • Answer using 200

    on '/'

    . The content doesn't matter.
  • Output an arbitrary url as a readiness probe to the containers supporting the service.

The Ingress controller looks for a compatible readiness sensor first, if it finds one, it accepts it as a GCE HTTP downloader loadbalancer to check. If there is no readiness probe, or a readiness probe is required special HTTP headers, the Ingress controller points the GCE to health check for the HTTP controller loadbalancer in '/'. This is an Ingress example that uses a readiness probe from endpoints as a health check.

The GLBC addons page mentions this in the Limitations section :

All Kubernetes services must serve a page 200

for '/'

or whatever custom value you specify through the GLBC --health-check-path

argument.

Without GLBC add-on



If you are not using an addon, Kubernetes currently requires you to service 200

for GET

requests on the way /

for successful health checks, otherwise the backend will not receive any traffic.

There is some background in this error.

Google Container Engine (GKE)

If you're on Google Container Engine (GKE), the same Kubernetes default health check requirements apply there too .

Services exposed through Ingress must respond to requests HTTP 200

for requests GET

along the way /

. This is used for health check. If your app isn't serving HTTP 200

for /

, the backend will be marked unhealthy and won't receive any traffic.

The answer to your real problem

Having said all this, as you (@mmoya) indicate in your answer, adding the same port used for the readiness check as one of the ports in the module fixes the problem for your case as the port itself is not exposed to the outer shell. This made Kubernes rely on the health check from /

.

+3


source






More articles:


All Articles