Health check for redis master discovery from google tcp load balancer
I am trying to set up Google's internal TCP load balancer. The group of instances behind this lb consists of redis-server processes listening on port 6379. Of these redis instances, only one of them is master.
Problem : Add TCP health check to detect redis master and make lb redirect all traffic for recovery only.
Approach : Added TCP health check for port 6379. To send a command role
to the redis-server process and parse the response, I use the optional parameters specified in the health check. Please see the screenshot here .
Result : No health check is performed for all. If I remove the optional request / response parameters, the health check starts transmission for everyone.
Debugging
- Connected to lb with netcat and issued the command
role
, it sends a response starting with*3
(for master) and*5
(for slave) as expected. - The re-server process was written to the instance and stopped. Started listening on port 6379 with help
nc -l -p 6379
to check what exactly was received on the instance side in health check. He gets itrole\r\n
. - After step 2, restart redis-server and run the command
MONITOR
in redis-cli to view the log of the commands received by this process. There is no magazine hererole
. This means that the instance is receiving data (role\r\n
) via tcp, but not receiving the redis-cli process (as per commandMONITOR
) or something else is going on. Please, help.
source to share
Unfortunately, the GCP TCP health checks are quite limited in what can be verified in the response. From https://cloud.google.com/sdk/gcloud/reference/compute/health-checks/create/tcp :
--response=RESPONSE
An optional string of up to 1024 characters that the health checker expects to receive from the instance. If the response is not received exactly, the health check probe fails. If --response is configured, but not --request, the health checker will wait for a response anyway. Unless your system automatically sends out a message in response to a successful handshake, only configure --response to match an explicit --request.
Pay attention to the word "exactly" in the help message. The answer must match the line provided in full . You cannot provide a partial search string in the response.
As you can see at https://redis.io/commands/role the redis ROLE command returns a bunch of text. Although the substring "master" is present in the response, it also has a bunch of other text that will differ from setting to setting (depending on the number of slaves, their addresses, etc.).
You should make sure to raise the feature request with GCP for regex matching in the response. A possible workaround until then is to have a small web application on each host that runs the redis-cli role | grep master, and returns a response. The health check can then be configured to monitor this web application.
source to share