Gke can not disable Transparent huge pages ... permission denied
I am trying to run a redis image in gke. This works, except I get the dreaded "Transparent huge pages" warnings:
WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Redis is currently too slow to be useful ... So I tied to turn off THP:
sheena@gke-projectwaxd-cluster-default-pool-23593a74-wxrv ~ $ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
sheena@gke-projectwaxd-cluster-default-pool-23593a74-wxrv ~ $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
-bash: /sys/kernel/mm/transparent_hugepage/enabled: Permission denied
sheena@gke-projectwaxd-cluster-default-pool-23593a74-wxrv ~ $ sudo echo never > /sys/kernel/mm/transparent_hugepage/enabled
-bash: /sys/kernel/mm/transparent_hugepage/enabled: Permission denied
These permission errors are confusing. Redis wants THP to shut down so that it can run normally.
I did some tinkering and found that google uses a special os-image that makes the / sys / path read-only. There is an alternate image based on Debian 7. It turned me on, but in the end I have exactly the same problem.
So how can I stop redis from executing THP in google container?
It doesn't look like I'm doing anything unique here. Running databases in containers is pretty normal. And it's okay for the database to malfunction when THP is enabled. So ... what am I missing here?
source to share
Your command is a bit incorrect: echo
runs as root, but the redirect ( >
) itself runs as a user, so it can't write /sys/
.
The following command works well and on-vm container (on debian-based) and gci (based on the chromosome)
sudo sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
Saving this setting on container-vm
Add this parameter to the kernel command line /etc/default/grub
(remember to run sudo update-grub
and sudo reboot
after):
GRUB_CMDLINE_LINUX="... transparent_hugepage=never"
Saving this setting to gci
First, using the cloud console, copy the instance template that is used by the node pool.
Second, in the metadata, change the value for userdata:
#cloud-config
write_files:
- path: /etc/systemd/system/hugepage.service
permissions: 0644
owner: root
content: |
[Unit]
Description=Disable THP
[Service]
Type=oneshot
ExecStart=/bin/sh -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
[Install]
WantedBy=kubernetes.target
...
runcmd:
- ...
- systemctl enable hugepage.service
- systemctl start kubernetes.target
Third, change the instance template to the newly created one:
gcloud compute instance-groups managed set-instance-template \
gke-YOUCLUSTER-YOURPOOL-grp \
--template=YOURNEWTEMPLATENAME \
--zone=...
Forth, recreate instace (s):
gcloud compute instance-groups managed recreate-instances \
gke-YOUCLUSTER-YOURPOOL-grp \
--zone=... \
--instances=...
Instances will lose all data and disable THP. All new instances will also have THP disabled (in this node pool).
source to share