Mremap (2) with HugeTLB to change virtual address?
Is the Linux mremap (2) function able to change the HugeTLB virtual address obtained from mmap () to a new fixed virtual address?
(Prerequisites: I want to reassign a virtual address based on the physical address of the memory I get. This is efficient to perform virtual and physical address translations by directly checking pointer addresses. I will be using DMA memory for hardware in user space.)
This doesn't look like my simple test program:
#define _GNU_SOURCE
#include <stdio.h>
#include <sys/mman.h>
#include <stdint.h>
#define LARGE_PAGE_SIZE (1024*1024*1024)
int main() {
void *p1;
void *p2;
p1 = mmap(NULL, LARGE_PAGE_SIZE, PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_ANONYMOUS|MAP_HUGETLB|MAP_LOCKED,
0, 0);
if (p1 == MAP_FAILED) {
perror("mmap");
return 1;
}
printf("p1 = %p\n", p1);
p2 = mremap(p1, LARGE_PAGE_SIZE, LARGE_PAGE_SIZE,
MREMAP_MAYMOVE|MREMAP_FIXED,
(void*)(((uint64_t)p1) | 0x500000000000ULL));
if (p2 == MAP_FAILED) {
perror("mremap");
return 1;
}
printf("p2 = %p\n", p2);
}
Mmap () failed with mremap () error:
$ gcc -o mremap_hugetlb mremap_hugetlb.c && sudo ./mremap_hugetlb
p1 = 0x2aaac0000000
mremap: Invalid argument
Note that the new address is calculated from the one that was obtained using the original mmap (). It is important. The desired address is unknown in advance and so I cannot simply pass MAP_FIXED to mmap ().
The workaround I'm currently using is to make the mmap () a backed file, so that then smap () it again at a fixed address and munmap () the old mapping. This is suboptimal because it requires finding the mounted hugetlbfs filesystem, and I don't like the complexity of this dependency.
Current code based on the workaround: https://github.com/lukego/snabbswitch/blob/straightline/src/core/memory.c#L56
source to share
Right now it looks like you need to use hugetlbfs.
If I am wrong, the problem occurs in the Linux kernel because it mm/mremap.c:mremap_to()
is causing mm/mremap.c:vma_to_resize()
that fails EINVAL
for huge pages .
Maybe the test is wrong or the function doesn't have any code to handle huge pages correctly. I am wondering if I should contact linux-kernel and linux-mm to see if this is a bug that should / can be easily fixed. However, this will not help users using modern (and older) kernels.
Remember that when used mmap()
in a file descriptor, a different code path is usually used, as each filesystem can specify its own handler mmap
. For hugetlbfs, the code is in fs/hugetlbfs/inode.c:hugetlbfs_file_mmap()
. And as you said, this code path seems to work well for you.
Note that it is best if you let the user configure the hugetlbfs mount point instead of scanning one of them /proc/mounts
, so sysadmin can configure multiple hugetlbfs mount points, each with a different configuration, for each service running on the server. (I hope your service doesn't require running as root.)
source to share