Linux DMA user space with iommu = VFIO enabled and disabled
Is it possible for a Linux userspace application to use its own memory for DMA without IOMMU locking and without using VFIO?
Our app works great when iommu is disabled (intel_iommu = off) or in transition mode (intel_iommu = passthrough). However, it does not work when IOMMU is enabled (intel_iommu = on) because the memory we are allocating in user space is not allowed for DMA.
The official solution would be to use the VFIO Linux interface to manage the IOMMU, however we think the VFIO feature is not very mature and would rather find a simpler solution.
Can we somehow instruct the IOMMU to allow DMA for the physical memory we have allocated? This would be great because then we wouldn't have to instruct our users to change the kernel boot parameters.
If there is a simple and reliable solution based on VFIO, then that will be interesting too.
For details on how we allocate memory, see the previous question: mremap (2) with HugeTLB to change the virtual address?
source to share
You can write a simple char device driver for this.
Internal driver (pseudo code):
static ssize_t device_write(struct file *filp, const char __user *buf, size_t count, loff_t * ppos)
{
struct page *page;
dma_addr_t DmaBusAddress;
copy_from_user(addrstr, buf, sizeof(addrstr));
uaddr = simple_strtoul(addrstr, NULL, 0);
/* Get page structure which describes your user space
memory area
*/
res = get_user_pages(...,uaddr,...&page);
/* (optional)Get the kernel virtual address for your user space page
*/
kernel_virtual_address = kmap(page);
DmaBusAddress = dma_map_page(...,page)
/*
or the function below if you use address instead of page
DmaBusAddress = dma_map_single(...kernel_virtual_address,count...);
*/
dev->dma_dir = DMA_TO_DEVICE;/* Write operation */
dev->dma_size = count;
dev->dma_addr = DmaBusAddress;
/* Assume you have already memory map your DMA controller I/O space
with remap_pfn_range() */
writeRegister(DMA_ADDRESS,dev->dma_addr);
........................
writeRegister(START_DMA_TRANSFER,1); /* enable DMA controller */
}
The above code shows how to use the shared DMA layer.
Quote from ldd3
"The IOMMU can arrange to map physical memory within the address range available to a device, and it can call physical scattered buffers to look at adjacent devices. Using the IOMMU requires a shared DMA layer; virt_to_bus is not up to the task."
and
"The overall DMA layer goes to great lengths to ensure that things work correctly on all architectures, but as we will see, the behavior requires a small set of rules."
The chapter "Mapping memory and DMA" can answer all your questions.
Here is the link: http://free-electrons.com/doc/books/ldd3.pdf
source to share