Accessing XServer host from Docker container

My app, running inside a Docker container, uses OpenGL for off-screen rendering.

If the container is running on a graphical host, it must use the XServer host for rendering.

I have configured such a container and it works fine on any system with mesa drivers installed (softpipe or nouveau), but does not work on a system with NVidia's own drivers.

How can I create a docker container capable of hardware accelerated rendering (where possible), but completely independent of the actual graphics hardware installed on the system? Or do I need different versions of containers for each graphical environment I can run into?