I actually need gpu, not the ui. I need it to do scientific computation. Video streaming service is another case. gpu has better video encoding capabilities.

I previously heard that docker has trouble loading device drivers.

Not the parent poster, but needing GPU isn't necessarily the same as having UI. You can use GPU for a variety of general purpose math (Example: mining bitcoins, or doing stuff like Folding@Home), or for offline rendering.

yes, I understand offline rendering. I'm looking into egl off screen rendering. But due to historical reason, the current gpu drivers (NVIDIA) need x server.