I will admit that it isn't as straightforward to get it to work as you might imagine at first. But once you've got the automation humming, its been a surprisingly easy to maintain. Would highly recommend that route! (but of course I'm biased since I've worked on it)
One interesting use case is that its straightforward to access the underlying hardware on baremetal machines with `SecurityContext: priviliged` (you can do a much more fine-grained security permissions; I'm just giving an example). So for instance, you can access GPU's, TPM (trusted platform modules) this way.
+1 for "regular" Ceph. Don't bother with that rook stuff. Just setup a regular Ceph cluster and go. Kubernetes handles its stuff and a (much more reliable and stable) Ceph cluster handles blocks and files.
NIO, the self-driving car company is doing this. They did a pretty detailed interview on their use case which includes a 120 PB data lake, and Cassandra, Kafka, TensorFlow, HDFS. You can read here: https://portworx.com/architects-corner-kubernetes-satya-koma... . (Disclosure, I work for Portworx the solution they use for container storage, but hopefully the content speaks for itself).
1200 core k8s cluster + 1.5PB ceph (shared nodes to some degree)
no issues with persistent disks etc, only "annoying" thing is to figure out RBAC initially
you just use a storagecontroller, then its no work whatsoever, ceph does the rest
I've been recently experimenting with getting lustrefs usable with kubernetes, and needed some way to natively integrate it into the cluster I had.
https://github.com/torchbox/k8s-hostpath-provisioner Proved to be useful, it allows you to use any mounted path on (all) a node(s) (hostpath method) to return satisfy a persistent volume claim and return a persistent volume backed by the mounted file system.
A similar set up could work using bare metal, are you using something like Openstack Ironic?
OpenSource: OpenEBS, Rook (based on Ceph), Rancher's Longhorn
Commercial options highly recommended if you want safety and support as storage is hard, although people seem to be running all of these options well enough. Portworx probably most highly developed with Quobyte a good option if you want shared storage with non-kubernetes servers.
We experimented with both using cephfs via rook and glusterfs via heketi and ran into enough operational issues and speed bumps that we're just using hostPath volumes for now.
I mean without cloud services like Google cloud persistent disks.