Mesos was a derivative idea from some sporadic idea from 2 level scheduling from inside Google based on mapreduce
Mesos was wrong from day one, they thought they have the right idea, but was really caught up by a Small group of Google engineers who happened to be from academia. These engineers were very good at having theoretically nice idea
In the time mesos was invented, Google had its own mesos, which is a similar project by learning wrong lesson from Borg. That thing is Omega.
Eventually everyone decided that Borg was right all along, thus the kubernetes
I have no idea what any of the google tech has to do with anything here.
Quoting from the original spark paper:
> Spark is built on top of Mesos [16, 15], a “cluster operat-
> ing system” that lets multiple parallel applications share
> a cluster in a fine-grained manner and provides an API
> for applications to launch tasks on a cluster
The RAD lab folks who built Mesos were aware of Borg and how it approached the problem of schedling a bunch of different jobs on a collection of disparate hardware. Prior to borg, most large-scale clusters were managed with batch queue software, while borg and mesos are more from the "service management"- a collection of jobs that run concurrently, with priority levels used to preempt lower-priority jobs to allow higher-priority jobs to schedule and run "immediately".
The need for this pops up for nearly every large scale data processing enterprise- with k8s replacing mesos, yarn, and other systems as the cluster scheduler du-jour.
One of the big advantages of a service scheduler versus a batch queue is that you can implement a batch queue on top of a service scheduler much more easily than you can implement a service scheduler on top of a batch queue.
