Hacker News new | past | comments | ask | show | jobs | submit login

How does the parallel "execution strategy" work? Do you specify a maximum concurrency level? Can you distribute the computation onto other machines?



Graph is currently just in-process (no cross-machine distribution), although it's definitely a possibility down the line. The currently released parallel strategy is also just a pedagogical example, not really meant to be used.

Here's some WIP on a more practical parallel compilation: https://gist.github.com/w01fe/4710008 Once the kinks get worked out this will go into the OSS project. Presumably concurrency level will be controlled by a parameter, and/or passing an appropriate ExecutorService.


It would be very interesting if you could easily convert a graph into a Storm[0] topology. An example of the Storm Clojure DSL is available in the storm-starter project[1].

[0] http://storm-project.net/

[1] https://github.com/nathanmarz/storm-starter/blob/master/src/...


I'm not sure how it's implemented in Graph, but the best thing you can do is to abstract out an execution to a general object.

From there, you can create all sorts of adapters and the "graph" engine just farms out the execution to the appropriate adapter. This could be anything from a job daemon on a batch farm, or, say, a typed ThreadPoolExecutor on a machine (which can either execute native clojure/java, or any sort of other script language.


This would be brilliant. Does any other framework support this? (outside Clojure)




Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: