Love prefect! but for workflows involving concurrency, Prefect code needs to get somewhat invasive.
Prefect relies on prefect.task()-wrapped methods as the lowest granularity of concurrency in a program, and requires you to use the (somewhat immature) prefect task APIs to implement that concurrency.
This is an excellent write up thank you for sharing! Yea, our concurrency API needs an upgrade - coincidentally this is going to be a theme of the next sprint or two so I hope I can report some improvements back soon.
Prefect relies on prefect.task()-wrapped methods as the lowest granularity of concurrency in a program, and requires you to use the (somewhat immature) prefect task APIs to implement that concurrency.
more on this complaint here: https://austinweisgrau.github.io/migrating-to-prefect-part-3...