I have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.
The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.
I have used Airflow as a product for over a year and I can say for sure that Airflow sucks for orchestrating ML.
The ability to pass state in Airflow absolutely sucks. It's interesting that Metaflow is trying to compile into Airflow but I am skeptical about it's scalability on Airflow especially with DAGs that have a larger task footprint.