A survey of ML Ops off-the-shelf solutions for building an automated machine learning pipeline<\/p>\n
At WAY2VAT we have been running a specialized\u00a0homebrew\u00a0machine learning pipeline for years.\u00a0Our pipeline is a piece of software that governs aspects of running\u00a0our\u00a0machine-learning-based\u00a0patented\u00a0product, the Automatic Invoice Analyzer\u00a0(AIA).\u00a0The AIA supports our everyday business by eliminating human processing time on all fronts of the VAT\\GST reclaim process \u2013 from extracting basic\u00a0transaction\u00a0information to determining whether an invoice is\u00a0eligible for submission in a claim for VAT\\GST return.<\/p>\n
The core of\u00a0the\u00a0AIA\u00a0technology is\u00a0composed\u00a0of\u00a0more than a dozen algorithms, each with specific training data and evaluation metrics. Together, the algorithms provide a coherent, detailed analysis of any invoice that we receive, from extracting fields to determining the language and currency. Many of the algorithms are co-dependent and run\u00a0sequentially, each following on the results of the last,\u00a0while the purpose of others is to correct the results of intermediate steps for getting a clearer picture.<\/p>\n
The complexity of our\u00a0product\u00a0demands we keep on top\u00a0of training and evaluating\u00a0the\u00a0models in production, as well as research into new methods. To that end, we use several tools:<\/p>\n
At one point\u00a0several years ago,\u00a0ML Ops \u2013 Machine Learning Operations<\/strong>\u00a0(cf.\u00a0a\u00a0definition by Google:\u00a0https:\/\/cloud.google.com\/solutions\/machine-learning\/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning<\/a>)\u00a0has become\u00a0a full-time occupation for our engineers.\u00a0Based on our research into the ML Ops world, we\u00a0offer a\u00a0survey\u00a0of\u00a0the\u00a0ML Ops\u00a0landscape\u00a0of\u00a0off-the-shelf solutions, especially around the major pain points of experiment management.\u00a0An off-the-shelf solution can dramatically reduce\u00a0upkeep and maintenance of the scripts that run and schedule\u00a0experiments and enable a tighter lock on\u00a0hyperparameter\u00a0searching and choosing, let alone the nice dashboards and graphs it produces.<\/p>\n After a round of sourcing and research, we converged (pun intended) on Apache AirFlow (https:\/\/airflow.apache.org\/<\/a>)\u00a0as the\u00a0orchestrator for our DAG.\u00a0AirFlow\u00a0is a generic tool for running custom-code DAGs, and we repurposed it for spinning up cloud machines with Docker images that load data and run experiments. This\u00a0may\u00a0prove\u00a0successful\u00a0for others, but\u00a0keep in mind\u00a0it\u00a0needs\u00a0constant, daily care, since data and experiments\u00a0are\u00a0constantly\u00a0changing with new features implemented.\u00a0While very versatile,\u00a0AirFlow\u2019s\u00a0code blocks in Python\u00a0may\u00a0end up being quite long\u00a0and\u00a0with\u00a0many dependencies such as the cloud API, therefore\u00a0making them\u00a0harder to maintain.\u00a0AirFlow\u00a0also needs\u00a0its own server\u00a0for\u00a0running, which adds some overhead cost\u00a0and maintenance care for consistent uptime.<\/p>\n AirFlow\u00a0does not give any tooling for managing datasets, experiments or models. It\u2019s strictly a DAG executor.\u00a0To\u00a0complement\u00a0AirFlow\u00a0one must\u00a0provide dataset, experiment and cloud machines management.\u00a0We\u00a0use a mix of\u00a0homebrewed solutions as well as\u00a0off-the-shelf\u00a0data and\u00a0experiment management systems.<\/p>\n As part of our surveying the landscape of ML Ops tools to automate ML pipelines, we compiled the following list of vendors and open-source solutions. While it is not exhaustive,\u00a0the list\u00a0is valuable for those running ML-based products\u00a0and looking for the right vendor\u00a0or project.\u00a0Note: we do not affiliate with any of these projects\u00a0and this list simply reflects our team\u2019s research.<\/p>\n Open-source\u00a0tools are marked with an open book: \ud83d\udcd6<\/p>\n Experiment Managers & Hyperparameter Tuners\u00a0<\/strong><\/p>\n These projects\u2019 primary goal is to run and track ML experiments. Often, they would offer a DAG orchestrator as well as dataset management features. The\u00a0outputs are usually accessible through actionable dashboards where experiments can be run, stopped,\u00a0logged\u00a0and tracked through their lifecycle (learning curves,\u00a0exploring intermediate results).\u00a0We also include in this bucket the Hyperparameter Tuners \/ Optimizers (HPO) that\u00a0frequently\u00a0go with experiment managers, since HPO\u00a0many times requires multiple experiments.\u00a0In this category we see many\u00a0open-source\u00a0tools, sometimes catering to the academic community and not just for industry.<\/p>\n <\/p>\n Dashboards\u00a0<\/strong><\/p>\n These projects primary goal is to provide visualization for ML experiments.\u00a0Usually,\u00a0this functionality is built into projects in the above list.\u00a0Since the application is fairly generic,\u00a0some tools such as Kibana are in fact tools for general purpose dashboarding however they can be applied very simply towards tracking\u00a0ML\u00a0experiments by\u00a0analyzing runtime logs.<\/p>\n All-in-One: Experiment,\u00a0Deploy\u00a0and Monitor\u00a0<\/strong><\/p>\n These projects are a one-stop-shop for ML Ops. They offer solid experiment management alongside dataset and model management. These are most often paid services, with prices in the $100s\/seat\/month\u00a0range,\u00a0rendering them\u00a0infeasible for smaller organizations. In return to a steep price point they offer\u00a0customer\u00a0support, guaranteed uptime through reliable hosting.<\/p>\n <\/p>\n Conclusion\u00a0<\/strong><\/p>\n The ML Ops landscape in 2020 has grown to great proportions. From general tools for DAG\u00a0execution, logging and dashboards it transformed to a multi-billion-dollar\u00a0industry that\u2019s driving AI in the\u00a0largest\u00a0companies\u00a0in any domain.\u00a0All-in-one tools can offer end-to-end features for rolling out a machine learning\u00a0pipeline but\u00a0are usually expensive and sometimes over-promise and under-deliver. Open-source point solutions for\u00a0experiment management are at zero\u00a0cost but\u00a0have overhead for maintenance and require building infrastructure. There are tradeoffs to selecting a path, however it is now clear that ML Ops is going to keep growing\u00a0and changing the way we productize machine learning.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" A survey of ML Ops off-the-shelf solutions for building an automated machine learning pipeline At WAY2VAT we have been running a specialized\u00a0homebrew\u00a0machine learning pipeline for years.\u00a0Our pipeline is a piece of software that governs aspects of running\u00a0our\u00a0machine-learning-based\u00a0patented\u00a0product, the Automatic Invoice Analyzer\u00a0(AIA).\u00a0The AIA supports our everyday business by eliminating human processing time on all fronts of […]<\/p>\n","protected":false},"author":1,"featured_media":3733,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[26,1],"tags":[],"class_list":["post-3730","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-vatgst"],"acf":[],"yoast_head":"\n<\/h2>\n
Apache\u00a0AirFlow<\/h2>\n
<\/h2>\n
The Current ML Ops Landscape<\/h2>\n
\n
\n<\/a>Experiment tracking, hyperparameter optimization,\u00a0model\u00a0and dataset versioning<\/li>\n
\n<\/a>Auto-Magical Experiment Manager, Version Control and ML-Ops for AI<\/li>\n
\n<\/a>A\u00a0community-oriented\u00a0data science platform for collaboration. Based on\u00a0Open-Source\u00a0tools and open formats.<\/li>\n
\n<\/a>Implementations of\u00a0recently published methods for optimizing hyperparameters of machine learning algorithms\u00a0to efficiently search for well performing configurations.<\/li>\n
\n<\/a>A\u00a0tool to help you configure, organize, log and reproduce experiments<\/li>\n
\n<\/a>Lightweight experiment management tool<\/li>\n
\n<\/a>collaborative and analytical training suite for insightful, fast, and reproducible modern machine learning<\/li>\n
\n<\/a>Machine Learning Toolkit for Kubernetes<\/li>\n
\n<\/a>From Spotify, Luigi\u00a0helps you build complex pipelines of batch jobs handling dependency resolution, workflow management, visualization, handling failures, and more.<\/li>\n<\/ul>\n\n
\n<\/a>Visualization and tooling needed for machine learning experimentation.<\/li>\n
\n<\/a>a web dashboard for the Sacred machine learning experiment management tool.<\/li>\n
\n<\/a>An open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases.<\/li>\n
\n<\/a>Compose observability dashboards. It has an\u00a0open source\u00a0\ud83d\udcd6<\/li>\n<\/ul>\n\n
\n<\/a>An\u00a0open-source\u00a0platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry.<\/li>\n
\n<\/a>An end-to-end machine learning platform to build and deploy AI models at scale.<\/li>\n
\n<\/a>helps data scientists and developers to prepare, build, train, and deploy high-quality machine learning (ML) models.<\/li>\n
\n<\/a>Automates everything from data extraction to model deployment.<\/li>\n
\n<\/a>Systematic control to machine learning to help you build better models faster. It’s freely available under the Apache 2.0\u00a0open-source\u00a0license.<\/li>\n
\n<\/a>Reproduce, automate, and scale your data science workflows with production-grade\u00a0MLOps\u00a0tools.<\/li>\n
\n<\/a>Track, compare, explain and optimize experiments and models.<\/li>\n
\n<\/a>End-to-end enterprise-grade platform for data scientists, data engineers, DevOps and managers to manage the entire machine learning & deep learning product\u00a0life cycle.<\/li>\n<\/ul>\n