Production-Grade Data & ML Infrastructure without Hiring A Team
As a managed Flyte solution, Union empowers ML, Data, and BioTech teams to deliver increased value. Union streamlines your workflow for 10x productivity by eliminating infrastructure constraints and complex setup processes.
Three steps to streamline & scale
Union is an easy-to-use orchestration platform for Data, ML, and BioTech teams. Utilize your existing code with added benefits of versioning, data lineage, and more, all powered by Kubernetes—no extensive learning required.
Supercharge your code
Turn your ETL and ML code into scalable tasks and workflows.
Move the slider to see the Flyte™ Decorators that identify your code as tasks and workflows.
Get ready to execute
Union is an out-of-the-box experience for machine learning engineers and data scientists who need to deliver at scale and often without any help from IT or infrastructure teams.
Observe & optimize
As your workflows become more complex, it becomes increasingly important to gain deeper insights into their performance.
Union’s built-in dashboards, logging, and task-level resource monitoring enables users to identify resource bottlenecks, long execution times, and simplify the debugging process, resulting in optimized resources and faster experimentation.
How Data, AI, & ML orchestration works
AI orchestration is the process of managing and automating tasks such as data preparation, model training, deployment, monitoring, and updating. As your operations scale up, orchestration becomes the essential fabric that runs your ML workflows and pipelines. Advanced orchestrators like Union manage your infrastructure and simplify integrations with various frameworks and platforms.
1. Create a Workflow
Begin by defining the workflow for your ML project. This involves outlining the sequence of tasks for the project, including data collection, preprocessing, model training, evaluation, and deployment. With Union, you can conveniently use Python to write and run your workflows.
2. Manage Resources
Use declarative statements in your Python code to manage resources and Union allocates the computational resources for each task. This includes hardware resources such as CPUs, GPUs, and memory, ensuring that each stage of your pipeline has the resources it needs, thereby minimizing wastage.
3. Schedule & Execute
By leveraging Kubernetes, orchestrators like Union can efficiently schedule, version, and execute containerized tasks. This fully automated approach not only ensures reproducibility but also enhances scalability.
4. Monitor & Observe
Once a workflow is deployed, advanced orchestrators like Union offer dashboards, logs, and task-level monitoring. These tools are essential for troubleshooting and provide valuable insights into resource utilization.
Write your Python code locally, execute it remotely
Enjoy the freedom to write Python code that runs both locally and remotely in your Kubernetes cluster. Take advantage of full parallelization and utilization of all Kubernetes nodes without creating Docker files or writing YAML.
Get started with Union
Union is immediately available for AWS and GCP. Union offers free trials for qualified users. To get started with Union check out the documentation or sign up to be an early user below.
An open platform for your entire team & stacks
Union is designed for data and ML teams who don’t want the overhead of maintaining and managing Flyte™ deployments, setting up Kubernetes infrastructures, and provisioning security and data policies.
Union, as a data orchestrator, empowers DataOps and data engineering in a modern data stack by providing advanced automation capabilities. Data and analytics professionals can leverage Union to create, deploy, and run fully automated and reproducible end-to-end data pipelines.
“Union runs my data pipelines and even connects with Apache Airflow.”
Data science teams benefit from Union’s data science capabilities, which enable efficient data science workflows. With advanced automation features, Union allows data scientists to create, deploy, and execute end-to-end pipelines that are fully automated and reproducible.
“From my research work to effective collaboration on one platform.”
ML engineers need a streamlined and scalable approach to their machine learning workflows. With its comprehensive set of tools and efficient design, Union enables ML engineers to easily build, deploy, and manage complex workflows, accelerating their development and delivering more robust and accurate models.
“Deploying and monitoring all my workflows around the clock.”
DevOps and engineers are responsible for managing a wide range of tools, frameworks, and services to build and maintain end-to-end data ecosystems. This can be a complex and time-consuming task, requiring significant expertise and resources. An orchestrator like Union greatly simplifies this process by providing a unified platform for managing and automating data and ML workflows, enabling DevOps and engineers to streamline their workflows and focus on delivering more value to their organization.
“Running our Kubernetes cluster while using Union to manage our data plane.”
Overcome the complex challenges of building scalable ML products
Union accelerates data processing and machine learning for businesses in every industry. It’s built on the trusted open-source project Flyte™, and combines the power and efficiency of Kubernetes with enhanced observability and enterprise features, all fully managed in your cloud account. Data and ML teams can more easily collaborate on optimized infrastructure, boosting their velocity.
Break down siloed teams & infrastructure
When data and ML teams work with distributed tooling and infrastructure, communication and collaboration can become difficult. Siloed teams use different tools, processes, data formats, and infrastructure, which can lead to delays, errors, or even scrapping projects due to a lack of alignment.
With Union, simplify the process of sharing work across teams and environments with reusable tasks, versioned workflows, and an extensible plugin system.
Infrastructure doesn’t have to be difficult
On-prem, hybrid cloud, multi-cloud, multi-region, the options today are endless for choosing the right infrastructure for your projects. These choices offer flexibility to users, but the use of multiple clouds can lead to issues with data consistency, networking, security, and service integrations. This can result in the failure of ML projects and the breakage of infrastructure and applications.
With Union, it is simple to consume resources and services across clouds in one unified platform.
Cost optimization for complex workflows
When infrastructure becomes distributed across different providers and instances, it can be a daunting task to track and forecast usage and spend. It is all too normal to see this lack of visibility leading to tremendous compute costs from underutilized resources with little understanding of when and how it happened.
Union provides real-time visibility and monitoring at the workflow and task level as well as a resource dashboard that includes all of your projects.
Increase control & oversight across teams and projects
Without proper governance, teams might not adhere to consistent standards or regulatory requirements for data management, model development, testing or deployment.
With Union, you can simplify the management and security of your platform through enterprise-grade Role-Based Access Control—allowing you to scope individual users to view and execute on specific Projects and Workflows.
Data, ML, research & production
These fine companies among many others create data and ML products with Union’s Flyte™ engine.