To compute a distinct version ID, Marquez applies a versioning function to a set of properties corresponding to the datasets underlying datasource. The latest version ID is updated only when a change to the dataset has been recorded. Each version can be read independently and has a unique ID mapped to a dataset change preserving its state at some given point in time. When a dataset change is committed back to Marquez, a distinct version ID is generated, stored, then set to current with the pointer updated internally.ĭataset Version: A read-only immutable version of a dataset. A version pointer into the historical set of changes is present for each dataset and maintained by Marquez. A datasource enables the grouping of physical datasets to their physical source. A dataset is contained within a datasource. Such associations catalog provenance links and provide powerful visualizations of the flow of data.ĭataset: A dataset has an owner, unique name, schema, version, and optional description. A job version associates one or more input and output datasets to a job definition (important for lineage information as data moves through various jobs over time). Job Version: A read-only immutable version of a job, with a unique referenceable link to code preserving the reproducibility of builds from source. Note that it’s possible for a job to have only input, or only output datasets defined. A job will define one or more versioned inputs as dependencies, and one or more versioned outputs as artifacts. Job: A job has an owner, unique name, version, and optional description. The diagram below shows the metadata collected and cataloged for a given job over multiple runs, and the time-ordered sequence of changes applied to its input dataset. Dataset changes are recorded at different points in job execution via lightweight API calls, including the success or failure of the run itself. A job run is linked to versioned code, and produces one or more immutable versioned outputs. Datasets are first-class values produced by job runs. Marquez’s data model emphasizes immutability and timely processing of datasets. The Metadata Repository serves as a catalog of dataset information encapsulated and cleanly abstracted away by the Metadata API. Metadata needs to be collected, organized, and stored in a way to allow for rich exploratory queries via the Metadata UI. The API allows clients to collect and/or obtain dataset information to/from the Metadata Repository. It’s a low-latency, highly-available stateless layer responsible for encapsulating both metadata persistence and aggregation of lineage information. The Metadata API is an abstraction for recording information around the production and consumption of datasets. OpenLineage provides support for Java and Python as well as many integrations. To ease adoption and enable a diverse set of data processing applications to build metadata collection as a core requirement into their design, Marquez implements the OpenLineage specification. Metadata UI: Used for dataset discovery, connecting multiple datasets and exploring their dependency graph.Metadata API: RESTful API enabling a diverse set of clients to begin interacting with metadata around dataset production and consumption.total runs, average runtimes, success/failures, etc). Metadata Repository: Stores all job and dataset metadata, including a complete history of job runs and job-level statistics (i.e.It consists of the following system components: Marquez is a modular system and has been designed as a highly scalable, highly extensible platform-agnostic solution for metadata management. Marquez enables highly flexible data lineage queries across all datasets, while reliably and efficiently associating ( upstream, downstream) dependencies between jobs and the datasets they produce and consume. Designed to promote a healthy data ecosystem where teams within an organization can seamlessly share and safely depend on one another’s datasets with confidence.RESTful API enabling sophisticated integrations with other systems:.Simple operation and design with minimal dependencies.Enforcement of job and dataset ownership.Easily collect metadata as OpenLineage events via the LineageAPI.Precise and highly dimensional data model.Centralized metadata management powering:.A reference implementation of the OpenLineage standard.Marquez was released and open sourced by WeWork. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more. Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem’s metadata. Collect, aggregate, and visualize a data ecosystem's metadata View on GitHub Quickstart Download Overview
0 Comments
Leave a Reply. |