Marquez

What it is

Marquez collects, aggregates, and visualizes metadata from various data sources to provide observability into complex data pipelines. It helps teams understand the lineage of their data and track dependencies across different systems.

Marquez stands out by providing a self-hosted, open-source solution for collecting and visualizing metadata from diverse data sources, making it ideal for teams that need deep insights into their complex data ecosystems without relying on cloud services.

When it fits — and when it doesn’t

Known weaknesses

Each one links to the public issue, changelog or docs page it came from, where we have one.

1. 01

   Steep learning curve for non-Java developers

   Primary language is Java, which may be unfamiliar or challenging for teams predominantly using other languages like Python or Go.

2. 02

   Limited documentation and community support

   The project's documentation lacks depth and examples, making it difficult to understand advanced use cases without significant trial and error. Community forums and Stack Overflow have limited activity compared to more popular tools.

3. 03

   Integration complexity with non-supported data platforms

   While Marquez integrates well with Apache Airflow, Kafka, and Spark, integrating it with other less common or proprietary systems can be complex and require significant custom development effort.

4. 04

   Performance issues at scale

   As the number of data sources and pipelines grows, Marquez's performance degrades due to its current architecture limitations. This can lead to slower response times for metadata queries and visualizations.

What it does well

• Metadata collection and aggregation from various data sources.

• Visualization of data lineage and dependencies.

• Integration with popular data platforms like Apache Airflow, Kafka, and Spark.

Pricing

Free tier

YesOpen source — free to use

Paid from

$0

Model

Flat rate

Enterprise plan

No

Pricing not independently confirmed — check the vendor’s page.