Humans generate a lot of data. Every two days we create as much data as we did from the beginning of time until 2003! The International Data Corporation estimates the global datasphere totaled 33 zettabytes (one trillion gigabytes) in 2018. The estimate for 2025 is 175 ZBs, an increase of 430%. This growth is challenging organizations across all industries to rethink their data pipelines. The nature of data usage is problem driven, meaning data assets (tables, reports, dashboards, etc.) are aggregated from underlying data assets to help decision making about a particular business problem, feed a machine learning algorithm, or serve as an input to another data asset. This process is repeated multiple times, sometimes for the same problems, and results in a large number of data assets serving a wide variety of purposes. Data discovery and management is the practice of cataloguing these data assets and all of the applicable metadata that saves time for data professionals, increasing data recycling, and providing data consumers with more accessibility to an organization’s data assets. To make sense of all of these data assets at Shopify, we built a data discovery and management tool named Artifact. It aims to increase productivity, provide greater accessibility to data, and allow for a higher level of data governance. The Discovery Problem at Shopify Data discovery and management is applicable at every point of the data process: Acquire Where is the data coming from? What is the quality of this data? Who owns the data source? Transform What transformations are being applied? How can this data be accessed? How often does this process run? Model What business logic is being applied? Is the model stale or current? Are there other similar models out there? Apply Who are the main stakeholders? How is this data being applied What is the provenance of these applications? High level data process The data discovery issues at Shopify can be categorized into three main challenges: curation, governance, and accessibility. Curation “Is there an existing data asset I can utilize to solve my problem?” Before Artifact, finding the
NayaRead shows a summary and links to the original article. The full article is hosted on shopify.engineering.