Metadata is one of those words we hear a lot but for those that still have doubts about the meaning, here it goes:
data that provides information about other data (Merriam-Webster dictionary)
In other words when we create tags and put them on pictures, when an app embeds the date and location into a file, all of this information is considered metadata. Back to data engineering, we use metadata to explain which information is written into a table, who is the owner, what a column means, etc. And at this point we start the journey to improve governance through metadata.
Lost Metadata Creates Unnecessary Governance Work
Usually businesses care about metadata only when it's very close to analytics because non-technical people will interact with it. Software engineers, data engineers, data analysts access this data daily and probably figure out the meaning of some columns even if the names are shortened somehow. But hey, not everyone is that close to data and just wants to use it to explain real-world facts. That said when data is close to the final report everyone starts to label data, provide owners, lineage and so on. However, much of this information is already travelling on the wire from the point where it's created until it reaches the datalake. Every data transformation is done by a person/team over data that was produced by another person/team. All of this information is just lost and in the end someone needs to go there and label everything all over again.
Capture and Propagate Metadata Across Data Pipelines
When we build an API that produces data there is a lot of useful information we can extract during build: the API owners, parse entities and extract column labels from ORM definitions, etc. For other artifacts like Declarative automation bundles it's possible to do the same. Given we inject metadata we need ways to consume this injected metadata and propagate it when data is produced to queues, databases, etc, and at this point we start to have metadata flowing together with data itself. Not only technical metadata like offset number, source table name but business metadata as well like data producer, column descriptions.
Turn Metadata Into Automated Governance Controls
At this point it's easier to understand how to use metadata to improve governance. For example: if we are able to inject and propagate data owners we can easily tag those people as owners in the datalake too. Same with column descriptions: we can infer column descriptions when they are only replicated to a new table. In the end governance needs fine-grained information to be automated otherwise we will go back to those times where each request needed to be evaluated by a specialized committee.
Everytime we capture metadata close to its creation we save time later with re-labelling, context understanding, miscommunication and all sorts of back and forth. Over time we can even start to integrate custom metadata with existing tools to automate tasks like column masking and row filtering. Imagine marking a specific column with a tag called PII and everywhere this data is flowing to we could enforce visualization based on the user profile logged in analytics tool.
Effective Data Governance Requires Enforceable Controls
Good governance is enforced through good tooling. We can create processes, forms, positions and follow frameworks but in the end if you can't enforce it somehow there will be breaches. Metadata is the fuel and tooling is the car. One without the other is just useless. We have out there plenty of tools, paid or free, that are capable of providing a lot of features out-of-the-box but if I could give an advice: prioritize flexible tools, capable of customizing at some level. Corner cases will appear and handling with less tools possible is also a sign of good governance.
