Data quality and metadata relationship

data quality and metadata relationship

Clarity of relationships Metadata helps resolve ambiguity and inconsistencies when determining the associations between entities stored throughout data. I have recently gained more interest in data quality as well as philosophy, and was struck by their relationship. Data quality is the extent to. Keywords Metadata, Data quality, Attributes, eMaintenance, . Data and relationships are represented in a at, two-dimensional table that.

It is often expressed in terms of dimensions such as accuracy, completeness, consistency, credibility, currentness, and understandability. Philosophy attempts to answer the deeper questions that we have, providing us with wisdom. This often relates to areas such as existence, knowledge, values, reason, mind, and language. There are two themes that are both central to data quality as well as philosophy: In this blog, I will explore other implications of the overlapping of themes.

data quality and metadata relationship

The latter is very frequent in spatial data, where positional accuracy is one of the most important measures of data quality. In data quality, the accuracy dimension is one of the most difficult dimensions to measure. An alternative could be to compare the data with a reference data set that is closer to reality; however, that is not the same as comparing it with reality.

Positional accuracy can be very difficult to measure when there are no clear boundaries e.

data quality and metadata relationship

From a philosophical perspective, reality is the state of things as they exist, rather than as they may appear or might be imagined.

Reality includes everything that is and has been, whether it is observable or comprehensible. The truth refers to what is real, while falsity refers to what is not.

Implementing Data Quality Through Metadata, Part 1

Some idealists say that we only know how things appear to us, not how they really are. An object may appear contradictory from different viewpoints, which would imply that there cannot be an object that unites these contradictory characteristics. Independent of which philosophers you believe, it is a fact that independent of whether there is one objective reality, the way we perceive reality is subjective.

It is based on how we interpret what we see, based on filters that we have built, given our personal experience. How can you be sure that the value observed by a person that has measured the data is not just their interpretation of reality? Related to the previous, we may be interested from a data quality perspective in determining the accuracy of unstructured data containing certain statements.

You could for example try to measure the data quality of policy documentation, containing all sorts of policy statements that you want to test. Minimalists say that truth is not a characteristic of a conviction or statement. Pragmatists say that something is true when it works. An interesting perspective on the truth of statements is provided by the Tractatus Logico-Philosophicus of Ludwig Wittgenstein.

The statements in this philosophical work have been discussed at length by a lot of people, and discussions remain about the truth of the statements. What does not really help is that the work is described only in terms of propositions that are not accompanied by any support; they are just stated.

The Relationship Between Data Quality and Philosophy |

Part two of the series will examine the beneficial technical meta data tags that can be incorporated into an architecture to measure data quality and provide flexibility to the system design. The warehouse developers use technical meta data as a method to build a tighter relationship between the repository and the data warehouse. This is accomplished by incorporating technical meta data directly into the data warehouse design and ETL processes.

This technique is used to extend the design and architecture of the data warehouse to provide increased processing optimizations for data acquisitions, maintenance activities, and data quality measurement opportunities. These technical meta data tags, unlike information stored in a meta data repository, are referenced at a row level of granularity in the data warehouse. This direct association of meta data to each row of information in the data warehouse is a key distinction of extending meta data into the architecture.

To select operators, each row of data is tagged from the source systems during ETL processing with technical meta data.

Metadata, Data Quality, and the Stroop Test — OCDQ Blog

The meta data tags on each row in the warehouse provide a clearer semantic meaning to the data by placing the information in context with the repository.

As an example, consider a client dimension table that derives its information from two operational sources. Client information is extracted in priority order from either from a sales force automation application, an enterprise resource planning application ERPor both, depending on availability and stability of the data. The absence of technical meta data in the dimension table would require use of the information without consideration of the source system s that provided it.

The Relationship Between Data Quality and Philosophy

Technical meta data tagging allows you to determine the origin s of information in the dimension table. Information originating from one or more sources can be easily and quickly determined through the technical meta data tag on that row.

data quality and metadata relationship

A clear, consistent method of tagging the data originating from the operational systems needs to be developed and agreed upon by both the technical and business users of the data warehouse. Any technical meta data tied to the row must be applicable to the entire row of data, not just the majority of columns in the table.

I like to keep technical data tagging to a minimum in a simple dimensional data model design.