Knowledge High quality dimensions
Taking a shopper viewpoint of information high quality is undoubtedly a precious preliminary step. But it surely may not cowl the completeness of the check scope. Intensive literature evaluations have addressed this problem for us, providing a variety of information high quality dimensions which can be related to most use instances. It’s advisable to evaluation the record with information customers and collectively decide which dimensions are relevant and create exams accordingly.
| Accuracy | Format | Comparability |
| Reliability | Interpretability | Conciseness |
| Timeliness | Content material | Freedom from bias |
| Relevance | Effectivity | Informativeness |
| Completeness | Significance | Degree of element |
| Forex | Sufficiency | Quantitativeness |
| Consistency | Usableness | Scope |
| Flexibility | Usefulness | Understandability |
| Precision | Readability | |
You may discover this record too lengthy and marvel the way to begin with it. Knowledge merchandise or any info system will be noticed or analyzed from two views: exterior view and inner view.
Exterior view
The exterior view is about using the information and its relation with the group. It’s usually thought of a “black field” with performance to signify the real-world system. The scale that fall into the exterior view are extremely business-driven. Typically, the analysis of these dimensions will be subjective, so it’s not at all times straightforward to create automated exams for them. However let’s take a look at a number of well-known dimensions:
- Relevancy: The extent to which information are relevant and useful for the evaluation. Contemplating a market marketing campaign geared toward selling a brand new product. All information attributes ought to instantly contribute to the success of the marketing campaign akin to buyer demographic information and buy information. Knowledge like metropolis climate or inventory market costs are irrelevant information on this case. One other instance is the extent of element (granularity). If the enterprise desires the market information to be on the day stage, nevertheless it’s delivered on the weekly stage, then it’s not related and helpful.
- Illustration: The extent to which information is interpretable for information customers and the information format is constant and descriptive. The significance of the illustration layer is commonly ignored when accessing information high quality. It contains the format of the information — being constant and user-friendly, and the that means of the information — being comprehensible. As an illustration, take into account a state of affairs the place information is anticipated to be accessible in a CSV file with descriptive column descriptions, and the values are anticipated to be in EUR foreign money relatively than in cents.
- Timeliness: The extent to which information is recent for information customers. For instance, the enterprise wants the gross sales transaction information with a most delay of 1 hour from the purpose of sale. It signifies that the information pipeline needs to be refreshed incessantly.
- Accuracy: The extent to which information is compliant with enterprise guidelines. Knowledge metrics are sometimes related to difficult enterprise guidelines akin to information mapping, rounding modes, and so on. Automated exams on information logic are extremely beneficial and the extra, the higher.
Out of the 4 dimensions, in terms of creating information exams, timeliness and accuracy are extra easy. Timeliness is achieved by evaluating the timestamp column with the present timestamp. Accuracy exams are possible by buyer queries.
Inside view
In distinction, the interior view is anxious with the operation that continues to be unbiased of particular necessities. They’re important whatever the use instances at hand. Dimensions within the inner view are extra technical-driven versus business-driven dimensions within the exterior view. It additionally implies that information exams are much less depending on customers and will be automated more often than not. Listed here are a number of key views:
- High quality of information supply: The standard of the information supply considerably impacts the general high quality of the ultimate information. The information contract is a superb initiative to make sure supply information high quality. As information customers of the supply, we are able to make use of an analogous method to observe the supply information as information stakeholders do when evaluating the information merchandise.
- Completeness: The extent to which info is retained in its entirety. Because the complexity of the information pipeline will increase, there’s a greater chance of data loss occurring throughout the intermediate phases. Let’s take into account a monetary system that shops buyer transaction information. The completeness check ensures that each one transactions efficiently traverse the complete lifecycle with out being omitted or unnoticed. For instance, the ultimate account stability ought to precisely mirror the real-world state of affairs, capturing each transaction with none omissions.
- Uniqueness: This dimension goes hand-in-hand with the completeness check. Whereas completeness ensures that nothing is misplaced, uniqueness ensures that no duplication happens throughout the information.
- Consistency: The extent to which information is constant throughout inner programs every day. The discrepancy is a typical information problem that always stems from information silos or inconsistent metric calculation strategies. One other side of the consistency problem happens between days when information is anticipated to have a gentle development sample. Any deviation ought to elevate a flag for additional investigation.
It’s price noting that every dimension will be related to a number of information exams. What’s essential is knowing the suitable software of dimensions to particular tables or metrics. Solely then, the extra exams employed, the higher.
Up to now, we’ve mentioned the scale of exterior views and inner views. In future information check designs, it’s vital to think about each the exterior and inner views. By asking the precise inquiries to the precise folks, we are able to improve effectivity and scale back miscommunication.
