Data quality initiative has always been the IT function.Therefore, IT has always looked at it from the technical perspective, eventually making it a data cleansing and data profiling operation. Interestingly, it is ‘Business’ who owns and uses the data. Now we have a situation, wherein the data owner hardly contributes or is heard. On the other hand, those who drive the initiative, neither use the data nor take any data driven decisions. Isn’t this a tricky situation?
Gap between the Business and IT Perspective
Most of the times, data quality rules are applied to the data processing systems and very less to the actual data element. The rules are configured to check the record counts, nulls, invalid formats and some basic reconciliation. In short, only two of the data quality dimensions i.e. Validity and Uniqueness are covered by IT and leaving out the remaining four –completeness, accuracy, consistency, and timeliness.
Let’s look at this business scenario.The data element, Policy Premium amount can be checked by IT team for its null, zero or negative values thus validity is checked. However, IT may not have enough information to verify the accuracy of the Policy Premium amount without business insights. Policy Premium amount depends on other critical data elements such as driving record, age, the type of a car, credit score, residence address etc. This cluster of data elements can be evaluated combined to check an accuracy and consistency of data.
Business already holds a lot of institutional knowledge about data elements quality. Subject matter experts can call-out some of the key dependencies of one data element to the another, key calculations and aggregations can be evaluated. IT needs to encompass these business artifacts and apply them as data quality criteria beyond simply checking the IT processes quality. Rule that takes care of business
Rule that takes care of business
IT managers have always reacted based on the issues reported by business community. Although business has reported the issue, they have not been so much involved in the root cause analysis and issue resolution. Data Quality rules are written in silo by technical teams and due to the less understanding of business knowledge, the scope of rules typically remains superficial i.e. limited to very basic rules like null checks, duplicate checks, record counts and few conformed value checks.
Second challenge is that, the rule is written purely on the technical ground and the logical conditions are buried inside the script. It’s difficult even for technology folks to conduct a peer-to-peer review of such rules especially when the rule logic is complex written code. There is very less or no business involvement in reviewing the rules. Due to the technical fabric of data quality rules, business carry no encouragement to collaborate and partner into building a true business empowering data quality solution.
Data that cannot be trusted cannot be used for business, and hence Data quality should be much more than a simple IT operation. This divergence between business and technology is addressed by a Business Rule.
- Rule that’s written by a business personnel
- Rule that’s non-IT
- Rule that captures business expectations and
- Rule that embeds the business criteria for checking quality of data elements
What good is the business data if you don’t know its business rule?
One of the factors to consider while writing Business rule is that it should not be confused with a business application specific data-entry rule.
Here is a scenario, Application data entry screen restricts a user to only enter a restricted phone number format. There is no value to replicate the same data entry rule in the form of a business rule for that data source. Another example of data entry rule enforcement by business application is Termination date field entry is enforced by an application data entry rule when the contract’s status is auto-selected as ‘terminated’.
Business rules definition should be captured beyond the application specific data entry Rules.
The operational domain knowledge about data that business creates and uses becomes very important aspect in defining business rules. Some of the compliance and privacy data policies can be transformed into business rules. This in turn will evaluate data elements on how much they comply to the policies by enforcing them to follow certain business rules.
For Example: A policy that allows a home-owner to wave the late fees only once in a year could transform into a business rule where the occurrence of customer’s late fee waived cannot exceed more than one in a given year.
Data Quality: Business and IT Co-ownership
If Business is responsible in formulating a business rule then the question is – where is IT team in this business rule activity? IT creates data quality rules based on the business rules specification for each data element or set of data elements. Data quality rule is the actual implementation of a business rule. There is a one to many relationship between a business rule and IT data quality rules. This unique relationship established by a business rule to IT data quality rule now brings business to the table as a partner with IT in developing a comprehensive data quality solution.
Any failures reported on the data quality rule executions are now traced back to a business rule. Business community will be in better position to link its processes, functions, business areas to a business rule and hence the impact from any data quality issue can be seamlessly traced back to the business. That’s a big win for business. Also, IT is now in a much better position to implement data quality solution that addresses combined needs of business and technical communities. A win- win combination of business and IT collaboration provides a big win to data sponsors and senior executives.
Have you faced any unique challenge in implementation of Data quality issue? How do you manage the partnership issues between Business and IT teams at your end? We would love to know more about your challenges, inputs and thoughts here. The very purpose of this Data Quality, Governance and Security focused series is to collaborate and contribute to the DQ Body of knowledge.
Ravindra B. Sutar and Dr Preetam Tiwari
Leave a Comment