Data cleansing, data cleaning or data scrubbing is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database. Used mainly in databases, the term refers to identifying incomplete, incorrect, inaccurate, irrelevant, etc. parts of the data and then replacing, modifying, or deleting this dirty data.
After cleansing, a data set will be consistent with other similar data sets in the system. The inconsistencies detected or removed may have been originally caused by user entry errors, by corruption in transmission or storage, or by different data dictionary definitions of similar entities in different stores.
Data cleansing differs from data validation in that validation almost invariably means data is rejected from the system at entry and is performed at entry time, rather than on batches of data.
The actual process of data cleansing may involve removing typographical errors or validating and correcting values against a known list of entities. The validation may be strict or fuzzy.
Some data cleansing solutions will clean data by cross checking with a validated data set. Also data enhancement, where data is made more complete by adding related information, is a common data cleansing practice.
Data cleansing may also involve activities like, harmonization of data, and standardization of data. Standardization of data is a means of changing a reference data set to a new standard, ex, use of standard codes.
IDN has a proven and effective methodology to assist clients in improving data quality on a standalone basis or as part of a data conversion project.
The majority of information companies use to make decisions is derived from data stored in application systems. Unless continual focus and investment is spent to maintain or improve data quality, it will degrade over time. The causes of data degradation are many:
IDN assists companies with the cleansing of data by ensuring that data pass a set of quality criteria. Those include: