Setting up data validation mechanism and alerts

When dealing with transactional datasets, or any other scenario where data always contains the same fields, and is coming in on a regular basis, it is useful to set up monitoring mechanisms that will automatically evaluate new data, screening it for different types of anomalies. These may include :

- data schema - screening both data fields and data types, e.g. missing fields or new, unexpected fields; Date field with text values entered by mistake

- cell value validation - screening for values outside the expected range - e.g. percent field > 100%, or age >100

This is by no means the only method to screen and deal with a rigid or an evolving dataset. In some scenarios you may prefer to use data mapping, and merge the raw data with a set of pre-defined values, that will replace 'dirty data', or deposit the regular or irregular values in an output file. 

You may wish also to combine the Validate data operation with the Record filter to set up a production logic, where all the records are going through regular screening, then some passed to the report, others flagged via email alert and saved in a file that someone will check (see image).

A number of settings allow the users to tailor the results - to accept/reject records that don't satisfy all/some criteria and also what happens when the records are outside the specified rules.

In a situation where no anomalies are detected, the data will pass through the Validation block. In case of any criteria not being met, different actions could be set up:

- Warning - the block will flag the fields and breached rules, but will not halt the execution and records passing through

- Error - an error message will be generated and the data will not pass through the block

- Email alert : email mechanism could notify selected users about any issues found, or send a note about the execution, regardless of the evaluation outcome.

Setting up an email account for notification messages

Email account details, such as email account of the sender, mail server, port number and credentials, should be configured in the Admin part of the Omniscope application. 

If the email account has 2-step verification set up, an actual application-specific password will be entered in the Password box.

See the example below.