Rule-based field pickers in Pivot, Aggregate and Compare blocks

When an Omniscope workflow is connected to data sources with variable schemas (missing or new fields from one execution to the next), there are settings that allow users to either detect these inconsistencies (using the Validate Data block) or manage them.

'Rule-based field pickers' in Pivot, Aggregate, and Compare blocks make data modelling easier and enable you to define logical and flexible field selection rules to handle variations in field schemas. These settings let you create change-tolerant workflow configurations, where fields can be chosen based on filter rules, such as all numeric fields or fields with names containing "price".

For example, let's say we have a company's spend data and want to create an aggregated table with one line for each combination of fields that contain "Category", while the block should pick up all resulting fields that are formatted as decimal and contain "amount" in their names.

Note: In both the Split and Result settings, you can select the data type rule and specify that the field name must contain a specific substring. The text box can contain one or multiple fields (or field rules) while sharing the same function (e.g., sum Field1, Field2, and rule X). This will have the same effect as having separate settings for the three fields/rules, with "sum" selected as the function for each one.

Each setting that has a chain icon next to it can be parameterised and this rule can be centrally managed for multiple blocks in the workflow in one location (either on the project level or the app level).

And here is the result of this aggregation - all category fields are picked up and all amount fields are summed up as result.