Nondeterministic data

In Omniscope, the workflow expects blocks to produce deterministic output during a single execution. Each time data is retrieved from a block, the workflow assumes the same sequence of records will be returned.

In rare cases, certain blocks may not meet this requirement. When possible, Omniscope detects this and automatically retrieves and stores a copy of the data for the duration of execution. However, this introduces a performance cost, and nondeterminism cannot always be inferred, so this safeguard is not applied universally. When nondeterministic output is retrieved multiple times, it can result in duplicated or incorrect downstream results.

If a nondeterministic block is only queried once, this is not a problem - and most workflows behave this way. Issues arise only when a nondeterministic block is combined with a workflow structure that triggers multiple data retrievals and does not implicitly cache results.

To help identify this situation, we’ve introduced a setting under Omniscope Admin > Settings > Advanced > Workflow called “Block repeatability checks.” This option is disabled by default because it adds slight processing overhead. When enabled, Omniscope verifies that repeated data retrievals from a block return identical results and will fail execution if inconsistencies are detected. (Internally, this is done using record hashing, so there is no additional storage overhead.)

To manually resolve nondeterministic blocks, add a Savepoint block immediately downstream and enable the “Temporary” option. This ensures the data is calculated once and then reused consistently throughout the workflow: