Workflow Execute and Refresh from source actions

Omniscope has improved workflow execution actions in projects compared to previous version:

• Permanently keep the execution data in full for blocks connected to Report blocks.
  
  

• Temporarily keep the execution data in full when

        (a) the data is underneath a small threshold (e.g. 500K cells); 

        (b) for all final blocks and 

        (c) for a single block open or selected - as long as there is disk space available under a user defined free space threshold.

• Omniscope will perform auto-cleanup of temporary data to keep workflow execution data only when necessary, giving priority to more recent workflow executions. Typically any just-executed data will be retained for about 15 minutes, to allow you to check it.

You can perform workflow execution in different "scopes":

• The entire workflow (using the main workflow Execute/Refresh buttons when no block is selected)
• One or more selected blocks (you can select by press-hold-drag, or ctrl-click), using the main workflow Execute/Refresh controls
• The currently opened block, using the Execute button at the top of the block dialog.
  
  
• From 2022.1 Rock build - you can now also set within the Report to auto-refresh on open. See Report Settings > Global tab > Auto-refresh section. 

There are two ways to execute your data workflow. These are available as buttons in the workflow (on the main toolbar / in the 3 dots menu) and from inside a Report.

Execute ("what's needed") 

When you click the Execute button in the Workflow App, Omniscope will now execute only the necessary workflow paths to provide the data downstream to the final blocks. The execution is more efficient and dynamic so you can execute and build data as you build your workflow.

Omniscope will work out what blocks to execute by taking into account previous execution data and the current blocks configuration status.

Note that if a block (say a File block) was already executed in full and its data was cached, and you execute a block downstream of it, Omniscope will reuse the previous execution input data, without refreshing the File block data.

When you select a block and click Execute button, Omniscope will disregard the previous execution data for the selected block, and re-execute the workflow upstream from it.

The UI will show whether a block has been executed and its execution data is available (cached) within the project stored in full (block will appear with fully coloured background) or when a block execution data is not present (block background will fade out).

Refresh from Source (force refresh all data from inputs)

When clicking "Refresh from source" Omniscope will trigger workflow execution from the block inputs (sources) all the way through to the final (furthest downstream) blocks, essentially discarding any previous execution data and re-executing the workflow fully, loading fresh data from the sources.

 
If a block is selected in the Workspace, clicking "Refresh from source" will only affect the workflow paths flowing to the selected block, making Omniscope execute the selected block upstream branches.

From 2022.1 Rock build - you can now also set within the Report to auto-refresh on open. See Report Settings > Global tab > Auto-refresh section.

Use cases

As you build up your workflow, adding blocks, exploring data, you typically use the Execute button on newly added blocks, to fill them up with data, reusing the previous step cached execution for speed.

Once your data workflow is setup, you typically use the Refresh from source button, for example in cases you just want fresh data to flow through you blocks, when refreshing a Report or publishing new data to an output file.