Performance and diagnostics - labs features

In Omniscope 2021.2 we have introduced some "labs features" relating to workflow execution performance and diagnostics:

These are not fully mature, and may be removed or altered before being considered production-ready, but are available for experimentation. The first two choices may improve performance in some use cases where you have complex workflows with many blocks and sequences thereof.

The last option enables collection of execution performance statistics, such as the CPU time spent executing each block. It is useful for the dev team to gather feedback on real-world execution performance, but also useful for our users to self diagnose performance bottlenecks. If enabled, every execution will write data about the workflow and about individual blocks within to a "workflow_exec_perf.csv" file within the omniscope-server/logs folder, which you should consume using a further workflow and report.

Here's an example (split across 2 screenshots):

For each row, you can see the project (IOX file), and whether it is reporting the workflow execution as a whole, or an individual block within that execution. "Identifier" tells you the block name, followed by the exact start/end times; for blocks, it's the time the block first started doing any work, or last stopped.

"Time, elapsed" is the time between those points.

"Time, active" is the total time spent working on the block; if the block pauses to wait for input data, it won't be counted. This is the best measure of how much the block is a bottleneck.

"Time, thread" is the total time per CPU spent working on the block. So if a block uses two CPU cores consistently, this would be double the active time. This is the best measure of the amount of CPU resource used for that block.

The thread counts let you know how many CPU cores were used during execution. Most blocks use a single core since they process data in a stream, but some (e.g. Join, Append, and more depending on Labs features) use multiple cores.

Finally, "Extra data" contains additional JSON metadata related to execution, and is populated mainly by Labs features, if enabled.

If you have a workflow that is performing poorly, please enable this labs feature, execute/refresh the workflow, and send us the workflow_exec_perf.csv file along with a screenshot of the workflow, for us to analyse and help advise upon.