For Each Block

In Brief

Function: The For Each block repeatedly executes a workflow, once for every record in the Iterations input, appending the results. Each execution will be customised by changing parameter values using fields in the Iterations data.

Typical Use Case: A company designed a workflow to prepare and analyse a dataset. The result of the workflow is used to power a dashboard highlighting key aspects of the analysis. The company would like to compare the analysis with other analyses of the same dataset where some options in blocks used in the workflow are changed. Without the For Each block, the company now needs to duplicate the workflow, change the options, run the analysis again and append the results to update the dashboard. With the For Each block, this can be automated.

Description: In the aforementioned example, it was described that the For Each block can automatically change options and re-run a workflow, or parts of it, and append the results of each iteration. This is done by utilising App or Project Parameters. After building a workflow, we need to identify block options for which we would like to execute the workflow with different values. Then, we create App or Project parameters with some meaningful initial values, and link the options to these parameters. Next, we create a tabular text input in which the fields correspond to the parameter names we just created. Each row is an iteration, and each cell in a row specifies the value we would like to give the parameter for this iteration.

Case Study

We are looking at Kaplan-Meier survival curves for patients of different hospitals. These curves visualise the probability that patients survive up to a certain time. The goal is to compare the curves of different hospitals to see in which patients have better chances of survival.

Workflow

The workflow is available here.

We are looking at a workflow which contains patient data in a Data Table Block. The data is filtered by a Record Filter Block which is configured to only let patients from one particular hospital pass. Next comes the Survival Analysis Custom Block which computes the Kaplan-Meier curves.

Initial Workflow

If we have a look at the Record Filter options, we see that it is currently configured to filter everything except patients from Saint Mary hospital.

Record Filter options

Without the For Each block, we would need to duplicate the Record Filter and Survival Analysis Block multiple times, once for each hospital. Using the For Each block, we automate these iterations.

We continue by parameterising the Record Filter with a Project Parameter. For this, we go to the 3-dots menu and select "Parameters".

3-dots menu

In the Parameters menu, we create a new Project Text Paramameter which we call "Hospital" and which we give the default value "Saint Mary".

Parameters menu

Now we can assign this parameter to the Record filter by clicking on the Parameter-link icon on the upper-right of the option and selecting "Hospital".

Parameterised Record Filter

It is now time to add a For Each Block and a Text Input Block to the workflow.

Final Workflow

The Text Input Block simply lists all the Hospitals over we wish to iterate. Notice that the Field name corresponds to the Parameter "Hospital" we previously created.

Text Input Block

Lastly, we have to configure the For Each Block. In this particular example, we don't actually need to change any options. However, the block allows us to configure various input and output options:

Iteration parameters:

Use all fields: If selected, all fields from the Iterations Input will be used to override parameters. If fields are defined which do not correspond to an App or Project Parameter, a warning is shown. When not selected, it is possible to choose which fields to use from a field selector.

Output:

Add field containing iteration number: If selected, a field with the given name is added to the output. If a field already exists with this name, a number is appended.

Include all Iteration input fields: If selected, all Iteration Input fields are included in the block output. If not, a subset can be chosen.

Include all Workflow input fields: If selected, all Workflow Input fields are included in the block output. If not, a subset can be chosen.

Advanced:

Some blocks produce a different output with different input data. For example, the Pivot Block adds new fields to its output data depending on the fields to pivot on from the input data. If such blocks exist in your workflow, it is possible each For Each iteration will produce a different output schema. If Require same schema is selected, and the schema changes from iteration to iteration, the execution is stopped. If Combine same schema is selected, the execution is not stopped, and all the different outputs are collected and combined. In this mode it operates the same way as the Append Block.

Output

The output of the block contains a new field with the default name "Iteration", or whatever was chosen in the options, as well as all, or a subset, of the Iteration input fields, followed by all, or a subset, of the Workflow input fields.

For Each outputs

Visualising the results

The results are visualised by creating a series of plots visualising the survival rates.



Old article about For Each Custom block

We will explain a specific use case where we populate a database using the For Each block, which can be found in the Custom blocks section in Omniscope Evo workflow menu.

Note: This Custom block is running a Python procedure behind the scenes, so you need to have Python installed on the machine that is running Omniscope.

In many use cases where there is a need to execute the same procedure for multiple parameters you may consider using this block to manage a looping mechanism.

Let's have a look at a scenario where the user wants to extract multiple csv files from a folder, then populate a database, where each csv file will become a new table in the database.

We will create one Omniscope project that will be tasked to extract the csv file metadata from the target folder, containing the csv details, and the second file (here 'File 2.iox'), that will publish the data to a database.  

Step 1

In the first file we are using the File Attributes source block to connect to the data folder, as we are interested in the file names/ file path only, rather than the Batch Append block, which would extract the data as well.

Following that we will use a Search/ replace block to remove any characters from the file names that might not be welcome in the database table naming convention (dashes, spaces etc.) 

Using the Field Organiser block we will name the two fields of interest [Table name] and [File path]. 

To configure the ForEach block we will use the url address of the File 2, and also reference the 2 parameters : Table name and File path (these will appear as options in the drop-down menu).

Step 2

In the File 2 we will create 2 project parameters with matching names - Table name and File path. There is no need to specify initial values and you can leave them blank for now.

We will connect a File block, which has a File path parameter in the Path field, to a Database Output block, which has the Table name param in the Table name field, and our target database credentials.

(There is a pink link symbol next to every block field that has an active parameter setting)

Before executing the first file, containing the ForEach block, it is useful to check 

Admin settings > Advanced >Workflow > Concurrent Workflow Jobs 

that Omniscope installation can execute 2 or more workflow jobs concurrently.

Now we can run and execute the first file, which will trigger the File 2, as our ForLoop block is running the execution remotely, going through the files one by one and populating the target database.

This is just one example where this block can be applied.