Macros in Omniscope Evo

Omniscope Evo 2022.1 adds the ability to decompose complex workflows into smaller "macros" and optionally share them with other users/folders/projects on the server via the Macros folder, using these 3 new types of block:

You can use macros without sharing them, as a way of reducing the complexity of a large workflow, by creating a smaller project alongside the main project, and using the "Macro" block to embed it within the main project.

Or, you can create projects within the "Macros" folder, and make them immediately available to all users and projects in other folders on the server, via the Add Block menu.

To get started, install Omniscope 2022.1 or later, and check out the new demo (Demos/Macro demos/Example macro). Pre-existing Omniscope installations will need to install updated demos from Admin first.

Example uses

• You administer a team server, and have a set of curated data sources, and want your users to be able to use these directly without worrying about database connections or CSV file formats.
• You have a large workflow with several different sections, and wish to simplify the main workflow (an alternative would be to use visual grouping of blocks).
• You have a common series of operations you wish to perform, such as reformatting some fields, validating a commonly seen schema, or merging a series of lookup files, and want these operations available as one-click blocks in your day to day workflows.

In pictures

In this example we've created a shared macro by creating a project "UK postcode lookup" within the Macros folder:

Opening it reveals the macro's composition. In this case it's the same macro demo we bundle in the Demos folder, with additional annotations explaining macros:

Here's a simplified close-up, showing the Macro Input and Macro Output blocks:

Elsewhere, you can now use this macro by adding it from the Add Block menu:

Alternatively, if the macro project isn't in the sharing folder, you can still use it via the Macro block:

The Macros folder

The Macros folder is typically "Macros" within the root of the sharing folder, i.e. .../omniscope-server/files/Macros. It is created automatically for new installations, but must be manually created (a regular folder called "Macros") for existing server installations.

If desired, this folder can be deleted to disable it, or relocated via "Admin > Web server > Settings > General > Macros folder" (visit /_admin_/wsgeneral/ on your server).

The folder stores "macro projects", i.e. reusable workflows, and is automatically and immediately shared with other users and folders on this installation. All projects in this folder will be available for use as Shared Macro blocks within the Add Block menu of the workflow in other projects elsewhere.

Subfolders can be optionally used to organise macros into categories within the Add Block menu. For example you might have a folder called "Preparation" (an existing block category) or "ACME Corp data sources" (a new block category). In the screenshots above, "UK postcode lookup" sits directly in the Macros folder, so is shown under Macros in the Add Block menu.

Security warning: all projects in the Macros folder can be used by any project anywhere on the server. See "Security and permissions" section below.

Macro definition

The macro project's workflow can be considered to provide the "definition" of the macro. (If it is a shared macro, end users might not have permissions to see this, and might only be able to use it as an opaque block within the Add Block menu.)

When a project using a macro executes, it's as if the macro workflow's blocks have been copied into the main workflow while execution proceeds. Inputs and outputs are mapped to Macro Input and Macro Output blocks.

Inputs and outputs

The macro workflow uses the Macro Input and/or Macro Output blocks to define the input(s) and output(s) available at point of use:

• A macro which transforms data will typically have one Macro Input and one Macro Output. 
• A macro which publishes data will typically have one Macro Input and no Macro Outputs. 
• A data source macro will typically have one Macro Output and no Macro Inputs. 
• More complex cases can have any number of inputs and outputs, as needed.

In this example macro definition, a single input data flow is transformed via a lookup into a single output data flow, providing the useful ability to add lat/lon coordinates for UK postcodes:

Macro workflows typically have 5 distinct sections:

• Specimen data (the workflows upstream of any Macro Inputs), which defines the fields needed by the macro, and provides test data for developing the macro. These blocks do not participate in macro execution.
  In the above example we're using a Text input block to define a few simple records of data. This macro requires you to supply a Postcode field (this field is required by the Field organiser block 'Extract first part'); any other fields will be propagated to the output, unchanged.
  
  
• Macro Input block(s), which define the input connectors available on the resulting Macro block at point of use. In the above example there is one Macro Input block, so at point of use, the Macro block will have one input connector, where the user will need to supply one data stream.

  If you have multiple inputs, you must configure their 'number' option to be uniquely numbered from 1 to n, and typically should configure appropriate titles and descriptions.
  
  

• Macro constituents, the block(s) which perform the macro's function. In the above example:
  A Field organiser block adds a "Postcode, first part" field which uses a regular expression to isolate the first part of the postcode;

  A Lookup data block brings in some bundled 4-letter postcode centres data;

  A Join block merges these to bring in lat/lon coordinates;

  And another Field organiser block deletes the field added by the first block.

  Note that a Field organiser block in macros should be used with "Include other input fields" ticked and "Auto-sync" unticked, to allow arbitrary fields to pass through.
  
  

• Macro Output block(s), which define the output connectors available on the resulting Macro block at point of use. In the above example case there is one Macro Output block, so at point of use, the Macro block will have one output connector, where the user will be able to consume a single output data stream containing input data augmented with lat/lon coordinates.

  If you have multiple outputs, you must configure their 'number' option to be uniquely numbered from 1 to n, and typically should configure appropriate titles and descriptions.
  
  

• Assertions/tests (the workflows downstream of any Macro Outputs). Each Macro Output block can optionally be followed by an arbitrary set of blocks that do not participate in macro execution.
  This is useful when designing a macro. In this case, we're using a Report block to visualise the output from the macro when run against the test data provided by Text input.
  You might also use blocks such as Validate data, to perform assertions on your macro's results given the test data.
  
  

Options 

To give your macro some end-user options, in the macro workflow, open Parameters (see Workflow parameters) in the 3-dots menu to see and configure some project parameters:

In this case we've added a single parameter "Include unmatched data", boolean, ticked by default, which we then reference in the Join block "Merge lat/lon" below.

This will let our users pick whether to include records for which the postcode wasn't found.

It's not currently possible to let users pick fields. Instead you must define your "specimen" data with appropriately named fields (in this example, "Postcode"), and at point of use, if necessary Field Organiser blocks can be used ensure such a field exists in the data flowing into the Macro block.

Side-effects

Executing a macro block has no effect on the macro definition - copies of the macro workflow's blocks are effectively executed as part of the main workflow. There are no hidden side-effects when a macro block is executed.

Be careful, however, about including blocks that have obvious external side-effects inside a macro. For example, a File Output block (which writes data to a file), or a Database Output block (which writes data to a database), or a Custom R/Python block (which executes arbitrary code with potential for uncontrolled side-effects). If two projects use the same macro, you could inadvertently execute them sequentially but in different orders, and get unexpected results. Or, worse still, execute them in parallel and have errors or data corruption.

Security and permissions

If needed, the shared Macros folder's permissions can be administered to hide it from the enclosing folder, or to limit access to specific users, using regular Omniscope folder permissions. Note that irrespective of these permissions, a shared macro (a project in the Macros folder) will be available as an opaque macro block in the Add Block menu to any user authorised to edit workflows; permissions on the Macros folder are used to restrict which users can view or edit the "definition" of the macro (i.e. open/edit the macro workflow itself).

Macros that are not shared and used via the Macro block can be accessed by any user or project that has permissions to access their enclosing folder. If this is not desired, you should delete the Macros folder on your server.

Omniscope 2022.1 introduces "Project Resource Access" settings - finer-grained controls over what resources can be accessed by projects. In addition to limiting access to data files in the sharing folder, you can now add a whitelist of additional folders outside the sharing folder (by default we do this for "administrative projects" for Omniscope's own logs folder), and can also limit projects to only access data files within a particular subfolder of the sharing folder. 

When using macros, the macro definition (the macro workflow, containing Macro Input and/or Macro Output blocks) determines what files can be consumed or created by blocks within the macro. This may have different restrictions to the project using the macro (at point of use). For example:

1. The macro definition might be <files>/Macros/My_macro.iox. You might configure the Macros folder such that projects inside it can access data files across the entire sharing folder, but only certain users might have permissions to author these macros.
2. At point of use, a project using the macro might be <files>/JoeBloggs/My_project.iox. The JoeBloggs folder might be restricted, with projects inside only able to access files within the JoeBloggs folder.

In this example, if there is a File block within My_project.iox, it will be able to browse and consume data files within the JoeBloggs folder (and subfolders) only.

But if the shared macro My_macro is added to My_project.iox, it will effectively have "empowered" My_project to access files across the entire server, via File (or File Output) blocks within My_macro.iox.

You can control this by deciding what blocks to put into macro workflows, and whether to let them be customisable at point of use via project parameters.