Best practise for optimising and maintaining data workflows in Omniscope

Modified on Fri, 3 Oct at 2:26 PM

When you're building or inheriting an Omniscope workflow, it's easy to end up with something that's big, complicated, and hard to maintain. A few simple habits can make the difference between a workflow that feels like a maze and one that's clear, efficient, and a joy to work with.

This guide shares practical ways to prepare your data, simplify workflow design, optimise performance, and maintain workflows over time.

1. Fix problems early (Schema first)

A messy schema at the start usually means pain later on.

Use a Field Organiser or Bulk Field Organiser block early to rename, re-type, or drop fields you don’t need.
Check field types: make sure dates are properly recognised as dates, numbers as numbers, etc.
- Fix input date formats if they're not ISO-standard, so Omniscope recognises them correctly.
- Set output date formats to trim unnecessary bits (e.g. milliseconds) before moving on.
Set fields metadata, for example group so that you can explore them hierarchically in a Report (Table View)

By tidying fields up front, you prevent schema errors and keep downstream blocks clean and predictable.
Think of it as laying a solid foundation before you build the house.

2. Pick fields dynamically, not manually

Avoid ticking boxes one by one every time you want to select fields. Especially for wide datasets, and when your input data changes over time:

Use rule-based field picking: for example, "select all fields containing score" or "exclude fields of type text."
This way, new fields are automatically included if they match your rules.

It's less error-prone, easier to maintain, more robust, and means you don't have to revisit the workflow every time the schema changes.

3. Simplify repetitive sections with For Each blocks

If you're duplicating the same sequence of blocks across multiple fields (like for product, location), there’s a better way.

Use a For Each block to parameterise the logic once and run it across multiple fields or datasets.
This keeps the workflow shorter, cleaner, and easier to maintain.

Instead of 10 copies of the same process, you have just one block that loops through the fields.

Please note the Control Flow section contain further blocks to build conditional flows

4. Prefer formulas over regex

Regex can be powerful, but for many cases it's more complexity than you need.

For date filtering, create a Boolean flag field with a formula like:
```
IF(DATEDIFF([Date], NOW(), "month")>1, "false", "true") 
```
This checks if a record is within the last month.
This approach makes your filters clearer, easier to read, and simpler to adjust later.

We know formula syntax can feel tricky at first. Alongside the articles in our Help Center, you can try our Ninja Assistant bot, which uses our KB to suggest formula snippets from plain language. (We're working on integrating it directly in the Formula Editor though... so you won't need to jump between tools.)

5. Keep an eye on performance

Omniscope handles large datasets well, but design still matters:

Use the Admin Dashboards to see which blocks are slowing you down.
Filter or aggregate data early so you're not carrying unused rows/fields through the whole workflow.
Use Savepoint blocks to optimise data execution time

Often, fixing one or two bottlenecks is enough to make the whole workflow snappy again.

6. Automate the boring stuff

Don’t waste time doing the same cleanup by hand in 20 places.

Use bulk field organisers to rename or tag fields in one go.
Build formula-driven flags (like "last 12 months = true/false") instead of manually re-applying filters.
Create standard macros for common transformations (date parsing, null handling, field naming).

The more you automate, the less room there is for mistakes.

7. Modularise with Macros and data outputs

Once you’ve cleaned up a workflow, you may find it's still big and sprawling. That's the time to modularise:

Macros: wrap repeated block sequences into a reusable block. You can parameterise them for different inputs. Macros are best done after other simplifications, as a final step. They are a way to build a "main" workflow that references smaller sub-workflows; a bit like functions in coding.
Data outputs: if a workflow is too large, you can break it into stages by outputting data as a file, a database table, or even a server-wide queryable storage block. Another workflow can then pick it up as input.

This gives you smaller, clearer workflows that are easier to maintain, and lets teams share responsibilities across different stages of data preparation and analysis.

8. Use AI features (but don’t just rely on them alone)

Omniscope is rolling out AI-powered features like Workflow Ninja to help you:

Summarise large workflows.
Spot bottlenecks or redundant blocks.
Suggest optimisations you might not have noticed.

Use AI as a helper. But remember, only you know the context and purpose of your data.

9. Document as you go

Finally, don't leave your future self (or your colleagues) guessing.

Rename blocks so it's obvious what they do.
Add notes in blocks for any tricky formulas or logic. (Markdown supported)
Add shapes like post-it, rectangles, arrows with comments if necessary
Group related sections visually in the canvas.

A well-documented workflow is faster to pick up, easier to troubleshoot, and much more shareable. Obviously don't over do it. As forgetting to updating the docs could lead to the worst situation: having documentation not in sync with the actual model misleading you in the first place.

10. Versioning and change management

Workflows evolve. A big risk is making changes without a way back.

Save versions before major edits, even if it’s just a duplicate of the project folder. Sure you can rely on the Project Version History within Omniscope, but data execution is not part of it.
Keep backups of stable workflows if you're experimenting with macros or outputs.
Use project folders to keep history and organise experiments.

This gives you a safety net when "optimisation" accidentally breaks something important.

11. Test and validate your workflows

Always test changes before rolling them out fully.

Run workflows on a small test dataset first to check schema handling and logic.
Add checkpoints (like Validate blocks) so you can verify at a glance whether data is flowing correctly, or alert / notify on errors / warnings.
After edits, check schema consistency, especially date fields, categories, and IDs.

Think of it as unit testing for your data. Catching issues early saves hours later.

12. Build for team reuse

Workflows are often shared. Make them easy for others to use.

Create a shared macro library for common transformations.
Use consistent naming conventions for fields, blocks, and outputs.
Add short notes in tricky places so colleagues know why something is done a certain way.

Reusable patterns mean your team solves problems once, not ten times.

13. Keep workflows clean

It’s easy for workflows to grow messy as you experiment. Schedule time for cleanup.

Remove old blocks you no longer need.
Delete or archive unused datasets and outputs.
Trim exploratory branches once you’ve learned from them.

A lighter workflow is always easier to maintain.

14. Monitor and review regularly

Even a great workflow can degrade over time as data changes.

Set up scheduled runs and alerts for failures.
Check logs and timing reports regularly; if a workflow is taking longer each week, degrading, something upstream may have changed.
Review key workflows quarterly (or whenever schemas change) to make sure they are still efficient and clean.

A little monitoring prevents workflows from quietly drifting into chaos.

To recap:

Fix schema issues early with Field Organiser.
Use For Each blocks instead of duplicating logic.
Pick fields dynamically with rules.
Prefer formulas to regex for clarity.
Optimise performance by filtering and aggregating early.
Automate repetitive steps with formulas and macros.
Modularise with macros and outputs to split stages.
Use AI wisely as a helper.
Document for the future.
Version, test, and validate your workflows.
Build for reuse and sharing across your team.
Housekeep regularly to keep workflows tidy.
Monitor over time so small problems don’t become big ones.

Follow these practices and - we promise :) - your workflows will be simpler, faster, and easier to maintain. Not just today, but in the long run.

And if you still need help navigate a monster data model, we're here to help.