NumPy / pandas incompatibility in custom Python blocks ("dtype size changed")

Custom Python blocks can fail after an Omniscope update (or after a custom block installs new Python packages) with an error caused by incompatible versions of the Python libraries numpy and pandas. This article explains why it happens and how to resolve it.

What the error looks like

A Python custom block fails to execute, and the error text contains one of the following:

• ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
• AttributeError: np.unicode_ was removed in the NumPy 2.0 release. Use np.str_ instead.

Both mean the same thing: the numpy and pandas versions in the Python environment your block is using are not compatible with each other. In practice this is an older pandas (before 2.2.2) together with numpy 2.x. NumPy 2.0 (released June 2024) was a breaking change, and pandas only became compatible with it from version 2.2.2 onwards.

Why it happens

Omniscope installs the Python packages your custom blocks need into a managed environment. Two things combine to cause the mismatch:

• When the Omniscope version changes (a major upgrade, a minor update, or a rollback), that managed environment is cleared and rebuilt from scratch. This has been the behaviour for years, to give each Omniscope version a clean set of packages.
• Custom blocks bring their own dependencies. A block can pull in a newer numpy (2.x) alongside an existing older pandas, or vice versa. Because the environment is shared across all of a server's Python blocks, once an incompatible pair is present, every Python block can fail.

Because the environment is rebuilt on each update, the problem can return after a later Omniscope update, even if you fixed it before.

How to fix it: Docker mode, or fix the host-mode packages

There are two routes. Docker execution mode avoids the problem and is an alternative to everything else here; the host-mode options below repair the shared Python environment when Docker is not available.

The recommended route: Docker execution mode. In Admin > R/Python & Blocks Library, switch custom block execution to Docker. Each block execution then gets its own isolated Python environment, so an incompatible package combination cannot arise and blocks cannot interfere with each other. This is our recommended approach for production servers. It requires Docker to be available on the server.

If Docker is not available (host mode), you instead need to get a compatible numpy/pandas pair into Omniscope's managed environment. A compatible pair is either an up-to-date pandas with numpy 2.x:

• pandas==2.2.2
• numpy==2.2.6

or, if any of your blocks require an older pandas, an older pandas with numpy held below 2.0:

• pandas~=2.0.0
• numpy>=1.24,<2.0

Apply one of those pairs in either of two ways:

• In a single block's Dependencies: open the block, choose Unlock Custom Block design from the 3-dots menu, and add the pair to Dependencies in the Design tab. Best when only one or a few blocks are affected.
• Across the whole environment, with the "fix packages" project: import and run the Fix python packages project attached to this article (see the attachments at the foot of the page). It is an empty custom block that pins a compatible numpy/pandas pair and installs it for every block in one step. The versions it pins (an older pandas with numpy held below 2.0) suit most cases; if your blocks need a different pair, edit them in the block's Design tab. Run it once after each Omniscope update.

Purging packages manually is a reset, not a lasting fix. You can locate Omniscope's managed Python package folder, delete it, and restart Omniscope so it reinstalls packages fresh (see the Custom Block Troubleshooting article for the location). But if the same workflow runs again it can reinstall the incompatible pair and the error returns. Use it to get a clean slate, then pin a compatible pair as above so it sticks.

Finding which projects use custom Python blocks

To see which of your projects contain custom Python blocks, so you can check what might be affected or needs attention, import and run the Scan projects for blocks project attached to this article (at the foot of the page), read its annotations, and click Refresh.

It can return after an update

Any Omniscope update rebuilds the managed Python environment, so a manual or pinned fix may need re-applying after you next update. If your blocks run on a schedule, verify a Python block after each update. We are improving how Omniscope manages Python libraries so that this becomes unnecessary; Docker mode already avoids it today.

A note on custom blocks

Where a core block covers your need, prefer it to a custom Python block. For example, the core Depivot block now offers the same functionality as the community Melt De-pivot custom block, using rule-based field selection, and is not exposed to Python package changes.