Data Engine errors troubleshooting

If you experience an error with Data Engine during app initialisation, Omniscope will warn you by showing a warning message in the app welcome page header and in the app project toolbar.

The message looks like this:

Data Engine was not initialised correctly.

You will not be able to query data in Views.

See app logs for more info about the error.

This error can happen after an Omniscope upgrade on some environments, e.g. an antivirus blocking the app, an app crash, a forced reboot, or some packages missing on Linux.

Error scenarios

Go to Admin web app -> Logs section and open the “main_log.txt” log (e.g. https://youromniscopeaddress/_admin_/logs/server )

Look for the “Could not complete Omniscope Data Engine initialisation” message and gather the information around that line. 

You may find yourself in one of these scenarios, looking at what the Omniscope log messages contains:

• A) "!FATAL: GDKlockHome: Database lock 'C:\Users\Username\AppData\Local\Visokio\Omniscope\MonetDb\dbfarm\visokio_DEC2016\.gdk_lock' denied"
  
  This means that after a crash or a forced reboot the .gdk_lock file was left there.
  To fix this:
  1) Stop Omniscope
  2) Delete the .gdk_lock file from the folder as seen in the logs
  3) Restart Omniscope.
  
  
• B ) “MonetDb server not found on your Linux OS.”
  
  This means you have to install the Linux packages, by following the instructions present in the logs / or by looking at the Linux installation setup guide.
  
  Stop Omniscope, install the packages required and start Omniscope again. You should be good afterwards.
  
  

• C ) “The Monetdb installation found is not the recommended version of Monetdb” (Linux)
  
  This means you have the wrong packages version on your machine and you have to install the recommended one. In the logs, you will find which is the recommended package version to install.
  
  Follow the Linux installation setup guide to install the Linux MonetDB packages.
  (The command to install the packages is something like:
  
  sudo apt-get install monetdb5-sql=MONETDB_VERSION monetdb5-server=MONETDB_VERSION monetdb-client=MONETDB_VERSION 
  
  where MONETB_VERSION is the recommended version number, e.g. 11.29.7
  As above, follow the setup guide, stop Omniscope, install the packages and restart Omniscope.
  
  

• D ) "Your system is configured to use HTTP or SOCKS proxy on localhost / 127.0.0.1"
  
  If you have network proxy enabled on all protocols and on any address, localhost included, Omniscope may not start correctly.
  
  To fix the issue you have to disable proxy to be used on localhost address, and surely on other protocols which are not HTTP / HTTPS / FTP.
  
  On Windows 7 in the proxy settings untick "use the same proxy server for all protocols", confirm the changes and then restart Omniscope.
  
  On Windows 10 you need to add "localhost" to the list of addresses to exclude:
   
  
  
  

• E ) Any other case (if MonetDB cannot start for another reason)
  
  Check the logs carefully, if Omniscope recognises a known issue you will find the solution explained step by step in the logs so you can solve it on your side.
  
  Otherwise, send us the logs file content, and we will be in touch.
  
  You can download all logs as zip by following the link in the app warning, or by visiting the admin app - data engine section at youromniscopeaddress/_admin_/dataengine/
  
  

Let Omniscope fix your tables

If you have installed Omniscope and MonetDB packages correctly, and Data Engine cannot be initialised when starting Omniscope especially after a major version upgrade, and you are in error scenario E and no solution was suggested in the logs,  you can try the following step to have Omniscope rebuild your tables:

1. Stop Omniscope.
   
   

2. Backup LocalStorageRegistry.xml file (Don't delete it! Just create a copy for safety).
   
   Find this:

   1. on Windows in the Visokio/Omniscope folder in the %LOCALAPPDATA% folder, e.g. from  C:\Users\MyUsername\AppData\Local\Visokio\Omniscope

   2. on Linux / Mac in ".visokioappdata" folder in the home directory.
      
      

3. Backup the MonetDB Data Engine folder, usually located in the same folder as above, but user customisable. Found the location in the Admin web app -> Disk manager section (e.g. at http://youromniscopeaddress/_admin_/diskmanager/ )
   
   

4. Rename the “dbfarm” folder located in the MonetDB Data Engine folder to “dbfarm_old”.
   
   

5. Start Omniscope
   
   

6. If Data Engine is running correctly proceed to step 7, otherwise if you still have the warning / error complaining about “Data Engine was not initialised correctly” stop here and contact us.
   
   

7. Go to the Admin web app
   
   

8. Go to Data Engine and Storage section
   
   

9. Click on “Fix all tables” button and wait until all tables have been fixed.
   
   This can take long time depending on how many tables and how much data Omniscope has to fix.

10. Once you are completely satisfied the data has been restored correctly, you can delete dbfarm_old from step 4, and backups created in steps 2 and 3.

About the dbfarm and related files/folders

Some advanced / detailed notes about files and folders used internally by Omniscope, which shouldn't normally be administered directly.

The <visokioappdata>/Visokio/Omniscope/localStorageRegistry.xml file tracks:

1. Storage Blocks, raw data; persisted as IODs and managed by the MonetDb db engine, at <visokioappdata>/Visokio/Omniscope/MonetDb/datastorage;
2. Storage Blocks, queryable data; persisted as MonetDb tables within …/dbfarm (corresponds to raw data IODs in the datastorage folder);
3. All other blocks, queryable data; persisted as MonetDb tables within …/dbfarm (corresponds to raw data IODs in the <project>.data folders).

If you delete the datastorage folder, you will permanently lose persisted Storage Block tables. You should never do this.

If you delete the dbfarm folder, you will lose queryable data for all blocks. This should be entirely safe to do, merely incurring a delay on opening reports when the queryable data for that report is lazily but automatically recreated.  Omniscope keeps queryable forms of data in addition to raw data, and can always rebuild it from the raw data, providing that hasn't been deleted (see above).

In the Admin app, the Fix all tables option will appear if queryable data is missing, and proactively do the same, instead of on-demand when a report is opened.

It's safe to delete this, but will incur a delay on first subsequently opening reports, and should only be done if you are encountering reliability or performance issues with report data.

If you delete the localStorageRegistry.xml file, you will lose all Storage Block data entirely and all queryable data for all blocks. You should never do this.

Note that it is possible to relocate the “MonetDB Data Engine” (<visokioappdata>/Visokio/Omniscope/MonetDb/) , containing the dbfarm and datastorage folders, in the Admin app. Check there before administering these folders manually.