MicroStrategy - Enterprise Manager - Data Loader Process

mstr-em

Done reading:
TN198868: MicroStrategy Enterprise Manager 9.x data load fails with the error 'unique constraint violated.INSERT INTO IS_SESSION ....'
TN16418: The "Custom time window" data load option is disabled in MicroStrategy Enterprise Manager 9.x
TN19826: How to troubleshoot problems with MicroStrategy Enterprise Manager data load

Not yet done reading:
http://community.microstrategy.com/t5/Server/TN237810-Enterprise-Manager-scheduled-data-load-does-not-trigger/ta-p/237810
http://community.microstrategy.com/t5/Server/TN38144-MicroStrategy-Enterprise-Manager-scheduled-dataloads-do/ta-p/188320
http://community.microstrategy.com/t5/Administration/MSTR-Enterprise-manager-data-Loader/td-p/33269
http://community.microstrategy.com/t5/Object-Development/Error-in-data-loader-in-Enterprise-Manager/td-p/157655
http://community.microstrategy.com/t5/Administration/Enterprise-Manager-failed-to-insert-null-value-through-operation/td-p/250328
http://community.microstrategy.com/t5/Server/TN34448-MicroStrategy-Enterprise-Manager-9-0-1-scheduled-data/ta-p/185096
http://community.microstrategy.com/t5/tkb/articleprintpage/tkb-id/server/article-id/1171
http://community.microstrategy.com/t5/Administration/Enterprise-manager-statistics-loading-issues/td-p/253573
http://community.microstrategy.com/t5/Architect/TN44835-How-to-use-MicroStrategy-Cube-Advisor-with-MicroStrategy/ta-p/194561
http://community.microstrategy.com/t5/Architect/TN12093-How-to-enable-and-use-the-MsiFileTable-Log-file-for/ta-p/172722
http://community.microstrategy.com/t5/Server/TN19711-What-is-the-difference-between-the-MicroStrategy/ta-p/179910
http://community.microstrategy.com/t5/tkb/v1/page/blog-id/server/label-name/enterprise%20manager/page/41?labels=enterprise+manager
http://community.microstrategy.com/t5/tkb/v1/page/blog-id/server/label-name/enterprise%2Bmanager/page/40?labels=enterprise+manager
http://community.microstrategy.com/t5/Server/TN30594-How-to-enable-the-SQL-executed-during-the-MicroStrategy/ta-p/181533
http://community.microstrategy.com/t5/Administration/statistics-table-in-EM/td-p/127510
http://community.microstrategy.com/t5/tkb/articleprintpage/tkb-id/server/article-id/1284
http://community.microstrategy.com/t5/tkb/articleprintpage/tkb-id/server/article-id/2789
http://community.microstrategy.com/t5/Administration/Em-Load-Tobuleshooting-ETL-Data-Not-loading/td-p/236378
http://community.microstrategy.com/t5/Server/TN5854-How-does-MicroStrategy-Enterprise-Manager-9-x-know-what/ta-p/166939
http://community.microstrategy.com/t5/Server/TN44869-How-to-confirm-the-status-of-an-Enterprise-Manager-Data/ta-p/194595
http://community.microstrategy.com/t5/tkb/articleprintpage/tkb-id/server/article-id/1266
http://community.microstrategy.com/t5/Server/TN20550-New-feature-in-MicroStrategy-Enterprise-Manager-9-0/ta-p/180712

http://community.microstrategy.com/t5/Server/TN47813-MicroStrategy-Enterprise-Manager-9-x-data-load-fails/ta-p/197443

Where is the log file for the data loader?

C:\Program Files (x86)\MicroStrategy\Enterprise Manager\MSTRMigration.log

Is there a way to monitor the Enterprise Manager data loader process?

There's not out-of-the-box tool in MicroStrategy to monitor the Enterprise Manager data load process, or a way to send an email if the data load fails. As the data load is running, information about the data load is written to the MSTR Migration log, so you can check this log periodically to check on the status of the data load. It may be possible to monitor this log with a third-party tool, but this is outside the scope of MicroStrategy and MicroStrategy Technical Support does not have a third-party tool that they recommend.

To see if a data load has failed you can look for keywords and phrases such as "There was a failure during the process", "Data Load process was canceled by user", "Previous incomplete migration found", etc.

What should we do if the loader fail with "unique constraint (xxx.IS_SESSION_PK) violated"?

Run the following query against the Enterprise Manager Warehouse:

SELECT * FROM EM_IS_LAST_UPDATE WHERE IS_STATUS=0

If the above SQL statement returns more than 1 rows, delete the duplicated rows such that the remaining row match what is described in TN16418.

Run a custom window data load for the data load window which was initially unsuccessful, i.e, the 1st data load which failed when inserting data into IS_SESSION_TMP1 (A full data load will first truncate the data in IS_SESSION_TMP1 before inserting new data. With a recovery data load, this truncation does not happen and duplicate rows end up in the table).

Run a manual data load so that the Enterprise Manager Warehouse data is in sync with the Statistics tables and metadata.

How can we track the time that the data loader process take to load the data?

The Enterprise Manager contains two reports for this:

  1. Enterprise Manager Project > Public Objects > Reports > Operations Analysis > Data Load -> Data Load Durations - Complete (Last Week)
  2. Enterprise Manager Project > Public Objects > Reports > Operations Analysis > Data Load -> Data Load Durations - Project Wise (Last Week)

How can we determine if the data loader process is running?

Using Windows Task Manager, we can check for the process MAEMETL.exe. This process indicates that an EM Data Load is running. Please note that this process is different from MAEMETLS.EXE, which is the scheduler for EM which is always running.

How can we speed up the data loader process?

  1. http://community.microstrategy.com/t5/Server/TN30401-Data-load-seems-to-hang-or-take-a-very-long-time-during/ta-p/181365

When do we need to run a custom window load?

Once the EM_IS_LAST_UPDATE table is updated, the users should run a Custom Time Window Load under the Data Load Options in the MicroStrategy Enterprise Manager 9.x console, which encompasses the time period for which it failed. If the users just update the EM_IS_LAST_UPDATE table without doing a following Custom Time Window Load, they can experience loss of data for the time period that begins with the timestamp in the IS_WIN_END column of the first updated IS_STATUS row of zero ending with the last such updated row.

In other words, we need to run a custom window load after a failed load.

How can we troubleshoot the data loading process?

  1. Look at the log file: C:\Program Files (x86)\MicroStrategy\Enterprise Manager\MSTRMigration.log
  2. Search Google for the particular error message.
  3. Open a support ticket with MSTR support team

What are some typical / common reasons why a data load may fail?

  • Data cannot be loaded from a project that is not currently loaded on Intelligence Server. Data also cannot be loaded from a project that is set to Request Idle, Execution Idle, or Full Idle mode. Before loading data from a project, make sure it is not in any of these idle modes and is set to Loaded status. For an explanation of the different project modes, including instructions on how to set a project’s mode, see Setting the status of a project, page 32.
  • If you have changed the password for the user that configured a project for Enterprise Manager, data cannot be loaded from that project until you reconfigure it in the Enterprise Manager console. In the 2. Configure step, remove the project source from the list of project sources being monitored, and then add it to the list again.
  • If you have deleted a project in Developer that is being monitored by Enterprise Manager, the data load process fails until you remove that project from the list of projects being monitored.
  • If the MicroStrategy Enterprise Manager Data Loader service has stopped, the scheduled data loads will not occur. Restart the Data Loader service to resume scheduled data loading.
  • If MicroStrategy Intelligence server was restarted while a data load process is running, the current data load process may fail, and the sub-sequent data load may also fail.

For additional possible causes of data load failure, see MicroStrategy Tech Note TN16852.

How can we run a custom window data load?

  1. Launch Enterprise Manager Console
  2. Click on Tools -> Data Load Options
  3. Specify a custom time window (in the middle of the screen) and click on Custom Time Window
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License