Analysis Services hung on Data Tier

While looking at our TFS 2008 Data Tier today I noticed that the msmdsrv.exe resource (i.e. Analysis Services) was using 99% of the Data Tier CPU. I knew that when the OLAP processing happened, the Data Tier process would spike. But this spike was consistent. Something was wrong.

When running the GetWarehouseStatus web service (i.e. http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx?op=GetWarehouseStatus) we got "ProcessingOlap". Typically this takes a minute or two for us. Oddlly, we were getting this status for over an hour (though I'm sure it was happening for a while longer than that).

We figured the process was hung and wanted to restart it. We have about 150 users on the system so we didn’t want to recycle IIS as it would drop users. So instead of recycling IIS on the App Tier, we shut down the Analysis Service on the Data Tier. This brought the Data Tier CPU usage back to normal.

Then to test things out, we invoked the data warehouse update (using the Run web service). When the OLAP processing ran, the Data Tier process neared 100% a number of times (as expected). After about a minute, the process finished and the Data Tier was back in good standing.

We're not sure what to take away from this, but it's something we'll have to monitor.