Slow "Get" issue resolved

Three or four months back we decided to move our build systems from a dedicated virtual machine (VM) farm to the company supported VM farm in the data center. The data center infrastructure had all the bells and whistles (support, backups, redundancy, etc.,).

Everything worked well with the exception of the Gets. Our Gets were taking between 5-10 times longer than in our dedicated farm. We thought it might be a hardware issue, but the new VMs had more memory, more processors, sat physically closer to our TFS server, etc. What the heck?

I spent a good 2-3 days running tests and engaging experts including Teamprise and Microsoft. (I also gained some gray hair.) No such luck getting it figured out.

Turns out my colleague ran into a guy who was having similar issues after migrating to the data center VM farm. Turns out, the data center VM farm was running a new version of our virus scanning software. The real-time scanning must have been checking every file. Once we turned that off, our Gets were significantly faster in the data center VM farm.

What I still don't understand is that the Gets were taking abnormally longer on JAR files (e.g. mac.jar) compared to other binary files (e.g. mac.msi). I'm thinking that since a Jar file could be setup as an executable (I think that is possible in Windows), the virus scanner was spending extra time on those. This is just a hunch.

So if you're running into performance issues with your TFS Gets, try disabling your virus scanner (just for testing of course) and see if that is the cause.

"Updating Work Item Data has encountered a problem"

If you get an error in Teamprise like "Updating Work Item Data has encountered a problem. Java.sql.SQLException: File input/output error: Unexpected Token:", just clean your local cache. It's probably some kind of synchronization issue with the local cache and TFS server.

On XP the Teamprise local cache data are stored under C:\Documents and Settings\%USERNAME%\Local Settings\Application Data\Teamprise\2.0. For this error you just need to delete the ./db directory. You will have to close Eclipse or Teamprise before doing so.

Finding the process locking files or folders on Windows 2008 Server.

If you ever need to find what process has a file/folder locked in a 64-bit Windows 2008 Server machine, check out Handle. I usually use WhoLockMe, but it didn't work on my 64-bit machine running Windows 2008 Server.

Unzip handle.exe on your machine and issue "handle c:\myFolderName". It will list out the process ids that have the file or folder locked.

Sysinternals has a ton of other tools that I've found very helpful in administering servers.

Disabling an ASP.NET Web Project from building in a Solution

We ran into a problem recently where we wanted to add an ASP.NET Web Project to our Solution but to not be built during the Team Build. Why? It's a long-story, but it seems to be working well for the group. We had this setup before and working fine, but then someone broke it and we just got around to fixing it again.

Unfortunately when adding the ASP.NET Web Project this time, we were getting errors in Team Build because Team Build couldn't find the project to build Hum? In the Solution's Configuration Manger we had unselected the "Build" check mark for the ASP.NET Web Project. Yet the build continued to run.

After scratching our heads, my colleague (my much smarter colleague) noticed that the Solution File had this in it.

{95153BD9-7C07-4754-8DB2-37512B95B43C}.Release|Any CPU.Build.0 = Debug|Any CPU

My understanding of this is that when we run the build via Team Build, we say "Release|Any CPU" in the TFSBuild.proj file. It's the default. When Team Build (actually MSBuild) sees this line, it assumes that we intend to build the ASP.NET Web Project. However, we don't and what I'm still missing is why unselecting the "Build" check mark didn't remove this line.

To remove the solution setting, my colleague Right Clicked on the ASP.NET Web Project, Selected Property Pages, Selected Build and under "Before running startup page," she changed it from Build Website to No Build. The check box Build Web Site as part of Solution was already unchecked though if it was checked, we would have unchecked it as well. Now the Solution build works fine as it does not build the ASP.NET Web Project (which we intentionally don't want to build).

In summary, do get the ASP.NET Web Project to not build, we 1) unchecked the "Build" check box in the Solution's Configuration Manager, 2) Changed the ASP.NET Web Project's "Before running startup page" to No Build and 3) made sure check box Build Web Site as part of Solution was unchecked.

Some combination of these removed the line {95153BD9-7C07-4754-8DB2-37512B95B43C}.Release|Any CPU.Build.0 = Debug|Any CPU from our solution file.

TFS 2008 SP1 upgrade nightmare

I'm typically very pro-TFS on this blog, but I have to share some negative news. Mainly because it cost me, and my colleagues, an entire evening away from our families while we upgraded. Here are the issues and resolutions from our nightmare. It took us six hours in total. We're done now and I'm off to my home where I can lie awake all night and stew.

Issue #1:
Issue: .NET Framework 3.5 SP1 install failed when installing Visual Studio 2008 SP1

Resolution: Installed framework separately.


Issue #2:
Connecting to SQL Server Reporting Services. Please wait...
Invalid namespace
Value cannot be null.
Parameter name: uriString
Configuring SQL Server Reporting Services failed.

Resolution: Removed the Standby Key from the database.
C:\Program Files\Microsoft SQL Server\90\Tools\binn\rskeymgmt -l
C:\Program Files\Microsoft SQL Server\90\Tools\binn\rskeymgmt -r asdfasfasdf-8968686-asfa-asdfdsf-9as7dfasfasd



Issue #3:
Issue: Restart attempt failed.

IIS Admin Service or a service dependent on IIS Admin is not active. It most likely failed to start, which may mean that it's disabled.

03/26/09 19:55:52 DDSet_Status: Process returned 1062
03/26/09 19:55:52 DDSet_Error: The Commandline '"C:\WINDOWS\system32\iisreset.exe" /Restart' returned non-zero value: 1062.
MSI (s) (AC!14) [19:55:52:046]: Product: Microsoft Visual Studio 2008 Team Foundation Server - ENU -- Error 32000.The Commandline '"C:\WINDOWS\system32\iisreset.exe" /Restart' returned non-zero value: 1062.

Error 32000.The Commandline '"C:\WINDOWS\system32\iisreset.exe" /Restart' returned non-zero value: 1062.
03/26/09 19:55:52 DDSet_CARetVal: 32000

Resolution: The install person watched the IIS Admin service and 'W3SVC' service. When they went down, he manually brought them back up. We tried this five times manually and never got it right. We finally reverted to issuing the command "iisreset & issreset & issrest …" continually. Eventually we got the timing right and it went through. This is the one that really gets me upset. I've never seen such a thing.

Issue #4:
Issue: Team System Web Access 2008 uninstalls, but you have to delete the IIS Website and App Pool manually through the IIS Manager. You also have to purge d:\Program Files\Team System Web Access 2008 as the uninstall does not delete that either.

Resolution: Delete the IIS data and installed files manually.


Issue #5:
Issue: We ran into the issue with .NET 3.5 SP1 and Fully Qualified Domain Names. Here are some of the error messages we found in the Event Viewer.

Exception Message: TF30063: You are not authorized to access tfs.mycompany.com

TF53002: Unable to obtain registration data for application Build.

Detailed Message: TF53005: Unable to retrieve the Team Foundation Server installed UI culture.

Detailed Message: TF53002: Unable to obtain registration data for application VersionControl.

Detailed Message: TF30040: The database is not correctly configured. Contact your Team Foundation Server administrator.


Resolution: See here for resolution steps. The first one (i.e. recommended) didn't work for us. We had to do the second one, which worked. http://support.microsoft.com/kb/926642. A special thanks to Brian and Buck for posting this on their blogs.

Honestly, if it wasn't for what people post on blogs, we would have had to rolled back. A special thanks to everyone in the community who shares their experiences. Hopefully some of the issues and solutions I've given here help others.

UPDATE (3/37/09): One thing to remember is if you use a FQDN (e.g. tfs.mycompany.com) is you have to run ActivateAT [MachineName] (e.g. activateat MyTFSAppTier) on the primary app tier before running the install. We knew about this as someone else at the company told us so it wasn't an issue that we ran into, but could be one that you do if you fail to do this.

Cross Platform Builds with Team Build

For some reason I seem to constantly find myself working with cross platform solutions (e.g. .NET, J2EE, C). Recently I've been assigned to figure out how we can compile custom Apache HTTPd modules without re-inventing our current build solution (which uses Team Build). I've created a diagram that describes the proposed solution.

The basic idea is we use Team Build for all the peripheral build tasks, outside of compile. Compile has to be done on platform specific machines (e.g. Linux 64). So to compile, we simply use Ant's SSH tasks to 1) transfer code to the platform machines, 2) compile via make and then 3) transfer binaries back to the Team Build server from the platform machines so the build can continue. We use Teamprise's Build Extensions to call Ant. In addition, in previous lives, we've used Teamprise for #1 and we'll probably look at doing the same in the near future. Here is the diagram.

Teamprise’s “Show the New Project ….” Feature

If a user Imports a folder from Source Control via Teamprise’s Eclipse plug-in, the plug-in will look for a server copy or local copy of the .project file. If that file is found, it’s used for the project creation EVEN if you select the “Show the New Project ….” select button. This becomes troublesome when a user walks through the import wizard, completes the import, then realizes they wanted to use the “Show New Project ….” feature in Teamprise. If they delete the Project from Eclipse, but don’t delete the local .project file (created during the first import), subsequent imports will always default to the local .project file, EVEN if the user selects the “Show the New Project ….”.

I can see both sides of almost everything, however, I think the “Show the New Project ….” select option should override the fact that a local or server .project file is found. Or at a minimum, a warning should be tossed to the user. I think this would have saved my good friends downstairs a headache or two!

Excluding Built Types from Alerts

Friday I got a question from one of my colleagues asking how to exclude a particular Build Type from their Alerts. I knew it could be done, but wasn't sure exactly what the proper filter would be. Thanks to Jason Prickett's blog post for some general guidance, I think we figured it out.

The easiest way to do this is to use Power Tools "Alert" interface which shows up under a Team Project in Visual Studio's Team Explorer. Simply add a 'DefinitionPath = '\TeamProject\TeamBuildType' clause. Add Or clauses if you have more than one Build Type in a Team Project you want to grab.

For those of you using Teamprise, you can install the free Visual Studio Team Explorer and use the command line tool called BisSubscribe. You can use BisSubscribe from the command line to run something similar to this:

bissubscribe.exe /eventType BuildCompletionEvent2 /server http://tfs.mycompany.com:8080 /address firstname.lastname@mycompany.com /deliveryType EmailHtml /filter "\"TeamProject\" = 'TeamProject' AND "DefinitionPath" = '\TeamProject\TeamBuildType'"

Hope this helps.

Using Command Line to search for Changeset comments.

Question: Mac, how can we search Changeset comments, recursively, for comments matching a certain string (e.g. "refactoring").

My Answer: I would just use the command line client and pipe the output to a string searcher.

Windows: tf history /server:http://tfs.mycompany.com:8080 "$/Team Project/Development" /noprompt /recursive | findstr refactoring

Linux: tf history /server:http://tfs.mycompany.com:8080 "$/Team Project/Development" /recursive | grep refactoring

*Linux is using Teamprise's Command Line Client software.

TF214007: No build was found with the URI

After making some changes to our build machine, we started getting the following error. I got asked to look into the issue and became immediately confused.

Exception Message: TF214007: No build was found with the URI vstfs:///Build/Build/14365. Either the URI does not exist, or TLR\svcTFSService does not have permission to access it. (type BuildNotFoundForUriException)

I was able to log onto the build machine and hit the PROD URL for build 14365 just fine. So I knew it couldn't be permissions. What was it? I said while scratching my head.

Six months back we migrated our TFS instance from DEV hardware to PROD data center hardware. As part of that process, we updated the DEV's Instance IDs so we didn't run into conflicts when trying to hit either site (we continue to use DEV for testing).

While any new connection to DEV would get the new Instance IDs, the legacy connections on our build servers still had the cached version of DEV's Instance IDs, which just so happen to be our PROD Instance IDs (as expected since we migrated from DEV to PROD).

So what was happing was, when someone was making changes to the build machine, they opened Visual Studio and got connected to DEV by mistake. DEV, using the cached Instance IDs (again which were PROD's), became the active TFS server somehow. So when team Build was firing off, it was actually hitting DEV, which obviously didn’t have a build URI 14365. Thus the error.

To fix this, we deleted the TFS cache on the build server. Then to test it, we purposely logged into DEV and checked our local cache. Sure enough, the new DEV entry in the cache had the new Instance IDs. All was well after that.