TFS 2008 SP1 upgrade nightmare

I'm typically very pro-TFS on this blog, but I have to share some negative news. Mainly because it cost me, and my colleagues, an entire evening away from our families while we upgraded. Here are the issues and resolutions from our nightmare. It took us six hours in total. We're done now and I'm off to my home where I can lie awake all night and stew.

Issue #1:
Issue: .NET Framework 3.5 SP1 install failed when installing Visual Studio 2008 SP1

Resolution: Installed framework separately.


Issue #2:
Connecting to SQL Server Reporting Services. Please wait...
Invalid namespace
Value cannot be null.
Parameter name: uriString
Configuring SQL Server Reporting Services failed.

Resolution: Removed the Standby Key from the database.
C:\Program Files\Microsoft SQL Server\90\Tools\binn\rskeymgmt -l
C:\Program Files\Microsoft SQL Server\90\Tools\binn\rskeymgmt -r asdfasfasdf-8968686-asfa-asdfdsf-9as7dfasfasd



Issue #3:
Issue: Restart attempt failed.

IIS Admin Service or a service dependent on IIS Admin is not active. It most likely failed to start, which may mean that it's disabled.

03/26/09 19:55:52 DDSet_Status: Process returned 1062
03/26/09 19:55:52 DDSet_Error: The Commandline '"C:\WINDOWS\system32\iisreset.exe" /Restart' returned non-zero value: 1062.
MSI (s) (AC!14) [19:55:52:046]: Product: Microsoft Visual Studio 2008 Team Foundation Server - ENU -- Error 32000.The Commandline '"C:\WINDOWS\system32\iisreset.exe" /Restart' returned non-zero value: 1062.

Error 32000.The Commandline '"C:\WINDOWS\system32\iisreset.exe" /Restart' returned non-zero value: 1062.
03/26/09 19:55:52 DDSet_CARetVal: 32000

Resolution: The install person watched the IIS Admin service and 'W3SVC' service. When they went down, he manually brought them back up. We tried this five times manually and never got it right. We finally reverted to issuing the command "iisreset & issreset & issrest …" continually. Eventually we got the timing right and it went through. This is the one that really gets me upset. I've never seen such a thing.

Issue #4:
Issue: Team System Web Access 2008 uninstalls, but you have to delete the IIS Website and App Pool manually through the IIS Manager. You also have to purge d:\Program Files\Team System Web Access 2008 as the uninstall does not delete that either.

Resolution: Delete the IIS data and installed files manually.


Issue #5:
Issue: We ran into the issue with .NET 3.5 SP1 and Fully Qualified Domain Names. Here are some of the error messages we found in the Event Viewer.

Exception Message: TF30063: You are not authorized to access tfs.mycompany.com

TF53002: Unable to obtain registration data for application Build.

Detailed Message: TF53005: Unable to retrieve the Team Foundation Server installed UI culture.

Detailed Message: TF53002: Unable to obtain registration data for application VersionControl.

Detailed Message: TF30040: The database is not correctly configured. Contact your Team Foundation Server administrator.


Resolution: See here for resolution steps. The first one (i.e. recommended) didn't work for us. We had to do the second one, which worked. http://support.microsoft.com/kb/926642. A special thanks to Brian and Buck for posting this on their blogs.

Honestly, if it wasn't for what people post on blogs, we would have had to rolled back. A special thanks to everyone in the community who shares their experiences. Hopefully some of the issues and solutions I've given here help others.

UPDATE (3/37/09): One thing to remember is if you use a FQDN (e.g. tfs.mycompany.com) is you have to run ActivateAT [MachineName] (e.g. activateat MyTFSAppTier) on the primary app tier before running the install. We knew about this as someone else at the company told us so it wasn't an issue that we ran into, but could be one that you do if you fail to do this.

Cross Platform Builds with Team Build

For some reason I seem to constantly find myself working with cross platform solutions (e.g. .NET, J2EE, C). Recently I've been assigned to figure out how we can compile custom Apache HTTPd modules without re-inventing our current build solution (which uses Team Build). I've created a diagram that describes the proposed solution.

The basic idea is we use Team Build for all the peripheral build tasks, outside of compile. Compile has to be done on platform specific machines (e.g. Linux 64). So to compile, we simply use Ant's SSH tasks to 1) transfer code to the platform machines, 2) compile via make and then 3) transfer binaries back to the Team Build server from the platform machines so the build can continue. We use Teamprise's Build Extensions to call Ant. In addition, in previous lives, we've used Teamprise for #1 and we'll probably look at doing the same in the near future. Here is the diagram.

Teamprise’s “Show the New Project ….” Feature

If a user Imports a folder from Source Control via Teamprise’s Eclipse plug-in, the plug-in will look for a server copy or local copy of the .project file. If that file is found, it’s used for the project creation EVEN if you select the “Show the New Project ….” select button. This becomes troublesome when a user walks through the import wizard, completes the import, then realizes they wanted to use the “Show New Project ….” feature in Teamprise. If they delete the Project from Eclipse, but don’t delete the local .project file (created during the first import), subsequent imports will always default to the local .project file, EVEN if the user selects the “Show the New Project ….”.

I can see both sides of almost everything, however, I think the “Show the New Project ….” select option should override the fact that a local or server .project file is found. Or at a minimum, a warning should be tossed to the user. I think this would have saved my good friends downstairs a headache or two!

Excluding Built Types from Alerts

Friday I got a question from one of my colleagues asking how to exclude a particular Build Type from their Alerts. I knew it could be done, but wasn't sure exactly what the proper filter would be. Thanks to Jason Prickett's blog post for some general guidance, I think we figured it out.

The easiest way to do this is to use Power Tools "Alert" interface which shows up under a Team Project in Visual Studio's Team Explorer. Simply add a 'DefinitionPath = '\TeamProject\TeamBuildType' clause. Add Or clauses if you have more than one Build Type in a Team Project you want to grab.

For those of you using Teamprise, you can install the free Visual Studio Team Explorer and use the command line tool called BisSubscribe. You can use BisSubscribe from the command line to run something similar to this:

bissubscribe.exe /eventType BuildCompletionEvent2 /server http://tfs.mycompany.com:8080 /address firstname.lastname@mycompany.com /deliveryType EmailHtml /filter "\"TeamProject\" = 'TeamProject' AND "DefinitionPath" = '\TeamProject\TeamBuildType'"

Hope this helps.

Using Command Line to search for Changeset comments.

Question: Mac, how can we search Changeset comments, recursively, for comments matching a certain string (e.g. "refactoring").

My Answer: I would just use the command line client and pipe the output to a string searcher.

Windows: tf history /server:http://tfs.mycompany.com:8080 "$/Team Project/Development" /noprompt /recursive | findstr refactoring

Linux: tf history /server:http://tfs.mycompany.com:8080 "$/Team Project/Development" /recursive | grep refactoring

*Linux is using Teamprise's Command Line Client software.

TF214007: No build was found with the URI

After making some changes to our build machine, we started getting the following error. I got asked to look into the issue and became immediately confused.

Exception Message: TF214007: No build was found with the URI vstfs:///Build/Build/14365. Either the URI does not exist, or TLR\svcTFSService does not have permission to access it. (type BuildNotFoundForUriException)

I was able to log onto the build machine and hit the PROD URL for build 14365 just fine. So I knew it couldn't be permissions. What was it? I said while scratching my head.

Six months back we migrated our TFS instance from DEV hardware to PROD data center hardware. As part of that process, we updated the DEV's Instance IDs so we didn't run into conflicts when trying to hit either site (we continue to use DEV for testing).

While any new connection to DEV would get the new Instance IDs, the legacy connections on our build servers still had the cached version of DEV's Instance IDs, which just so happen to be our PROD Instance IDs (as expected since we migrated from DEV to PROD).

So what was happing was, when someone was making changes to the build machine, they opened Visual Studio and got connected to DEV by mistake. DEV, using the cached Instance IDs (again which were PROD's), became the active TFS server somehow. So when team Build was firing off, it was actually hitting DEV, which obviously didn’t have a build URI 14365. Thus the error.

To fix this, we deleted the TFS cache on the build server. Then to test it, we purposely logged into DEV and checked our local cache. Sure enough, the new DEV entry in the cache had the new Instance IDs. All was well after that.

Check in comment to dynamically enable or disable CI builds.

I just got done adding in some custom MSBuild changes to automatically check in a Jar file to another build type (i.e. MainBuild in this example) after the Jar file build type (i.e. CommonBuild in this example) is finished.

Buck showed me a really cool way to "enable" or "disable" continuous integration (CI), which is turned on for us, in the MainBuild. That is, when CommonBuild checks in the Jar file to MainBuild, we can control whether we want to fire off a MainBuild.

First we set a Property called ExecuteMainBuildCIBuild to true|false. If true, we set the check in comment equal to /comment:"AUTO CHECKIN FROM TFS FOR COMMON JAR BUILT IN $(BuildNumber)". If false, we set the check in comment to /comment:"$(NoCICheckinComment)". $(NoCICheckinComment) resolves to the string ***NO_CI*** which tells Team Build to not fire a CI build.

End users can also use ***NO_CI*** if they want to control the CI build behavior. For example, if you're wishing to check in a change and not execute a CI build, you can simply put ***NO_CI*** in the Comments section. I'm not sure if management will enjoy us sharing this back door, however I've found it to be a valuable tool for solving some unique problems.

Update on our TFS adoption

Hello again! Yes, I'm still alive and kicking. Sorry for not writing any updates lately. I've been working on some other things outside of TFS. In addition, I've got a number of colleagues who are taking over the primary role as TFS administrators. I still get involved at times, but it's good to see others taking an interest in supporting the product.

I do have to share with you some tough news about TFS. TFS is kind of taking a public relations beating in my new group. In fact, so much so, there has been a "TFS Improvement" team created to look at changes we can make to both, our use of the tool, and possible suggestions for the vendor on how to make TFS better.

I'm a bit confused on where the movement is coming from, as I'm not part of the "improvement team," however, in reviewing the meeting notes, it looks like Work Items used to support an Agile development effort are a primary target. Here is a sampling of some of the complaints.

- Need a hierarchical representation of work items that shows partial completion
- Enable users to find and track dependencies more easily
- Need to map work items to current priorities, in/out list
- Need to make it easier to set up alerts on work items for yourself and others
- People should be able to create a Team Query
- TFS should auto-complete stories when all tasks are marked as completed. Or, depending on workflow, it should auto-magically assign the story to the business partner for final validation
- TFS should do the same to backlog stories when the associated tech stories in the work stream are completed

Looking through this list, I think there are many things we could do to help solve these, somewhat vague, issues people are complaining about. For example, Alerting is now enhanced with the new Power Tools. And I see TSWA also has a way to setup Alerts. This may have come from the Teamprise people who still are having some trouble with Alerting. We've showed them the TSWA and I think that is helping. The Team Query thing is simply a permission deal that we can work out internally.

However there is one primary feature that I think would solve a number of problems we're having. And that is a hierarchy that can perform "Actions" when parents or children are updated. For example, you can see the two bottom "ideas" (from a former Version One user who recently joined our company) are regarding a parent Work Item State action to take place when Child Task(s) are completed/updated. This, very vocal, person would also like to see Field roll-ups. That is, if the Child Tasks had "Hours Worked On Task," those values should be rolled up into the Parent Work Item so you can see the rolled up values of "Hours Worked On" for the Child Tasks of the Parent (i.e. Product Backlog Item in our case).

We'll see where this goes. There is a lot of movement towards looking at Version One right now. And I see they have a TFS interface so I'm sure that will be talked about at the management ranks. Personally, I like what we have and wouldn't suggest a change. However, I often think decisions get made because a few squeaky wheels start to complain. I guess we'll see what comes of this all. And hopefully Rosario will provide a number of nice new features for Work Item tracking. Personally, I think it will.

Team Companion

I'm actually on vacation today, but wanted to share this. So the business and management folks don't like to use an IDE when using TFS. Turns out even going to TSWA is a struggle for some.

My colleague was doing some digging and came across Team Companion, which is a TFS plug-in to Outlook. We've been piloting it for a few weeks. The users most likely to never use TFS, have been enthralled with having Work Items within their Outlook. From what I hear, they can't say enough great things about it. We're actually getting people interested in normalizing the paper work behind software development.

If you're having a hard time getting management and business folks working in TFS, give Team Companion a try. With TFS being embedded in the only software package they use (i.e. Outlook), TFS becomes even more critical in our business process.

"Mac, how do I compile Java with Team Build?"

In the past week, I've fielded a number of questions from people within my company regarding how to get started compiling Ant based Java projects within TFS. Here is what I tell them. And it would be what I'd tell you if you asked.

My Colleagues: "Mac, can you provide us some information on how to get started using Team Build to compile our Java projects?"

Mac: "There are a number of different ways to solve the problem, but I'd suggest looking at Teamprise's Team Build Extensions. This is an open source project, supported by Teamprise, to run Ant scripts via Team Build (which is nothing more than MSBuild behind the covers). They have a very good tutorial (as part of the distribution if I remember) and support forum.

I'd start by trying to get one of your simpler Java projects to build. Then once you build that, you can get a better understanding of how the tooling can best solve your problem. What we do is have one parent Ant script that calls all the other tier Ant scripts. So basically Team Build is just the high-level driver and we let Team Build Extensions and Ant take care of the rest. This seems to work pretty well for both [MY CURRENT BUSINESS UNIT] and [MY OLD BUSINESS UNIT]. Also, I think if you have JUnit tests that you run as part of the build, and follow some simple rules, Teamprise Build Extensions will publish the results back to TFS. Pretty cool!
"

So if you're looking for help getting started, consider this my response to you. Let me know how it goes.