Automatic Checkout with Teamprise

June 2007 - I'm doubt this posting will help anyone in trouble, but I thought that I'd post it anyways as I think it's a pretty cool feature. One of the really nice things I like about Teamprise's Eclipse Plug-in is how you can open up a file, start to edit it while Teamprise automatically checks the file out for you.

Unfortunately I don't see this feature in Team Explorer. In Team Explorer I find myself making a few changes, clicking Save and getting the "Save As, Overwrite, Cancel" button. I then "Overwrite" the file, check it out, and check it in. With my math, Teamprise saves you at least two clicks with the automatic check out. Below is a high level (very high level) workflow.

Teamprise Eclipse Plug-in
Open file (click 1)
Save file (click 2)
Check in (click 3)

Team Explorer
Open file (click 1)
Save file (click 2)
Overwrite file (click 3)
Check out file (click 4)
Check in file (click 5)

"Code Coverage Analysis Service" in Stopped Status

June 2007 - We've been looking into an "issue" of our "Code Coverage Analysis Service" service sitting in a "Stopped" state in both our DEV and PROD environments. Our Sys Admins alerted us of the potential "issue" as they were getting paged every time the service stopped. I'm not 100% sure how their monitoring works, but from what I understand they are paged when ever a service set to "Automatic" stops. Once these servers went out of maintenance mode, the Sys Admin pagers wouldn’t stop buzzing!

When starting the service we'd periodically get a message stating "The Code Coverage Analysis service on Local Computer started and then stopped. Some services stop automatically if they have no work to do, for example, the Performance logs and Alerts service." So the Sys Admins set the service to "Manual" until we got this figured out.

While working with Microsoft support on a different issue we asked about the service being stopped. According to our support representative this is normal behavior. Apparently other services do this as well. I'm surprised our Sys Admins never ran into it before given we run over a thousand Windows boxes.

Anyway the services are put back to "Automatic" and the Sys Admins have excluded this particular service from their "Started/Stopped" monitoring.

I'd be interested to see if anyone has similar experiences.

RSKEYMGMT command for failover to standby TFS server

We're a daring group of folks who have added a standby TFS App Tier server to our topology. We'd like to see TFS support true failover, but for now we're comfortable with a warm backup that with some manual intervention we can get up and running if need be.

We followed the How to: Activate a Fail-Over Application-Tier Serverto a tee and as it turned out, that was our downfall. I'm not saying the documentation is wrong, all I'm saying is we had to modify step 4) under Reporting Services to get our standby up and running.

After a number of hours, we broke down and called Microsoft Support who directed us to a very competent TFS support engineer. He correctly diagnosed the issue as being Reporting Services related and joined us with another very competent Reporting Services support engineer. Microsoft should be proud of the work they both did. Great support is one of the primary reasons we buy commercial software/support.

The issue was regarding the command "RSKEYMGMT –a –i <instance ID for AT2> -f c:\backups\My_RSBackup_TFS_AT01 -p aPassword". We got the error "Unable to locate the Report Server Windows service for instance <instance ID>". From what I understand, since the TFS install requires Reporting Services to be installed as a "default" instance (see step 10 in "How to: Install Microsoft SQL Server 2005 Reporting Services for Team Foundation Server (Dual-Server Deployment)" which is located in the TFS install documentation.) you can't activate it by a named instance name. Thus the "–i " switch in the command was throwing the error.

Removing the –i and running the command "RSKEYMGMT –a -f c:\backups\My_RSBackup_TFS_AT01 -p aPassword" on the standby App Tier worked just fine.

Although the process is not perfect, we're now able to failover. Hope this helps if you run into the same issue.

TFS Data Warehouse Not Being Updated by TFS TFSServerScheduler

May 2007 - We had an ongoing issue where the TFS Data Warehouse was not updated by TFS TFSServerScheduler but could be invoked manually. What we think happened is when we installed TFS to a different port (i.e. 8888) and went through the process of putting our App Tier behind a Fully Qualified Domain Name, not all of the Registry Entries were updated as needed.

With the help of this posting and some sleepless nights we sifted through the App Tier's registry and found a number of references to AppTier-B01:8080 instead of AppTier-B01:8888 or better tfs.int.mycompany.com:8888 which is a DNS entry for AppTier-B01:8888. We also found a number of duplicate references where both AppTier-B01:8080 and tfs.mycompany.com:8888 were listed. We logged on as the TFSInstall, TFSService, and TFSReports account and did the clean-up as some of the settings are under HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\8.0\TeamFoundation\Servers.

The lesson learned is to make sure all your registry settings are correct for ALL USERS. Because our AppTier name did not change (we just put it behind DNS), we think the most important piece was the port number which we changed from 8080 to 8888. Below are our current settings on the App Tier.

Sys Admin Account:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\Servers
tfs.mycompany.com = http://tfs.mycompany.com:8888

HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\Servers
tfs.mycompany.com = http://tfs.mycompany.com:8888

TFSService Account:

HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\Servers
tfs.mycompany.com = http://tfs.mycompany.com:8888

Can you have TFS states and groups with the same name?

May 2007 - We've been dealing with the very informative user error message "TF26212: Team Foundation Server could not save your changes. There may be problems with the work item type definition. Try again or contact your Team Foundation Server Administrator." for about two weeks now and we finally think we have an answer.

The error seemed to show up when we had a state (e.g. Testing) named the same as a group (e.g. Testing). Here are our steps and final thoughts on how we resolved the issue.

Start by editing the Web.config file under \Program Files\Microsoft Visual Studio 2005 Team Foundation Server\Web Services by changing

name="traceLevel" value="1" to name="traceLevel" value="4"

and

key="traceWriter" value="false" to key="traceWriter" value="true"

This will start logging myriad amounts of information under C:\WINDOWS\Temp\TFLogFiles (or some other directory if you change the defaults).

After you make these changes (you don't need to bounce IIS) try to get the error again. If you see an error like below in the TFLogFiles you may be in the same boat as us.

[WI] [Error, 2916, 5, 11:31:33.016] SvrEx: Microsoft.TeamFoundation.WorkItemTracking.Server.ValidationException: Forcing rollback ---> System.Data.SqlClient.SqlException: Forcing rollback
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.HasMoreRows()
at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
at System.Data.SqlClient.SqlDataReader.NextResult()
at Microsoft.TeamFoundation.WorkItemTracking.Server.PayloadTableCollection.Populate(SqlDataReader reader)
at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(IRequestContext context, String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate, String connectionString)
--- End of inner exception stack trace ---.

While we've had differing results with a number of use cases, the root cause seems to be related to having a state and group named the same. We saw the issue with a state called CCB and group called CCB and the same issue with a state called Testing and a group called Testing.

What's even more alarming is they don't even have to be in the same project. We're pretty sure if you have a Testing group in Project1 and a Testing state in Project2, you will get a conflict.

Again we're not 100% sure this should be written in stone, but our error has disappeared now that we renamed the CCB state to Change Control Board and Testing group to Testers. And removed all references to the CCB state and Testing group from all the projects in our instance. This seemed to work for us.

Good luck!

Report Server Windows Service (MSSQLSERVER) cannot connect to the report server database

May 2007 - We just got our standby server setup and failed over for the first time. I'll send out the steps we took in a forthcoming post as we're still working out some minor details.

We were a bit concerned as after setting up the standby server we were getting the following error messages (below) in Event Viewer. The first error message just happened once (or so we can tell). The second one was recorded every minute in Event Viewer on the standby.

When we failed over the to standby both errors went away and started to appear on the primary. I posted a message to the MSDN forums and Mr. Chen confirmed this was as expected. I'm not sure I followed him on this, but he said the reason was "This is usually used to improve production quality in the future." Again I'm not sure I follow him on why this will help to improve quality, but I'll take his word for it.

Happy failover!

Error Messages:

Report Server Windows Service (MSSQLSERVER) has not been granted access to the catalog content.

Report Server Windows Service (MSSQLSERVER) cannot connect to the report server database.

HTTP Status 400 With Reporting Services

May 2007 - There are probably a myriad of reasons to get the error "The request failed with HTTP status 400: Bad Request" in Reporting Services running with TFS, but here is why we got the error.

While trying to get our TFS instance behind a fully qualities DNS name we added tfs.int.mycompany.com to host header entry on the Default Web Site on our TFS App Tier. After adding this entry users could still see Reports in Visual Studio, but when Right Clicking and selecting Show Report Site Internet Explorer would pop up and give us the error ""The request failed with HTTP status 400: Bad Request". As a side note, adding this entry in the host header did nothing for us while trying to get our App Tier behind DNS.

Once we removed that value, the error went away. Hope this helps anyone else.

Our own Dog Food Statistics

May 2007 - What a month this has been. We had some issues getting TFS up and running (see my previous post for some of the bigger ones), but finally we have a PROD system that is capable of taking on users. For those of you who are thinking about implementing TFS, make sure you have dedicated resources to work on it. We're fortunate that our company is funded a full time administrator to get this up and running for 200+ users (1 admin --> 200 users). If you're a Sys Admin who, in addition to your day-to-day responsibilities, is supposed to get TFS up and running, good luck!

I'm a bit of a statistics nut so while we currently wait for Brian Harry to release the TFSServerManagerTool (http://blogs.msdn.com/bharry/archive/2007/01/22/tfsservermanager-powertool.aspx) I thought I'd try to pull some simple statistics that we might find valuable and post them. Being our first project was created today, it seems like an apropos time to make this posting. I'm a bit pressed for time so I didn't have the opportunity to put together the myriad of statistics that Brian has. I'll wait for him to release the TFSServerManager PowerTool to get those crazy numbers. That being said, we find this information helpful so here it goes.

Team Projects = 1; This was a pretty easy one to find!

Users = 6; This one was a bit tougher. I started by running the TfsSecurity tool (http://msdn2.microsoft.com/en-us/library/ms252504(VS.80).aspx) with the "/imx all:" parameter. Unfortunately this gave us back all the users plus the 50 Sys Admins that also have access to the machine, but would never use TFS with their administration accounts (e.g. MyDomain\M0000001). Consequently we don't want to count them. The good thing is our user accounts all start with \U (U for user they tell me) so I was able to find actual TFS users by running "TfsSecurity /server:tfs.int.mycompany.com /imx n:"[SERVER]\Team Foundation Valid Users" find "MyDomain\U"". This gave us back the number of MyDomain\U users (e.g. MyDomain\U000001) which are the only users we add to TFS projects.

Work items = 16; This was also easy as each new work item creates a new incremented id. There are 15 standard CMMI process template Tasks that get created. After that we created one to change a Change Request state which was number 16.

Files/Folders = 1859/31; I got these values using the following commands. For files I ran "tf dir /server:tfs.int.mycompany.com $/ /recursive" less the value I got for the number of folders, which is next. For folders I ran "tf dir /server:tfs.int.mycompany.com $/ /folders /recursive".

Changesets = 7; Another easy one for now. I got this value by running "tf changeset /server:tfs.int.mycompany.com /latest /noprompt"

That's it for now. If you find additional or better ways to find your own Dog Food statistics before Brian releases TFSServerManagerTool, let us know.

Trials and tribulations of a TFS install

Ah it's spring and we're still plugging away on getting TFS installed on our DEV environment. In TFS's defense most of the big delays are big company bureaucracy. That being said, installing TFS is not as easy as installing MS Word on my grandmother's PC. Here are some of the issues we ran into and the resolutions. I've also included some big company standard deviations we did. By big company standard deviations I mean things like our system administrators make us install "third-party" software on the D: drive instead of the C: drive. I can't imagine the number of hours upper management spent on that decision.

First a bit of background. From what I can tell our company supports Windows based servers with two groups. For this discussion let's call the database group the "DBAs" and the system administrators "Sys Admins." Upper management actually refers to them as something totally different, but the average sane person can't understand upper management's terminology (or thought process for that matter). The DBAs and Sys Admins work like this: In the morning the DBAs get coffee on the second floor, Sys Admins the fourth. DBAs go out to lunch at 11AM. Sys Admins eat at their desk around noon. DBAs drink Diet Coke, Sys Admins drink regular Pepsi. To an outsider one could conclude the two groups try to avoid each other at all costs. Thus, it's with these two groups we must coordinate the temperamental TFS install.

• The DBAs like to use a standard company SQL install image for all data tier machines. Unfortunately the big company image does not include Integration Services and Analysis Services. Apparently we're the first team that needs such "advanced technology." Lesson number one, don't install a base SQL 2005 64-bit image on a cluster and then expect to install Integration Services and Analysis services afterwards. After three days working with Microsoft, our DBAs gave up and re-installed SQL 2005 64-bit with Integration Services and Analysis Services selected on the initial install. I'm not sure why adding these two components after the cluster is setup is such a big deal, but the amount of vulgar language the DBAs used it must be a big deal.
• Our DBAs have "Big Company Approved" Reporting Services install instructions that includes "Configuration." We had the DBAs do the basic install with their instructions, then tackled them before they could run through the "Configuration" section. From what we're told, the TFS install does the configuration for us.
  
• We installed the TFS App Tier software to the d: drive. That is a requirement from the Sys Admins. No big deal here.
  
• Here is a big one. The entire datacenter has some kind of "flash copy agent" software running on port 8080. Conventional wisdom would lead you to believe one can simply change the TFS msiproperty.ini to some other port (e.g. 8888) and install TFS. There I go again thinking conventional wisdom is logical. Changing the misproperty.ini file resulted in the following error: "Error 28925.TFServerStatusValidator: Calling the Team Foundation Server ServerStatus Web service failed. Additional details about the problem can be found in the setup log. Verify your network configuration. For more information on troubleshooting this error, see the Microsoft Help and Support Center. Log file: TFServerStatusValidator - Team Foundation Server Status Validator tool (C) Copyright 2006 Microsoft Corporation. All rights reserved." The details are outlined in my forum posting (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1411189&SiteID=1&mode=1). After giving up and calling Microsoft, the support folks from India had the Sys Admin shut down the agent software running on 8080, install TFS to port 8080, step through a number of manual steps which included making a database change to switch URL values from 8080 to 8888 (steps are in my forum posting above), restart all TFS related applications/services, and finally start the flash copy agent software back up on 8080. The look on our Sys Admin's face after all this was priceless!
  
• Since our DBAs don't let the Sys Admins on the database machines and vice versa, we had some trouble installing the App Tier. From what we found out, the TFSSetup account used to install the App Tier needs to be a local administrator on the App Tier and a member of the local administrator group on the Data Tier. After bringing in a United Nations negotiator, we got the DBAs to "temporarily" add the Sys Admin's TFSSetup account to the local administrator group on the Data Tier. The install went fine after that.
  
• The TFS administrator (i.e. me) won't be a local administrator on either the App Tier or Data Tier. That would be against company policy. So we followed the three sections outlined by Microsoft (http://msdn2.microsoft.com/en-us/library/ms253096(VS.80).aspx) and added our team's group name to the appropriate locations in TFS, SharePoint, and SQL Reporting Services. Amazingly this seems to work just fine.
  
• Lastly we had to add the TFSReports user to the local administrator group on the App Tier to get around the error "The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID {BA126AD1-2166-11D1-B1D0-00805FC1270E} to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20). This security permission can be modified using the Component Services administrative tool." The end user error was "Reporting Services Error --- An error has occurred during report processing. (rsProcessingAborted) Get Online Help Cannot impersonate user for data source 'TfsOlapReportDS'. (rsErrorImpersonatingUser) Get Online Help Logon failed. (rsLogonFailed) Get Online Help For more information about this error navigate to the report server on the local server machine, or enable remote errors --- SQL Server Reporting Services." Being the install instructions say "This account should not be an administrator on Team Foundation Server computers" I'm pretty sure this is not correct. However at the time of this posting, no one from Microsoft can give us any guidance on better way to get Reporting Services to work. I'll keep this forum (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1345776&SiteID=1) updated if we ever figure out a better way to handle this.
  

And there it is. Our take on the joys of installing TFS on an App Tier with a clustered Data Tier. Happy installing!

What we dislike about TFS

Overall we really like TFS, but here are a few things that are less than desirable.

• "Friendly Names" (actually unfriendly for us) for fields such as "System.AssignedTo" or "System.ChangedBy" is a major oversight on Microsoft's end. Let's say you have an employee John Doe who's Active Directory name is "Doe, John (Company 1)". John then moves to company 2 and the Active Directory name changes to "Doe, John (Company 2)" This causes major issues in TFS as because it's integrated with Active Directory the work items are invalid and can't be modified. Fields like "System.AssignedTo" can simply be updated by the user. Hidden fields like "System.ChangedBy" can't be changed by the end user. For this issue call Tech Support and get the TFSLocalize utility. For one user it's not a bad solution, but let's say you have 200 users (like we do) and Company 2 decided to be called Business Unit 2. Now we have 200 users who need to be changed. TFSLocalize stinks as you have to map out old user (i.e. Doe, John Company 2) to new user (i.e. Doe, John Business Unit 2). While we're not up and running yet, this potentially is a major, major issue!!!!

• Administrating TFS, Share Point, and SQL Reporting Services in three different spots is laborious. In fact without the TFS Administration Tool, we'd be in a ton of trouble. Trying to keep track of permissions and users in three separate spots for 200+ users is unobtainable. So if you're new to administrating TFS, make sure to visit Codeplex (http://www.codeplex.com) and download the TFS Administration Tool (http://www.codeplex.com/TFSAdmin). Without this tool, administrating TFS is a nightmare.
  

• The out of the box Alerts stink. First of all there is no easy way (we know of) to "subscribe" all the users. Instead each user has to "subscribe" themselves. And worse yet, they need to know their SMTP address. Imagine the pain of telling your Vice President they have to type in first.last@company.com. They barely know how to tie their shows for goodness sakes. To get around this we're simply avoiding the concept of subscribing yourself and using Mariano Szklanny's web service to do all notifications (http://staff.southworks.net/blogs/mariano/archive/2006/04/14/411.aspx). I'd suggest a similar approach if you're a large team like us.
  

• No web interface for work item tracking. Developers love the work item and source control integration located in their IDE. On the other hand, managers hate having to open up Visual Studio 2005 just to approve a Change Request. While this seems like something trivial, a nice bundled web interface sure would be nice. Hey Mr. Gates, buy Teamplain (http://www.devbiz.com/teamplain) and make it part of your solution!
  

• Code policies on the client side instead of server. Out of our 200+ users, 100-125 are Java developers using Teamprise (http://www.teamprise.com). We like Teamprise, but wish we could enforce a policy like "Require Associated Work Items." Nope can't. All the policy code is local to the Team Explorer install. Dumb! If you've never used Teamprise give it a shot. It's a very nice tool with WONDERFUL technical support.
  

• No hierarchy for the work items. Rhetorical question; if Tasks are children of Change Requests, why can the Change Request close before the Tasks are closed? A nice hierarchy is needed. And Microsoft, don't give us "TFS is totally extendable to do all of these things you request." We're not in the business of developing configuration management tools. You are, so add hierarchy in for your customers.
  

• Can't remove or rename work items. For some management reason that supersedes any input I have, we're not using the Requirement work item for a Team Project that is already created. Instead of leaving it there to confuse users, we would sure like to remove it. Nope can't remove it. Not possible. Or how about when you create a new work item called WrognName. Opps I made a typo, this should be called WrongName. Nope, can't rename it. Not possible.
  

That's enough ranting for today. Again we like the product, but are excited to see the next release and what new offerings it has to address TFS's current limitations.