Custom Response Header For Our TFS 2010 App Tier Farm

I’m curious to see if people think this is a good idea or not. In our TFS 2010 farm we have four application servers. In the process of testing load balancing and fail over, I was having trouble trying to figure out which box my requests were going to. Our load balancer sets a cookie so we persist to the same App Tier machine, but it’s an obfuscated value that no one seems to be able to provide any insight into how to un-obfuscate it. (And I’m not sure if devenv.exe is passing that cookie as we seem to be bouncing across all four App Tiers when doing things like “Get.” Not a big deal, but it would be nice if there was a way to persist.)

To help me out, I set a custom response header called “x-tfs-machine” that resolves the name of the App Tier machine. Via Fiddler or Wireshark, I know can inspect what App Tier I’m hitting. It seems to be working quite well for me.

While I’m just using this for testing, I’m thinking about leaving it on when we go live so we can see what server a user is hitting if they are running into issues.

Team Explorer Everywhere Options for Users Wanting a Standalone Client

We’ve had some users who have shied away from using Eclipse plug-ins and have become comfortable with Teamprise Explorer (aka Fat Client). When Team Explorer Everywhere (TEE) was launched, Microsoft discontinued delivering the Fat Client. Thus, those users have continued to use Teamprise. We’d like to get those users cut over so Brian has showed a pretty good work around that should work for these users. Here is what we sent to our teams:

If you’re used to using the Teamprise Explorer for development, here are some options you have when migrated to Team explorer Everywhere (TEE).

1) Switch to using the TEE plug-in in Eclipse.
2) Use the free Team Explorer plug-in to the free Visual Studio 2010 shell. (This only works for Windows users)
3) Per Brian Harry’s post, use the Eclipse Platform Runtime Binary standalone client.

If you decide to go with the last option, here are some general instructions for using the Eclipse Platform Runtime Binaries.

1) Download and install Eclipse Platform Runtime Binary- Install TEE
2) Window Show View Other Team Foundation Server Team Explorer
3) In the Team Explorer panal, select the Add Existing Team Project dialog button
4) Add your server and connection information
5) Select Team Projects you’re involved with
6) Double click on Source Control to get the friendly Source Control tree view
7) Get yourself a cup of hot coffee

Windows Process Activation Service – Error 183

This is not related to TFS, but I need a place to write this down. On one of our runtime machines, we had an issue where II 7 stopped running. It turns out that the Windows Process Activation Service (WAS) was down. When trying to restart WAS, we got an Error 183 – Unable to create file (or something like that). When trying to open Event Viewer or any other snap-ins to MMC, we got a “could not load snap-in error.”

Through hair pulling and swearing, we finally tracked down a machine.config change that one of my colleagues made. In the process of doing some tuning, the I haven’t reviewed the schema, but I’m guessing this is not valid as it does not make sense.

After removing the autoConfig, we were back up and running. Thanks to Carlo for pointing us in the right direction.

<processModel maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" memoryLimit="90"/>

<httpRuntime minFreeThreads="12"/>

<processModel autoConfig="true"/>

Windows "Loopback bug" when using a FQDN

We just got our new TFS 2010 hardware! Unfortunately, my smile was quickly brought to its knees by the "loopback bug" where you can access a IIS site (in our case it's a Share Point site running under IIS 7.5) via fully qualified domain name (FQDN) from a client machine, but can't use the FQDN on the server looping back to the server. When trying to use the FQDN locally on the server, we continually get the username/password prompt.

Here is the work around if you loose a day on this like I did. Option #1 worked for us. Thanks for the post at Server Fault for reminding me.

Spring: Ambiguous constructor argument types

This is not TFS related, but build related and this is my only public blog where I document build-like stuff for future reference if I was to ever need it.

In Spring for Java, if you’re going to use “names” for wiring up things like constructor arguments, the class files have to be built with javac –g, which is an entry-level debug mode for javac. This will compile the classes, keeping such things as variable names. If you don’t use javac –g, then Java takes the liberty to modify variable names. I’m guessing they do this for, among other things, keep the class sizes down.

Below is a poorly formatted, convoluted series of knowns and tests to come up with a hypothesis and eventual solution to Spring issue we were running into.

Knowns

1) The Spring configuration is using variable name for mapping constructer arguments in the spring configuration to the set of constructor parameters.

bean id="SetManager" class="com.mycompany.manager.SetManager" lazy-init="false"

constructor-arg name="courtSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="pubSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="jurSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

bean

2) When deploying the application out, the error we’re getting is this.

SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'SetManager' defined in class path resource [spring-manager.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.springframework.core.io.Resource]: Ambiguous constructor argument types - did you specify the correct bean references as constructor arguments?

at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:684)

at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:192)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:984)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:886)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:479)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450)

3) When creating a WAR via Eclipse’s Export feature, the bindings are wired up fine. When creating the WAR via Ant (or using javac from the command line, which is what Ant is doing), we get the error. When diff-ing the WAR files, they are the same with one exception: The class files created via Eclipse are larger.

Hypothesis Given all of this, our hypothesis is that running the build via Ant, the method names are getting truncated (or modified) such that using the name attribute in Spring won’t work.

Tests

1) To test the use of indexes, instead of names we used something like this: “index="0"”. Using this got us around the initial issue, but ran us into a very similar problem loading another bean. At this point, we knew we were on to something. Or on something. Here are the Spring configuration snippets of using names verses indexes.

Names

bean id="SetManager" class="com.mycompany.manager.SetManager" lazy-init="false"

constructor-arg name="courtSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="pubSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="jurSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

bean

Indexes

bean id="SetManager" class="com.mycompany.manager.SetManager" lazy-init="false"

constructor-arg index="0" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg index="1" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg index="2" type="org.springframework.core.io.Resource" value="classpath:..."

bean

We used JAD to decompile both classes. Bingo, the issue is highlighted! When using javac, the default settings truncate the variable names. If Spring is to use variable names for wiring up arguments or properties, this won’t work very well ;)

Ant

public SetManager(Resource resource, Resource resource1, Resource resource2)

{

loadCourtSetMaps(resource);

loadPub(resource1);

loadJur(resource2);

}

Eclipse

public CustomDigestCollectionSetManager(Resource courtSetCSV, Resource pubSetCSV, Resource jurnSetCSV)

{

loadCourtlineCollectionSetMaps(courtSetCSV);

loadPublicationCollectionSetMap(pubSetCSV);

loadJurisdictionCollectionSetMap(jurSetCSV);

}

Solution

1) What parameter change in Ant’s javac Task could we change to get the accurate parameter names?

a. Set debug = true. Turning debug on will make sure the class files are compiled with the same variable names (and other things) as what is in the source code. This will increase the class file size, but from what we’ve noticed so far, does not cause great performance degradation.

Why a Web.config change does not result in a new process id

A colleague asked me why he sees Application_Start events after modifying his Web.config but does not get a new process id for the worker process. While this article is addressed to IIS 5 and 6, I think the same principle must apply for IIS 7.

From what I can tell, a change to the Web.config will result in tear down of the Application Domain, but not the Application Manager. The Application Manager is what controls the worker process.

If anyone disagrees with this, let me know.

“The Network Path Was Not Found” When Installing our .NET Node Agent

We ran into a problem installing some custom software we call the .NET Node Agent on machines that were in a new COMPANYQA Windows domain. After some troubleshooting, we found the issue to be related to how we figure out what domain the machine is in. We do this logic so we can insert the correct .NET Node Agent service account (COMPANYMGMT|COMPANY\svcNodeAgnet) into the machine’s local Administrators group. If a machine is in COMPANYMGMT, we need to add COMPANYMGMT\svcNodeAgnet. If the machine is in COMPANY, or now COMPANYQA, then we need to insert COMPANY\svcNodeAgnet. (We could have asked a human to make this distinction during install time, but asking a human to do anything around here causes more trouble than it’s worth.)

We use .NET’s System.DirectoryServices namespace to surf Active Directory (AD). We use the machine’s parent to grab Properties. When trying to get Properties from the parent (DirectoryEntry.Parent.Properties) we continually got the message “The Network Path Was Not Found.” After some in-depth troubleshooting, we found that we had to add companyqa.mycompany.com to the DNS suffix list. The only values were company.mycompany.com and so we couldn’t find the LDAP.companyqa.mycompany.com AD server.

In Windows 2008 Server, this setting is under Control Panel - Network Connections - Local Area Connections – Internet Protocol Version 4 (TCP/IPv4) - Properties – DNS – Append These DNS Suffixes. After we added companyqa.mycompany.com we were on our way.

Our TFS 2010 Topology

Grant Holiday (a blog I highly recommend) posted DevDiv’s 2010 hardware topology. Oddly enough, our proposed solution looks much the same.

With TFS 2008, we’re currently making use of F5’s Big/Ip for controlling access to our primary and standby server. We switch between the two during failover situations. With 2010 we’ll be enabling Round Robin and opening up load balancing between 4 VMs running the TFS 2010 App Tier on 2008 Server R2.

Moreover, like DevDiv, we’re taking the opportunity to break out our Warehouse and Analysis Services to a separate server. Our plan is to run an Active/Active cluster. Active Node 1 will run things like TFS_Configuration while Active Node 2 will run the Tfs_Warehouse and Reporting Services databases. If we have hardware failure, we’ll have a single Node running both while the failed node is repaired. We were thinking of doing a 3-node Active/Active/Passive cluster, but the data center didn’t feel comfortable with that because they don’t currently support that configuration.

Let me know what you think of our solution.

Contents of a Changeset when Accessing via a Work Item Link

UPDATE: I'm smoking crack. There is a slight change to the use case below. I left the original use case in there for an example of what works fine. The behavior I was seeing works the same in both VS 2008 and VS 2010. Sorry for the trouble of this post.

Using Visual Studio 2010 connecting to a TFS 2008 Server I can’t seem to get the nice dialog box to look at the contents of a Changeset that is linked to a Work Item. When I double click on it, I get a IE window that shows the contents in an HTML page, but I don’t have a dialog open up where I can do things like Right Click and compare to Previous Version. This is annoying as I use this feature all the time for code reviews and bug fix approvals.

I tried putting on TFS 2010 Power Tools, but this too did not give me the option. Tomorrow I’ll try to connect to a TFS 2010 Server instance to see if that changes the behavior. We have a number of people that use this procedure for code reviews and I can see some upset faces if we don’t have a better solution. Here is the exact use case again:


1) Open a Work Item
2) Click on Links
3) Double Click on Changeset.
4) This opens IE with the contents instead of a dialog box.


1) View History on a file in Source Control
2) Double Click on Changeset
3) Click on Work Item
4) Click on Links
5) Double Click on Changeset
6) Here is where I get IE popping up.

Oddly, I get the Changeset dialog box fine when I go directly to a Work Item to the linked Changeset. It's when I got to a Changeset to a Work Item to its linked Changesets that I get IE popping up.

Again, in VS 2008 this would give you a dialog box where you can do things like compare to previous version. In VS 2010, this seems to give me only a popup to IE.

Maybe I’m missing something? Yes you are Mac. This works the same in VS 2008 and VS 2010.

Browser Cache - Your Friend and Foe

Because we use a ton of Static Content (e.g. Java Script, CSS, HTML) in our new application, the last few years I've been involved in dealing with Internet Browser Caches. We cache our Static Content locally so the users don't have to download it every time they hit our site. While this made us very fast, we were having perpetual problems with invalidating browser cache when Static Content files change.

We had a number of architectural meetings about it with a number of suggestions on how to deal with it. URL re-writing, ETAGs, unique file names were all brought up. What we ended with was a directory versioning scheme where we inject a unique number into the base URI path and then update an authority file that orchestrates the pages, injecting the unique number. During deployment the files are staged under the unique number on the web server file system.

If you look a look at Google's home page, you can see they do something similar. Here is a Java Script file they are including on their home page. Looks like they took the idea and put it on steroids to me.

http://www.google.com/extern_js/f/CgJlbhICdXMrMAo4aEAILCswDjgPLCswFjgWLCswFzgHLCswGDgFLCswGTggLCswHTgyLCswJTjKiAEsKzAmOAssKzAnOAQsKzAqOAQsKzArOAwsKzA8OAIsKzBAOBAsKzBBOAUsKzBFOAEsKzBOOAUsKzBROAIsgAIT/0_HiDVMVc9k.js

Bing did something similar, but is more similar to what we do. Bing's number kind of looks like a TFS changeset number ;)

http://www.bing.com/fd/sa/0311190804/Shared.js

Since I've spent most of my life on the server side, I never realized how many gains you get from using Browser Cache effectively. Here are a couple of stats I took off our site while looking at the caching.

With the browser cache empty.
One 18.1 KB File = 172 MS

With the browser cache populated with the file.
One 18.1 KB File = 31 MS

I guess the lesson learned is using Browser Cache is great if you know enough on how to invalidate the cache when you need to push a new version of the code.