Windows Process Activation Service – Error 183

This is not related to TFS, but I need a place to write this down. On one of our runtime machines, we had an issue where II 7 stopped running. It turns out that the Windows Process Activation Service (WAS) was down. When trying to restart WAS, we got an Error 183 – Unable to create file (or something like that). When trying to open Event Viewer or any other snap-ins to MMC, we got a “could not load snap-in error.”

Through hair pulling and swearing, we finally tracked down a machine.config change that one of my colleagues made. In the process of doing some tuning, the I haven’t reviewed the schema, but I’m guessing this is not valid as it does not make sense.

After removing the autoConfig, we were back up and running. Thanks to Carlo for pointing us in the right direction.

<processModel maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" memoryLimit="90"/>

<httpRuntime minFreeThreads="12"/>

<processModel autoConfig="true"/>

Windows "Loopback bug" when using a FQDN

We just got our new TFS 2010 hardware! Unfortunately, my smile was quickly brought to its knees by the "loopback bug" where you can access a IIS site (in our case it's a Share Point site running under IIS 7.5) via fully qualified domain name (FQDN) from a client machine, but can't use the FQDN on the server looping back to the server. When trying to use the FQDN locally on the server, we continually get the username/password prompt.

Here is the work around if you loose a day on this like I did. Option #1 worked for us. Thanks for the post at Server Fault for reminding me.

Spring: Ambiguous constructor argument types

This is not TFS related, but build related and this is my only public blog where I document build-like stuff for future reference if I was to ever need it.

In Spring for Java, if you’re going to use “names” for wiring up things like constructor arguments, the class files have to be built with javac –g, which is an entry-level debug mode for javac. This will compile the classes, keeping such things as variable names. If you don’t use javac –g, then Java takes the liberty to modify variable names. I’m guessing they do this for, among other things, keep the class sizes down.

Below is a poorly formatted, convoluted series of knowns and tests to come up with a hypothesis and eventual solution to Spring issue we were running into.

Knowns

1) The Spring configuration is using variable name for mapping constructer arguments in the spring configuration to the set of constructor parameters.

bean id="SetManager" class="com.mycompany.manager.SetManager" lazy-init="false"

constructor-arg name="courtSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="pubSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="jurSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

bean

2) When deploying the application out, the error we’re getting is this.

SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'SetManager' defined in class path resource [spring-manager.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [org.springframework.core.io.Resource]: Ambiguous constructor argument types - did you specify the correct bean references as constructor arguments?

at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:684)

at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:192)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:984)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:886)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:479)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450)

3) When creating a WAR via Eclipse’s Export feature, the bindings are wired up fine. When creating the WAR via Ant (or using javac from the command line, which is what Ant is doing), we get the error. When diff-ing the WAR files, they are the same with one exception: The class files created via Eclipse are larger.

Hypothesis Given all of this, our hypothesis is that running the build via Ant, the method names are getting truncated (or modified) such that using the name attribute in Spring won’t work.

Tests

1) To test the use of indexes, instead of names we used something like this: “index="0"”. Using this got us around the initial issue, but ran us into a very similar problem loading another bean. At this point, we knew we were on to something. Or on something. Here are the Spring configuration snippets of using names verses indexes.

Names

bean id="SetManager" class="com.mycompany.manager.SetManager" lazy-init="false"

constructor-arg name="courtSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="pubSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg name="jurSetCSV" type="org.springframework.core.io.Resource" value="classpath:..."

bean

Indexes

bean id="SetManager" class="com.mycompany.manager.SetManager" lazy-init="false"

constructor-arg index="0" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg index="1" type="org.springframework.core.io.Resource" value="classpath:..."

constructor-arg index="2" type="org.springframework.core.io.Resource" value="classpath:..."

bean

We used JAD to decompile both classes. Bingo, the issue is highlighted! When using javac, the default settings truncate the variable names. If Spring is to use variable names for wiring up arguments or properties, this won’t work very well ;)

Ant

public SetManager(Resource resource, Resource resource1, Resource resource2)

{

loadCourtSetMaps(resource);

loadPub(resource1);

loadJur(resource2);

}

Eclipse

public CustomDigestCollectionSetManager(Resource courtSetCSV, Resource pubSetCSV, Resource jurnSetCSV)

{

loadCourtlineCollectionSetMaps(courtSetCSV);

loadPublicationCollectionSetMap(pubSetCSV);

loadJurisdictionCollectionSetMap(jurSetCSV);

}

Solution

1) What parameter change in Ant’s javac Task could we change to get the accurate parameter names?

a. Set debug = true. Turning debug on will make sure the class files are compiled with the same variable names (and other things) as what is in the source code. This will increase the class file size, but from what we’ve noticed so far, does not cause great performance degradation.

Why a Web.config change does not result in a new process id

A colleague asked me why he sees Application_Start events after modifying his Web.config but does not get a new process id for the worker process. While this article is addressed to IIS 5 and 6, I think the same principle must apply for IIS 7.

From what I can tell, a change to the Web.config will result in tear down of the Application Domain, but not the Application Manager. The Application Manager is what controls the worker process.

If anyone disagrees with this, let me know.

“The Network Path Was Not Found” When Installing our .NET Node Agent

We ran into a problem installing some custom software we call the .NET Node Agent on machines that were in a new COMPANYQA Windows domain. After some troubleshooting, we found the issue to be related to how we figure out what domain the machine is in. We do this logic so we can insert the correct .NET Node Agent service account (COMPANYMGMT|COMPANY\svcNodeAgnet) into the machine’s local Administrators group. If a machine is in COMPANYMGMT, we need to add COMPANYMGMT\svcNodeAgnet. If the machine is in COMPANY, or now COMPANYQA, then we need to insert COMPANY\svcNodeAgnet. (We could have asked a human to make this distinction during install time, but asking a human to do anything around here causes more trouble than it’s worth.)

We use .NET’s System.DirectoryServices namespace to surf Active Directory (AD). We use the machine’s parent to grab Properties. When trying to get Properties from the parent (DirectoryEntry.Parent.Properties) we continually got the message “The Network Path Was Not Found.” After some in-depth troubleshooting, we found that we had to add companyqa.mycompany.com to the DNS suffix list. The only values were company.mycompany.com and so we couldn’t find the LDAP.companyqa.mycompany.com AD server.

In Windows 2008 Server, this setting is under Control Panel - Network Connections - Local Area Connections – Internet Protocol Version 4 (TCP/IPv4) - Properties – DNS – Append These DNS Suffixes. After we added companyqa.mycompany.com we were on our way.

Our TFS 2010 Topology

Grant Holiday (a blog I highly recommend) posted DevDiv’s 2010 hardware topology. Oddly enough, our proposed solution looks much the same.

With TFS 2008, we’re currently making use of F5’s Big/Ip for controlling access to our primary and standby server. We switch between the two during failover situations. With 2010 we’ll be enabling Round Robin and opening up load balancing between 4 VMs running the TFS 2010 App Tier on 2008 Server R2.

Moreover, like DevDiv, we’re taking the opportunity to break out our Warehouse and Analysis Services to a separate server. Our plan is to run an Active/Active cluster. Active Node 1 will run things like TFS_Configuration while Active Node 2 will run the Tfs_Warehouse and Reporting Services databases. If we have hardware failure, we’ll have a single Node running both while the failed node is repaired. We were thinking of doing a 3-node Active/Active/Passive cluster, but the data center didn’t feel comfortable with that because they don’t currently support that configuration.

Let me know what you think of our solution.

Contents of a Changeset when Accessing via a Work Item Link

UPDATE: I'm smoking crack. There is a slight change to the use case below. I left the original use case in there for an example of what works fine. The behavior I was seeing works the same in both VS 2008 and VS 2010. Sorry for the trouble of this post.

Using Visual Studio 2010 connecting to a TFS 2008 Server I can’t seem to get the nice dialog box to look at the contents of a Changeset that is linked to a Work Item. When I double click on it, I get a IE window that shows the contents in an HTML page, but I don’t have a dialog open up where I can do things like Right Click and compare to Previous Version. This is annoying as I use this feature all the time for code reviews and bug fix approvals.

I tried putting on TFS 2010 Power Tools, but this too did not give me the option. Tomorrow I’ll try to connect to a TFS 2010 Server instance to see if that changes the behavior. We have a number of people that use this procedure for code reviews and I can see some upset faces if we don’t have a better solution. Here is the exact use case again:


1) Open a Work Item
2) Click on Links
3) Double Click on Changeset.
4) This opens IE with the contents instead of a dialog box.


1) View History on a file in Source Control
2) Double Click on Changeset
3) Click on Work Item
4) Click on Links
5) Double Click on Changeset
6) Here is where I get IE popping up.

Oddly, I get the Changeset dialog box fine when I go directly to a Work Item to the linked Changeset. It's when I got to a Changeset to a Work Item to its linked Changesets that I get IE popping up.

Again, in VS 2008 this would give you a dialog box where you can do things like compare to previous version. In VS 2010, this seems to give me only a popup to IE.

Maybe I’m missing something? Yes you are Mac. This works the same in VS 2008 and VS 2010.

Browser Cache - Your Friend and Foe

Because we use a ton of Static Content (e.g. Java Script, CSS, HTML) in our new application, the last few years I've been involved in dealing with Internet Browser Caches. We cache our Static Content locally so the users don't have to download it every time they hit our site. While this made us very fast, we were having perpetual problems with invalidating browser cache when Static Content files change.

We had a number of architectural meetings about it with a number of suggestions on how to deal with it. URL re-writing, ETAGs, unique file names were all brought up. What we ended with was a directory versioning scheme where we inject a unique number into the base URI path and then update an authority file that orchestrates the pages, injecting the unique number. During deployment the files are staged under the unique number on the web server file system.

If you look a look at Google's home page, you can see they do something similar. Here is a Java Script file they are including on their home page. Looks like they took the idea and put it on steroids to me.

http://www.google.com/extern_js/f/CgJlbhICdXMrMAo4aEAILCswDjgPLCswFjgWLCswFzgHLCswGDgFLCswGTggLCswHTgyLCswJTjKiAEsKzAmOAssKzAnOAQsKzAqOAQsKzArOAwsKzA8OAIsKzBAOBAsKzBBOAUsKzBFOAEsKzBOOAUsKzBROAIsgAIT/0_HiDVMVc9k.js

Bing did something similar, but is more similar to what we do. Bing's number kind of looks like a TFS changeset number ;)

http://www.bing.com/fd/sa/0311190804/Shared.js

Since I've spent most of my life on the server side, I never realized how many gains you get from using Browser Cache effectively. Here are a couple of stats I took off our site while looking at the caching.

With the browser cache empty.
One 18.1 KB File = 172 MS

With the browser cache populated with the file.
One 18.1 KB File = 31 MS

I guess the lesson learned is using Browser Cache is great if you know enough on how to invalidate the cache when you need to push a new version of the code.

Enabling the “Send Selection in Mail”

After installing VS 2008 and Team Explorer, a colleague of mine asked why she couldn’t see the “Send Selection in Mail” feature as the rest of us could. Here is why.

1) First, you need VS 2008 SP1. SP1 is key here.
2) Second, you need Power Tools.
3) Third, you need to make sure “.NET Programmability Support” is configured.

If you do 1-2, but not three, you’ll get an error that says “TF242403: To send a query or work items, you must install Outlook 2002 or later” even if you’re running Outlook 2003 or Outlook 2007.

“A version of Microsoft Visual Studio that is incompatible with this one”

A trusted colleague of mine recently tried to open a VS 2008 solution on his machine. When going to File – Source Control – Open from Source Control and double clicking on the Solution File, he got the message "the project [solution name] cannot be opened from source control because it was created by a version of Microsoft Visual Studio that is incompatible with this one."

We first though it was that he didn’t have VS 2008 SP1 installed. (I have SP1 installed and it worked fine for me.) However, even after installing VS 2008 SP1 he continued to have the problem.

We’re not sure what was corrupted, but after deleting his workspace and recreating it, the error went away. So it must have been a corrupted workspace, or something along those lines, that caused the issue.

It’s funny in technology how often, when in doubt, nuke and pave works.

Visual Studio 2008 SP1 with Team Explorer Hanging

I’m not sure when it started, but my Visual Studio 2008 SP1 has been hanging on my lately. Looking in Task Manager, I see devenv.exe*32 is consuming 99% of my CPU. My life has been within Java lately (via Teamprise’s Eclipse Plug-in) so I haven’t opened VS much, however, winds are changing a bit and so I have it open more often now than before. And the hanging has been getting on my nerves.

I’m not 100% sure of the cause, by I think it might be the Team Members (working….) that I see sporadically on a few of the ~24 Team Projects I have on my list. My guess is that is causing some issues.

I don’t use the Team Members feature of TFS so I’ve disabled it following these instructions. I just did the first step and it seems to have stopped the hanging for now.

So far so good, so hopefully I’m not speaking too soon ;)

Connecting Visual Studio 2008 SP1 to TFS 2010 RC

After my colleague installed TFS 2010 RC to our DEV system, I mustered up the courage to try and connect via Visual Studio 2008 SP1. Not much luck starting out.

I don’t remember the exact error, but it was something in regards to "Unable To Connect." Thanks for letting me know.

After reading through a few blogs and discussion boards (most of them referenced the need for Visual Studio 2008 SP1 which I already had), I came across this one which at the bottom shows you a registry edit you can make to add the TFS 2010 to your VS 2008 SP1 server list. After trying this, I got a bit farther and finally a meaningfully error message.

When trying to connect, we got "You Must Update Your Client with the Forward Compatibility Update.." I couldn't copy/paste the link in the dialog window (sigh), so I got out my pencil and paper to write it down. Here it is for you so you don't have to repeat the same archaic steps.

After installing this package, I was able to connect by putting in http://$server-name:8080/tfs/$collection-name

Out of Memory Exceptions in Ant

While trying to get some new build machines cooking, a colleague of mind forced me to notice that Java 1.6 on Windows must default it’s heap size to 64 megs. When running an Ant job, he was getting and Out of Memory Exception when trying to compile JUnit reports via the JUnitReport Task.

I figured out the default heap size by opening JConsol and checking Max setting on the Memory Tab. Sure enough, without any Max Setting it was 64m.

What got this whole thing confused was that we were setting a larger heap (1.6gig) on the JUnit execution. That is, when JUnit forked out for its own VM to run on, it used 1.6 gigs which turns out to be more than enough. However, the Ant process was running at 64m (the default remember) and giving us fits when Ant’s JUnitReport, which does not fork itself, was trying to compile the HTML reports.

To fix this, we simply upped Ant’s Max Heap Size to something larger (e.g. –Xmx512m) by setting ANT_OPTS=–Xmx512m as an Environment variable on the server. (As a side note, remember you need to refresh the environment of any tooling needing to use this new variable. Examples are recycling a process like Team Build or closing and restarting your command prompt). Once we did this all was better.

Free coffee is coming my way!