Running Tests and getting Code Coverage for our .NET projects.

As I wrote about a last week, the entire testing portion of Visual Studio is a mystery to me. I've never been a tester (for good reason) so I've really never spent the time to dig into Microsoft's offering. That was until I was forced to figure some stuff out, due to my new team wanting it.

From what I've found, TFS 2008 makes executing user's tests at build time very easy. All we had to do is add the following line to our Team Build project file and the magic happened. We had test results!!

<TestContainer include="$(OutDir)\Unit Tests.dll">

The code coverage was missing though so we had to track that down. Thanks to Benday, all we had to do is update our Team Project file again and magically, we had coverage results.

<RunConfigFile>$(SolutionRoot)\LocalTestRun.testrunconfig</RunConfigFile>

Of course this assumed our testrunconfig was setup for Code Coverage, which in our case it was.

So far, even a testing novice has found it pretty easy to get tests to run at build time. Feel free to share you experiences, both good and bad.

Publishing Test Results using the Testers Edition of Visual Studio

I fully admit, I don't know anything about the Testers Edition of Visual Studio. However, since I know a bit about TFS, I've been asked a number of questions about how the two talk to each other. Thus it's been a learning experience for me!

Here is the way I understand the Test Results publishing process to TFS for the Tester Edition of VS. Again, this is all really new to me so if I have something wrong, please correct me.

Per the document here, I see that a Tester will run a test on their local machine or using the "rig" to run on remote machines. After the tests are run, test results are stored in *.trx file on the tester's machine. Testers then can open that file and "Publish" them to TFS's operational store (which must be the TfsBuilds database maybe?). That data then gets moved to the TfsWarehouse per the warehouse schedule. At this point, you can Report on it in the warehouse.

Now there seems to be one small deviation in that Load Tests need to be loaded to a local SQL database which is defined outside of TFS. That is, the Load Tests database stores Load Test data before it's published to TFS. There must be someway then that the Tester Edition of VS can look in that local database where the Load Test results are stored, and publish the results to TFS. I think this is an important detail as there may be some ambiguity on the difference between the Load Test database and the standard databases that make up TFS (e.g. TfsBuilds, TfsWarehouse, etc).

If I have anything wrong here, I'd love for you to share your insight as we're trying to put the big picture all together.

Teamprise 3.0 is released!

If you didn't see, Teamprise 3.0 has just been released. As I've written a number of times, Teamprise is a great offering if you're looking for a cross platform solution for Team Foundation Server. NO I do not work for Teamprise or sell their product for a living. I'm simply a customer at a large software firm who uses Teamprise for our Java development groups.

We use every piece of their suite including the Ant scripts and Teamprise Build Extensions which allows you to call your Ant scripts from Team Build.

Teamprise has a great product, but even a better support group. If you have not checked them out, do it soon. You'll find it being one of the best products around.

The "Copy" task could not be initialized with its input parameters

As I wrote about last week, we've started to use a few Tasks from Sdc. Per the instructions we started by importing the entire list of Tasks. Something like this.

Import Project="$(TasksPath)\Microsoft.Sdc.Common.tasks"

Doing this led to an error in our Team Build though.

...
(EndToEndIteration target) (1)
(CoreDropBuild target)
C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(1310,11): error MSB4064: The "SourceFiles" parameter is not supported by the "Copy" task. Verify the parameter exists on the task, and it is a settable public instance property.
C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(1309,5): error MSB4063: The "Copy" task could not be initialized with its input parameters.
...

I took a look in the Team Build targets file and sure enough, the Team Build targets file uses Copy, but needs the MSBuild one not the Sdc one. I'm guessing this is an conflict error in Sdc.

Using guidance from the forums, we had to change our global import to just import the TaskNames we needed. This helped us get around the issue with the Copy conflict.

UsingTask TaskName="Microsoft.Sdc.Tasks.Tools.PsExec" AssemblyFile="$(TasksPath)\Microsoft.Sdc.Tasks.dll"

Check out SdcTasks

We've taken a mass adoption of the SdcTasks found on Codeplex. When comparing MSBuild to Ant, I've always been disappointed with the Core tasks in MSBuild. The default list seems to be limited. NOTE: I have not done a task for task comparison, but only a cursory compare.

The limited task set in MSBuild is what has really excited me about SdcTasks. There is a ton of stuff in this add in. Want to talk to Active Directory, Email, or talk to SQL? Take a look at SdcTasks for help.

Since I'm lazy, I'd like to see these rolled into the standard install for Team Build or MSBuild.

Other tools we use in conjunction with TFS

While TFS can do a lot of things, I thought I'd share a list of tools we're using (or just looking at) in conjunction with TFS for our full "Build and Deploy" solution.

- PSTools - We use this fine set of tools for executing deployment scripts on remote machines. Once Power Shell allows this ability (I think it's coming in 2.0) we might look at using Power Shell for our remote communication. In the mean time PSTools will work though we are getting some output from Psexec which is hanging MSBuild's Exec task.

- Power Shell 1.0 - While I'm not a shell expert, from what I've found Power Shell seems to be a great upgrade from CMD. We've changed all our deployment scripts to use Power Shell for the basis.

- AppCMD - We use AppCMD (pronounced App Command by the Microsoft person I talked to last week) is a command line interface for IIS 7.0. Instead of using our home grown tool for writing to the IIS meta-base (which we did with IIS 5/6) we're using AppCMD. Now since IIS 7.0 does not have the concept of a meta-base, we could just xcopy the applicationHost.config file over and call it a day. However, we're currently hosting multiple applications on the box and don’t want one deployment to bring down all apps. By using AppCMD to purge/pave our IIS 7.0 configuration, we scope the downtime to the application we're trying to deploy.

- IIS 7.0 and Tomcat - Our ASP.NET apps are running under IIS 7.0 (as you would guess) and our Java applications are running under Tomcat.

- Altiris Deployment Sever and Altiris Software Delivery System - So our deployment runs fine one a machine or two, but we need something that will execute the deployment on 1000+ machines all at the same time. Altiris has a couple of different products we're looking at to do this. Basically, this would remove our need for the Psexec tool from PSTools and our FTP Tasks in Ant we use to deploy a WAR file to Tomcat.

I know, I know this has very little to do with TFS, but we do hook the tools into TFS (and vice-versa) so I thought I'd share it with you.