We're a daring group of folks who have added a standby TFS App Tier server to our topology. We'd like to see TFS support true failover, but for now we're comfortable with a warm backup that with some manual intervention we can get up and running if need be.
We followed the How to: Activate a Fail-Over Application-Tier Serverto a tee and as it turned out, that was our downfall. I'm not saying the documentation is wrong, all I'm saying is we had to modify step 4) under Reporting Services to get our standby up and running.
After a number of hours, we broke down and called Microsoft Support who directed us to a very competent TFS support engineer. He correctly diagnosed the issue as being Reporting Services related and joined us with another very competent Reporting Services support engineer. Microsoft should be proud of the work they both did. Great support is one of the primary reasons we buy commercial software/support.
The issue was regarding the command "RSKEYMGMT –a –i <instance ID for AT2> -f c:\backups\My_RSBackup_TFS_AT01 -p aPassword". We got the error "Unable to locate the Report Server Windows service for instance <instance ID>". From what I understand, since the TFS install requires Reporting Services to be installed as a "default" instance (see step 10 in "How to: Install Microsoft SQL Server 2005 Reporting Services for Team Foundation Server (Dual-Server Deployment)" which is located in the TFS install documentation.) you can't activate it by a named instance name. Thus the "–i
Removing the –i and running the command "RSKEYMGMT –a -f c:\backups\My_RSBackup_TFS_AT01 -p aPassword" on the standby App Tier worked just fine.
Although the process is not perfect, we're now able to failover. Hope this helps if you run into the same issue.
Friday, May 25, 2007
RSKEYMGMT command for failover to standby TFS server
Thursday, May 24, 2007
TFS Data Warehouse Not Being Updated by TFS TFSServerScheduler
May 2007 - We had an ongoing issue where the TFS Data Warehouse was not updated by TFS TFSServerScheduler but could be invoked manually. What we think happened is when we installed TFS to a different port (i.e. 8888) and went through the process of putting our App Tier behind a Fully Qualified Domain Name, not all of the Registry Entries were updated as needed.
With the help of this posting and some sleepless nights we sifted through the App Tier's registry and found a number of references to AppTier-B01:8080 instead of AppTier-B01:8888 or better tfs.int.mycompany.com:8888 which is a DNS entry for AppTier-B01:8888. We also found a number of duplicate references where both AppTier-B01:8080 and tfs.mycompany.com:8888 were listed. We logged on as the TFSInstall, TFSService, and TFSReports account and did the clean-up as some of the settings are under HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\8.0\TeamFoundation\Servers.
The lesson learned is to make sure all your registry settings are correct for ALL USERS. Because our AppTier name did not change (we just put it behind DNS), we think the most important piece was the port number which we changed from 8080 to 8888. Below are our current settings on the App Tier.
Sys Admin Account:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\Servers
tfs.mycompany.com = http://tfs.mycompany.com:8888
HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\Servers
tfs.mycompany.com = http://tfs.mycompany.com:8888
TFSService Account:
HKEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\Servers
tfs.mycompany.com = http://tfs.mycompany.com:8888
Wednesday, May 23, 2007
Can you have TFS states and groups with the same name?
May 2007 - We've been dealing with the very informative user error message "TF26212: Team Foundation Server could not save your changes. There may be problems with the work item type definition. Try again or contact your Team Foundation Server Administrator." for about two weeks now and we finally think we have an answer.
The error seemed to show up when we had a state (e.g. Testing) named the same as a group (e.g. Testing). Here are our steps and final thoughts on how we resolved the issue.
Start by editing the Web.config file under \Program Files\Microsoft Visual Studio 2005 Team Foundation Server\Web Services by changing
name="traceLevel" value="1" to name="traceLevel" value="4"
and
key="traceWriter" value="false" to key="traceWriter" value="true"
This will start logging myriad amounts of information under C:\WINDOWS\Temp\TFLogFiles (or some other directory if you change the defaults).
After you make these changes (you don't need to bounce IIS) try to get the error again. If you see an error like below in the TFLogFiles you may be in the same boat as us.
[WI] [Error, 2916, 5, 11:31:33.016] SvrEx: Microsoft.TeamFoundation.WorkItemTracking.Server.ValidationException: Forcing rollback ---> System.Data.SqlClient.SqlException: Forcing rollback
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.HasMoreRows()
at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
at System.Data.SqlClient.SqlDataReader.NextResult()
at Microsoft.TeamFoundation.WorkItemTracking.Server.PayloadTableCollection.Populate(SqlDataReader reader)
at Microsoft.TeamFoundation.WorkItemTracking.Server.SqlAccess.ExecuteBatchPayloadImpl(IRequestContext context, String sqlBatch, List`1 parameterList, Boolean& errorOnBulkUpdate, String connectionString)
--- End of inner exception stack trace ---.
While we've had differing results with a number of use cases, the root cause seems to be related to having a state and group named the same. We saw the issue with a state called CCB and group called CCB and the same issue with a state called Testing and a group called Testing.
What's even more alarming is they don't even have to be in the same project. We're pretty sure if you have a Testing group in Project1 and a Testing state in Project2, you will get a conflict.
Again we're not 100% sure this should be written in stone, but our error has disappeared now that we renamed the CCB state to Change Control Board and Testing group to Testers. And removed all references to the CCB state and Testing group from all the projects in our instance. This seemed to work for us.
Good luck!
Friday, May 11, 2007
Report Server Windows Service (MSSQLSERVER) cannot connect to the report server database
May 2007 - We just got our standby server setup and failed over for the first time. I'll send out the steps we took in a forthcoming post as we're still working out some minor details.
We were a bit concerned as after setting up the standby server we were getting the following error messages (below) in Event Viewer. The first error message just happened once (or so we can tell). The second one was recorded every minute in Event Viewer on the standby.
When we failed over the to standby both errors went away and started to appear on the primary. I posted a message to the MSDN forums and Mr. Chen confirmed this was as expected. I'm not sure I followed him on this, but he said the reason was "This is usually used to improve production quality in the future." Again I'm not sure I follow him on why this will help to improve quality, but I'll take his word for it.
Happy failover!
Error Messages:
Report Server Windows Service (MSSQLSERVER) has not been granted access to the catalog content.
Report Server Windows Service (MSSQLSERVER) cannot connect to the report server database.
Thursday, May 03, 2007
HTTP Status 400 With Reporting Services
May 2007 - There are probably a myriad of reasons to get the error "The request failed with HTTP status 400: Bad Request" in Reporting Services running with TFS, but here is why we got the error.
While trying to get our TFS instance behind a fully qualities DNS name we added tfs.int.mycompany.com to host header entry on the Default Web Site on our TFS App Tier. After adding this entry users could still see Reports in Visual Studio, but when Right Clicking and selecting Show Report Site Internet Explorer would pop up and give us the error ""The request failed with HTTP status 400: Bad Request". As a side note, adding this entry in the host header did nothing for us while trying to get our App Tier behind DNS.
Once we removed that value, the error went away. Hope this helps anyone else.
Tuesday, May 01, 2007
Our own Dog Food Statistics
May 2007 - What a month this has been. We had some issues getting TFS up and running (see my previous post for some of the bigger ones), but finally we have a PROD system that is capable of taking on users. For those of you who are thinking about implementing TFS, make sure you have dedicated resources to work on it. We're fortunate that our company is funded a full time administrator to get this up and running for 200+ users (1 admin --> 200 users). If you're a Sys Admin who, in addition to your day-to-day responsibilities, is supposed to get TFS up and running, good luck!
I'm a bit of a statistics nut so while we currently wait for Brian Harry to release the TFSServerManagerTool (http://blogs.msdn.com/bharry/archive/2007/01/22/tfsservermanager-powertool.aspx) I thought I'd try to pull some simple statistics that we might find valuable and post them. Being our first project was created today, it seems like an apropos time to make this posting. I'm a bit pressed for time so I didn't have the opportunity to put together the myriad of statistics that Brian has. I'll wait for him to release the TFSServerManager PowerTool to get those crazy numbers. That being said, we find this information helpful so here it goes.
Team Projects = 1; This was a pretty easy one to find!
Users = 6; This one was a bit tougher. I started by running the TfsSecurity tool (http://msdn2.microsoft.com/en-us/library/ms252504(VS.80).aspx) with the "/imx all:" parameter. Unfortunately this gave us back all the users plus the 50 Sys Admins that also have access to the machine, but would never use TFS with their administration accounts (e.g. MyDomain\M0000001). Consequently we don't want to count them. The good thing is our user accounts all start with
Work items = 16; This was also easy as each new work item creates a new incremented id. There are 15 standard CMMI process template Tasks that get created. After that we created one to change a Change Request state which was number 16.
Files/Folders = 1859/31; I got these values using the following commands. For files I ran "tf dir /server:tfs.int.mycompany.com $/ /recursive" less the value I got for the number of folders, which is next. For folders I ran "tf dir /server:tfs.int.mycompany.com $/ /folders /recursive".
Changesets = 7; Another easy one for now. I got this value by running "tf changeset /server:tfs.int.mycompany.com /latest /noprompt"
That's it for now. If you find additional or better ways to find your own Dog Food statistics before Brian releases TFSServerManagerTool, let us know.
Monday, April 09, 2007
Trials and tribulations of a TFS install
Ah it's spring and we're still plugging away on getting TFS installed on our DEV environment. In TFS's defense most of the big delays are big company bureaucracy. That being said, installing TFS is not as easy as installing MS Word on my grandmother's PC. Here are some of the issues we ran into and the resolutions. I've also included some big company standard deviations we did. By big company standard deviations I mean things like our system administrators make us install "third-party" software on the D: drive instead of the C: drive. I can't imagine the number of hours upper management spent on that decision.
First a bit of background. From what I can tell our company supports Windows based servers with two groups. For this discussion let's call the database group the "DBAs" and the system administrators "Sys Admins." Upper management actually refers to them as something totally different, but the average sane person can't understand upper management's terminology (or thought process for that matter). The DBAs and Sys Admins work like this: In the morning the DBAs get coffee on the second floor, Sys Admins the fourth. DBAs go out to lunch at 11AM. Sys Admins eat at their desk around noon. DBAs drink Diet Coke, Sys Admins drink regular Pepsi. To an outsider one could conclude the two groups try to avoid each other at all costs. Thus, it's with these two groups we must coordinate the temperamental TFS install.
- The DBAs like to use a standard company SQL install image for all data tier machines. Unfortunately the big company image does not include Integration Services and Analysis Services. Apparently we're the first team that needs such "advanced technology." Lesson number one, don't install a base SQL 2005 64-bit image on a cluster and then expect to install Integration Services and Analysis services afterwards. After three days working with Microsoft, our DBAs gave up and re-installed SQL 2005 64-bit with Integration Services and Analysis Services selected on the initial install. I'm not sure why adding these two components after the cluster is setup is such a big deal, but the amount of vulgar language the DBAs used it must be a big deal.
- Our DBAs have "Big Company Approved" Reporting Services install instructions that includes "Configuration." We had the DBAs do the basic install with their instructions, then tackled them before they could run through the "Configuration" section. From what we're told, the TFS install does the configuration for us.
- We installed the TFS App Tier software to the d: drive. That is a requirement from the Sys Admins. No big deal here.
- Here is a big one. The entire datacenter has some kind of "flash copy agent" software running on port 8080. Conventional wisdom would lead you to believe one can simply change the TFS msiproperty.ini to some other port (e.g. 8888) and install TFS. There I go again thinking conventional wisdom is logical. Changing the misproperty.ini file resulted in the following error: "Error 28925.TFServerStatusValidator: Calling the Team Foundation Server ServerStatus Web service failed. Additional details about the problem can be found in the setup log. Verify your network configuration. For more information on troubleshooting this error, see the Microsoft Help and Support Center. Log file: TFServerStatusValidator - Team Foundation Server Status Validator tool (C) Copyright 2006 Microsoft Corporation. All rights reserved." The details are outlined in my forum posting (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1411189&SiteID=1&mode=1). After giving up and calling Microsoft, the support folks from India had the Sys Admin shut down the agent software running on 8080, install TFS to port 8080, step through a number of manual steps which included making a database change to switch URL values from 8080 to 8888 (steps are in my forum posting above), restart all TFS related applications/services, and finally start the flash copy agent software back up on 8080. The look on our Sys Admin's face after all this was priceless!
- Since our DBAs don't let the Sys Admins on the database machines and vice versa, we had some trouble installing the App Tier. From what we found out, the TFSSetup account used to install the App Tier needs to be a local administrator on the App Tier and a member of the local administrator group on the Data Tier. After bringing in a United Nations negotiator, we got the DBAs to "temporarily" add the Sys Admin's TFSSetup account to the local administrator group on the Data Tier. The install went fine after that.
- The TFS administrator (i.e. me) won't be a local administrator on either the App Tier or Data Tier. That would be against company policy. So we followed the three sections outlined by Microsoft (http://msdn2.microsoft.com/en-us/library/ms253096(VS.80).aspx) and added our team's group name to the appropriate locations in TFS, SharePoint, and SQL Reporting Services. Amazingly this seems to work just fine.
- Lastly we had to add the TFSReports user to the local administrator group on the App Tier to get around the error "The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID {BA126AD1-2166-11D1-B1D0-00805FC1270E} to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20). This security permission can be modified using the Component Services administrative tool." The end user error was "Reporting Services Error --- An error has occurred during report processing. (rsProcessingAborted) Get Online Help Cannot impersonate user for data source 'TfsOlapReportDS'. (rsErrorImpersonatingUser) Get Online Help Logon failed. (rsLogonFailed) Get Online Help For more information about this error navigate to the report server on the local server machine, or enable remote errors --- SQL Server Reporting Services." Being the install instructions say "This account should not be an administrator on Team Foundation Server computers" I'm pretty sure this is not correct. However at the time of this posting, no one from Microsoft can give us any guidance on better way to get Reporting Services to work. I'll keep this forum (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1345776&SiteID=1) updated if we ever figure out a better way to handle this.
Sunday, March 04, 2007
What we dislike about TFS
Overall we really like TFS, but here are a few things that are less than desirable.
- "Friendly Names" (actually unfriendly for us) for fields such as "System.AssignedTo" or "System.ChangedBy" is a major oversight on Microsoft's end. Let's say you have an employee John Doe who's Active Directory name is "Doe, John (Company 1)". John then moves to company 2 and the Active Directory name changes to "Doe, John (Company 2)" This causes major issues in TFS as because it's integrated with Active Directory the work items are invalid and can't be modified. Fields like "System.AssignedTo" can simply be updated by the user. Hidden fields like "System.ChangedBy" can't be changed by the end user. For this issue call Tech Support and get the TFSLocalize utility. For one user it's not a bad solution, but let's say you have 200 users (like we do) and Company 2 decided to be called Business Unit 2. Now we have 200 users who need to be changed. TFSLocalize stinks as you have to map out old user (i.e. Doe, John Company 2) to new user (i.e. Doe, John Business Unit 2). While we're not up and running yet, this potentially is a major, major issue!!!!
- Administrating TFS, Share Point, and SQL Reporting Services in three different spots is laborious. In fact without the TFS Administration Tool, we'd be in a ton of trouble. Trying to keep track of permissions and users in three separate spots for 200+ users is unobtainable. So if you're new to administrating TFS, make sure to visit Codeplex (http://www.codeplex.com) and download the TFS Administration Tool (http://www.codeplex.com/TFSAdmin). Without this tool, administrating TFS is a nightmare.
- The out of the box Alerts stink. First of all there is no easy way (we know of) to "subscribe" all the users. Instead each user has to "subscribe" themselves. And worse yet, they need to know their SMTP address. Imagine the pain of telling your Vice President they have to type in first.last@company.com. They barely know how to tie their shows for goodness sakes. To get around this we're simply avoiding the concept of subscribing yourself and using Mariano Szklanny's web service to do all notifications (http://staff.southworks.net/blogs/mariano/archive/2006/04/14/411.aspx). I'd suggest a similar approach if you're a large team like us.
- No web interface for work item tracking. Developers love the work item and source control integration located in their IDE. On the other hand, managers hate having to open up Visual Studio 2005 just to approve a Change Request. While this seems like something trivial, a nice bundled web interface sure would be nice. Hey Mr. Gates, buy Teamplain (http://www.devbiz.com/teamplain) and make it part of your solution!
- Code policies on the client side instead of server. Out of our 200+ users, 100-125 are Java developers using Teamprise (http://www.teamprise.com). We like Teamprise, but wish we could enforce a policy like "Require Associated Work Items." Nope can't. All the policy code is local to the Team Explorer install. Dumb! If you've never used Teamprise give it a shot. It's a very nice tool with WONDERFUL technical support.
- No hierarchy for the work items. Rhetorical question; if Tasks are children of Change Requests, why can the Change Request close before the Tasks are closed? A nice hierarchy is needed. And Microsoft, don't give us "TFS is totally extendable to do all of these things you request." We're not in the business of developing configuration management tools. You are, so add hierarchy in for your customers.
- Can't remove or rename work items. For some management reason that supersedes any input I have, we're not using the Requirement work item for a Team Project that is already created. Instead of leaving it there to confuse users, we would sure like to remove it. Nope can't remove it. Not possible. Or how about when you create a new work item called WrognName. Opps I made a typo, this should be called WrongName. Nope, can't rename it. Not possible.
Friday, February 02, 2007
Work Items, Work Items, Work Items
February 2007 - Well this month has been interesting. I spent the first half on the beech in Hawaii with a cocktail, great wife, and a good book (actually four of them). And the second half here at work making up for the two weeks I had off. Makes you wonder if vacation is really worth it?
In January we looked into work item tracking as that is much of upper management's focus. They like the states and transitions and have hours of fun drinking coffee, taking golf, and designing elaborate workflows that no one understands. We're using the CMMI process templates and then making our (i.e. upper management) modifications from there.
In the Change Request work item we're changing the default states to Submit, Estimate/Evaluate, Implement, Verify, Close. Why? Not sure, but this is what the big wigs wanted. At our company we call Bugs, Defect Reports. From what we can tell changing the name of Bug is impossible if a Team Project is already created. So we negotiated and decided to call a Defect Report a Bug when in TFS. We changed a few fields and changed the states to Submit, Evaluate, Implement, Verify, Close. Again, we're at upper management's mercy.
Both Issue and Risk we left pretty much the same. We added a few fields, but for the most part they're out of box. We're not 100% sure on how to use Requirement. Another group is looking at purchasing what they call a Requirements Management tool so we'll probably wait until that effort fails before we start looking at the use of our Requirement work item. That being said, my boss just stopped by my cube and during a lively chat we might have found a use. More on that to come in future months. The Review work item we plan on making optional if they want to use it for code/document reviews.
Task is the most complicated. Tasks are where all our release related work items are stored. If our release manager wonders what are all the changes going into release 7.7, they simply look at all the Tasks with the Release Number (i.e. changed the name of Iterations to match our terminology) set to 7.7. If our other application tiers (e.g. Application Tier) who are dependent on us needs to know all the changes, they typically look at the Tasks. Change Requests are typically to high level for developers. Using Tasks we also get a good idea when it's time to baseline. I won't get into all our states and transitions (far to many to explain here), but basically we have a state called Code Complete. Before we baseline a release (e.g. 7.7) we look to see how many Tasks are in Code Complete. Ideally all of 7.7 Tasks should be, but as it happens a few are always still in Active or Code Review. We typically move forward with the baseline, but know there is work to be done on the release before a comprehensive round of testing can start.
Development is also supposed to create Links so we have a good understand of what Tasks created Bugs, and what Change Requests or Requirements, Tasks are assigned to. In a perfect world (which we're far from) a Change Request or maybe Requirement (in the future) is submitted. One or many Tasks are assigned to Change Requests or Requirements. Then during the Task's Testing state, Bugs are created and Linked to the Task. Taking this further, Reviews, Issues, and Risks can also be Linked to the Task. This all being said, without rules to enforce a Linking hierarchy, it's really hard to make sure people follow these guidelines.
We also created two new work items. The first is called Build where we log all build results. We're a Java shop and not using Team Build (currently using Anthill which we also wrote a TFS adapter for http://bugs.urbancode.com/browse/ANTHILLOS-261) but still want to keep all our build results (e.g. build failed/succeeded) in TFS. One stop shop for all information. We simply create a Build work item, update it with the needed build information, and close it when the build is done. Seems to work alright. Then we also created Environment CR, which is used to track environment changes. These are handled differently then other Change Requests so that is why we created a different work item for it.
Anyway, probably enough information for the month. We'll touch base in March when all of what I said above changes. Happy winter!
Friday, December 22, 2006
TFS Migration Errors
January 2007 - Ah one month down in our Team Foundation Server (TFS) implementation! We've actually been quit busy even though upper management might not think so. Little do they know, while we wait for our PROD hardware to arrive we've had a demo copy running for months on some antiquated hardware we found under old C++ programming guides.
This past month we've been concentrating on designing workitem types for our CMMI process improvement implementation that we're so proud of, and spent a lot of time doing test migrations from Visual Source Safe (VSS) to TFS. It's in the VSS to TFS migration spirit we'd like to concentrate this month's post on. We've run (actually more like stumbled) into some valuable information that we're hopeful future (or current) TFS users will find valuable.
First some statistics from the VSSConverter tool. So far we've migrated 55,389 files with 137,547 "actions" in around 7 hours and 45 minutes running on desktop machine (996 MHz processor with 512 Mb of RAM). From what we can gather an "action" is any history item (e.g. edit or add) in VSS. We feel this is pretty acceptable and have no issues with the migration duration.
Now onto the errors and warnings. During our test migrations of VSS to TFS we've run into multiple instances of the following errors and would like to share our observations and thoughts of why the errors might have happened. Now I'm skeptical of most hypothesis, including the ones I come up with. So take this as is, and feel free to provide feedback if you agree or differ with our findings.
Error/Warning 1:
"Unable to scan history due to VSS Error: File or project not found"
So far we've found this error to present itself when a file has been "Destroyed" in VSS. We are guessing that the act of "Destroy" is recorded in the VSS history, but the actual file/project is missing and thus VSSConverter can't do the migrate and simply flags it as something it can't do. We guess that kind of makes sense, but once in TFS you do loose the fact that something at one time was "Destroyed." For hard core configuration management pundits this could be like arguing if a tree falls in the woods does it make a noise if no one is around to hear it? That is, was something ever created if there is no history of it?
Error/Warning 2:
"TF60096: Unable to migrate due to Team Foundation Error: The item
The most common cause of this error (we think) is when you have project name conflicts. For example, we once had a source tree called $/prod3 that was created, filled with source, deleted, then created again, and filled with source again. Don’t even get me started on why! This was resolved for the most part by taking smaller chucks lower in the hierarchy. See this posting one the background and some examples of how we got around it. http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=973933&SiteID=1
Error/Warning 3:
"TF60096: Unable to migrate due to Team Foundation Error: The item
"TF60096: Unable to migrate due to Team Foundation Error: TF14022: The deleted item
"Unable to migrate.
We've seen these errors when no file history was migrated at all. We get a message in the comments saying "Tip Version match Checkin---- DO NOT MODIFY. VSSConverter comment. Verification complete ----" which think is due some combination of deleting a file, recovering a file, and then renaming it (as a side note it's also a pretty lame error message). For example this is what we saw in the VSS history. I've purposely XXXXXXX'ed out the intern ;) who exercised themselves with this renaming experiment.
***************** Version 63 *****************
User: XXXXXXXX Date: 10/14/05 Time: 10:23a
Renamed application_STABLE.xml to application.xml
***************** Version 62 *****************
User: XXXXXXXX Date: 10/07/05 Time: 12:04a
Renamed application.xml to application_STABLE.xml
***************** Version 61 *****************
User: XXXXXXXX Date: 10/07/05 Time: 12:04a
Renamed application_STABLE.xml to application.xml
***************** Version 60 *****************
User: XXXXXXXX Date: 10/07/05 Time: 12:04a
Renamed application.xml to application_STABLE.xml
***************** Version 59 *****************
User: XXXXXXXX Date: 10/06/05 Time: 2:20p
Renamed application_STABLE.xml to application.xml
***************** Version 58 *****************
User: XXXXXXXX Date: 10/06/05 Time: 1:39p
Recovered application.xml
***************** Version 54 *****************
User: XXXXXXXX Date: 10/06/05 Time: 11:02a
Deleted application.xml
In conclusion, we hope you find this information valuable. Feel free to comment if you have additional information or disagree with our conclusions.