What is a Reverse Proxy

During lead developer roles of previous ASP.NET projects, I’ve needed to deploy into some fairly heavy enterprise infrastructure - once needing to deal with some unexpected behaviour caused by reverse proxy’s. I’ll describe in this article some things to keep in mind when doing this to ensure everything still runs smoothly and correctly all the way from the development environment to production.

A reverse proxy has a few functions, one function is acting as the network load balancer as I talked about in my previous post Web Farm Considerations. It can also be the point that processes SSL encryption.

Many people might already familiar a standard “proxy” that they may have needed to configure their browser to use to make connections to the internet (typically from inside a company network). More specifically this is known as a “forward proxy” and is to do with outgoing traffic, i.e. instead of the browser making a connection directly to a website, it will send the request first via a proxy server somewhere in the company’s network. The proxy server may be the only machine on the company network that can access the internet and gives the company further control over blocking of certain sites, monitor outgoing requests, virus scanning, caching etc.

A reverse proxy on the other hand sits on the server side – and is essentially the opposite of a forward proxy.

Application Path Issues

The scenario described here is when your ASP.NET application (ApplicationX) is to be deployed to a subdirectory of an already existent company wide domain (www.company.com), i.e to http://www.company.com/ApplicationX.

The user’s browser will connect to the reverse proxy which has a public address such as https://www.company.com/ApplicationX (so essentially connecting to it without realising it as opposed to the forward proxy that needs to be configured) – which will then forward the request to a server internal to that network such as http://internal05/ApplicationX to do the real processing of the request and dishing up the web content to the user. Acting as a NLB (Network Load Balancer) the request could also be redirected to http://internal0[1-4]/ApplicationX unbeknown to the user. These internal0[1-5] servers are only accessible within the company network, behind the firewall and not accessible directly from the cloud/internet.

Now – here is the issue to keep in mind when deploying in this type of scenario – it’s quite possible for servers internal01-05.company.com to be servers dedicated to ApplicationX – in this case it would seem natural to simply set the default website on these to serve ApplicationX i.e directly at http://internal05.company.com.

The problem with this is ASP.NET will send virtual addresses such as WebResources.axd (some more info about WebResource.axd in my post Securing Static (Non-ASP.Net) Files) back to the client browser as the absolute address “/WebResource.axd”. This will cause the client browser to make requests to https://www.company.com/WebResource.axd – not good ! Instead it should be https://www.company.com/ApplicationX/WebResource.axd .

To achieve this, the option is to create a virtual directory on the internal server called ApplicationX so that it is served from http://internal05.company.com/ApplicationX (matching the reverse proxy’s URL structure). ASP.NET will now return the absolute address “/ApplicationX/WebResource.axd” back to the client browser which will resolve correctly behind the reverse proxy.

In terms of linking to resources yourself, ensure that you don’t use something like Request.Url. This will work in your development environment, but when deployed behind the reverse proxy – Request.Url will actually return http://internal05.company.com/ApplicationX (as far as the ASP.NET application is concerned, it is serving content from this location as the reverse proxy has made a request to the internal server with this URL). Obviously from the client browser however any references to this address will not work since this server is behind the company firewall.

Instead, make use of the special character “~” wherever possible (in conjuction with ResolveUrl where necessary) which will resolve to “/ApplicationX”. Here are some examples (note that server side controls “~” can be specified and interpreted directly within the properties themselves without use of ResolveUrl):

<asp:Image runat="server" ID="image" ImageUrl="~/Penguins.jpg" /> 
<img alt="Penguins" src='<%=ResolveUrl("~/Penguins.jpg") %>' />

This will mean the client browser will request items back to http://www.company.com/ApplicationX as well as ensure everything still works in the development environment.

Using Request.ApplicationPath instead of “~” will probably cause issues if you are running your Visual Studio web server development environment from the root directory. Linking to resources from the root directory - Request.Application path returns “/” and you end up with with a request to //Penguins.jpg, which will fail, such as below: 

<img alt="Penguins" src='<%=Request.ApplicationPath + "/Penguins.jpg" %>' />

The solution is to either use “~” or alternatively run your development environment from a virtual path instead of the root directory if you do prefer using Request.ApplicationPath.

In summary, before deploying to a more complicated infrastructure than a simple single server scenario, it pays to think through any potentially surprising behaviour you might see. To save potential delays further down the track it can be wise to put the time aside to setup a test environment first that includes the reverse proxy, load balancing servers etc to ensure everything still works as expected.

Good luck! Hamish

I have recently had an interesting experience debugging some ClickOnce issues with my WPF Inspirational Quote Management System.  Essentially, I would deploy a new version of the application - but some users after choosing File -> Check For Updates, would receive a message saying that the latest version is already installed - which I know was not the case.

I could not figure out why it would not be picking up the new version of the application. One user being a friend, I was able to work with them using TeamViewer to try a few things on their computer, including inspecting the contents of the ClickOnce installer file (http://www.hamishgraham.net/files/WPFQuoteManagement/WPFQuoteManagement.application) that they receive on their computer after browsing to the URL. I noticed that the contents of the file downloaded was quite old compared to what I would get if I downloaded from the same URL on my computer. Straight away this was smelling of caching problems. 

No amounts of Control-F5 would fix this – only appending a question mark on the end of the URL would retrieve the latest version (common trick to bypass web caches), but this still does not help the WPF application which browses to the aforementioned URL (under the hood using ClickOnce) without any question mark for the update. 

I needed to confirm that this was indeed a web cache issue, so I did this by firing up Fiddler, and running the WPF application’s check for updates function again. Fiddler intercepts all HTTP traffic to/from the computer from all applications and not just a web browser - very handy in this case.

On my friends computer, the WPF application would receive the following HTTP headers for the ClickOnce .application file when checking for updates:

HTTP/1.1 200 OK
Content-Type: application/x-ms-application
Last-Modified: Tue, 01 Jun 2010 05:21:36 GMT
Accept-Ranges: bytes
ETag: "aeb494e4a1cb1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Fri, 25 Jun 2010 04:20:46 GMT
Via: 1.1 bc5
Content-Length: 5769
Connection: Keep-Alive
Age: 343128

On my computer on the other hand, it would look like:
HTTP/1.1 200 OK
Content-Type: application/x-ms-application
Last-Modified: Mon, 28 Jun 2010 05:41:10 GMT
Accept-Ranges: bytes
ETag: "eec57e838416cb1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Tue, 29 Jun 2010 04:08:22 GMT
Content-Length: 5765
Connection: Keep-Alive
Age: 0

I’ve highlighted the problem areas on my friend’s HTTP response. The file was being served from “1.1 bc5” which is the name of TelstraClear’s web cache (my friend being with TelstraClear), which by the looks of it holds a version that I last modified on the 1st June. The date field (25th June) indicates when the web cache last checked that the file was still valid.

Now, here is the problem, without explicitly setting a cache expiry for the file, the web cache is at its own liberty to decide when it deems appropriate to check whether the file is still valid. Without any explicit cache expiry existent in the HTTP header, web caches use heuristics are used to determine when to check validity again by doing calculations on Date and Last-Modified… i.e if the gap between the time I last checked validity of the cached file and when the file was last modified is large, it probably means the file isn’t changing very much, so wait even longer before checking for validity again.

I found someone who has already solved this issue http://www.codeproject.com/KB/dotnet/ClickOnceContentExpiratn.aspx which I implemented, however slightly differently by adding the following line instead into the web.config under the <handlers> node:

<add name="ClickOnce" verb="*" path="*.application" type="CustomHttpHandlers.ClickOnceApplicationHandler" resourceType="Unspecified" preCondition="integratedMode"/>

I then needed to enable the IIS7 Integrated Pipeline on the webserver which ensures all file types are routed through ASP.NET runtime for the HTTP Handler to be executed for the .application file, and the cache expiration explicitly set.

Retrieving the ClickOnce .application file now comes back with the following HTTP headers:

HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/x-ms-application
Expires: -1
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Tue, 29 Jun 2010 09:49:23 GMT
Content-Length: 5765
Connection: Keep-Alive

This should hopefully tell any web caches ISP’s are using that this file mustn’t be cached – meaning latest versions of the application can be deployed promptly to all users. It will take a few days until the web caches finally refresh before I can test this works as expected, but hopefully this will be a win.

Extra Resources

• Caching Tutorial or Web Authors and Webmasters
• Resource Expert Droid
• HTTP/1.1 Header Field Definitions

For years now I’ve had songs in my music collection that are so terrible I shudder when they come on – but the process of actually deleting the file so I’d never hear it again has always been so painful I never do it. So end up listening to the thing again and again.

Applications like iTunes, VLC and Windows Media Player all work the same way by deleting the song from the play list and not the file system itself. To delete from the file system you usually have to go through some complicated process to find the file path, open windows explorer and then delete it from there.

I’ve googled a bit over time for plugins and so on that let you actually delete the currently playing song but I’ve never really had much luck so thought I would bite the bullet and make something up.

I had grand plans of having a lovely little WPF application that sits in the system tray with an assigned keyboard hotkey to delete the currently playing song. I thought I would do this by just scanning the file handles the media playing application has hold of, find the mp3 / wma file and then delete it. Since .NET isn’t very capable at getting low level file handle information without going into a whole lot of unmanaged code I thought I would just use the output from the sysinternals Handle application. This needs administrative privileges however to run, and I subsequently found out that it is impossible to run an external process from .NET with elevated privileges (contrary to many posts around the internet) – on Windows 7 with UAC enabled anyway. This guy also found this out the hard way.

So I’m sorry, I didn’t end up making a pretty application but I did end up getting something going that works (a console application) – but is a bit geeky for the average user unfortunately.

However I thought I would share it anyway. Here are some steps to set it up:

1. First of all, download DeleteCurrentlyPlaying.zip and extract to a directory. If you get an error when running it later, you may not have the .NET Framework 3.5 installed. 
2. Open a command prompt with Administrative rights. This guide tells you how to do this. On Windows XP you won’t need to worry about this.
3. Navigate to the folder with the files in it e.g “cd c:\path\to\downloads\DeleteCurrentlyPlaying”
4. Now run the following if your song is currently playing in Windows Media Player, VLC Player or iTunes respectively (this is just the process name changing):
   handle –p wmplayer | DeleteCurrentlyPlaying
   handle –p vlc | DeleteCurrentlyPlaying
   handle –p itunes | DeleteCurrentlyPlaying
5. For your piece of mind, the file isn’t actually deleted, it is moved to a subfolder “To Delete” for you to delete at some point. Also if you would rather download the source and compile it yourself, you can find it here.

The application will wait until the file has been released by the application before it is moved (in the case of Windows Media Player it seems it can be moved while it is being played).

It should look something like this:

You can just leave this console window open, and then hit the up arrow to repeat the last command and hit enter again when the next one to go comes on.

In .NET 3.5 there are two new delegate types Action<T> and Func<T, TResult>. 

There are other overloads for Action<T> and Func<T, TResult> if you need to provide more than one parameter, e.g:
- Action<T1, T2, T3>
- Func<T1, T2, T3, TResult>

I wanted to test my understanding of these by making a quick console app that uses them. The example below is extremely contrived and something you would not do in practice of course but I believe it to break down the core essence of their use, so you can use your imagination from there as to how you would use them.

It actually started off much more simple but then I intentionally beefed it up including plenty of generics, and multi-lined / multi-parameter lamba’s so if you understand this snippet you know you also have a good grasp of those too.

        static void Main(string[] args)
        {
            int result = DoStuffToSomething<DateTime, DateTime, int>(
                DateTime.Now, 
                DateTime.MaxValue,
                date => {
                        Console.WriteLine(date.ToShortDateString()); 
                        Console.WriteLine(date.ToLongTimeString());
                    }, 
                (date1, date2) => (date2 - date1).Days);

            Console.WriteLine(result.ToString() + " days before .NET DateTime explodes."); 

            Console.Read();
        }

        static R DoStuffToSomething<T1, T2, R>(T1 something, T2 somethingElse, Action<T1> myAction, Func<T1, T2, R> myFunction)
        {
            myAction(something);
            return myFunction(something, somethingElse);
        }

Prints:
  22/04/2010
  12:48:12 p.m.
  2918175 days before .NET DateTime explodes.

Good to know I guess.

Also note Action<T>’s use in the framework such as the ForEach method on a generic List. This can be used like so (as taken from http://msdn.microsoft.com/en-us/library/bwabdf9z.aspx):

    static void Main()
    {
        List<String> names = new List<String>();
        names.Add("Bruce");
        names.Add("Alfred");
        names.Add("Tim");
        names.Add("Richard");

        // Display the contents of the list using the Print method.
        names.ForEach(Print);

        // The following demonstrates the anonymous method feature of C#
        // to display the contents of the list to the console.
        names.ForEach(delegate(String name)
        {
            Console.WriteLine(name);
        });
    }

    private static void Print(string s)
    {
        Console.WriteLine(s);
    }

I would also add to the MSDN example to show how the same can be done using a lambda expression:

    names.ForEach(name => Console.WriteLine(name));

When these new language features come out, I think it is good practice to at least get an understanding of them, so when the right situation presents itself you have a new and possibly more efficient/readable option to use.

For my blog, I use SyntaxHighlighter to format code snippets (which I believe to be the best code formatting solution out there for a few reasons). The problem is however that it requires you to wrap your code in <pre class="brush: c#"> your code goes here </pre> tags – which can be a pain.

The WMD Editor control (or the stackoverflow open source port) on the other hand has a convenient button for "code" (or using the Ctl-K hotkey) that will indent the snippet in such a way that when rendered by Markdown (whether in the WMD Editor’s preview pane with the server side MarkdownSharp for example) it will be wrapped in <pre><code>your code snippet</code></pre> tags.

This format won't be highlighted by SyntaxHighlighter by default however.

Without wanting to modify SyntaxHighlighter my solution to this was to use jQuery to transform all instances of:

<pre><code>code snippet</code></pre>

rendered on the page to:

<pre class="brush: c#">code snippet</pre>

After this has been done, SyntaxHighlighter will be able to format the code snippet as it usually would.

I’ve created the following jQuery snippet to do this (which also goes to demonstrate the power of jQuery):

	//Find all <pre> tags that have a <code> tag child, and add the "brush: c#" class to the <pre> tag.	
	$('pre:has(code)').addClass('brush: c#'); 
	//Find each <code> tag (with a <pre> parent), and replace it’s contents with itself leaving only the <pre> tag behind. 
	$('pre>code').each(function(index) { 
		var cnt = $(this).contents(); 	
		$(this).replaceWith(cnt);    
	});  

The end result being you will have beautiful syntax highlighting or your code snippets with a simple click of the WMD Editor's 'code' button (or the Ctl-K hotkey).
If you wish to use another brush, you can still explicitly define a <pre> tag as normal specifying a different SyntaxHighlighter brush.

ASP.NET Wiki Control with Markdown

I am hosting an open source project that wraps the WMD Editor and MarkdownSharp projects together into a convenient wiki control over Linq to SQL that you can embed in any ASP.NET website (also with special instructions for installing into BlogEngine.NET for a better page editing solution).

I have just updated the latest version to include SyntaxHighlighter integration and will use javascript to detect for the presence of jQuery and SyntaxHighlighter and, if found, will run the jQuery code mentioned above to format your code snippets automatically through SyntaxHighlighter.

More information on the control can be found on the ASP.NET Wiki Control with Markdown download page.

Background

I have been involved with ASP.NET deployments for some high load scenarios where the same exact copy of a web application is deployed to multiple servers and placed behind a network load balancer (NLB) – also known as a web farm.

Requests to the website first arrive at the NLB, and are then directed off to a web server that is available to handle the request (the web server allocated depends on a number of load balancing algorithms the NLB server can use).

When deploying an ASP.NET application to a web farm for load balancing there are a few things to consider.

Session State

Session information in ASP.NET simply speaking is the ability to associate some information against a particular user as they are clicking through your site by using the .NET Session object. Behind the scenes the user session is tracked through a single non-persistent cookie (removed when the browser is closed) named ASP.NET_SessionId. The cookie contains a single ID that associates that user back to their session information on the server.

The actual information can be stored on the server in one of three ways – but only two of which will work when deploying to a web farm:

InProc – This is the default. InProc stores the session information in memory of the ASP.NET worker process on the server. This cannot be used with load balanced servers since if the user is sent to another server on a subsequent request, that server will not have that user’s session information available in its local worker process memory space.

StateServer – With this option, all web servers in the web farm store session information in the memory of a single process (a Windows service called “ASP.NET State Server”) preferably on a separate machine. This option works for web farms, but is 15% slower than InProc. This however of course is offset by having the extra servers processing requests.

SQLServer – Using SQL Server, session information is serialized and stored to a SQL Server database. This is 25% slower than InProc, however SQL Server can be setup as a failover cluster which has reliability advantages over StateServer which is a single point of failure. This option can also survive a SQL Server restart; if StateServer on the other hand is restarted all session information will be lost.

Whether you choose StateServer or SQLServer is a trade-off between reliability and speed.

Machine Key

ASP.NET hashes ViewState by default to ensure that it hasn’t been tampered with before being sent back to the server. Out of the box the key used to generate the hash is a random one generated when the application pool starts. There is also a decryption key that is used to encrypt forms authentication tokens (also auto generated).

This has two problems:

1. If the Application Pool restarts (by sysadmin or automatically through inactivity or scheduled recycling) a new validation/decryption key will be generated, and a user on a subsequent post back will receive a ViewState validation error or be logged out.
2. If the user is sent to another server in the web farm on post back, the other server will have it’s own validation/decryption key, and the user will receive a ViewState validation error or be logged out.

So in order to work with a web farm it is necessary that all servers in the farm use the same validationKey and decryptionKey values and not have them auto generated.

A good codeproject article describes how to do this here http://www.codeproject.com/KB/aspnet/machineKey.aspx

<machineKey 
validationKey="E36B… CE0260028"
decryptionKey="DEF1F… 7F7C0BCCF85"
validation="SHA1" decryption="AES"/>

Also note that it is possible to encrypt ViewState (instead of just making it tamper proof) by setting the validation property to 3DES. This however incurs a performance hit so it is better to store sensitive information in the Session to avoid having to do this.

Storing / Sharing Files

Another problem exists when dealing with file uploads / downloads (when using the file system and not storing files in the database). All servers in the web farm need to be pointing to a common directory for the file store, otherwise if they are pointing to local directories the files will be split across them and sometimes exist and sometimes not depending on what server the user was sent to.

This can be overcome by setting up IIS Virtual Directories on the websites in the web farm that point to a common network share (on a file server somewhere on the network).

In order to test this concept locally for this blog post, I first created a directory c:\Test and turned on network sharing on this directory so that I can now browse to it through the UNC path \\HAEMOGLOBIN-PC\Test (in practice this would point to a share on another machine).

Within IIS7, the virtual directory “Uploads” can then be added that points to this share and will look something like this:

This allows the network share location to be maintained and controlled by system administrators – the only other thing to be aware of is setting up the correct permissions for the file share to be accessed and written to. This will depend on whether identity impersonation is enabled or what account the anonymous user is running as.

Coding against this is simple, saving a file is simply a process of using MapPath and the virtual directory name (this allows the system administrator to change the UNC location at a later point through IIS):

FileUpload1.SaveAs(MapPath(@"/Uploads/" + FileUpload1.FileName)); 

The above will be saved by IIS to \\HAEMOGLOBIN-PC\Test (as configured before and this could be on another machine) which in turn resolves to c:\Test on that machine.

In terms of providing a link to download a file - it would simply look like:

<a href="/Uploads/uploadedFile.zip">Uploaded File</a>

This will (once again through the IIS virtual directory mapping) download the file from the common file share location on the network and work for any server the user happens to be on in the web farm.

Hope this helps :)

I don’t know why I had the idea, but recently I thought it would be nice to just jot down a list of all those little interesting life memories I have stored away.

Sort of like a really zoomed out diary of your life, it’s been quite fun to do. I don’t really have time to do it, so every now and then I’ll just write a few more things down (it is a task in my @Downtime list in my previous post). I haven’t put any particular structure around it and it doesn’t have sentences at all, just one word bullet points to remind me of some interesting thing that has happened that I remember fondly.

For example, maybe just the name of a school you went to – and that is enough to bring back all those memories about the place that you wouldn’t even normally think of, and possibly even forget one day I guess??

If there are further particular little interesting memories I want to include about something I might have a little sub-list of things under that. One word is normally enough to trigger everything else related to that funny/fun/interesting/nice memory.

I’ve been trying to remember past holidays I’ve been on, jobs I’ve had, fun memorable little trips around home I’ve been on, places I’ve been to with cool / fun people. Something I’ve achieved or a hobby I used to do.

Best thing is not to think it’s a big task, just have fun while you remember stuff.

If you decide to give it a go - enjoy :)

In a similar vein to my previous post Project Managers Bridge to Agility I have recently stumbled across another nice article – this one from Pete Deemer at www.scrumtraininginstitute.com.

These things always ring so true – yet in practice it’s so hard for a company to do let go and trust in their team’s ability, especially when under pressure.
You really need to have all players sold on the belief that agile development does work.

As Jeff Sutherland mentioned in a seminar of his I went to last night, these things are so simple, but not necessarily easy.

Excerpt from MANAGER 2.0: THE ROLE OF THE MANAGER IN SCRUM below:

The traditional role of the manager in the corporate world is based on a model known as “command and control”. Here, the job of the manager is to identify what needs to be done, to issue detailed instructions to the employee, and then to ensure the employee completes the work according to the instructions. The role of the employee in this model is simply to follow the directions as given, trusting the judgment and wisdom of the manager to ensure that the right work is being done in the right way.

In complex, dynamic environments such as software development, this approach tends to break down. First, it is difficult and time-­-consuming for a manager to understand every requirement in full detail and issue precise instructions to guide the work of every employee. Within a software development Team, the work is highly interconnected, with intricate dependencies, and frequent change and surprise. To expect a single manager to do all the basic thinking for his or her team is unrealistic, and it often constrains the team’s productivity to the manager’s ability to give instructions. In addition, this approach tends to be demoralizing for employees; their role is simply that of “order follower”, and they often feel little sense of ownership of their work. Accountability is limited to answering the simple question, “did I complete the orders I was given?” If the answer is “yes”, the job has been done – regardless of whether the right thing was built, built well, or built to satisfy the business goals of the customer.

Scrum is based on a different approach: The Self-­-Organizing Team.

The Galaxy Team had been doing Scrum for several months, and the Team was
well on its way to being truly self-organizing. Their motivation was high, they were
focused, and after a few Sprints of under-delivery, they were now showing a
pattern of making reasonable commitments and delivering them 100% each
Sprint. Morale was high, and there was a real sense of “flow” in the work they
were doing. The engineering manager Francis had come a long way – once a
habitual micromanager, he was now acting like much more of a mentor and coach
for the Team. Unfortunately, though, in the eighth Sprint, the Team encountered
some unexpected difficulties, and about halfway through the Sprint, they were
significantly behind in their progress. The VP of the group, Simon, had ventured
into the Team’s work area to see their Sprint Burndown Chart, and called Francis
to his office.

“Francis, it looks like this Sprint is a disaster. What’s going on?” he asked.
Francis responded, “Well, the Team hit some bumps along the way, and they’re
trying hard to get everything done that they committed to, but it’s a bit touch-andgo
right now.”

Simon grimaced.

“Francis, this project is critical, and we can’t let it fall behind. I’m counting on you
to make sure the Team finishes what they commit to, this Sprint and every Sprint.
As a manager, your job is to make sure the Team gets it done; if things are going
well, then you can back off a bit, but the minute the going gets tough, I want you in
there making sure that no time is being wasted, and everyone is doing exactly
what needs to be done.”

Francis was exasperated. Simon had been too busy to attend the in-house Scrum
trainings, but Francis had emailed him a Powerpoint presentation about selforganizing
Teams and the new role of the manager, and Simon hadn’t voiced any
disagreement. Francis spoke up:

“But what about the self-organizing Team, Simon? What about our shift away from
micromanagement?”

There was a glimmer of recognition, as Simon recalled a Powerpoint he’d seen a
few months before.

“Yes, the Team is responsible, but when they start to fail, I hold you responsible.
We want maximum accountability, so I’m holding them accountable and I’m
holding you accountable. In our department, everyone is accountable! Now make
it happen.”

At that, Simon spun his chair around and started typing. Francis took the hint and
left the office.

The next day, Francis showed up at the Daily Scrum Meeting.

“Guys, we’re going to do a different format for the meeting today. Due to the
criticality of this project, Simon has instructed me to more actively… uhhh…
‘facilitate’ your self-organization during the Sprint. So what I’d like to do this
morning is get a status update on each of the features you’ve committed to –
whose done what so far, and what’s left to be done – and I’m going to be giving
some more detailed feedback so hopefully we can get everything 100% finished
by the end of next week.”

The Team looked at each other. Philip, the ScrumMaster of the Team, spoke up.

“Francis… uhhh… does this mean that the Team is no longer responsible for
managing itself?”

Several Team members nodded in agreement.

Francis replied, “Guys, we’re all responsible. You’re responsible for managing
yourselves, and I’m responsible for making sure you get everything done. We’re
all responsible together!” Francis didn’t see the eyeballs subtly roll.

As the Sprint proceeded, Francis was more and more involved. The Daily Scrum
became an update meeting for the Team to tell Francis what they’d been able to
complete, and for him to assign them the next day’s tasks. The mood of the Team
shifted; motivation seemed to go down, and Team members seemed to be
reverting to their previous role, what they used to sarcastically call “servants-of-
Francis-the-Great”. By the end of the Sprint, the Team was fully back into “orderfollowing”
mode, and Francis was directing their efforts task-by-task.

At the Sprint Review, the Team was surprised when Simon joined the meeting just
as it was starting.

“So…” Simon announced, “Did we get our commitment done?”

The Team looked at each other. Francis answered.

“Simon, unfortunately there are a couple backlog items that weren’t finished.”

There was a flash of anger in Simon’s eyes.

“How did this happen? Who is responsible for this?”

The Team was silent, but their heads all turned slowly to Francis.

Simon continued. “Francis, I told you to get it done. Next Sprint, I don’t want to
see this happen again. If it does, there will be hell to pay…”

Upon hearing this, everyone on the Team made a mental note to think very
carefully about just how much to commit to in the next Sprint. The last thing they
wanted was to get shouted at again two weeks from now.

As the Sprints passed, Francis became more and more involved in directing the
Team at every stage of their work. Gone was any semblance of self-organization,
and with it disappeared the improved motivation, drive, and focus that the Team
had started to display. Morale had plummeted, and so too had productivity. Lunch
breaks were getting longer, coffee-breaks more frequent, and Francis felt like he
was spending more and more of his time just making sure people were at their
desks working. Those amazing few Sprints, when the Team was truly selforganizing,
and performing at the level they were really capable of, were
becoming more and more of a distant memory. The return to micromanagement
was made all the worse because they’d had a taste of the self-organization “good
life”.

Information overload – something that is very real when you are in the tech industry. One way to manage this and solve this problem is to simply ignore it completely. For instance, don’t read RSS feeds, don’t have a twitter account, poke into your email inbox once in a while to see if anything new and interesting is in there and use your inbox as a way of reminding yourself of things that you have to do.

Maybe there isn’t anything wrong with this approach. These people tend to live outside of technology and run their lives from conversations and what they have on their mind that they believe is the right thing to do at any one time. You quite often wouldn’t hear back from them if you sent them a message – so you may need to rely on ringing them for a definite response, which can be frustrating. But as I said, maybe there isn’t anything wrong with it. People just operate differently – and that’s fine, we just have to acknowledge that people do things differently and we have to change our approach to fit with how others go about doing business depending on who we are dealing with. You can’t make the world conform to you, unfortunately. (For example, I wish more people used twitter which I think is a great tool. The uptake in Australia / New Zealand is poor even between tech people, but there is not much you can do except maybe wait :)

I’ve decided myself that I don’t want to simply block/ignore information as a way of surviving, but I also don’t want to be swamped by it, so over the years I’ve been trying to refine my daily workflow process in an attempt to manage “information flow” (a.k.a information overload) effectively and efficiently.  It’s based a lot of GTD (Getting Things Done) which I highly recommend anyone read – the book is obviously written in fairly general terms (so everyone can benefit) and it’s really left to everyone to find their own concrete “implementation” of it. The concepts you come to realise in the book really turn on a few light bulbs that help you go forward with for the rest of your life. David Allen has since written a sequel to this book, “Making It All Work” which is definitely on my to-read list.

Through trial and error I’ve tried a few different things but am starting to get to the point now where I am pretty happy with the whole process – although I think it will always be a continually evolving thing, especially as your life changes. I have mentioned the tools that I am using to implement the process also, but tools always change too – even more so than the process. I realise that there are many other valid tool alternatives out there and new ones are always cropping up – this is just to document what I’m using right now.

My Daily Workflow       

Diagram Explanation

“Stuff” is everything that bombards you from day to day.

• Email – I tend to just process these throughout the day. For people who feel a sense of email overwhelm though, some suggest disabling automatic send/receive and processing these at certain times of the day (to avoid distractions). I believe in “inbox zero” philosophy – where read emails do not sit in your inbox. Once read, a decision is made whether it is deleted, it requires an action that is done right there (i.e a reply < 2 mins), converted to an action (or put in a @ToReply folder), delegated, filed, or deleted. The key concept is that e-mail is simply a mode of transport, nothing more and that the inbox isn’t for storage. I also make use of the MS Outlook follow up function for sent emails where you want a reminder where it is important that you receive a reply.
• RSS Feeds – These are a great way of keeping up with what’s happening out there in the world around you, especially with technology. There is a problem with them though – they are overwhelming, especially as your number of subscriptions go up (I’m at 233 subscriptions right now which I actually quite easily keep on top of). If you don’t read them for a week – you are more and more disinclined to want to even look at them because of the share number of updates. I use Google Reader which I have in “list” mode – this is where you only see the titles of each post. I also have each feed in an appropriate category. What I do now is literally spent 5-10 minutes of the day in the mornings (sometimes less) going through each category and scanning the titles. If one interests me, I may expand it and have a quick read – if it’s something that will take a bit more involvement to read (and I still want to read it) I will add it to ReadItLater with its Google Reader plug in. Aside from that, I will simply hit the “Mark all as read” button. That way if there is something really good, or something new going on – you will see it, but you aren’t going to spend your life reading RSS feeds. Important difference! I’ve recently added the free MobileRSS application to my iPod Touch that syncs with Google Reader and allows you to read offline, as well as add items directly to ReadItLater.
• Facebook / Twitter – I manage both these through TweetDeck, but especially for twitter. At the moment I still tend to prefer the facebook webpage interface in conjunction with www.fbpurity.com in order to hide all those farmville and fortune cookie requests etc. Nevertheless it’s still handy to be able to see updates from TweetDeck if you wish. You can manage the level of notifications you want TweetDeck to give you – depending on what sort of distractibility level you want. You can filter a twitter “column” for example for your close friends that will display a full popup notification, and have the rest display a smaller notification or none at all. Things like that. I also use Echofon on the iPod touch for twitter processing.
• Life – Life of course has many other inputs, phone calls, random thoughts, talking with people etc. The main notion with GTD is to keep things out of your head and stored in one location – where you can review everything in context with everything else. If you try to keep everything in your head you will feel stressed, constantly be thinking about things in order to “remember” them, and become less effective.

The key point to the process is - every input requires a decision to be made, right there and then.

Is it somewhere I need to be?
Then the calendar is where it goes and then the item deleted. A lot of events are organised through facebook these days so I make use of the “Export” functionality on the facebook event page that takes it straight through and into Outlook as an appointment.

Is it something that I need to do something with – i.e an Action?
If not, I decide whether I want to trash it, or store it somewhere for future reference. Where I store this depends on what it is. If it’s a bit of paper it will go in my filing cabinet, an electronic file on my file system (backed up with DropBox if it’s non-media), stored as a reference item in MyLifeOrganized (which I will talk more about soon), in an email subdirectory if it’s an email, or if it’s a link I will use Delicious to store it, which will then sync back to a few pages on this site. If it’s a web article that I want to read (which may or may not result in a further action) but don’t have the time right now, I add it to ReadItLater. Once upon a time I let this grow to over 100 items. This is bad. Very bad. Like the RSS feeds that you leave to fester, you will never want to look at this list if it is this big. I keep these processed and back down to 0 weekly the best I can.

If this item is something that I need to do something with, GTD philosophy is to do it right then if I can do it in less than 2 minutes. Otherwise, it is going in my “to-do” list and prioritised alongside everything else. For this I have been using MyLifeOrganized. This tool is very flexible and there are many ways in which you can use it. I have taken to using a combination of Weekly/Monthly/Yearly goals as well as Contexts (another GTD term).

My first decision is what goal I should assign to the item:

• Week – Items assigned a weekly goal are essentially the ones that I have in view each day. This isn’t a hard goal where I punish myself if don’t have them all done by the end of the week, but allow them to roll over. It’s more of a focus thing than anything else. For agile methodologists, I guess this is more like the Kanban/Lean way of doing things with flow rather than Scrum’s fixed iterations. For work projects though I think fixed iterations are still good.
• Month – Items which will most likely become weekly goals within a month (or two). There is still a sense of time importance with these.
• Year – Items that I really don’t need to think about for quite a while, still want to do at some point but definitely have on the back burner right now.
• Downtime – For items that I don’t assign a Week/Month/Year goal, I have set these up to be visible in a view I call Downtime. These are typically tasks that are non-important and non-urgent – but still things that you would probably do before Year goals. Could be things that you can do while watching TV and may tend to be more @Fun – it does not matter if these are not done in any particular time frame.

Doing this is a way of prioritising what you think is important for you to be looking at and what should be shielded from your thoughts for now.

The next step is to set a context on the item. These are very personalised in my opinion. Everyone is going to have these looking different. The GTD book suggests these are things like @Phone, @Car, @Shops, @Home, @Work…. You get the picture. The idea being you filter the list based on the context you are in at the time. At this stage in my life I didn’t find much benefit from this so my contexts at the moment are more to do with the area of my life they belong in (which still helps me choose items off the list when they are grouped this way). It’s all very flexible, these vary a lot depending on your lifestyle.

MyLifeOrganized operates by allowing you to define different “views” on your master all encompassing task list. I have created a week view that displays anything that has a weekly goal assigned to it, or any item that is now “due” with the due date property (the items are also grouped by the context I assigned to them). The monthly view shows all monthly goal items and the yearly view in this case I have grouped by project, another GTD/MyLifeOrganized concept (projects are made up of tasks). 

I essentially only ever have the weekly view open at any one time (or maybe the Downtime list depending on the time of day / energy levels), unless I’m doing a “weekly review” which is an important GTD task conducted weekly where you look over all of your items. This is where I shuffle weekly / monthly / yearly goals to make sure my weekly list is current. It is important to empty anything that you have on your mind into the system at this point so it is all there in front of you to view and prioritise alongside everything else.

It sounds complicated but I’m feeling quite effective now. I would be delighted to hear how other people go about dealing with “stuff” in case there is something I can borrow and use myself :) We are all in this fight against this world that’s heading for information overload meltdown together :) I believe, with the right simple process that you eventually internalise and don’t need to think about, it becomes quite easy to manage information without having to block it out entirely.

Note that I’m not necessarily saying that you should use or invest in any of the tools that I have mentioned in this article. I just wanted to share what I’m currently doing with the idea that it might give you (the reader) some ideas about how you could tweak your own setup. Everyone’s circumstances / lives are different and everyone is going to want to do things slightly differently. This article just gives you my version :)

  A lot of the systems that I have worked on have involved more than one component. For example, instead of just an ASP.NET website, there would also be some sort of windows service, maybe a windows forms component, or independent modules and libraries working within those components – all working together to form a final solution.

This tends to create a bit of a versioning and deployment nightmare and you need to have a good solid strategy around it before launching into things. I have tried a couple of approaches.

For one project, I had an automated build script that would go through and modify the version number of all the assemblies, build all the components and label the version from the root in source control. All components would then be redeployed for every release, regardless of whether they have changed or not. This removed human error in deployments as everything was automated, and the deployment steps were always exactly the same for each release so even those could be automated.

This worked pretty well for this project, because there were so many developers that it would have been too time consuming to do a full version to version diff to figure out which components have changed for each release and to deploy only those. Yes there would be release notes on what has changed for each of the components as written by project management or the developer, but that would be reliant on how dutiful Mr developer happened to be at the time, and it’s possible little “tidy” ups were done, or refactorings in components that do not end up appearing on release notes. So one answer is just to test and redeploy everything (this is where automated testing can be very useful also to test for regressions across the system). The only problem with it is you potentially end up deploying a lot of unchanged stuff – albeit just with updated version numbers, all for potentially one small change in one of the components.

For another project – there were far fewer developers, and it was quite easy to just know what components were being changed for each release – so it was possible to just deploy those that were being updated. I came up with the concept of a system version.

Say if you have the following structure in your source control repository for your software solution:

• MySolution
  • ASP.NET Website
  • Windows Forms Application
  • Windows Service
  • Reporting Services Reports
  • Database Scripts

When you deployed the first version to production, imagine everything was versioned at 1.0.0. All assemblies / exe’s would have been deployed as version 1.0.0 (or using other techniques for database scripts like updating the database version row to this), and the label Version_1_0_0 (or similar) then applied to the MySolution root in source control.

This is in contrast to labelling each subcomponent separately in source control and keeping a configuration management register updated with what versions are deployed where and what versions are all working and tested with what other component versions – uggh.

Now – say that a bug in the windows forms application is reported – it makes sense to just send out a new version of just that, not everything. You make the change, and update the windows forms application’s version to 1.0.1. Now, the solution is is labelled with Version_1_0_1 from the root – where 1.0.1 is effectively the new system version. Note however that the rest of the sub-components are still set at version 1.0.0 (literally within the AssemblyInfo.cs classes) – but the 1.0.1 label spans across them, effectively grouping the various current sub-component versions under the 1.0.1 system version umbrella.

Now – lets say a whole lot of work is done across all the different components to create a new feature asked for that will become version 1.1.0. During this time, another small issue is found in production – this time in the windows service. Since we are in the middle of development of version 1.1.0, which is subsequently unstable – we must branch the solution from version 1.0.1 (using the source control branching functionality). Here we make our change to the windows service, update it’s assembly version to 1.0.2 and then label the root of the branch as system version 1.0.2. We then deploy just the windows service to the client.

If we were deploying this solution to multiple clients, who for some reason were running on different versions, you can easily recreate in the development environment the version that they have running in their environment as long as you know their system version. Note that this must be recorded somewhere – but in this case it’s just one number per system deployment, as opposed to a complex set of versions of all the components installed. Pulling that label out of source control will retrieve all the subcomponents in their correct version as deployed for that system version, and consequently retrieve exactly what is running in production for each component.

The only danger with this technique is the possibility of a component being changed (maybe a developer deciding to re-factor some previous work), and not having this component deployed along with the next system version. This would effectively create a mismatch between what is deployed and running on the client’s system and the code associated with that system version in source control. This creates unreproducible bugs and other issues. To solve this, a quick and easy technique should be devised to query source control to see what components have been “touched” since the last release.

I would be interested to hear how others go about versioning in these sorts of situations. I may make a follow up post at some point about database versioning, and UAT deployment techniques.