Yesterday's news

I keep getting asked, why can’t the Kanban board be electonic? “That’s more flexible” they say, but what I think they mean is “we feel funny having to stand-up, write stickies, and paste them onto the wall”. Since most information management tools today are electronic (excluding, perhaps, only meeting notepads), it’s natural to assume that electronic Kanban board is more practical.

However, the physical nature of the board is designed to address the shortcommings inherent into electronic todo lists and status tracking systems:

1) With electronic lists, sooner or later somebody will create another list, for the stuff that “doesn’t really belong on the main one”. This may immediately kill the Kanban idea of prioritizing all work items against each other and visualizing all work in progress. With physical board, there’s only one place the items are tracked on: the board. The burden of creating another physical board is high enough to prevent accidental proliferation. You could say that the physical board does not stop people from creating electronic todo lists. That is true, but their transient nature will make them so obviously secondary that primary status of the board will not be doubted. At the end of the day, we may still have to mandate the use of the single tool, whether physical or electronic, for all prioritization and tracking activities.

2) Electronic todo list, or a work item tracking system, has unlimited space, which means creation of too many pending items is not as discouraged as is the case with the limited physical space of the board. Limited space models limited capacity. When bottlenecks caused by limited capacity are not visualized, nothing can stop proliferation of low-priority pending work items, most of which are never worked on, or spend most of their life time in waiting-to-be-gotten-to state. Truth be told, some of the tools, e.g. agilezen.com, help you solve this problem by allowing you to define limits on number of tasks in each state.

3) With electonic todo lists, as with other digital tools, people spend most of their time alone, sitting at their desks, staring at the monitor. Planning and prioritization-related communication is, effectively, discouraged, since it’s either done via the limited media of the email, or requires scheduling team meetings. With physical board, the team members will find themselves having to go to the board, which encourages spontaneous face-to-face communication. Hell, you may even meet other people there, either talking about the same planning/prioritization problem (after all, it’s the same queue), or being able to give you a hint on yours. The only way this can be replicated in the e-world is by making it accessible only via some kind of giant touch screen. But that defeats the purpose of having the board be digital in the first place. Also, why pay thousands for what you may get for free?

4) Finally, with electronic lists, once the status is discussed, and prioritization decisions are made, people have to go back and spend more time to update their items, only to loose some details,or to find that some of the other pressing priorities were left out of the conversation. With physical board, the stickes are moved around, or added, as people speak, for immediate status update. Again, with electronic list this could be solved by having the manager update the status and shuffle the items around during the meeting, but this is not very scalable and quickly gets out of hand in practice.

To summarize, today there are tens, if not hundreds, products out there simulating Kanban board in electronic form (see agilezen.com for just one example). Some of them help you solve problem #2 above. But do you really want to pay money for the system that replaces the warmth (and effectiveness) of live conversation with lonely clicking of the mouse? I hope not!

I just had an epiphany. I realized how prioritization problems in IT organizations are related to people’s tendency to avoid personal responsibility.

First, let me describe the problem. In IT organizations I’ve worked in, or with, or heard about (and I can’t think of any specific reason to question my assumption that most other companies are the same) there’s a great deal of juggling going on. Also known as firefighting and multi-tasking. It’s not just multi-tasking that is indicative of an issue (that’s a subject for another rant), but its ad-hoc nature. By “ad-hoc” I mean that there’s usually no well-defined process nor guidelines nor accountability structure for making prioritization decisions.

You could argue that some prioritization is always done. We (sort-of) prioritize PROD tickets against each other based on their urgency level. We (sort-of) prioritize enhancements to be included / excluded in the release based on LOE. We even (sort-of) prioritize different projects against each other as they converge on single “resource” (developer). Yeah, that’s kinda true, but only to a certain degree.

In the simplest case, when developer works on single project, and all work is funneled to that dev through his direct manager, prioritization is usually done. Let’s leave aside exact methodology for now, at least it is being done. Although, let’s remember that even in this simple environment typical manager will try to get away without making explicit prioritization decisions. If dev asks, which one should he work on, the manager will try to respond with something vague, like “this one is more important, but you should not stop your work on another one”. If dev asks which specific task should he work on now, the manager will finally be forced to make the call, so he will make it. More on the how later.

In more complex cases, when developer switches between maintaining old projects and working on 1-2 new ones, the manager is no longer the only channel the work arrives on. There are usually some requests for assistance from colleagues. Some action items from meetings with stakeholders of other projects, especially if meeting does not include one’s direct manager. Some PROD tickets and/or emails directly from Help Desk or Support Desk. Some requests directly from users (this is typical if users are internal “business” users). Some personal requests from manager’s manager and other higher-ups (“hey, Jack, can you think about this in your spare time?”). Et cetera. This multitude of work gives manager good cover to avoid making prioritization decisions. Indeed, unless developer is aware of, concerned with, or has burned himself on, the topic of prioritization, it’s much easier for him to assume priorities based on his own limited understanding of matters than to go and ask his manager every time such question arises. Add to this that managers are usually busy in meetings, and that many of the requests are of “we need this yesterday” kind, and you will understand why developer might rather shut up and work, than go seek his manager to make that prioritization call.

Even if he is full of determination to catch his manager and press him to make an actual prioritization decision (instead of “they’re all equally important, just split your time between them”), once he has tried it several times, he will eventually fall back to the implicit way. Why? Because from his perspective the decisions are made in a very inconsistent fashion. What looks like random guessing though, is in fact the application of three most popular prioritization methods: “squeakiest wheel gets most grease”, “cover my butt”, and “kiss-up to boss”. Which gets us to the actual topic of this post: avoiding personal responsibility.

– “Need to decide between these two? Let’s ask the business.” — transition the risk of decision making to the other department’s head
– “This one is clearly more important. The client complained about it” — who will argue that the client’s needs are foremost?
– “This is Bill’s personal request” — clearly the CIO won’t fire you for making his request a priority
– “Everybody knows this is a pain, so let’s do it first” — something widely known always wins over something obscure, just because it’s easier to justify and find support for

Why do this happen? Very simple: people are afraid. Afraid to show their lack of professionalism, afraid of being laughed at, afraid at their decisions being challenged, afraid of making a wrong decision and getting reprimanded for it. People are much more interested in their job security than in the performance of their organization. Naturally, taking responsibility for decision is percieved as risky.

How can we fix it then? In my opinion, by introducing a well-defined, almost mechanical, process that formalizes decision-making procedure. Most importantly, making sure this process gets buy-in and support at the highest levels of the organization (i.e. “the business” executives). Some people get freaked out at this point, thinking that taking responsibility off people and defining the rigid process kills accountability and agility. I say: it does not have to, if the process itself is agile and adaptable, and the people are made accountable for the overall performance of the process, not for individual prioritization decisions. More on this in a future post.

Making real prioritization decision is hard. It requires evaluating both tactical as well as strategical factors. And what is the most important strategical factor? Pleasing the client and the boss. Just kidding! It is, evolving the platform for stable sustainable growth of the business. Does it involve PR and politics? Yes. But don’t let the tail wag the dog.

Trying to enumerate major classes of issues caused by lack of coordination in migrating SQL changes between the environments:

1. “Incompatible Changes”: Adam makes a schema change for release X. Bob makes a schema change for release Y. When release X is deployed to Testing, Adam’s change is applied fine but when release Y goes to Testing, Bob’s change fails since it was not designed to be applied on top of Adam’s. If two releases are tested in different environments, this issue may not be found until the integration testing of release Y.
2. “Broken Script Sequence”: As schema change scripts are developed, they may go through multiple rounds of modifications. If the sequence of scripts is never ran in its entirety, any incompatibilities between the individual scripts may not be discovered until entire sequence is deployed to PROD. A variation of this issue occurs when all or some scripts are not numbered and therefore may run out of order.
3. “Reversed Change”: Adam changes SP1 for project X but only stores it to project X-specific script folder. Bob changes the same SP1 for project Y. Adam’s change is deployed first. When Bob’s change is deployed, it reverses Adam’s change since Bob’s change used older version of SP1 as its basis.
4. “Premature Change”: Adam changes SP1 for project X and stores it in the common location called “BASE”. Bob takes SP1 from BASE and adds changes for project Y. This time around Bob’s change is deployed first. However, since it also includes Adam’s changes, it breaks project X, since related binary changes are not yet deployed.
5. “Incompatible Changes (Direct PROD Modification)”: Adam develops a schema change using DEVDB as the base. Meanwhile DBA makes some changes directly in PROD. When Adam’s change is deployed, the script fails since it’s not compatible with changes made directly in PROD.
6. “Change Fails When Applied Second Time”: As changes are deployed and redeployed to Test, some of them may be applied more than once. Usually this is solved by adding IF conditions to the script that verifies whether the column or table already exist. However if you think about it, this scenario is really a specific case of Broken Script Sequence.
7. “Duplicate Metadata”: As issues are found and fixed in Test, some scripts may end-up applied more than once. While schema-changes tend to fail fast, and SP changes can usually be applied on top, meta data changes end-up getting applied twice, and thereby duplicated. Usually this is solved by adding IF conditions to the script that verifies whether the meta-data already exist. However if you think about it, this scenario is really a specific case of Broken Script Sequence.
8. “Script Partially Completes”: Each individual script file usually has multiple schema or data modifications in it. If one of those modifications fails due to Incompatible Changes or Broken Script Sequence, the script keeps running on with some of subsequent scripts failing. This results in the database being in an unknown state, and requires manually intensive labor to research and repair.
9. “SQL Server Dies of SP Recompilation”: If all SPs in the database, whether modified or not, are simply (re-)applied during deployment, SQL Server may become slow or irresponsive due to SP recompilation.
10. “Lost Change (manual version)”: As issues are found during Testing and fixed, new scripts may be created by developers and added to the script sequence. The person deploying the scripts may mistakenly think that a script has been deployed before, and not deploy it, resulting in a lost change.
11. “Lost Сhange (automated version)”: The team is using an automated tool that tracks which scripts have been deployed. As issues are found during Testing and fixed, some scripts may require modifications. If developer modifies a script that has been applied before (instead of adding a new one), the change will not get deployed, since this script will be skipped by the tool as previously deployed.
12. “QUOTED_IDENTIFIERS”: If some scripts do not specify values of certain SQL options (e.g. QUOTED_IDENTIFIERS etc.), or specify different values, the tables and queries may be created with conflicting settings, which may result in hard to debug errors.

Will add more here as I discover them.

If you’re building a web app or web site project using an automated MSDeploy script, in addition to binaries, you want your content and config files also published to the output folder. On the desktop, this is done by using Visual Studio’s Publish wizard. After two days of searching the web, here’s how I solved this:

<MSBuild Projects="$(teamcity_build_checkoutDir)\XXXSource\xxxServices.sln"
Properties="Platform=$(Platform);Configuration=$(Configuration);
DeployOnBuild=true;
DeployTarget=Package;
PackageLocation=$(teamcity_build_checkoutDir)\Output\$(Configuration);
PackageAsSingleFile=False;
AutoParameterizationWebConfigConnectionStrings=False">

As you can see, I’m using MSBuild, specifying the platform and configuration. The Publish part is done by setting DeployOnBuild and DeployTarget properties, and specifying PackageLocation. Other parameters are specific to my situation so you may not want to set them (read on for more details).

Basically, this solution uses new VS2010/.NET 4.0 WebDeploy-based publishing methodology, you can read more about it here and here. The first link describes generating WebDeploy package via MsBuild script. The second link describes using DeployOnBuild parameter to deploy the package to the web server as part of the normal build process (as opposed to invoking MsBuild second time, as described in the first post).

In my case, I don’t want to deploy the package right away, but I do want to build the package as part of my normal build process. The novelty part in my solution is combining the DeployOnBuild approach with the Package target to get the best of both worlds.

Now, regarding other parameters, PackageAsSingleFile=false indicates that the package content should not be compressed to a zip-file (you can also set this directly in your web project’s Package/Publish properties). This is useful to me because I don’t intend to actually install these using WebDeploy, all I need is a set of binaries/content/config I can take and deploy onto my servers.

The last parameter (AutoParameterizationWebConfigConnectionStrings) is used to disable auto-parameterization. I need this since my script updates Web.Config with environment-specific settings in between the multiple configurations being built. So if I don’t disable auto-parametrization, the web.config file will be locked, and I won’t be able to write to it. Here’s the error I was getting before I disabled auto-parametrization:

error MSB3021: Unable to copy file "C:\BuildAgent\work\WebService\web.config_build"
to "C:\BuildAgent\work\WebService\web.config".
The process cannot access the file 'C:\BuildAgent\work\WebService\web.config'
because it is being used by another process.

Received a comment this morning on tiny iterative releases… Want to clarify something:

The comment was: if our typical release these days is likely to take a whole month from inception to production (half-a-week for analysis/design, week for dev, week for testing, week for uat/bug correction, half-a-week for deployment) then making it any smaller won’t be practical as it would put too much pressure on everybody.

My answer to that is twofold.

Firstly, I’m not suggesting making releases short and keeping the same scope. If release contains only one incremental enhancement it should not take entire two weeks for testing, UAT and bug correction.

Secondly, even if releases are somewhat longish, it does not mean we can’t tile them! This way, from dev perspective, every week is a development week, plus dealing with bug from the previous week, plus deploying something developed two weeks ago, plus helping with analysis of a feature for the next week. This is how we already work today, all I’m saying is let’s legitimize/institutionalize it and put some structure/guidance/limits around, specifically:

By making conscious effort to limit the size of each “tile”, we’re ensuring that fewer number of the “tiles” will end-up getting overlapped with subsequent “tiles”, thereby reducing the multitasking overhead and helping improve the quality.

In this post I want to address most typical questions about Kanban. Entire content of this post was an email I sent to my IT group after the meeting in which I introduced Kanban.

1. why limit work-in-progress to only a few simultaneous tasks?
2. why limit releases to one feature per release?
3. how does the board help us?
4. this is all fine but what if an urgent issue comes up, won’t this game be abandoned?
5. what’s in it for me? (a developer)
6. what’s in it for me? (a PM)

1. Limiting WIP helps control multitasking. Multitasking costs 20% overhead per each extra task due to context switching, induces stress (“so much on my plate”), and lowers quality (things fall through the cracks, everything’s done in a rush). Lower quality causes rework (fixing the issues), the rework in its turn feeds the multitasking cycle. Having less simultaneous tasks than people also means people start helping each other (teamwork!).

Next natural question is: “but if we limit WIP how do we make progress on all those initiatives” — and the answer is: by making each trackable task as small as possible developers can take turns working on tiny useful items in each competing initiative. From higher level this will still be (pseudo)multitasking, but at the lower level each developer will never have more than one, small item on his plate.

2. “Why limit releases to one feature per release?”

Because it 1) solves the multitasking issue, 2) allows each feature to be tested right after it’s developed (if bugs found they will be easier to fix since dev will still be “in the context”) and 3) makes deployment simpler, faster, and less likely to break. Also, it makes prioritization and planning less big of a deal, since other initiatives don’t have to wait months to get some action. With one-feature-per-release we can make incremental improvements to each of our initiatives EVERY DAY. “So what if this feature was misprioritized as high priority? It will be done in a day or two and then we can work on the right one.” Finally, it will eventually enable Disney Line Planning, based on the average time each tiny item spends in the works. (Actual MEASURED time, not predicted time). So if we succeed in chunking our work in pieces that take 10 days each, from conception to PRODuction, and there are 10 other items in front of X, WE KNOW that X will be completed in 100 days. “Nice theory” you may say and I will say: Aren’t computers a theory? Aren’t logistics a theory? Aren’t finances a theory?

3. “How is the board gonna help?”

By making our current process visible, by making the issues obvious, by forcing all requestors check their assignments to devs against each other, since they will collide on the physical space of the board.

“But why is it a board on the wall, couldn’t it be a spreadsheet?” — Because there may be multiple spreadsheets but only one Wall. Because working with stickies is simpler. Because the Wall is in your face, therefore harder to ignore. Because writing with a pen on a media you can’t edit, helps ensure that only well-understood stuff is written (= no B.S.). Because working with physical cards activates spatial memory (=B.S.). Because the Wall’s limited space nicely models our team’s limited capacity.

4. “This is all fine but what if an urgent issue comes up, won’t this game be abandoned?”

Urgent issues will be scheduled on the queue, just like everything else. The only difference, they will skip directly to the head of the queue without waiting to be promoted from low level to medium and then to high. They will still have to wait for work-in-progress to complete. If we agree on the small work item concept, this will be less of a an issue cuz the wait time will be smaller.

True emergencies will be handled outside of this system, just like they always are.

5. “what’s in it for me? (a developer)”

You will be protected from craziness and mess, you won’t have to do estimates, your status meetings will no longer be painful.

6. “what’s in it for me? (a PM)”

If this fails, you will know you were right. If this succeeds, you will have learned something.

Most PMs in software development industry are slave drivers doubling as astrologists. Well, truth be told some of them also have client-management skills as well as specialized domain knowledge. The biggest problem, though, is that they quite often optimize their work not for long-term company success, but merely for their own promotion in the hierarchy. Indeed, as one lean software development expert once said:

One of the problems with the American style of business is not that it pursues profit above all else, but that it does not. The traditional style of corporate America is the pursuit of hierarchy above all else, and profit only appears as a necessary side effect. This is much of what I find compelling about lean and ToC: they place sustainable and profitable production as the highest goal, and organization adapts to serve that goal. Corporate America gets this backward: business exists as a franchise to fund social hierarchy and the executive lifestyle, and massive waste is created in the process.

Whether you agree or not, you can’t deny that the Lean is coming. In a few years you simply won’t find a PM job unless you know and can employ what today is still considered cutting-edge by the ignorant masses. Here are some fundamental documents on the topic.

• Famous no-bullshit no-nonsense introduction to Scrum. Describes actual process of an actual company in great details. Very inspiring and practical at the same time: Scrum And XP From The Trenches (PDF, you’ll have to create a free account on InfoQ in order to download it)
  

• Gentle introduction to Kanban, building up on Scrum fundamentals: 1. Striking a different bargain with the business.  2. Spreadsheet example for a small Kanban team and 3. Scrumban.

• A few years after he wrote the first item on our list, the same author wrote this comparison of Scrum with its older brother, Kanban (yes, Kanban has been known in Japan for at least 50 years now). In-depth analysis of subtle differences between the two helps understand when to apply which methodology and what elements from the two can be combined to achieve better result:  Kanban and Scrum, making the most of both (a PDF again)

• A 21-page official summmary of Scrum’s roles, activities and artifacts, directly from the creators of Srum. Since Scrum is often mixed with XP, Kanban, and adapted to enterprise’s situation, this document serves as the standard model to check your implementation against. The idea is to prevent people from getting disappointed with Scrum after they watered it down with other elements while still expecting the same great results: The Scrum Guide (PDF)

• A simple and well-defined methodology for prioritizing the backlog: Priority Filter

• How to handle bugs in a Kanban system: Accounting for Bugs and Rework

How often in SQL do you have to get the most recent child record of a given master record? Pretty damn often. The simplest solution is usually to use a correlated subquery (basically, a subselect inside the column list of the SELECT clause) with a TOP 1 / ORDER BY. However, this won’t work if you need multiple columns from the child table. What to do? Resort to joins and group-by’s, or perhaps, the mighty ROW_NUMBER? Not so fast. There’s a neat intermediate solution, using CROSS APPLY.

The idea is to use obscured and badly documented ability of CROSS APPLY to take a correlated subquery as its right argument. Yes, that’s right, CROSS APPLY is not limited to table-value functions. Here’s how.

Let’s say you have two tables: Products and ProductPrices. Your task is for each Product to retrieve the most recent ProductPrice. Your query would look like the following:

SELECT
   p.ProductID,
   p.ProductName,
   pp.PriceDate,
   pp.Price
FROM Products p
CROSS APPLY (SELECT TOP 1
                PriceDate,
                Price
             FROM ProductPricess pp
             WHERE pp.ProductID = p.ProductID
             ORDER BY PriceDate DESC) pp

Cool? Yeah, I thought so too!