Yesterday's news

Today I moved my Russian translation of W(p)GtR from Godaddy to my shiny new Linode VPS.
What I needed though, is to redirect people visiting the old site, to the new site.
My old site was running on Windows / IIS / ASP.NET so I could not use Godaddy’s Redirect URL feature in Control Panel which only works on Linux accounts.
To redirect those visitors hitting the folder (i.e. zvolkov.com/poignant_ru), I added default.asp with following content:

<%@ Language=VBScript %>
<%
Response.Status="301 Moved Permanently"
Response.AddHeader "Location","http://www.poignant.ru"
%>

This was easy. The trick was to redirect visitors to individual static .html pages — and here’s how I solved it.
Basically, you need to add a web.config to your folder with httpRedirect tag in it, like this:

<configuration>
<system.webServer>
  <httpRedirect enabled="true" exactDestination="true" httpResponseStatus="Permanent">
     <add wildcard="*ch1.html" destination="http://www.poignant.ru/ch1.html" />
     <add wildcard="*ch2.html" destination="http://www.poignant.ru/ch2.html" />
     <add wildcard="*ch3.html" destination="http://www.poignant.ru/ch3.html" />
     <add wildcard="*ch4.html" destination="http://www.poignant.ru/ch4.html" />
     <add wildcard="*index.html" destination="http://www.poignant.ru/index.html" />
     <add wildcard="*copyright.html" destination="http://www.poignant.ru/copyright.html" />
     <add wildcard="*help.html" destination="http://www.poignant.ru/help.html" />
     <add wildcard="*WpGtR_rus.pdf" destination="http://www.poignant.ru/WpGtR_rus.pdf" />
  </httpRedirect>
</system.webServer>
</configuration>

Works like a charm! You will notice that the site is significantly faster now.
This is because it is served by nginx, a relatively new super-speed web server.
Next week I will be moving my blogs as well.

Ok, I must admit, I’m just enjoying spending money. Then again, what kind of computer is complete until you spend a fortune? Here’s what I bought since the last post:

DVD — $70

The tough part about choosing a DVD-reader-writer is to find one that would match the front panel. I have settled on this LG WH12LS30. It’s a multiformat Blu-Ray / DVD / CD reader and writer. Perhaps more importantly, it’s front panel more-or-less matches my Corsair 650D’s black brushed aluminium front.

HDD — $65

Considering that I have an SSD as my primary drive, the main factor in the purchase of an HDD for me was noise. I hate those bird chirping sounds most hard-drives emit, but more than the chirping, over years I came to hate the hiss of a rotating spindle. That’s why I decided to go with this 1 Tb Western Digital Caviar Green WD10EARX. According to these tests by silentpcreview.com, its 2Tb brother is the quietest of all 3.5″ hard drives on the market today. Now, these tests were done before the 1Tb version was available. Given that the 2Tb version has 3 actual disks inside, the 1Tb is likely to have only 2 disks. This guess is supported by the WD spec sheet, which says that WD10EARX is 40 grams lighter than WD20EARX, requires 0.10VDC less power, and emits 1dbA less noise. From first hand experience I must say this is the quietest hard drive I have ever seen in my life.

Total cost - $1376 (plus shipping)

That’s it for today. The next up is the Monitor.

This is fun. Researching, assembling, testing my own baby computer.

Dimensions

Make sure your components will actually physically fit together.

First, make sure your case is big enough for your Power Supply and your motherboard. You can get dimensions of both before you by them.

If you have a non-standard CPU cooler, like I do, make sure it will fit in the case height-wise. Make sure the memory will fit with the cooler installed.

If you’re planing on a big-ass video-card, make sure it will fit length-wise into the case. Make sure it won’t obstruct other slots on the motherboard that you are going to use.

If you’re buying *high* memory DIMMs (the new style, with heat radiators), make sure they won’t bump into your video card or your CPU cooler.

Luckily, I had very little issues with the above. I ordered a huge case, so my CPU cooler (barely) fit. But I had to install it facing the backpanel, to resolve the conflict with my DIMMs.

Noise

Standard fans that come with case — any case, even a $175 one — are crappy. Meaning, they produce lots of noise.

First, if your case has 3 or more fans — you may not even need them all. Two good fans should be plenty enough.

Next, I’m replacing the exhaust fan (the one on the backpanel) with this $17 Scythe. Since it is 4 pin (Pulse Width Modulation), I can plug it into MB and have the BIOS slow it down when cold. It also comes with a tiny knob of its own which I can rotate to adjust the lowest speed.

My Scythe will only produce 7 decibels of noise on its lowest setting. This is very good, but remember, multiple fans add up. If you have a fan on the top of the case (I do), see if you can replace that one too. I’m getting a 200mm Coolermaster for $16.  It may look obnoxious with those red LEDs, but it will push the air at 110 Cubic Feet per Minute while making only 19 decibels of noise. If worse comes to worse, I think I can cut those leds off.

Again, measure twice, cut once. The fans come in  different diameters (120mm, 140mm, 200mm are most common), but most importantly they have different distances between their mounting holes, and different depths (=thickness) too.

Hard Drive

After long hesitations I decided to buy an 128GB Crucial m4 SSD for $200. Why SSD? Because it’s been more than 3 years since they became common and I must check whether those super-speed myths are true. Why Crucial m4? Because Crucial is an old-school memory company that cares about quality, unlike OCZ who make super-fast SSDs that crash. That simple.

Total Cost

This brings my total cost from $1008 to $1241 (plus shipping, so probably more like $1300). The only thing I’m missing is a video card. Another $200. So the total cost is going to be $1500. Not bad for a rig made off of top of the [mainstream] line components.

This is it for now, and since I personally know one person who reads my blog, I can tell you: you should totally do it, Igor!

When it comes to learning things, I’m a big fan of layered approach. By this I mean a methodology of study that emphasizes the history of evolution of whatever it is we are learning. Recently I’ve been applying it to Unix text editors. This post is a document of my research into this topic.

As you may or may not know, in Unix operating systems there is a line of text editors that lead to creation of famous vim. The full line looks as follows:

• ed — “the standard text EDitor” written by Ken Thompson for the first Unix back in 1971
• em — “the ed for Mortals” — a mod of ed featuring one-line visual editing, developed by George Coulouris in 1976
• ex — an “eXtended” version of em written by Bill Joy at Berkley University in 1977, featuring the full-screen visual editing mode
• vi — a hard link to ex 2.0 that starts the editor with the visual mode turned ON by default
• vim — a “vi iMitation” developed in 1988-91 for Amiga computers. Supports syntax highlighting, various plugins, mouse, unlimited undo, word-completion etc.

You can find more vi history in wikipedia.

What I want to look at today is the great-great-grandfather of vim: the venerable ed. Using ed is like writing with ink using a goose pen. Its Zen character hides itself under the surface of false simplicity. It requires concentration and focus. It teaches you the origins of the commands you will see in vim. Try using ed for a day to edit all your text files.

If you’re running Mac or any flavor of Unix, you already have it. If you are on Windows, you can get it from here.

Open it on command line like this: “ed 1.txt”.

At first you can be shocked: No only you don’t see scrollbars, there is no fucking text visible anywhere on screen! That’s because ed is a command-centric editor. You issue commands to move in the files and to edit text, one line at a time. Here, to make your experience less shocking I’ve organized ed commands into 8 layers, each building up on the previous. Once you have tried them all, you will not only understand vim better, but also get the taste of what computers felt like back in 1972.

Layer 1. Moving around

• h — help explain why ed said “?”
• q — quit
• + — go to next line
• - — go to prev line
• +10 — go 10 lines forward
• -10 — go 10 lines backward
• 123 — go to line 123
• $ — go to last line
• [Enter] — move through text printing one line at a time
• z — move through text printing one screen at a time

Layer 2. Editing text

• a — append text after current line
• . — exit the text mode and go back to command mode
• p — print the current line
• d — delete current line
• c — change the text of the current line
• i — insert text before current line
• Q — quit even if there are unsaved changes
• w — write the file back to disk

Layer 3. Search and replace

• ?abc —  go to the previous line containing “abc”
• /abc — same as ? but searches down, and wraps over to the top of the file
• s/abc/xyz — if current line contains “abc”, substitutes it for “xyz”
• s — substitutes the next match on the same line
• s/abc/xyz/3 — substitutes third match on the current line, instead of the first one
• s/abc/xyz/g — substitutes all matches on the current line

 Layer 4. Line addressing

• 123a — append text after line 123
• 123i — insert text before line 123
• 10,20d — delete lines from 10th to 20th
• 10,20c — delete lines from 10th to 20th and enter the text that will go in their place
• 10,20p — print lines from 10th to 20th
• 10,20n — print lines from 10th to 20th with their line numbers
• %p — print all lines
• 10,20s/abc/xyz/g — substitute all occurrences of “abc” with “xyz” in lines 10 through 20
• %s/abc/xyz/g — substitute all occurrences of “abc” with “xyz” in the entire file

Layer 5. Regular expressions

The substitute command “s/abc/xyz” supports regular expressions in its “abc” part:

• . means any character. To search for dot, use \.
• .* means any sequence of any characters
• ^ means beginning of line
• $ means end of line

Also, it supports special characters in the “xyz” part:

• & means the actual substring matched by the “abc”
• \[Enter] means a new line

Layer 6. Advanced editing

• j — join the next line to the end of the current line
• m123 — move current line in front of line 123
• t123 — copy current line in front of line 123

(the above commands also work with multiple lines)

• u — undo the last command. doing the  “u” again redoes the undone command.
• 123kx — mark line 123 with label “x”
• ‘x= — print line number of line marked with label “x”
• ‘x — go to line marked with label “x”

Layer 7. Interaction with external world

• !dir — execute shell command “dir” without exiting to shell
• r !dir — execute shell command “dir” and append its output to the end of the current text
• 10,20w 2.txt — write lines 10 through 20 to file 2.txt
• 10,20w !cmd — send lines 10 through 20 to standard input of shell command cmd
• f 2.txt — changes current file name to 2.txt

 Layer 8. Interactive search

• G/abc — for each line matching “abc”, makes it current and allows user to enter a command

Lately I felt slightly bored, and so I decided to entertain myself by assembling a custom “rig”. I took Jeff Atwood’s latest system as a starting point, and made a few tweaks.

Here’s what I’ve ordered so far:

CPU — $315

I knew that Intel Sandybridge is the latest and greatest processor technology at the moment. I went to the wikipedia article to research the models and arrived at the same Core i7 2600K model which Jeff has used in his build. This basically is the fastest Sandybridge currently available (the 2700K is not available yet as I’m writing this). I did not even consider AMD, not sure why.

Motherboard — $200

Next I went to wikipedia’s article for LGA 1155, which is socket for the CPU I’ve chosen. I’ve also looked at this table of chipsets. From these two I figured I want a P67 or Z68 chipset. From there I went to ASUS web-site (I knew I wanted an ASUS or Gigabyte motherboard), and looked at my options. Both P67 and Z68 had some very good boards but Z68 model line had more choices and looked more “enthusiast-oriented”. Finally I’ve settled on Asus P8Z68-V for $180. It is a full-size ATX board, with the full selection of slots, and all the modern technologies built-in, like USB 3.0, UEFI BIOS and 6Gb/s SATA. It also had standard stuff like the Audio card, the NIC etc. But when I saw that Jeff’s rig uses the Pro version of the same board I decided to upgrade… it only costed me $20 more. The Pro version has 2 extra SATA 6Gb/s ports, two 1394 ports, and an extra USB 3.0 back panel plate. I should have probably stayed with the non-Pro version since I won’t use any of these extra features.

Memory — $118

After reading this wonderful comparison of memories, I’ve decided there’s no point in wasting money on fancy memory, and went ahead with Corsair Vengeance 16 GB kit, the same one Jeff had in his rig. Being 1.5V this particular memory uses less power (=emits less heat), and runs almost as fast as G.Skill Ripjaws-X which costs twice as much.

CPU heatsink/cooler — $75

I wanted a cooler that would have a PWM fan (aka 4-pin fan), because my motherboard can control the speed of CPU fan based on CPU temperature. After some hesitation I’ve settled on Thermalright Archon, one of the best air-based (as opposed to liquid-based) coolers available. It is a little tall at 170mm, and I hope it will fit in my case.

Power Supply Unit — $127

I’ve noticed Jeff’s Seasonic X-760 was too powerful for his needs (he was only using 128W-332W of the available 760W), and since my plans were even less ambitious, I’ve decided to go with Seasonic X-560, a 560W version. Judging by the reviews I’ve seen, Seasonic X series are one of the best PSUs out there in terms of noise and power consumption.

Case — $173

First I’ve ordered NZXT Tempest 410 Elite – a nice-looking $100 case. In the specs they said it provided 170mm of space for the CPU heatsink. It also had some unique features like detachable front-fans and proper USB 3.0 header. But after reading some very nasty reviews I changed my mind and went with Corsair Obsidian 650D — a much more professional case with even more space inside, better cable management and overall higher build quality. I’m still on the edge as to which I should keep — must decide in the next few days, before they arrive, so I can refuse shipment on one of them and get my money back. I will most likely keep the Corsair, don’t want to open the NZXT and realize my CPU cooler does not fit by 1 millimeter.

Videocard — $0

I will not buy an external video-card until next year, the Intel HD 3000 graphic card being built right into the CPU. Out of all the cards available now I’m looking at NVidia 560. My main concern (besides price) is the power consumption, since my Power Supply is only 560 watt I can’t get a super-hungry card. This wikipedia article offers nice comparison of NVidia cards, which includes their TDP. According to my power calculations I need to get a card that would eat no more than 160 watt at its peak.

Total — $1008

This is a little more than $1000 spent so far. This is all I need for now to build my own PC. For harddrive, I’m planning to reuse an old IDE HDD (will buy an IDE-to-SATA adapter for ten bucks) coupled with a 16GB USB drive I have somewhere. For the monitor, I will use my Sony 52-inch LCD TV. The only other thing I will need to get is a keyboard.

Since I already have 5 computers at home (Matthew’s gaming desktop, 3 laptops, an 24-inch iMac — I’m not counting a Playstation 3, a Wii, and 3 iPhones) I’m not sure what I’m gonna do with this one once I’ done building it. Maybe I will sell it on Ebay… Or install Google Chrome OS and run a superfast internet browser on a 52-inch screen… Any ideas?

Some of you might have noticed a few posts in Russian lately that were eventually deleted. These were my early attempts at translating one of the most famous programming books, Why’s (Poignant) Guide to Ruby. For those of you who read Russian, the work in progress is available at  zvolkov.com/poignant_ru/ – please check it out, you can leave your comments on this post.

If you want to participate in the translation project, the raw texts are hosted on github.com/zvolkov/poignant_ru – feel free to fork the repository and send me your pull requests.

• Is the team dedicated to single product? How many “projects” (independent timelines) are there for the team?
• The product(s) that you develop, where do they get deployed and by whom?
• Do you provide production support (troubleshooting, hot fixes)? How does that work?
• What’s your source control branching policy? Do you follow the mainline model with maintenance branches / feature branches?
• How flexible is your working hours? Do you allow working from home?
• Do you use any legacy software / tools?
• Are there projects in use not covered by unit-tests?
• What is your dress code?
• What is the escalation path for technical debates / conflicts? For business (schedule / scope / priority) conflicts?
• How do you track bugs? How do bugs triaged / tracked / resolved?
• Is blogging / tweeting allowed on corporate time?
• What kind of work stations developers have (RAM, Number of monitors)
• How much overtime is typically expected per week?
• What is the policy for purchasing a development tool (Resharper, BeyondCompare etc.)?
• What training and improvement opportunities the company provides (free time, books, dev conferences, in house training)?
• Do you practice Continuous Integration (in the narrow sense of running builds off of checkins)?
• Would I have admin rights on my development workstation?

1. Unit-Testing
   • The Art of Unit Testing
   • Growing Object-Oriented Software Guided by Tests
   • Working Effectively with Legacy Code
   • xUnit Test Patterns: Refactoring Test Code
2. Professionalism and Craftmanship
   • Clean Code: A Handbook of Agile Software Craftsmanship
   • The Clean Coder: A Code of Conduct for Professional Programmers
3. Project management
   • Scrum and XP from the Trenches
   • Kanban, by David Anderson
   • Scrumban – Essays on Kanban Systems for Lean Software Development
   • Peopleware: Productive Projects and Teams
   • The Mythical Man-Month
4. Object Oriented Design
   • Design Patterns. Elements of Reusable Object-Oriented Software.
   • Head First Design Patterns
5. Miscelaneous best practices
   • Agile Principles, Patterns, and Practices in C#
   • Brownfield Application Development in .Net
   • Foundations of Programming, by Karl Seguin
6. .NET technology specific
   • C# in Depth, Second Edition
   • Illustrated C# 2010
   • Pro ASP.NET MVC 3 Framework
   • Inside the Microsoft Build Engine: Using MsBuild and Team Foundation Build
   • Applied Microsoft .NET Framework Programming
7. Inspirational and mind expanding
   • Why’s (poignant) guide to Ruby
   • The Best Software Writing I: Selected and Introduced by Joel Spolsky
   • The Art of UNIX Programming
   • Seven Languages in Seven Weeks

Everybody is familiar with indexes, used by RDBMSes to speed up filters and joins. But what about aggregates? Normally, aggregating values requires a full scan though the data. Let’s take, for example, AVERAGE. In order to calculate average price for the entire table, we must do a complete index scan. At least, it’s a single value, so we may easily cache it if needed to increase the performance. Now, imagine we need to aggregate a subset of data from the very first date in the table to an arbitrary user-selected date. This time we must do a partial index scan, and we must cache a value per each date…

Now, most of the time we’re simply adding the same values again and again, only the stop date differs. Wouldn’t it be nice if there was a way to reuse the aggregated data, so that subsequent queries could somehow benefit from the previous queries?

It turns out there is such a way. I learned it when reading the CouchDB guide. CouchDB employs a cool pattern that can be used in a multitude of similar scenarios. For those familiar with CouchDB, I’m talking about the persisted B-tree index of map/reduce results. For others, here is the idea:

The idea is to precalculate the aggregated data and store it at different levels of the B-tree index. The index can then be used to efficiently query the aggregate without having to reaggregate all the data all the time. Then, if any leaf-level value changes, only the ascending path through the tree has to get recalculated.

In our average price example, the index could store the SUM and the COUNT of items at day, month, and year levels. Then, if anybody wants to query average price year-to-date all you had to do is sum up all the SUMs and COUNTs for all the full months since year start, plus all the days available for the last month, then divide total SUM by total COUNT. If a past price has to change, the change has to propagate through the index, but only corresponding day’s and month’s and year’s values have to be updated, and even then the values for other days and other months within the year can be reused for the calculation!

Not only simple calculations can be indexed this way. More complex, lossy calculations like Standard Deviation can be composed from partitioned data too. Here is an example of how Standard Deviation can be calculated on chunks of data, and then combined to form the final value:

--geting row data
;with q1 as (select TradeDate, TransactionValue from Transactions where AccountID = 64337 and TransactionValue is not null),
--splitting the date into Year / Month / Date
q2 as (select day(TradeDate) as d, month(TradeDate) as m, year(TradeDate) as y, TransactionValue from q1),
--calculating sum, sum of squares, and count for each DAY
q3 as (select d, m, y, sum(TransactionValue) as T, sum(TransactionValue*TransactionValue) as S, count(*) as C from q2 group by d, m, y),
--combining days into months
q4 as (select m, y, sum(T) as T, sum(S) as S, sum(C) as C from q3 group by m, y),
--combining months into years
q5 as (select y, sum(T) as T, sum(S) as S, sum(C) as C from q4 group by y),
--combining years into single value
q6 as (select sum(T) as T, sum(S) as S, sum(C) as C from q5),
--calculating the mean
q7 as (select T, S, C, T/C as M from q6),
--calculating the variance
q8 as (select (S - T*M)/(C-1) as Variance from q7)
--calculating the std deviation
select SQRT(Variance) from q8

Sometimes you want to ensure that multiple instances of your application cooperate nicely when using a shared resource. Maybe you want to ensure that only one user at a time can edit a sales order. Maybe you want to ensure that only very first instance of certain event is recorded in a table.

In the past I either used native SQL transactions (beware of deadlocks!), or had to implement my own locking table (easy to get wrong as complexity grows). Luckily, there is an alternative solution that is both easy to use and flexible. I’ve only learned about it today but apparently it was available since SQL Server 2005! The name is sp_getapplock.

Basically, sp_getapplock allows you to place a logical lock on virtual “resource” identified by a varchar key. It supports Shared/Exclusive semantics, timeouts, implicit or explicit lock release and more. Here’s the simplest scenario, assuming you want to implement a global equivalent of simple critical section (i.e. a lock that works across multiple instances of your application):

1. begin tran
2. exec sp_getapplock @resource=’my locking key’, @LockMode=Exclusive
3. …access the protected resource here…
4. commit tran

In this scheme the lock is implicitely released when transaction commits or rolls back. If you want your lock to live longer that typical transaction, you can increase its scope to SQL Connection level by setting @LockOwner = ‘Session’. The lock will be auto-released when SQL Connection is closed. (There’s no way to make the lock live accross multiple connections. At first this may sound like a deal-breaker, but if you think about it, this eliminates the need to worry about garbage-collecting the old locks remaining after your client application dies. As Stalin used to say: “There is no man — there is no problem”)

What I want to show today is how easy it is to use sp_getapplock from NHibernate to achieve safe alternative to ISession.BeginTransaction(IsolationLevel.Serializable). All you have to do is use ISession.CreateSQLQuery to create a wrapper around sp_getapplock:

using System;
using NHibernate;

namespace Sample
{
    public static class ApplicationLevelDatabaseLock
    {
        public static void ObtainApplicationLevelDatabaseLock(this ISession session, string resourceName, string lockMode, int timeoutMilliseconds)
        {
            int status = (int)session.CreateSQLQuery(
                    @"DECLARE @status INT;
                      exec @status = sp_getapplock @resource=:resourceName, @LockMode = :lockMode, @LockTimeout=:timeout;
                      SELECT @status as status")
               .AddScalar("status", NHibernateUtil.Int32)
               .SetParameter("resourceName", resourceName)
               .SetParameter("lockMode", lockMode)
               .SetParameter("timeout", timeoutMilliseconds)
               .UniqueResult();

            if (status < 0) throw new ApplicationException("Could not obtain application-level " + lockMode + " lock on " + resourceName);
        }
    }
}

Now you can use it like this:

using (var tran = session.BeginTransaction())
{
    session.ObtainApplicationLevelDatabaseLock("my exclusive resource 1", "Exclusive", 5000);
    // do your thing here, only one instance of your app will run this code at any given time.
}

Isn’t this nice? As one of my friends says (in a very dry tone, with almost British accent): “I’m all tingling with excitement”