Yesterday's news

Here’s what I’ve been up to lately: FreeBSD, vim, tmux, command line twitter client (ttytter) and unicode-capable graphic terminal (jfbterm). No X-Windows involved!

I have a table of Prices: ProductID, Date, Price and Source. My application needs to load new prices but if the price already exists and it is from a “better” source, it should update existing record.

The orignal app loaded all data in SQL temp table and did a bulk insert to the target table, followed by a bulk update. Kinda simple and effecient even if too SQL-centric. The first version of my app used to check every ProductID+Date combination for existance using regular stateful NHibernate session, then load new prices using SQL Bulk load, and update each existing price using regular NHibernate session. Needless to say, not only it was slow but the gap between the read and the insert allowed new prices from other sources (my app is not the only source) to sneak into the table causing duplicate/ambigous records to appear. I needed a solution that would allow me to Insert-or-Update records fast, without resorting to temptables+SQL mess. (I have a habit of trying to minimize the amount of SQL code in my application but if nothing else worked, I would have to fall back to the old ways.)

Here’s the outline of my new solution:

• Stateless NHibernate session (IStatelessSession) for extra efficiency. No need to spend CPU cycles keeping that first-level cache growing.
• Single explicit transaction wrapping entire load operation for all records. This reduces stress on SQL Server by not having it commit every record. If my volume becomes too high I can always limit the transaction size to say 10000 rows.
• Custom sql-insert script defined in the entity’s mapping file that implements the read-update-or-insert logic . The idea is to be able to completely process one record without going back-and-forth between SQL Server and the .NET
• NHibernate batching enabled, in order to push the records hundreds at a time to the SQL Server.

Combined, these techniques are designed to make the whole operation less chatty and, in a way, achieve the effect of having the processing done on the SQL Server side.

Here’s what I discovered while implementing the above:

• IStatelessSession does not support .BeginTransaction(IsolationLevel) overload. Instead, do session.Transaction.Begin(IsolationLevel.Serializable) which does exactly same thing (I looked at NH source code)
• You can further optimize the logic by not doing session.Get for each many-to-one-property of your entity. In my example, instead of doing price.Product = session.Get<Product>(ProductID) I can create a dummy Product object once and simply set its Id to a different value for each price. Stateless session does not care that the object is transient.
• The custom sql-insert script is sensitive to the order of parameters (they’re simply represented by “?”s). Turn on NH debug-logging and look at NH log to see the exact order in which object’s properties are mapped to SQL params. Do this before you implement sql-insert.
• The custom sql-insert script is sent to the server once for every inserted record. You may want to move it to an SP to minimize the traffic, and execute the SP from sql-insert. (Note that prepare-sql setting has no effect on batched statements)
• In order for NH batching to work, the entity’s Identity must be assigned before the insert. Can’t use generator=native (aka SQL Server IDENTITY).
• Since your Insert is now in fact Insert-Update-Or-Do-Nothing you want your entities Identity be its natural key, in my case ProductID + Date. That requires implementing Equals and GetHashCode but other than that it’s real easy.
• NH batching requires check=”rowcount” setting on the sql-insert. This means your script must return number of records affected, whether it actually inserted, updated or did nothing.
• NH batching uses global settings for its CommandTimeout, not the one defined in your SessionFactory. This means you must add hibernate-configuration section to your App.Config (or Web.Config).

This summary turned out to be more elaborate than I expected. No details for you then. You can figure it out, I have trust in you.

So a Java developer, a web developer and a UNIX admin go to a bar to get a drink. The Java developer and the web developer get into an argument over who’s job is more important. “Thousands of people see my work” claimed the trendy web developer. “My code runs the banks!” retorted the tidy Java developer. They both look to the UNIX admin, expecting him to have the final word. He puts down his drink slowly. He then smashes his glass on the table, jumps on it and rips off his pants. He takes a shit and rubs it in the Java devs face, and while the web dev makes a run he jumps him, ripping his eyes out and eating them. He then goes back to the cave where he lives.

In order to make bash shortcuts like Alt+. work in virtual terminals, you need to modify the keyboard mappings to map the left Alt to work like Unix special “Meta” key.

You can either start with one of the existing mappings, like this:

sed 's/lalt/meta/g' /usr/share/syscons/keymaps/us.iso.kbd >  /usr/share/syscons/keymaps/local.kbd

Or, if you’re not sure which keymap file you’re currently using you can get your current keybindings using kbdcontrol:

kbdcontrol -d | sed 's/lalt/meta/g' > /usr/share/syscons/keymaps/local.kbd

Then, assuming you want this change to be global, edit your /etc/rc.conf file and add the following line:

keymap="local"

Just a random selection of keyboard shortcuts and other productivity tricks for bash.

Keyboard shortcuts: (use bind -p to see all current mappings)

• Ctrl+a — begining of line
• Ctrl+e — end of line
• Ctrl+u — cut head
• Ctrl+k — cut tail
• Ctrl+y — paste
• Ctrl+_ — undo (may require pressing Shift, depending on your keyboard mappings)
• Ctrl+L — clear screen
• Alt+b — skip to next word boundary (if Alt doesn’t work on all terminals, see the following post or try Esc followed by the char instead)
• Alt+f — skip to prev word boundary
• Alt+d — delete next word
• Alt+backspace — delete prev word
• Ctrl+w — cut word
• Ctrl+R — search command history
• Alt+. — repeat last command’s argument

Add following to .inputrc:

• “\e[3~": delete-char -- enable normal Delete function (as opposed to both Backspace and Delete working as Backspace)
• "\C-?": delete-char -- same as above but works on virtual terminal (not just on SSH)
• "\e[1~": beginning-of-line -- enable Home button
• "\e[4~": end-of-line -- enable End button
• "\e[1;5D": backward-word -- enable Ctrl+Left (only works on SSH terms)
• "\e[1;5C": forward-word -- enable Ctrl+Right (only works on SSH terms)

History tricks:

• alias h = 'history 25' -- add to .bashrc (call .bashrc from .bash_profile using the ".")
• PS1 = "\!:\W\$" -- add to .bash_profile to set prompt to [History#]:[CurrentDir]$
• !$ — substitute last argument of last command
• !123 repeat history number 123
• export HISTIGNORE=$’[ \t]*:&:[fb]g:exit:ls’ — add to .bashrc, this way exact dups, ls, fb, fg, exit, and lines starting with space won’t be added to history
• PROMPT_COMMAND=’history -a’ — add this and next line to .bashrc to make bash save history immediately, this way history will work right with two simultaneous terminals
• shopt -s histappend

Tab-completion:

• set completion-ignore-case on — add to .inputrc to enable case-insensitive completion
• set show-all-if-ambiguous on — display all completion options on single Tab rather than on double-Tab
• case $- in — install bash-completion and add this to .bashrc to get Tab-completion for command arguments (the path can be different)
     *i*) [[ -f /etc/bash_completion ]] && . /etc/bash_completion ;;
  esac

grep:

• grep -options ‘word’ filename — general syntax
• -A4 -B4 — show +- 4 lines of context
• -v — NOT match
• -i — case insensitive
• -w — separate word
• -n — show line numbers
• -r — recursive

Other useful stuff:

• alias cd=’pushd > /dev/null’ — add to .bashrc to map cd to pushd, this way the next alias bd can be used to go back
• alias bd=’popd’
• find . -type f -exec grep “word” /dev/null {} \; –find word recursively in all files
• sed ‘s/old/new/g’ filename — find and replace

Quick update: the learning is going on really good. MINIX turned out to be a piece of crap, even as a learning tool. FreeBSD is much more solid. Unix System Administration Handbook turned out a better book than I expected it to be. Less about administration and more about fundamentals. The Unix Time-sharing System has been a godsend. An Introduction to the UNIX Shell was very helpful too. These 3 have my topmost recommendations to anyone starting with *nix.

I’m currently running FreeBSD 8.0 under Windows 7′s Virtual PC (the one that comes with XP Mode). The biggest pain was getting DHCP to work. The trick was to force full-duplex on the virtual Ethernet adapter and set DHCP client to run synchronously at the boot time. Here’s how my /etc/rc.conf looks now:

font8x8="cp437-8x8"
font8x14="cp437-8x14"
font8x16="cp437-8x16"
allscreens_flags="MODE_32"
synchronous_dhclient="YES"
ifconfig_de0="DHCP media 100baseTX mediaopt full-duplex"

The top 4 lines configure support for 30-vertical-lines screen, as opposed to default 25-lines one.
The bottom 2 lines configure DHCP.

Another pain was adding second virtual harddrive and mounting it as /home — I finally did it using the “dedicated” method from FreeBSD Handbook.

Finally, in order to get VM to feel more responsive I had to add following setting to my /boot/loader.conf:

kern.hz=50

This is pretty much it, everything else was by-default. This was tons of fun!