Jun 26 2010

Learning UNIX

Category: Uncategorizedzvolkov @ 3:14 pm

My master’s thesis project was written in PHP + MySQL. Back then I could install Linux, rebuild its kernel and do basic administrative work. All of that is mostly forgotten by now. Lately I’ve been contemplating Windows culture,  how it affects my quality of life as a programmer, and speculating about virtues of UNIX universe. Arguably, even such remote offsprings as Java and Ruby can be related to UNIX culture. And who of Fortune 100 companies run Microsoft? Google? Facebook? Wikipedia? None!

Thus I decided to refresh my UNIX. I started with quick review of linages and distributions. Of hundreds of them, I centered on Debian, FreeBSD, and MINIX. Debian — because it’s the biggest and longest surviving truly non-commercial project. FreeBSD — because it seems to be the most authentic descendant of the original UNIX. And MINIX — because it is a minimalistic UNIX-like OS that has a companion book explaining low-level kernel design, complete with source code.

At the moment, I’ve installed MINIX in a VM and am playing with the shell. I’ve also identified a few key topics and did some significant googling to find best articles to quickly get myself up to speed (the ones that I both read and loved are highlighted in bold).

Of all the books praised on the Internet my local library had Unix System Administration Handbook, A Practical Guide to Linux, by Mark G. Sobell  and Mac OS X Tiger for Unix Geeks — all three are highly praised books.

Let the fun begin!

 

 


Jun 23 2010

Testing MSMQ from Powershell

Category: Uncategorizedzvolkov @ 4:01 pm

Just a quick note to myself on how to ping a queue using PowerShell (note how verbose the syntax is!)

[Reflection.Assembly]::LoadWithPartialName( "System.Messaging" )
$msmq = [System.Messaging.MessageQueue]
$mq = New-Object $msmq("FormatName:DIRECT=OS:SERVERNAME\Private$\QUEUENAME", $False, $False, [System.Messaging.QueueAccessMode]::Peek);
$mq.Peek([System.TimeSpan]::FromSeconds(1))


Jun 21 2010

Goomgum – my first ever OSS project

Category: Uncategorizedzvolkov @ 4:00 pm

Opsource Cloud Restful API is a RESTful web service interface that allows users to control their cloud environment over HTTPS.

Goomgum is a .NET library that can be used to simplify access to Opsource Cloud API from .NET applications.

Goomgum wraps REST-style API operations in an OOP-style class library that can be used as a base for creating your own Opsource Cloud automation tools in .NET. Goomgum currently features a small but growing subset of the parent API, a set of unit-tests that does not require access to the live API, and a sample command line utility that consumes the library. With goomgum you can use LINQ and other .NET niceties to manage swarms of VMs with ease:

using OpSource;
using System.Linq;

var c = new Cloud("login", "password");
Account a = c.Authenticate();
var servers = a.GetServers().Where(s => s.Name.StartsWith("test"));
foreach(var s in servers)
    s.Start();

Feel free to download the binaries and try for yourself.

Goomgum is currently developed in C# for .NET Framework 4.0 Client Profile. The client library itself does not have any third-party dependencies, but the Unit Tests and the Command Line App depend on a few free libraries & tools such as Rhino Mocks, Command Line Parser Library and ILMerge.

If you want to participate I will be glad to give you write access to the repository. The project is currently hosted on Google Code.


Jun 18 2010

Environment-aware Configuration with DNS-based Environment Determination

Category: Uncategorizedzvolkov @ 3:36 pm

Inspired by this article, I decided to try DNS-based Environment Determination (DBED).

DBED is an Configuration Management technique by which you can minimize the effort/overhead of maintaining mulitple Technical Operation Environments (DEV, TEST, QA, UAT etc.). The problem that DBED intends to solve is proliferation of configuration changes growing as carthesian product of number-of-Environments times number-of-Dependencies-Between-Services. So if you have 2 environments with 2 web services and a SQL Server, you’d have to manage total of 4 configuration entries. This quickly grows out of hand when you have to manage 20 interrelated services across 5-6 environments. The promise of DBED is to deploy same exact configuration of a service to each of the environment-specific servers and have it know which servers its dependencies live on. Hence the name: environment-aware configuration.

The approach is to create a subdomain for each of the environments (e.g. dev.internal.company.com, qa.internal.company.com etc.) and then in each subdomain create bunch of aliases to various servers, (e.g. mainsql.dev.internal.com –> DB123.internal.company.com, mainsql.qa.internal.com –> DB456.internal.company.com etc.). These aliases (unqualified by the full domain suffix) is what you then point to in your config files. But how to “route” the request to a different subdomain based on the requesting server’s environment? By modifying that server’s DNS suffix sequence! Here’s how (assuming your DNS is on Windows):

First, get your IT/NOC/Admin department’s buy-in and support. Indeed, you’ll need permissions to access your Domain Controller (this is where DNS Server typically runs on) and/or Active Directory (if you take the Group Policy route, more on this later). Alternatively, if you can bring up your own DNS Server and have enough rights to point your non-PROD servers to it, you may be able to stay self-sufficient a bit longer.

Second is to connect to the DNS Server and create new Primary Zone, under one of the subdomains you control. For example in my case we had internal domain called Tech1.Corp.Local under which I created Cloud1.Tech.Corp.Local, Cloud2.Tech.Corp.Local etc. You should also create “Delegation” records in the parent subdomain (Tech.Corp.Local) to point to the same server (unless your CloudX zones will be on a different DNS Server). Simple scenarios (e.g. a client PC directly asking for name resolution) will work even without Delegation records, but more advanced scenarios like recursive queries made by other DNS servers may be broken. Then again, if you are in a complex DNS environment you’ll probably require specialist’s help anyway. Alternatively you can create the environment subdomains directly in the existing zone (Tech.Corp.Local) as opposed to creating bunch of new zones. If you do that you don’t need to create delegation records. It actually does not make hell lot of a difference whether to do subdomains or delegations. With delegations you’ll have an option of easily moving the separate zones to another DNS server whereas with subdomains you’ll have to recreate them by hand.

Third is to create CNAME (Canonical Name) records in your environment zones (or domains) to map your aliases to the actual names of the servers. The reason we’re creating CNAME and not A records is to avoid creating a hard link to the servers’ IP addresses. Instead, CNAME simply redirects DNS lookup from the alias to an existing name. This way your aliases will keep working even if the servers’ IP addresses change. Once you’re done you should have something similar to the picture above. Repeat this for each environment.

Finally, you will need to modify the DNS suffix sequence on each of the servers in each environment. In the Network settings, under TCP/IP 4, Advanced, DNS tab; there’s an option for searching various domains. By default it’s set to use only the primary connection suffix (domain); but you can switch it to use a list you set (make sure you include all the domains you want it to search). When resolving “maindb” without any domain, it’ll iterate through the list until it finds a resolution. Of course, this doesn’t work if you specify a full FQDN in your config file. In my cloud example, my suffix list needs to have Cloud1.Tech1.Corp.Local at the top, followed by Tech1.Corp.Local (followed by my cloud provider’s suffix). With this setup an unqualified name will first try to resolve in the Cloud1, and then in the Tech1. This allows our aliases to map to corresponding environments, while still keeping all other names functional. If RDPing into each server and modifying its DNS settings feels like a pain (which it is!) you may try to use Active Directory’s Group Policies to push the suffix list to corresponding servers. I was told it’s possible but as I have not tried it myself I will “leave it as an exercise” for you. UPDATE: As explained here DNS Suffix Search List can be modified remotely from command line by doing wmic /node:[machine] nicconfig call SetDNSSuffixSearchOrder (Cloud1.Tech.Corp.Local,Tech.Corp.Local) and you can also use wmic /node:[machine] nicconfig get DNSDomainSuffixSearchOrder to verify it.

Once all of the above is done, restarting either of the servers should not be required, just try pinging both your new aliases as well as regular names and make sure everything works. Now point your configs to aliases and enjoy environment-aware configuration!

 


Jun 15 2010

Memcached 1.4.4 and NHibernate 2.1.2

Category: Uncategorizedzvolkov @ 3:55 pm

Looks like Memcached ClientLibrary used in the Memcached Cache Provider that shipped with NHibernate 2.1.2 (and older) has issues with Memcached 1.4.4 (and newer)

Here’s the error I’m getting:

Error deleting key: XXXX@YYYY#121271.  Server response: CLIENT_ERROR bad command line format.  Usage: delete <key> [noreply]

From what I read in memcached release notes 1.4.4, looks like Memcached 1.2.x used to support so called lingering delete. This is when the client sends the delete command with an optional timeout parameter set to a non-zero value. This causes the server to flag the item as deleted but still keep in the cache (until the timeout expires) — this way the subsequent add operations would fail. This rather obscure feature was removed completely in 1.4.0, but 1.4.4 introduced a backwards compatibility change that allows the timeout value of 0 (i.e. non-lingering delete), while continuing to reject others.

Looking at NHibernate.Caches.MemCache.MemCacheClient’s method I see the following code:

this.client.Delete(this.KeyAsString(key), DateTime.Now.AddSeconds((double) this.expiry));

So we have two options: keep NH wrapper supplying an expiration date and making Memcached.ClientLibrary.MemcachedClient ignore it OR modify NHibernate.Caches.MemCache.MemCacheClient to not supply expiration date. I’m inclined towards second option, but can it be hardcoded or should I make it configurable?

Here’s my post to NHContrib group: http://groups.google.com/group/nhcdevs/browse_thread/thread/14630c7fe6f9a18e

And here’s JIRA ticket: NHCH-27


Jun 10 2010

Automating web tests with RobotFramework

Category: Uncategorizedzvolkov @ 12:39 pm

I knew for a while I could automate my tests using Selenium. What stopped me from actually doing it is the way you had to write your tests: Normally, you either record tests using Selenium IDE (a Firefox add-on) or write unit-tests using Selenium API from C#. What I did not know is that I could write my test scripts in human language!

The problem with recording the tests, you can’t do it before your page is written, so you’re limited to regression testing only. The problem with writing the tests in C#, you can’t have your QA team writing tests (unless you can afford QA team that knows C#!). With RobotFramework you can have your QA team develop tests before developers are done coding, and use them for both acceptance- and regression-testing.

The script basically looks like this:

Open Browser  http://google.com/  firefox
Title Should Be  Google
Input Text  q  zvolkov
Click Button  btnG
Page Should Contain  The Way Of The Future

You save this to a .txt file, feed it to RobotFramework and it spits out a nicely formatted html report. This means you can automate your tests and have them executed on a regular basis by your build server!

There are bunch of predefined commands like “Click Button” and “Page Should Contain”, plus you can define your own using same highly readable text-based syntax. For example the following defines new command “Search For” that combines two built-in actions:

Search For  [Arguments]  ${query}
    Input Text  q  ${query}
    Click Button  btnG

As you can imagine, with custom commands you can define reusable domain-specific libraries to significantly reduce the effort required for writing new tests.

If you want to invest 15 minutes and try it for yourself, follow these steps:

  1. Make sure you have Java 1.6.x installed by going to http://www.java.com/en/download/installed.jsp
  2. Install latest Python 2.6.x (not 3.x!) Windows installer from http://python.org/download/
  3. Add C:\Python26 and C:\Python26\Scripts to PATH (press Win+Pause, select Advanced System Settings/Advanced/Environment Variables/System Variables/Path/Edit and add “C:\Python26;C:\Python26\Scripts” add the end)
  4. Install robotframework-2.5.win32.exe (run as administrator!) from http://code.google.com/p/robotframework/downloads/list
  5. Install Selenium Library Windows installer 2.4 (run as administrator!) from http://code.google.com/p/robotframework-seleniumlibrary/downloads/list
  6. Make sure you have Firefox installed as my demo uses Firefox by default.
  7. Unpack robotFramework_demo.zip to a local folder.
  8. Double-click on test.cmd

This demo shows-off slightly more robust script with parametrized commands in a separate include file etc. I don’t know about you but I like it. RobotFramework that is.


Jun 04 2010

Tell, don’t ask: an OOP design principle

Category: Uncategorizedzvolkov @ 2:58 pm

Just discovered this yesterday myself (don’t laugh!):

Procedural code gets information then makes decisions. Object-oriented code tells objects to do things.

OO is a tell-don’t-ask paradigm. Yes, I know people don’t always use it that way, but one of Kay’s original concepts was that objects [...] do not ask any questions. They simply tell each other what to do. [...] in original concept for OO communication was half-duplex.

The problem is that, as the caller, you should not be making decisions based on the state of the called object that results in you then changing the state of the object. The logic you are implementing is probably the called object’s responsibility, not yours. For you to make decisions outside the object violates its encapsulation.

The main purpose of this exercise is to ensure a correct division of responsibility that places the right functionality in the right class without causing excess coupling to other classes.

The biggest danger here is that by asking for data from an object, you are only getting data. You’re not getting an object – not in the large sense. Even if the thing you received from a query is an object structurally (e.g., a String) it is no longer an object semantically. It no longer has any association with its owner object. Just because you got a string whose contents was “RED”, you can’t ask the string what that means. Is it the owners last name? The color of the car? The current condition of the tachometer? An object knows these things, data does not.

The fundamental principle of Object Oriented programming is the unification of methods and data. Splitting this up inappropriately gets you right back to procedural programming.

From http://www.pragprog.com/articles/tell-dont-ask and http://blog.objectmentor.com/articles/2010/06/03/tdd-in-clojure