Binding a DOM element to an observable using Knockout.js

I've been doing a bit of experimenting with Knockout.js recently. Having written quite a bit of .NET code over the years, I am very used to thinking in (and I prefer thinking in) the MVVM pattern when it comes to UIs, and so Knockout.js is nice.

And, as always, when writing code and living in the real world you sometimes need to do things that do not quite fit the pattern. In this case, I needed to draw on a canvas from my view model whenever some observables on it changed. The pragmatic thing to do when you need to draw on a canvas is to just grab the canvas by ID or whatever, but since this canvas lived in a template, identifying it through selectors quickly got messy (and hard).

Binding to it seemed like a much better idea, so I created the simple `element` Knockout.js binding. This is a one-way-to-source binding (.NET guys will be familiar with this terminology), meaning that the binding only writes to the view model from the view. In addition, the binding only does this once (when it is initialized), as the element will stay for the lifespan of the page.

So, in all its glory, here is my Knockout.js `element` binding:


Use it as you would any other binding:

<canvas width="100" height="60" data-bind="element: yourObservable"></canvas>

Simple as that. Now go use it!

How to combine Git, Windows, and non-ASCII letters

The default Git installation in Windows works really bad if you're using `cmd.exe` and have non-ASCII letters in your commit information and/or code.

Thankfully, Git is highly configurable, and the fix is rather easy:

1. Set i18n.commitencoding to the codepage you're on in cmd.exe (I'm on windows-1252)
2. Set i18n.logoutputencoding to the same codepage.
3. Set the LESSCHARSET environment variable to a proper name for the code page you're on (I'm on latin1), either by adding a user environment variable in Control Panel > System and Security > System > Advanced System Settings > Advanced > Environment Variables..., or setting it your cmd.exe session (e.g. set LESSCHARSET=latin1).

Boilerplate version for your copy-paste convenience (replace encodings as necessary):

    git config --global i18n.commitencodig windows-1252
    git config --global i18n.logoutputencoding windows-1252
    set LESSCHARSET=latin1

The first setting tells Git how your commit messages, including your author information, are encoded. The second tells Git what encoding it should use when writing output from a command like git log. The third and final setting tells less, the pager that Git runs git log output through, what encoding to use.

Diffie-Hellman support in Node.js

Yay! My patch implementing support for Diffie-Hellman key exchange in Node.js has finally been merged into the Node.js master branch. This will simplify the OpenID for Node.js codebase a lot. It will also make the OpenID association phase run a lot faster, since the current code does Diffie-Hellman in Javascript while the Node.js crypto version does it all in native code using OpenSSL.

A brief API overview:

• crypto.createDiffieHellman(prime_length) creates a Diffie-Hellman key exchange object and generates a prime of the given bit length. The generator used is 2.
• crypto.createDiffieHellman(prime, encoding='binary') creates a Diffie-Hellman key exchange object using the supplied prime. The generator used is 2. Encoding can be 'binary', 'hex', or 'base64'.
• diffieHellman.generateKeys(encoding='binary') generates private and public Diffie-Hellman key values, and returns the public key in the specified encoding. This key should be transferred to the other party. Encoding can be 'binary', 'hex', or 'base64'.
• diffieHellman.computeSecret(other_public_key, input_encoding='binary', output_encoding=input_encoding) computes the shared secret using other_public_key as the other party's public key and returns the computed shared secret. Supplied key is interpreted using specified input_encoding, and secret is encoded using specified output_encoding. Encodings can be 'binary', 'hex', or 'base64'. If no output encoding is given, the input encoding is used as output encoding.
• diffieHellman.getPrime(encoding='binary') returns the Diffie-Hellman prime in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.getGenerator(encoding='binary') returns the Diffie-Hellman prime in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.getPublicKey(encoding='binary') returns the Diffie-Hellman public key in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.getPrivateKey(encoding='binary') returns the Diffie-Hellman private key in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.setPublicKey(public_key, encoding='binary') sets the Diffie-Hellman public key. Key encoding can be 'binary', 'hex', or 'base64'.
• diffieHellman.setPrivateKey(public_key, encoding='binary') sets the Diffie-Hellman private key. Key encoding can be 'binary', 'hex', or 'base64'.

NOTE: The API is still subject to change.

I would appreciate getting a note if you actually do something useful with it. :) Play around with it and let me know what you think!

Mocking HtmlHelper in ASP.NET MVC 2 and 3 using Moq

Still having trouble mocking HtmlHelper? This is an update to my previous post on mocking HtmlHelper way back when ASP.NET MVC RC1 was released. Eric notified me through a comment on the post and a question on StackOverflow that the code for ASP.NET MVC RC1 did not work with ASP.NET MVC 2. The code in this post should work with ASP.NET MVC 2 and ASP.NET MVC 3 Preview 1.

Minimalistic MapReduce in .NET 4.0 with the new Task Parallel Library(TPL)

Among the news in .NET 4.0 are several additions by the [Parallel Computing Platform Team](http://blogs.msdn.com/pfxteam/). As I wandered through the documentation of the Task library with cloud computing and parallelism buzz in the back of my head, I got the idea of using tasks to create a minimalistic MapReduce. Here's the result, a rather crude and simple, but efficient MapReduce for you to play with and utilize!

What is MapReduce? 

For those of you who don't know what MapReduce is: MapReduce is a simplified interface for parallel data processing. MapReduce was initially described by the Google engineers Jeffrey Dean and Sanjay Ghemawat in the 2004 paper titled [MapReduce: Simplified data processing on large clusters](http://labs.google.com/papers/mapreduce.html).

MapReduce processes data by splitting the processing in to a set of transformations (in functional programming, this is called the "map" function (it maps or transforms an input to an output)). The results of the transformations are then combined into a single result (in functional programming, this is called the "reduce" function (it reduces a set of values to a single value)). On a sidenote, Linq has equivalent functions, but the names are different, presumably to make them more familiar to people with SQL knowledge. In Linq, map is called `Select`, and reduce is called `Aggregate`.

Shortly put, to process a huge set of data, you split the data into chunks and process each chunk in parallel. This eventually creates a new set of intermediary results, which is reduced to a single result.

Implementing a minimalistic MapReduce in .NET 4.0

The signature of my MapReduce function is static Task Start(Func map, Func reduce, params TInput[] inputs);

In other words, to start a MapReduce run, you supply a map function, a reduce function, and a set of inputs. Each input will be turned into an intermediate result (of type TPartial). Inputs are transformed concurrently. When all inputs are transformed, the reduce function is called to transform the partial results into a final result (of type TResult). Cool!

The map part is implemented by starting a task for each supplied input using Task.Factory.StartNew(() => map(input)).

The reduce part is implemented as a continuation of all the map tasks, meaning that the reduce task waits for all the map tasks to complete, and then executes. This is achieved using Task.Factory.ContinueWhenAll(mapTasks, tasks => PerformReduce(reduce, tasks)).
As you can see, the implementation is minimalistic and simple, and usage is likewise.

Here's a simple example using MapReduce to calculate the root mean square (MSE) of a set of values:

Actual applications of MapReduce are of course far more interesting than this simple example.

Applications of MapReduce

MapReduce can essentially be applied to any problem where you need a number of things to be done in parallel. It can even be applied in cases where you don't need a final result. Just return an arbitrary value as the result (or even better, implement a variant of my MapReduce which uses Action).

A few obvious use cases:

• Distributed search
• Distributed sort
• Tokenization
• Indexing
• Log processing
• Machine learning
• General artificial intelligence
• General data mining
• Large scale image processing
• ...

The list goes on and on, these are just a few things off the top of my head.

You can grab the source code for MapReduce here (requires .NET 4.0 or later):


As usual, play around with it, have fun, and let me know if you find it useful!

RSA using BouncyCastle

Trying to do RSA using BouncyCastle, but struggling to find your way around the API? In a previous post (see [here](/posts/why-cripple-the-net-rsa-implementation)) I pondered why the RSA implementation in `System.Security.Cryptography` is restricted to only the most common usage scenarios. I mentioned [BouncyCastle](http://bouncycastle.org) as an alternative for those who wanted a more flexible API, but never got around to providing examples where BouncyCastle was used. By request, this post provides usage examples by building a crude and simple, but efficient set of methods for RSA key generation, encryption, and decryption, all built on top of BouncyCastle.

NOTE: The general cryptographical security of the presented method is beyond the scope of the article. The code presented is not cryptographically secure for large data sets. If you're here looking for a way to do cryptographically secure RSA in the general case, you should look into more complicated approaches including padding, blinding, and more sophisticated block cipher modes. Cryptography is a topic undergoing constant research, so stay up to date and be sure to evaluate the strength of your solution for the scenarios in which you apply it.

BouncyCastle provides flexibility and control over your encryption approach, which comes at a cost. The BouncyCastle API might be a bit hard to cope with at first, but if you know encryption in general you should be able to find your way around the API without too much effort. This post will be focusing on RSA, since that was my original need, but it should be mentioned that BouncyCastle provides many other asymmetric (and symmetric) algorithms for which the usage is similar to what you find below.

Creating RSA keys

Creating RSA keys is a simple task. The method below lets you specify the key size in bits, and creates a key pair for you.

That's all there is to it.

Encryption

Now that we have a key pair, we are ready to encrypt and decrypt using RSA. In the example below, we use a key (public or private) to encrypt a byte sequence. To encrypt a string, simply convert the string to a byte array using Encoding.GetBytes.

The approach above uses a list to gather output for the sake of simplicity. Note that the RSA engine can only process a limited block size at a time (block size depends on the key size). The approach above processes a data set of an arbitrary size.

The above method does not impose constraints on which key you use for encryption. Use the public key or the private key as you see fit for your solution.

Decryption

The Decrypt method is very similar to the Encrypt method:

Again, it's up to you which key you choose to use. If you want to use the common approach, encrypt using a symmetric cipher, hash the data, and sign the hash with your private key using the above Encrypt method. If you want to use another approach like encrypting the actual data using your private key, you are of course free to do so.

I hope this post helps those of you who want to apply RSA (or any other asymmetric cipher) to more subtle cases than those supported by the .NET framework.

Mocking HtmlHelper in ASP.NET MVC RC1 using Moq

For those of you trying to mock HtmlHelper, but finding it difficult, here's a mock that works in ASP.NET MVC RC1.

The ViewDataDictionary that is passed to the HtmlHelper can be empty, or made to contain the data you want for your test.

Why cripple the .NET RSA implementation?

I just found out that `RSACryptoServiceProvider`, the RSA implementation in .NET, does not allow you to use a private key to encrypt data. I'm no cryptographic expert, but I do know how asymmetric key algorithms like RSA work, and that you can use a private key for encryption. That's how signing works. But why cripple the implementation and limit it to just signing?

Strongly typed data binding in Windows Forms

Windows Forms data binding is a great tool for model-view-style applications, where the connection between the model and its view is easily declared. However, data binding is also an error-prone and tedious process with no IntelliSense support where properties are specified as strings.

Tired of the need of looking up property names when you declare data bindings? Sick of mistyping a property name when you bind and not discovering the mistake until you run your application? Have a look at Strongbind.

Strongbind vs traditional data binding

Traditional data binding is typically declared like this:
Obviously, several errors can arise from this declaration: Your control may not have a Text property, or you might have spelled it wrong. The same goes the Description property of your source.

The same declaration in Strongbind is written as follows:

As you probably understand, the risk of mistyping is removed, and we get IntelliSense support out of the box.

Behind the scenes of Strongbind

To achieve this strongly typed data binding, Strongbind uses a technique known as proxying. Strongbind dynamically generates a proxy for your business object and your control, and uses the proxies to intercept the calls to the property getters during runtime to declare the data binding. Hence, you need to declare a bindable source and bindable target first to create the proxies, and then use these proxies during the binding declaration. You will get a runtime error if you try to use your real objects when declaring data bindings.

Limitations of Strongbind

Although Strongbind makes data binding a far more declarative process, the library does have its limitations.

Controls containing ActiveX components are not supported. If the control containing an ActiveX component is a custom control created by you, you can get around it by declaring an interface for
the control and specifying that as the type to use when declaring your binding source:

Also, binding to concrete binding sources with non-virtual properties is not supported. Again, the recommended workaround is to create an interface for your binding source and use that when declaring the data bindings. (You always want to create these interfaces, since decoupling your objects' interfaces from their implementation is recommended for testability, maintainability, and is generally A Good Thing(tm).)

Apart from these two issues, which can be worked around in most cases, Strongbind should work flawlessly. If it does not, please let me know.

Where do I get it?

Strongbind is an open source project hosted at Github. To get the latest version, check out the code from its repository.

Strongbind is still in an early development stage, so no releases have been created yet. I still encourage you to check out and start using the library as soon as possible, though. A beta will be released as soon as I feel comfortable doing it.

If you want to contribute to Strongbind, I happily welcome you to do so.

Bind away!

AJAST - Cross-domain REST calls using JSON injection

The typical (and original AJAX) approach to calling web services asynchronously from a browser uses the `XMLHTTPRequest` object to request data asynchronously. However, as most of you probably already know, requests made using this object are restricted to the same domain as the script they originate from. This means that in order to request data from services like Google Maps, Flickr, etc. you need to implement a server-side proxy on your domain to use the `XMLHTTPRequest` object. But what if you want to stay on the client side? Enter JSON injection.

JSON injection, or actually script tag injection, is a rather common technique that circumvents the `XMLHTTPRequest` limitation by dynamically injecting script tags into the calling page. A script tag can have any domain as its source, which means that cross-domain calls are possible. The technique is also referred to as JSON callbacks, although it really is not limited to JSON payloads. The technique is also referred to as JSONP, although [the original JSONP](http://ajaxian.com/archives/jsonp-json-with-padding) is a bit more extensive than just callbacks using script injection.

It really is a neat technique without a cool term. JSONP is a term for a superset of JSON injection. A more precise term than JSON injection would be Javascript injection, but that's already used to describe vulnerabilities in web pages where malicious Javascript code is injected through e.g. links from external sites. I hereby propose the term AJAST - Asynchronous Javascript And Script Tags. At least it'll be the name of my implementation. If it doesn't catch on for anything but that, we'll even be a bit more confused than we already are. Now, what do we require to do AJAST?

AJAST requirements

The requirements an AJAST request lays on the server-side are the following:

1. The server must provide its services through HTTP GET requests.
2. The client must be able to supply the name of a callback function that the response will be wrapped in.
3. The server is expected to provide a response on the form callback(payload), where callback is the name of the callback function supplied by the client, and payload is the payload returned by the server. The payload can be XML, JSON, or any other form of data that the Javascript callback function can accept as a single argument.

These requirements are already fulfilled by many REST services, but they are still hard to use in an AJAST fashion due to client side challenges. The two main requirements for an AJAST library are:

1. Complete handling of requests. Nothing more than a URL and a callback should be needed to create a request.
2. Timeouts is a show stopper for AJAST. With script injection, it is difficult to know if a call completes, and from an AJAST usage perspective it is essentially impossible to create a decent solution without knowing if requests complete or not.

Security is another obvious challenge, although in my view it is a challenge for the Internet in general rather than AJAST in particular. Developers creating cross-domain applications should be aware of the security risks involved, and take measures to prevent security breaches accordingly. There is no silver bullet.

Although several examples for implementing an AJAST request are found around the web, I found no fully functional stand-alone implementations. Dan Theurer's article on script requests provides code that can be used to create an implementation, but leaves the timeout problem unsolved. Toolkits such as Dojo also implement variations on the approach using IFrame requests, but they are a lot more hairy, and I really don't want a framework (or parts of it) bloating the web site I am creating just to be able to do AJAST. I want a library that can perform just the task that I want it to perform, and perform it well.

An AJAST library

So, I decided to create my own AJAST library, OX.AJAST, complete with the following features:

• A fully encapsulated mechanism for making AJAST calls. You simply supply a URL, the name of the callback parameter that will be appended to the URL, and a callback function.
• Support for timeouts. Remote requests can of course time out, and time outs need to be handled. Apart from the obvious security challenges involved with using AJAST (note that I'm not saying they're defects, they're challenges for us developers to handle) , this is the hardest challenge for AJAST requests. Without the ability of specifying timeouts, we're essentially in the dark with regards to whether or not a request will complete. OX.AJAST neatly supports timeouts by wrapping the supplied callbacks, putting on a timer, and checking for completion when the timer times out.
• Guarantee that the callback function will be called. Whatever happens, and as a direct consequence of the timeout support, the library guarantees that the callback function will be called. For this reason, the callback function must accept two arguments, the first a boolean indicating if the request succeeded or not, the other a string containing the response from the call. If the first argument is false, the call may have timed out or failed.

Here it is in all its glory:

Using the AJAST call function

There are two ways of using OX.AJAST. The simplest is to use the call function.

The call function will execute the request by appending &callback=wrapper to the URL and injecting a <script> tag with the final URL as the src attribute. This will add a call to the DOM as soon as the data is received, which the browser will execute.

The function called from the injected script tag is a wrapper around the callCompleted function provided to the call function. The wrapper function is created by the call function, and handles timeouts and deletion of the script tag after the callCompleted function completes. As mentioned, by using this wrapper, the AJAST library can guarantee that callCompleted will be called, which significantly eases the handling of asynchronous calls for users of the library.

The function also allows you to specify how long the request will wait for a response before it times out. The default timeout is 5 seconds. Finally, you can pass an argument specifying if you want the response to be automatically decoded from JSON before it is passed to your callback function.

As stated above, all callback functions must be on the form callbackfunction(success, data){}, where success indicates whether or not the asynchronous call succeeded, and data is any data that was received from the call. It is also important to note that data may be undefined, but success will always be true or false.

Using the AJAST broker

The AJAST broker encapsulates a common pattern for REST requests using HTTP GET. Many RESTful services found online typically use some kind of root URL of the form http://xampl.com/rest as the base URL for all their REST services. The query string determines which service is requested, as well as the arguments for the service.

For the services that follow this pattern, the AJAST library provides a Broker class that encapsulates the process of calling the REST services.

The example below shows how the request from the first example can be made using the broker.

The broker also supports the specification of a timeout limit, automated JSON decoding, and also provides the option of passing a set of default arguments that will be passed with every request, such as a Flickr API key.
Now let's do something useful with it.

A real example: Flickr using AJAST

To keep the example as simple as possible, we'll create the functions necessary for a page which fetches the most recent photos from Flickr.

Luckily, Flickr supports REST and JSON callbacks in a lovely manner, so we'll use the broker for our calls.

We've told the broker to call a function named recentFetched`when the recent photos have been fetched, so let's implement that as well. To keep the example simple, we'll just append the photos to the body of the document.
Now we just have to call flickrGetRecent from somewhere in a document, and the most recent photos will be appended to the document.

As you can see, the OX.AJAST library is really easy to use, and enables you to do pure client-side REST service calls across domain boundaries with hardly any effort. I hope you find it useful. Drop a comment if you have problems or suggestions, or if you create improvements to it. Now start using AJAST!

Object-Relational Mappings Considered Harmful

Creating an Object-Relational Mapping (ORM) has become the de facto way of handling persistence in the object-oriented programming paradigm. Almost all systems require some form of persistent state, and relational databases have become the de facto place to put that state. Relational databases are proven, scale well, and organize data in a tabular manner suitable for many of the real world problems that we try to solve, so they are an obvious choice. Choosing them, however, means we have a new problem at our hands, known as the object-relational impedance mismatch.

The problem is that a relational database is not suited for storing an object-oriented model. An object is almost always a non-scalar value, meaning that it won't fit well in a table row. Hence, we have to create a schema suitable for persisting our objects to a set of tables. This schema will be different for each type of object we have, so it's a rather tedious task, but it solves the problem.

As always, we developers tend to dislike repetitive tasks, so we try to simplify and automate the extra work involved with creating an ORM. This has led to various design patterns such as DAO, and in recent years a set of fully automated ORM tools such as Hibernate (Java), NHibernate (.NET), SQLAlchemy (Python), and Propel (PHP) have become very popular. These tools offer a highly transparent solution to the ORM problem, at the addition of a cost that varies heavily with the nature of the ORM problem. Still, their presence moves us closer and closer to a solution - we are creating an abstraction that fully encapsulates the difference between a relational database as our storage for persistent objects, and the objects themselves. Hopefully, we will soon be able to create an ORM that induces a linear or perhaps even constant cost on our solution, regardless of the nature of the problem we are trying to solve.

But wait. The real problem we are trying to solve is how to persist the state of our objects, remember? The database is indeed a store, and hence a candidate solution, but using it introduces an impedance mismatch, which is another problem we need to solve. It really is like trying to fit a square block in a round hole.

Let's stop trying to solve the wrong problem of creating an ORM, and start finding a solution to our initial and real problem of persisting our objects. Recent innovations such as LINQ have taken us a step in the right direction by making persistence an integral part of the language, but we're still a long way from automated persistence for our objects. I am certain, though, that moving focus away from the challenges of an ORM and on to the challenge of persistence in general will take us further.

Let's do that.

(And I'm sorry for reusing Dijkstra's already overused] phrase, but I couldn't resist.)

Extension methods for copying or cloning objects

C# 3.0 includes a new feature known as extension methods, and fiddling with it triggered the idea of creating a mechanism for copying or cloning (virtually) any .NET object or graph of objects. The manifestation of that idea has become a rather decent little framework for copying objects. It performs a deep copy as automatically as it possibly can, and provides mechanisms to easily solve many of the cases which cannot be covered automatically. It is great for copying your custom object hierarchies, and saves you the pain of a solution like implementing ICloneable for an entire hierarchy of objects.

Let's start off with a few words on extension methods. They are best explained through an example. Let's say we want to be able to calculate area given size. Wouldn't it be nice to be able to add GetArea to the already existing Size class? Well, let's do so!

As you can see, the new syntax simply allows you to tell the compiler that the this of this method is a Size. This means that the method is an extension of the Size class.

As mentioned, I had the idea of extending the very base of the C# class hierarchy (System.Object) with a method for copying or cloning "any" object. Obviously, the method cannot automatically copy _any_ object, since it cannot possibly know how to construct an object from an arbitrary class. Hence, a small framework needed to be created. The goals were to:

• Enable copying of many objects automatically.
• Enable copying of virtually any object with very little effort.
• Automate and hide away as much as possible (The KISS Principle).

The result is Copyable (pun intended).

The Copyable framework

Copyable is a small framework for copying (or cloning, if you will) objects. The straightforward way of using it is to just reference the assembly it's in from your project, and start copying!

The instance copy is now a deep copy of instance, no matter how complex the object graph for instance is. The relations in the copy graph is the same as in instance, but all objects in the copy object graph are copies of those in instance.

For the automated copy to work, though, one of the following statements must hold for instance:

• Its type must have a parameterless constructor, or
• It must be a Copyable, or
• It must have an IInstanceProvider registered for its type.

Besides the Copy method, The Copyable class and IInstanceProvider interface are the two major building blocks of the Copyable framework. Each of these blocks enable copying of objects that cannot automatically be copied.

The Copyable base class

Copyable is an abstract base class for objects that can be copied. To create a copyable class, you simply subclass Copyable and call its constructor with the arguments of your constructor.

This code above makes MyClass a copyable class. Note that if MyClass had had a parameterless constructor, subclassing Copyable would not be necessary.

MyClass can now be copied just like the previous example, e.g. MyClass b = new MyClass(1, 2.0, "3").Copy().

The introduction of the Copyable base class solves many problems, but not all. Let's say you wanted to copy a System.Drawing.SolidBrush. This class does not have a parameterless constructor, which means it cannot be copied "automatically" by the framework. Also, you cannot alter it so that it subclasses Copyable. So, what do you do? You create an instance provider.

The IInstanceProvider interface

An instance provider is defined by the interface IInstanceProvider. As the name clearly states, the implementation is a provider of instances. One instance provider can provide instances of one given type. The Copyable framework automatically detects IInstanceProvider implementations in all assembies in its application domain, so all you need to do to create a working instance provider is to define it. No registration or other additional operations are required. To simplify the implementation of instance providers and the IInstanceProvider interface, an abstract class InstanceProvider is included in the framework.

This implementation will be used automatically by the Copyable framework. NOTE: To be usable, the instance provider MUST have a parameterless constructor.

The instance provider pattern does not solve the case where you want different initial states for your SolidBrush instances depending on which context you use them for copying. For those cases, an overload of Copy() exists which takes an already created instance as an argument. This argument will become the copy.

Limitations and pitfalls

Although this solution works in most cases, it's not a silver bullet. Be aware when you copy classes that hold unmanaged resources such as handles. If these classes are designed on the premise that their resources are exclusive to them, they will manage them as they see fit. Imagine if you copied a class which holds a handle, disposed one of the instances, and continued using the copy. The handle will (probably) be freed by the original instance, and the copy will generate an access violation by attempting reading or writing freed memory.

That's it! The Copyable framework can be downloaded from Github. For those interested in reading more on extension methods, MSDN provides an excellent explanation in the C# Programming Guide, and Scott Guthrie has an introduction article here.

Enjoy Copyable, and please let me know if you find it useful or come across any problems with it.

UPDATE 2009-12-11: Due to popular demand, I have made the source code for Copyable available under the MIT license.

UPDATE 2010-01-31: The requirement of parameterless constructors has been removed in the latest version of Copyable available on Github. A new release will follow soon.

Five advices on implementing a cache

I've spent the last few days at work implementing a cache in the data access layer (DAL) of one of our services. The cache works great, and speeds up our service very much in some cases, and somewhat in all cases. I've implemented caches before, and experienced many of the difficulties that arise when introducing a cache. It always seems rather easy, and always has unwanted side effects. The general advice is of course not to do it (and the advice from the database guys are always not to do it), but here are my five best advices on what you should consider if you decide to do it.

1. Make sure the cache is transparent.

The system shall not in any way notice that there is a cache handing objects to it instead of the database, nor should the introduction of a cache require changes in any layers above the layer where the cache resides. If you decide that changes are required, be aware that you are making changes to code that does work, and that re-verifying its behavior is a hard and expensive task.

2. If your system is transactional, make sure that the lifetime and mutability of objects in your cache matches your transaction isolation level.

The obvious solution for a cache in a transactional system is a cache that lives per transaction, but this is not guaranteed to work. As an example, let's say that transaction 1 reads some data and starts operating on them. Meanwhile, transaction 2 reads, alters, and commits some data that partially or fully depends on the data read by transaction 1. After the commit, transaction 1 reads some other data that depend on the new data committed by transaction 1. In this case, a cache in transaction 1 alters the system behavior if a too low isolation level is used (the alteration is most likely correct, but does not need to be, and it most certainly changes the behavior nonetheless) (see [this Wikipedia page](http://en.wikipedia.org/wiki/Isolation_(computer_science) "Transaction isolation") for an explanation of transaction isolation levels). Be aware of your isolation level, and know also that the default isolation level is different from RDBMS to RDBMS. As an example, MySQL uses repeatable read as the default isolation level, while MS SQL Server uses read committed. With an isolation level less than repeatable read, a cache with any mutable data in it is essentially useless.

3. Use the cache for immutable data as much as possible.

Also, use the cache for mutable data as little as possible, since this significantly increases the difficulty of the cache implementation, and with it the risk of errors.

4. Give the objects in the cache an as short lifetime as possible.

When implementing a cache, you want the objects in your cache to live as long as possible, since accessing the cache is much faster than accessing the database. Well, think about this: A cache with objects that live forever is actually a replacement for your database, which is not what you want to achieve. To avoid an implementation that is difficult, hard to verify, and unnecessary complex, make objects live as short as possible, while maintaining an increase in speed.

5. Be absolutely certain that the keys you use in your cache identify objects uniquely and unambiguously.

This sounds obvious, but with complex object hierarchies and caching of different parts of the hierarchy at different levels, it suddenly becomes very hard. In general, cache either the top-most or the lower-most objects in your hierarchy. The choice of an approach depends on how you access your data. The best way to decide which approach to take is to do a thorough analysis of your data and the objects that represent them, how you access these objects, how you use them, and in which cases you are most likely to gain speed by introducing a cache.

LINQ vs Loop - A performance test

I just installed Visual Studio 2008 beta 2 to see what the future holds for C#. The addition of LINQ has brought a variety of query keywords to the language. "Anything" can be queried; SQL databases (naturally), XML documents, and regular collections. Custom queryable objects can also be created by implementing IQueryable. Sadly, like every abstraction, these goodies all come at a cost. The question is how much?

I decided to create a simple test to see how much of a performance hit LINQ is. The simple test I deviced finds the numbers in an array that are less than 10.

Initially, I assumed the performance impact would not be too large, since its equivalent is the straightforward imperative loop, which should not be too hard for a compiler to deduce given static typing and a single collection to iterate across. Or?

    LINQ: 00:00:04.1052060, avg. 00:00:00.0041052
    Loop: 00:00:00.0790965, avg. 00:00:00.0000790

As you can see, the performance impact is huge. LINQ performs 50 times worse than the traditional loop! This seems rather wild at first glance, but the explanation is this: The keywords introduced by LINQ are syntactic sugar for method invocations to a set of generic routines for iterating across collections and filtering through lambda expressions. Naturally, this will not perform as good as a traditional imperative loop, and less optimization is possible.

Having seen the performance impact, I am still of the view that LINQ is a great step towards a more declarative world for developers. Instead of saying "take these numbers, iterate over all of them, and insert them into this list if they are less then ten", which is an informal description of a classical imperative loop, you can now say "from these numbers, give me those that are less than ten". The difference may be subtle, but the latter is in my opinion far more declarative and easy to read.

This may very well be the next big thing, but it comes at a cost. So far, my advice is to create simple performance tests for the cases where you consider adopting LINQ, to spot possible pitfalls as early as possible.