When is it not worth writing a Unit Test?

I’m currently working on a e-commerce site based on ASP.NET MVC 5 for a new venture (in stealth mode, so I can’t say what it is). I have been working on it for over six months and we set a target date to get a new Beta2 version of the site up by end of September (yesterday).

That puts a bit of pressure on development and I had a couple of interesting decisions around Unit Testing that I thought I would share.

I also include some quotes from one of the giants of Software Design, Martin Fowler, and at the end I give some criteria to help you decide what to test, and maybe what you might not test.

Setting the scene

I expect everybody who comes to this article will know about Unit Testing (here is link to a definition in case you are new to this). I also expect you have an opinion, ranging from “you need to have 100% test coverage” through to “Unit Testing is a waste of valuable development time”.

All my articles are based on my real-life experiences. In this one I found, just before the deadline, that a small piece of code took maybe 10 minutes to write and 1+ hours to write the Unit Tests. On the other had I realised two days earlier I had written another piece of code which took about the same time to write, but I didn’t write any Unit Tests for it.

On reflection I decided that both decisions were correct. In this article I go through my reasoning to see why I think this way. You may disagree with my reasons (please do!), but hopefully I will make you think.

Before I start let me give you my first quote from Martin Fowler. If you read my history you will see I came back to programming in 2009 after a 21 year break. I gobbled up all the reading I could find, and Martin Fowler was a great help to put me on a better track.

It is better to write and run incomplete tests than not to run complete tests.

Martin Fowler in the book ‘Refactoring’

1. Code I tested

Below is the code I wrote in about 10 minutes, which I then took over an hour to test. To save you reading it the name of the method says what it does, i.e. if the justification of a field is changed, say from Left to Centre then it changes the x position to match.

private void HandleChangeOfJustifyAndCompensatePosition(
    FieldJson fieldJson, CrudAdvancedStylesDto dto)
{
    if (fieldJson.styleInfo.justify != dto.Justify)
    {
        //The justification has changed, so we compensate the position

        if (fieldJson.styleInfo.rotation != 0 || dto.Rotation != 0)
            throw new NotImplementedException(
            "HandleChangeOfJustifyAndCompensatePosition does not currently support rotation.");
        if (fieldJson.styleInfo.shear != 0 || dto.Shear != 0)
            throw new NotImplementedException(
            "HandleChangeOfJustifyAndCompensatePosition does not currently support shear.");

        var leftX = 0.0;
        switch (fieldJson.styleInfo.justify)
        {
            case JustifyOptions.Left:
                leftX = fieldJson.xPosMm;
                break;
            case JustifyOptions.Center:
                leftX = fieldJson.xPosMm - (fieldJson.widthmm/2);
                break;
            case JustifyOptions.Right:
                leftX = fieldJson.xPosMm - fieldJson.widthmm;
                break;
            default:
                throw new ArgumentOutOfRangeException();
        }

        switch (dto.Justify)
        {
            case JustifyOptions.Left:
                fieldJson.xPosMm = leftX;
                break;
            case JustifyOptions.Center:
                fieldJson.xPosMm = leftX + (fieldJson.widthmm / 2);
                break;
            case JustifyOptions.Right:
                fieldJson.xPosMm = leftX + fieldJson.widthmm;
                break;
            default:
                throw new ArgumentOutOfRangeException();
        }
    }
}

Now, you might ask, why did it take me so long to write a test for such a small method. Well, it is a private method so I can only get to it via the business logic that uses it (Note: I really don’t want to change the natural design the the software to make this method easier to test. I have found that leads to all sorts of other problems).

So, why did the tests take so long? Here is a quick summary:

  • The new method added to an existing existing piece of business logic that already had quite a few tests, so it wasn’t a green field site.
  • The previous tests needed a significant amount of set-up, especially of the database, to  run. These needed changing.
  • Because the original tests had been written some time ago they didn’t use the new database mock I had developed, which would help with the new tests. I therefore decided to move all the tests in this group over to the mock database.
  • The Mock database didn’t have some of the features I needed so I had to add them.
  • … I could go on, but I am sure you get the picture.

This leads me onto a key point.

I find 80%+ of the work of writing Unit Tests in about getting the right set-up to allow you to exercise the feature you want to test.

So let me explore that.

Set up of the test environment

In any significant software development there is normally some sort of data hierarchy that can be quite complex. Often to test say one bit of business logic you need to set up other parts of the system that the business logic relies on. When this is stored on a database as well then it makes setting up the right environment really complex.

This might sound wrong for Unit Testing, which is about isolating sections of code and just testing that, but I find in the real world the lines are blurred. In the example code about I really don’t want to make a separate class, which I inject at runtime, just to isolate that method for testing. IMHO that leads to bad design.

Over the years I have tried lots of approaches and here is what I do:

1. Write ‘Set-up Helpers’ to create data for Unit Tests.

My test code is has a Helpers directory with a range of Helper classes (27 so far in this six-month+ project) which create data suitable for testing specific parts of the system. In the past I skimped on these, but I soon learnt that was counter-productive.

Most build the data by hand, but a few load it from json or csv files held in the Test project.

2. Use mocks where you can.

Mocks, or fakes, are pieces of code that replace a more complex subsystem or class. Mostly they replace a system library, but sometimes replace some of your own code so you can trigger thing that the normal code would rarely do, i.e. error conditions.

Mocks/fakes are great and I used them as much as I can. They can provide really good control of what goes on for a small about of work. I personally use Moq, which I find good.

However there are some limits to replicate all the feature of a big system like the Entity Framework ORM (EF), or some of the MVC Controller systems. I do have a mock for Entity Framework, which is pretty good, but it doesn’t do relational fix-up and current doesn’t support .Include().

2. Decide on how you will use a real database

I have found out the hard way that I need to use a real database for testing. There is down side to that of testing taking a lot more time to start (EF6 has a fairly long start-up time). However I have found, up to now (see comment at end of this section), the only way to truly check some code out.

There are a number of ways you can go, from clearing and reloading the database for every test up to just adding to the database. I have found clearing and reloading pretty difficult/time consuming, especially with complex foreign keys, so I use unique strings, via GUIDs, in strategic places to create a unique set of entries for a test.

Whatever way you go you need to think through a system and keep to it, otherwise thinks get messy.

Possible future options on EF?

  • Two separates readers of this blog have recommended Effort Library, which provides an in-memory EF database. That sounds like a great idea, but once I am in a project I try not to make big changes to the system so I haven’t tried it yet, although EF 7 may be better (see next).
  • Entity Framework 7 will have an in-memory database, which sound perfect for Unit Testing. I definitely plan to look at that.

Another Martin Fowler quote:

I’ve often read books on testing, and my reaction has been to shy away from the mountain of stuff I have to do to test. This is counterproductive…

Martin Fowler in the book ‘Refactoring’

The Unit Test framework has a BIG effect

I have found that the Unit Test framework is crucial. I started using NUnit many years ago and found it great, especially when run from inside Visual Studio via Resharper. However, when I started testing JavaScript I found other Unit Testing frameworks had some really useful features.

I will only summarise my thoughts on this because I wrote a long article called Reflections on Unit Testing in C# and JavaScript, which gives more detail. However let me give you the places where NUnit skews my Unit Testing approach.

NUnit does not have nested set-up calls

As I said earlier I find most of the work goes into setting things up to run a test. NUnit has a FixtureSetup/FixtureTearDown, which is run once at the start/end of a group of tests in a class, and a Setup/TearDown that is run before/after each test. These are useful, but not sufficient.

JavaScript Unit Test frameworks, like like Mocha and Jasmine , have a BeforeEach and AfterEach, which can be nested. Now why is that useful? I find that for some of the more complex methods to test I want to check multiple things when it is finished, e.g. was it successful, did it return the right data and did it change the database in the way I expected.

In Mocha and Jasmine I would have a group of tests with maybe an outer BeforeEach that sets up the environment and then an inner BeforeEach which calls the method, followed by separate tests for each of the parts I want to check. This means that a) the setup code is in one place, b) the call to the item under test is in one place and c) the tests are separately shown.

In NUnit I tend to find I either duplicate code or test lots of things in one test. (Note: If this doesn’t make sense then there is an example of nested set-ups in my article Reflections on Unit Testing in C# and JavaScript).

Possible future options:

One of my readers,

NUnit does not support async Setup/TearDowns

.NET 4.5’s async/await is a big change – see my full article on async/await for more. I find I use async/await in 90%+ of my  database calls and maybe 80% of my business logic. NUnit supports async tests, but not the FixtureSetup/FixtureTearDown or Setup/TearDown. That causes me to have to put more setup code in the actual tests.

2. The code I didn’t test

We have been off on a journey into why testing can be difficult. Now I’m back to the decision whether or not to Unit Test. Below is a fairly simple piece of business logic. It most likely took a bit longer than the first bit of code, as there was a DTO class to set up an the normal interface plumbing. What is does is find a customer order connected to something called a FOL.

public class GetFilledOutLabelLinksAsync : IGetFilledOutLabelLinksAsync
{
    private readonly IGenericActionsDbContext _db;
    private readonly IGenericLogger _logger;

    public GetFilledOutLabelLinksAsync(IGenericActionsDbContext db)
    {
        _db = db;
        _logger = GenericLibsBaseConfig.GetLogger(GetType().Name);
    }

    /// <summary>
    /// This returns information on the FOL and any associated order
    /// </summary>
    /// <param name="folId"></param>
    /// <returns></returns>
    public async Task<ISuccessOrErrors<FolAssociatedData>>
         RunActionAsync (int folId)
    {
        var status = new SuccessOrErrors<FolAssociatedData>();

        var fol = await _db.Set<FilledOutLabel>()
              .SingleOrDefaultAsync(x => x.FilledOutLabelId == folId);
        if (fol == null)
        {
            //very odd, but we fail softly
            _logger.WarnFormat(
                 "Could not find the FOL of id {0}", folId);
            return status.AddSingleError(
                  "Could not find the Label");
        }

        //Note: of FOL should only be associated with one LabelOrder,
        //but we have written this in a 'safe' way
        var associatedOrder = await _db.Set<LabelOrder>()
            .Include(x => x.Order)
            .OrderByDescending(x => x.Order.DateCreatedUtc)
            .FirstOrDefaultAsync(x => x.FilledOutLabelGuid == fol.GroupBy);
            
        return status.SetSuccessWithResult(
             new FolAssociatedData(fol, associatedOrder), "Got data");
    }
}

So why didn’t I write a Unit Test for this, and more importantly why am I happy with that decision? Well let me start my list of decison points with another quote from Martin Fowler.

 The style I follow (for Unit Testing) is to look at all the things the class should do and test each one for any condition that might cause a class to fail. This is not the same as “test every public method”, which some programmers advocate.

Testing should be risk driven; remember, you are trying to find bugs now or in the future.

Martin Fowler in the book ‘Refactoring’

My Criteria for Testing/Not Testing

I don’t have any hard and fast rules for Unit Testing, but over the years I have found, sometimes the hard way, what works for me. Here are some guiding principals, starting with the most important, that I use to decide if I need to test something.

Will it fail silently?

If the code could ‘fail’ silently, i.e. it goes wrong but doesn’t throw an Exception, then it needs watching! Any method that does maths, manipulates strings etc. can get the wrong answer and you need to check it, as it could cause problems and you don’t even know.

What is the cost of it going wrong?

It is a risk/reward thing. The more important/central the piece of code is to your application then the more you should consider Unit Testing it. (I can tell you that the payment side of my application is covered with Unit Tests!)

Will it change in the future?

If some code is likely to be changed/refactored in the future then adding Unit Tests makes sure it doesn’t get broken by accident. I love Unit Tests when it comes to Refactoring – I can change vast chunks of code and the Unit Tests will check I didn’t break it in the process.

Does it have lots of conditional paths?

Method with lots of conditional paths or loops are complex by nature. The logic could be wrong and its hard to test all the paths with functional testing. Unit Tests allow you to exercise each path and check that the logic is correct.

Does it have special error handling?

If the path through a method is only triggered on errors then Unit Testing may be the only way to check them out. If you added error handling then it must be important so maybe it should be Unit Tested.

I’m sure there are more, but I’ll stop there. You can always leave a comment and tell me what I have missed.

I should say there are some negative points, mainly around how hard it is to test. For me this isn’t the deciding factor, but it does play into the risk/reward part.

Final quote from Martin Fowler.

You get many benefits from testing even if you do a little testing. The key is to test the areas that you are most worried about going wrong. That way you get the most benefit for your testing effort.

Martin Fowler in the book ‘Refractoring’

Conclusion

So, if you look at the criteria and look back at the two pieces of code you will see that the code I tested a) could fail silently, b) will change in the future (see the NotImplementedExceptions), and c) had lots of conditional paths. It was hard work writing the tests but it was a no-brainer decision. (Note: I did find one bug around the tests for the not implemented ‘rotate’ and ‘shear’ checks).

The second piece of code wouldn’t fail silently, is unlikely to change and has one error path and one success path. You might also note the defensive style of coding as this is about a potential order a customer wants to make, so I don’t want it going wrong. Time will tell if that was a good decision or not, but I am happy for now.

Please feel feed to add your own comments.

Using .NET Generics with a type derived at runtime

I have used .NET Generic types a lot and found them really useful. They form a core part of the solution in my open-source GenericServices library and my private GenericActions library. This article explores why and how I use Generics and some of the techniques I use to hide the more complex type definitions.

Even if you are not interested in my particular use of Generics this article has some useful information on:

  1. How to create an instance of a generic type where the type is defined at runtime.
  2. The two approaches to calling methods or read/write properties of the created generic instance.
  3. The performance issues around the two approaches to calling methods or read/write properties.

Only interested in how to use Generics with a runtime-derived type? Just skip to here in this article.

What are Generic Types

Generics are everywhere – if you use List<T> etc. you are using Generics. If you are not familiar with Generics then look at this introduction. The primary thing about Generics is it allows you to write one code implementation that can then work on a range of types. In particular they produce efficient code because they are ‘type safe’, i.e. the type is known, so the code can access what it needs.

Without Generics we would either:

  • Have to write the same code for each type, which is bad practice.
  • Write one method but use Reflection to access the items we want, which is not type-safe.

The down side of Generics is you (normally) have to know the type at compile time. However I will show you later how to overcome this.

Where I have found Generic types really useful

I developed a web application called Spatial Modeller that does mathematical predictive modelling for healthcare. The modelling is complex and I used a  Domain-Driven Design (DDD) approach, which said I should keep the database/business classes focused just on the modelling problem. That worked really well, but it meant that the database/business classes weren’t right for display to the user.

I therefore had a Service Layer which had classes more attuned to the display and visualisation of the data. These are typically called Data Transfer Objects (DTOs), or in ASP.NET MVC they are also called ViewModels. The Service Layer code transformed the data passing between the Data/Business Layers and the Presentation/WebApi Layer.

The problem was I ended up with writing very similar code, but with different data types, for the transformation of the database/business classes to DTOs. That is both bad software practice, time consuming and boring!

After that project I was determined to create a solution for that. It took a bit of experimenting, but Generics was the answer. However the final solution wasn’t quite a straightforward as you might think.

The problem of a complex Generic type definitions

Let me give you a real example of where my presentation layer wants to run a piece of business logic. In Spatial Modeller to model a scenario the following happens:

  1. The user picks the potential hospital locations for dropdown lists and fills in various selections.
  2. These are handed to the business layer modeller as primary keys for the hospital locations and various enums/parameters.
  3. The modeller does its work, writing some data to the database.
  4. The modeller returns the results as primary keys.
  5. These need to be looked up show the user.

So, in that process we have five classes:

  1. The presentation input DTO, called DtoIn.
  2. The business modeller data input, called BizIn.
  3. The class/method to call in Business Layer, called IBizAction.
  4. The output of the business modeller, called BizOut
  5. The presentation output DTO, called DtoOut.

So, if we define a Generic class to handle this its class definition would look like this:

var service = new ActionService<DtoIn, BizIn, IBizAction, BizOut, DtoOut>
     (db, new BizAction(db));
var dataIn = new DtoIn(some data here );
var result = service.RunAction(dataIn);

That is a) quite hard to write and b) not easy to use with Dependency Injection and c) downright ugly! In fact in both of my libraries its even more complex than that, with different types of input. Even worse the code will be the same, but have different transform parts. In GenericActions I calculated there were sixteen versions, all of which would need separate implementations.

GenericServices was much easier, but still produced ‘ugly’ generic type definitions, which needed special handling at dependency injection time.

Getting rid of the ‘Ugly’ type definitions

OK, so Generics got me so far but I needed to do something else to combine and simplify the type definition of the Generics. I did two things:

  1. For both libraries I hide the complex type definition via a outer, non-generic helper class where the called method does a decode of the types and creates the correct Generic class instance with the right types.
  2. In the case of GenericActions, which has so many versions, I did a further decode of the in/out types inside the Service Layer code, which reduced the sixteen possible versions down to six. I could have gone further, but this was the right level to keep performance up.

Before I describe the solution I will show you the improved code for the example in the last section (see below). I think you will agree that is much nicer, and the creation of the service is now very Dependency Injection friendly.

var service = new ActionService<IBizAction>(db, new BizAction(db));
var dataIn = new DtoIn(some data here);
var result = service.RunAction<DtoOut>(dataIn);

So, let us go through the steps to achieve this.

Part 1: Decode the underlying types

I’m going to use an example from GenericServices, as it is open-source and I can therefore link to the actual code. So the example for GenericServices is:

var service = new UpdateService(db);
var dataIn = new DetailPostDto(some data here);
var response = service.Update(dto);

The steps are:

  1. service.Update<T>(dto) is a top-level method (see first class) that can take either a database class, like Post, or a dto.
  2. If it is a dto then it is derived from abstract EfGenericDto<TEntity, TDto>. This forces the dto definition to include the database class. See the DetailPostDto.cs as an example.
  3. The Update method calls DecodeToService<UpdateService>.CreateCorrectService<T> (see this class, first method). It gets a bit complicated in this class because of sync/async versions of DTO, but the bottom line is it:
    1. Finds if it is a EfGenericDto or not. If not then it assumes its a direct access.
    2. If it inherits from EfGenericDto it finds the database class and the Dto class.

Bookmark

Part 2: Create the Generic Class from the runtime types

Ok, the last process found whether it was a database class, or a dto class. There are different classes to handle these two cases. The direct update has one Generic Type parameter, being the database class. The dto version has two Generic Type parameters: the database class and the dto class. However in the example below I look at a very simple case to make it easier to understand. GenericServices is a bit more complicated, but follows the same approach.

The code to create the instance of the Generic class is petty straightforward. For example if I wanted to create a List<string> at runtime I would

  1. Produce an array of the type(s) need to form the generic type, in this example ‘string’
  2. Get the generic type, in this example ‘List’
  3. Combine them using the ‘.MakeGenericType’ method
  4. Create an instance of that type using ‘Activator.CreateInstance’

The code below shows an example of creating ‘List<string>’ at runtime.

var dataType = new Type [] { typeof(string)};
var genericBase = typeof(List<>);
var combinedType = genericBase.MakeGenericType(dataType);
var listStringInstance = Activator.CreateInstance(combinedType);

Part 3: Calling methods from the created instance

You should be aware that ‘Activator.CreateInstance’ method returns an object, so you can’t just ‘listStringInstance.Add(“hello world”)’ as the compiler will something like “‘object’ does not contain a definition for ‘Add’“. You have two choices:

1. Use Dynamic Type

You can place the output of the ‘Activator.CreateInstance’ into a dynamic type (see start of line 4). This turns off compile time type checking which allows you to call any method, or access any property, with the type checking done at runtime. So in our List<string> case it would look like this:

var dataType = new Type [] { typeof(string)};
var genericBase = typeof(List<>);
var combinedType = genericBase.MakeGenericType(dataType);
dynamic listStringInstance = Activator.CreateInstance(combinedType);
listStringInstance.Add("Hello World");

Dynamic is easy to use, and allows much better freedom. However the dynamic runtime library takes a lot of time on the first call of the method. See Performance section for more detailed analysis.

2. Use Reflection

Reflection allows you to find methods, properties by name. You can then call the method or access the property via different Reflection methods. In our example we are calling a simple method ‘Add’ in the instance type (would be null if no method of that name existed) and then ‘Invoke’ that method, i.e.

var dataType = new Type [] { typeof(string)};
var genericBase = typeof(List<>);
var combinedType = genericBase.MakeGenericType(dataType);
var listStringInstance = Activator.CreateInstance(combinedType);
var addMethod = listStringInstance.GetType().GetMethod("Add");
addMethod.Invoke(genericInstance, new object[]{"Hello World"});

The reflection approach does have some complications, such as method or property accesses returns an object, which can be a problem if you need to type it. Also if the method had an output you need to build a generic Method using ‘MakeGenericMethod’. All in all reflection is harder to handle that dynamic.

However the reflection approach has a tiny first-use cost compared to the dynamic approach. However for lots of subsequent calls to the method then dynamic is quicker. See Performance section for more detailed analysis. (below)

Performance issues – simple types

The way you access the method or property does have a big effect on the performance of the command, as we are doing more. I care about performance so I have studied this in detail. There are two costs, one-off  first-use compute time and per instance compute time.

Let me start with a simple table giving the times for my simple List<string> example. What I did was run the same test on three types:

  1. Normal code, i.e var list = new List<string>(); list.Add(“Hello World”);
  2. Using reflection, i.e. listStringInstance.GetType().GetMethod(“Add”);
  3. Using dynamic, i.e. listStringDynamicInstance.Add(“Hello World”);

To take out any first-use costs I ran it on List<string> twice, followed by List<DateTime> three times, as it did seem to change. The List<DateTime> is there to check if building a different type has the same first-use cost. Here are the results, which were measured using the ANTS Profiler.

BE WARNED: This table is a bit misleading by implying refection is always faster than dynamic – it isn’t. See the ‘Performance issues – GenericAction library’ section where a real-life example shows that dynamic wins in the end.

Type Order Compiled Reflection dynamic
List<string> 1. First 0.8 ms 37.08 ms 5.0 ms
(was 600ms)
List<string> 2. Second < 0.001 ms 17.05 ms 1.0 ms
List<string> 2. Third < 0.001 ms 0.01 ms 0.6 ms
List<DateTime> 3. First (after List<string>) < 0.001 ms 0.03 ms 2.7 mS
List<DateTime> 4. Subsequent < 0.001 ms 0.03 ms 0.5 ms

 

When I first started testing I was getting 600ms for the first-use cost on the first dynamic method call, of which about 400 ms comes from the Dynamic Language Runtime. However once I installed .NET 4.5.2 on my windows system this dropped to 5ms. I cannot confirm that as the release notes do not say there is a change there. Maybe it was just reinstalling the .NET library. Anyway, be warned that things might have changed.

The way that .NET handled dynamic types is complex and I don’t claim to understand it properly. I would refer you to Simon Cooper’s series on ‘Inside DLR’ (Dynamic Language Runtime) starting with ‘Inside the DLR – Callsites‘ and then look at ‘Inside DLR – Invoking methods’ which describes the caching of methods.

Performance issues – GenericAction library

In GenericServices and GenericActions I have used the dynamic approach. Seeing the figures above made me wonder if that was the best way. I therefore forked a version of GenericActions and changed it to use a Refections approach with no use of dynamic anywhere. Here are my findings:

  1. When comparing the dynamic-based and the Reflection-based versions of GenericActions the first-use costs of dynamic are much less than the table above suggests. The figures are:
    1. Refection-based: 300 ms first-use cost due to AutoMapper first-use costs.
    2. dynamic-based: 700 ms first-use cost, which, if you exclude the AutoMapper part, means that the dynamic part is only 400 ms. I haven’t measured after installing .NET 4.5.2 but it now seems quicker.
  2. If a created instance is used many times then dynamic wins in the end. As an example my test of system on a 1000 accesses then dynamic was between 120% and 200% faster than reflection (on my tests anyway).
  3. Dynamic is much easier to use. I found it really hard to do everything with reflection, mainly because you cannot cast something. My altered GenericActions version worked, but some of the error checking on incorrect types did not work any more. I might have fixed them, but it wasn’t worth it.

So, lets look at the different parts of the performance problem.

1. One-off first-use compute time costs

With dynamic there is a cost on first decode and call of the method in my libraries which use dynamic – about 0.2 seconds or more on my system. As I explained earlier this because the first time you call a method in a dynamic type its needs to be created, which takes some time. However the method is then cached so later calls are very fast.

Any first-use performance cost is a pity, but I have other first-use time costs, like MVC, Entity Framework, AutoMapper etc., so I accepted , mainly because I can mitigate it (see next section) and overall the dynamic approach is faster.

Mitigating first-use costs

One thing that really helps is keeping the application alive so that the first-use cost only happens when you start/restart your application. I mainly develop web applications and one great feature introduced in .NET 4.5.1 was ‘Suspend‘ on an ASP.NET web site.

On shared hosting (maybe other hosting as well, not sure) when a web site has no traffic, i.e. all user sessions have timed out, then it stops. This means for low-traffic sites the web application would stop and the next user coming to it would have to wait for it to start. The ‘Suspend’ feature keeps the application alive so that this does not happen.

Suspend support does depend on your hosting provider setting up the feature in IIS so please read this post on it. My non-Azure shared hosting provider WebWiz supports this. Azure web sites doesn’t support ‘Suspend’ but fixes this with an ‘Always On’ feature which polls the site regularly enough that it stays up.

2. Per instance compute time

The whole decode and create takes about 0.0024 ms on my system, excluding first-use compute time. The actual creation of an instance of the generic type isn’t that costly (I haven’t measured it separately), but it is the decoding of the types etc. that take the time.

In the case of GenericServices the database call is so large, > 1 millsecond in all measured cases, that its not worth trying to improve the decode/create time.

However in GenericActions, which may call a very simple business method which could return very quickly, it is worth looking at. I therefore implemented a system caching system using a ConcurrentDictionary in my GenericActions library. It took quite a bit of tuning but the effect was worthwhile, as it brought the worse case, which has multiple decode calls, down from 0.037 ms to 0.018 ms. These are small figures, but I think worth the effort to make calls to the business logic have a low performance cost.

Conclusion

I have described not only a way of creating a Generic Type from a runtime type, but a whole system for using complex Generic Types, yet still having type definitions that are simple to use and Dependency Injection friendly. I have also looked how you call the methods or access the properties in the runtime created type, focusing on ease of use and detailed performance figures. I hope that is useful to people.

Happy coding!

Announcing NuGet release of GenericServices

I am pleased to say that I have finally released a version of the GenericServices on NuGet!

For those who are not aware of GenericServices it is an open-source .NET library designed to help developer build a service layer, i.e. a layer that acts as a facard/adapter between your database in the data layer and your User Interface or HTTP service.

GenericServices makes heavy use of Entity Framework 6 – EF6 and .NET 4.5’s async/await commands. Its aim is to make the creation of the service layer simple while providing robust implementations of standard database access commands. You can get a summary of its features in from the Why GenericServices? page on the GenericServices documentation site or the README page on the GenericServices’ GitHub home page.

Link to NuGet Version

The library is known as GenericServices on NuGet, so simply search with that name in the ‘Manage NuGet Packages’ in Visual Studio. You can also find it on NuGet at https://www.nuget.org/packages/GenericServices/ .

Why have you released on NuGet now?

I did not feel confident to release GenericServices onto NuGet, where it is likely to get much more use, until I had good documentation and examples. In the past two weeks I have:

  1. I have written quite a bit of documentation and added a useful index in the sidebar. Do have a look at the documentation in the GenericServices’ Wiki.
  2. I have made available a much more advanced example called SampleMvcWebAppComplex. SampleMvcWebAppComplex is a ASP.NET MVC5 web site which was built as a ‘stress test’ of how well GenericServices library and Microsoft’s Entity Framework V6 could cope with an existing SQL database AdventureWorksLT2012.
    1. The example web site is at http://complex.samplemvcwebapp.net/.
    2. The source code is available as a reference source. Note: this project is not fully open-source because it uses a paid-for library, Kendo UI MVC.
  3. There are two articles I wrote on the Simple-Talk web site that describe in detail the process I went through to build SampleMvcWebAppComplex. They are:

I still have the original, more basic example web site called SampleMvcWebApp. You can access this via:

  1. The live example web site is at http://samplemvcwebapp.net/
  2. The source code, which is fully open-source, is available too.

Happy coding!

Alpha release of GenericServices

Quick Summary
This post introduces my new GenericServices framework designed to lighten the development load of building a  service layer.
This post summarises what GenericServices is about and the motivation behind it. It also provides links to code source and the example web site.

I am pleased to announce the alpha release of my new Open Source project called GenericServices available on GitHub. It will also be available on NuGet when it is has a more stable interface (watch this space).

GenericServices is a .NET class library which helps a developer build a service layer, i.e. a layer that acts as a facard/adapter between your business/data service layers and your User Interface or HTTP service.

It does this by providing standard database CRUD (Create, Read, Update and Delete) methods and a standard way of calling business methods, each with clear extension points. The aim is to cut out the standard boiler-plate code so the user only has to write the data or business specific code.

What application frameworks can GenericServices work with?

GenericServices is designed work as a service layer framework in any .NET application, such as  ASP.NET MVC, Widows Azure Web apps, etc. It assumed a disconnected state model, e.g. a web site or HTTP RESYful service where the read of data prior to update is disconnected from the update of the data.

I have also assumed a horizontal scaling model, e.g. scale by having multiple web instances,  as this is how Azure and most web sites scale. I have therefore not thought about serialisation of objects for vertical scaling, i.e. where each layer of the application are run on a separate server and remote calls are used between layers.

GenericServices uses the following .NET frameworks/systems.

  • It needs .NET 4.5 because it implements all commands in normal and the new async/await Tasking format introduced in .NET 4.5
  • It uses Entity Framework 6 (EF6) for database access, again because it supports async commands.
  • It also makes use of the open source AutoMapper library for transforming data and business classes to/from the user interface oriented DTOs.

What is the motivation behind building GenericServices?

I design and develop fairly complex analysing, modelling and data visualisation web applications (see Spatial Modeller). These require a Domain-Driven Design approach to the data and business objects, while the visualisation needs a comprehensive user interface which can include a Single Page Application (SPA) fed by a REST interface. This means there often a mismatch between the business/data layers classes and the data needed by the user interface and SPA.

My experience is that the Service Layer, plus Data Transfer Objects (DTOs), is the best way to solve mismatch. However I have found that the service layer is often filled with lots of code that is very similar, with just the data being different. Its also pretty boring to write. I therefore researched a number of approaches to handle the standard code and finally came up with a solution using C#’s Generic classes. I have therefore called it GenericServices.

Q: How would I use it? A: Look at example web site!

I have taken the trouble to produce a live example web site. This has examples of all the GenericService commands, with documentation explaining how they work – for example see the Posts Controller code explanation page.

As well as showing the use of GenericService it also contains another companion project of mine; code for executing long-running methods with progress and cancellation controls in ASP.NET MVC using SignalR.

This web site is an ASP.NET MVC5 application and is itself an open source project called SampleMvcWebApp. The source of is available on GitHub.

Feedback

While in alpha phase I suggest you leave comments here of contact me via this web site’s Contact Page. Once it is up on NuGet I will try and set up a Stack Overflow group and monitor that.

Mocking SignalR Client for better Unit Testing

Modal dialog showing task progress.
Modal dialog showing task progress. Uses SignalR.

Why I needed to Mock SignalR

I build geographic modelling applications and they have lots of tasks that take a long time, sometime many minutes, to run. I am currently  developing an open source framework to help speed up the development of such ASP.NET MVC applications. Therefore part of the framework I have includes modules for handling long running processes, with a progress bar, messages and user cancellation. Click on the image on the left to see a a simple example of a model panel with a green progress bar at the top and a list of sent messages as the task works its way through the calculations (well, in this case a test code so the messages aren’t that exciting) .

I have used SignalR for the communication channel between the JavaScript and the MVC5 server. I have found SignalR to be excellent and makes two-way comms really easy.

However my application is fairly complicated because of all the things that can happen, like errors, user wanting to cancel, losing connection, etc. In particular the JavaScript client uses a state machine to handle all the options, and that needs checking. For this reason I wanted to unit test both ends. (Read my blog on Unit Testing for an in-depth look at how I use Unit Testing).

The C# end was fairly straight forward to test, as it was partitioned well. However for the JavaScript end I needed to Mock the SignalR JavaScript library. I could not find anything online so I wrote something myself.

Mocking the SignalR JavaScript Client

I turns out that is wasn’t that hard to mock the SignalR Client, although I should say I don’t use the autogenerated SignalR hub scripts, but use the createHubProxy(‘HubName’) as I find that easier to manage that loading a dynamically created script. I have listed the code mocked SignalR Client code below:

//This is a mock for a small part of SignalR's javascript client.
//This code does not mock autogenerated SignalR hub scripts as the
//ActionRunner code uses the connection.createHubProxy('HubName') method,
//followed by .on or .invoke to setup the receive and send methods respectively

var mockSignalRClient = (function ($) {

    var mock = {};

    //first the items used by unit testing to see what has happened
    mock.callLog = null;
    mock.onFunctionDict = null;
    mock.doneFunc = null;
    mock.failFunc = null;
    mock.errorFunc = null;
    //This logs a string with the caller's function name and the parameters
    //you must provide the function name, but it finds the function arguments itself
    mock.logStep = function (funcName) {
        var log = funcName + '(';
        var callerArgs = arguments.callee.caller.arguments;
        for (var i = 0; i < callerArgs.length; i++) {
            log += (typeof callerArgs[i] === 'function') ? 'function, ' : callerArgs[i] + ', ';
        };
        if (callerArgs.length > 0)
            log = log.substr(0, log.length - 2);
        mock.callLog.push(log + ')');
    };
    mock.reset = function() {
        mock.callLog = [];
        mock.onFunctionDict = {}
        mock.doneFunc = null;
        mock.failFunc = null;
        mock.errorFunc = null;
    };

    //doneFail is the object returned by connection.start()
    var doneFail = {};
    doneFail.done = function (startFunc) {
        mock.logStep('connection.start.done');
        mock.doneFunc = startFunc;
        return doneFail;
    };
    doneFail.fail = function(failFunc) {
        mock.logStep('connection.start.fail');
        mock.failFunc = failFunc;
        return doneFail;
    };

    //Channel is the object returned by connection.createHubProxy
    var channel = {};
    channel.on = function (namedMessage, functionToCall) {
        mock.logStep('channel.on');
        mock.onFunctionDict[namedMessage] = functionToCall;
    };
    channel.invoke = function (actionName, actionGuid) {
        mock.logStep('channel.invoke');
    };

    //connection is the object returned by $.hubConnection
    var connection = {};
    connection.createHubProxy = function (hubName) {
        mock.logStep('connection.createHubProxy');
        return channel;
    };
    connection.error = function (errorFunc) {
        mock.logStep('connection.error');
        mock.errorFunc = errorFunc;
    };
    connection.start = function () {
        mock.logStep('connection.start');
        return doneFail;
    };
    connection.stop = function () {
        mock.logStep('connection.stop');
        return doneFail;
    };

    //now we run once the method to add the hubConnection function to jQuery
    $.hubConnection = function() {
        return connection;
    };

    //Return the mock base which has all the error feedback information in it
    return mock;

}(window.jQuery));

I think you will find most of this fairly easy to understand. Lines 8 to 34 are all the variables and methods for Unit Testing to use. The rest of the code implements the methods which mock the SignalR methods I use in my code.

How did I use this Mock SignalR?

SignalR works by adding .hubConnection() to jQuery so it was simple to make the mock SignalR client do the same (see line 71 above). My actual code checks that jQuery is present and then that $.hubConnection is defined, which ensures SignalR is loaded. Here is a piece of code from my ActionRunner.comms.js that does the initial setup to see how it uses SignalR and therefore what I needed to Mock.

//This deals with setting up the SignalR connections and events
function setupTaskChannel() {

    actionRunner.setActionState(actionStates.connectingTransient);

    actionRunner.numErrorMessages = 0;

    //Setup connection and actionChannel with the functions to call
    var connection = $.hubConnection();

    //connection.logging = true;
    actionChannel = connection.createHubProxy('ActionHub');
    setupTaskFunctions();

    //Now make sure connection errors are handled
    connection.error(function(error) {
        actionRunner.setActionState(actionStates.failedLink);
        actionRunner.reportSystemError('SignalR error: ' + error);
    });
    //and start the connection and send the start message
    connection.start()
        .done(function() {
            startAction();
        })
        .fail(function(error) {
            actionRunner.setActionState(actionStates.failedConnecting);
            actionRunner.reportSystemError('SignalR connection error: ' + error);
        });
}
Jasmine Unit Test checking what SignalR functions were called
Jasmine Unit Test checking what SignalR functions were called

Using this mocking framework

There are two main ways I use it. Firstly you get a log of each method called, which helps ensure the right methods are called.

Secondly most of the calls to SignalR link functions to certain SignalR events. By capturing these functions the unit test can call them to simulate SignalR messages, errors etc. That allows a very good level of checking.

Getting the whole picture

In case you are interested in downloading the code or seeing how it was used then here are a series of links to various bits of code. These are taken form an open source project that I am currently working on, so the code is subject to change. I have listed all the various parts of the testing. UPDATE: These links had broken because the git repository had changed – sorry about that. Now fixed.

Conclusion

As I said in my previous post ‘Unit Testing in C# and JavaScript‘ I find mocking in JavaScript very easy and helpful. The ActionRunner is complex enough to need unit testing and I found mocking the various parts I wanted to replace fairly quick to implement.

I hope this helps you with SignalR and encourages you to mock other frameworks to help you test more easily. Happy coding.

 

 

 

Reflections on Unit Testing in C# and JavaScript

It has been a while since I wrote my first blog post called ‘Why I’m not scared of JavaScript any more‘ and since then I have written a fair amount of JavaScript. One of the things that I have found interesting is how I use different approaches to Unit Testing in each language. I’m not saying I have the ‘best’ way of Unit Testing each language, in fact I think some areas are a bit weak, but maybe you will find my experiences and reflections useful.

If you want to get to the meat of the post skim the next three paragraphs that set the scene and jump to ‘The differences in how I Unit Test between the two languages’ section.


First, respect to Gilles Ruppert…

Before I start I should say that I was heavily helped on the JavaScript side by an excellent programmer called Gilles Ruppert. As I was new to JavaScript I sort help and my son recommended Gilles. Gilles worked for me for about 10 weeks and set up the architecture of the single page application that I needed. He also set up an excellent JavaScript development environment using Karma test runner, Mocha test framework, Expect assertions and finally Sinon stub/spy/mocks. I am building on the shoulders of a giant.

Setting the scene

have just finished the first release of Spatial Modeller™ , a medium sized ASP.NET MVC web application using ASP.NET MVC4 written in C# and a large single page application written in JavaScript using backbone with marionette.

When Gilles finished we had just under 300 JavaScript Unit Tests, a handful written by me. Now I have added a lot more code and written an additional 400+ Unit Tests, taking the JavaScript tests to over 700.

On the C# side I use NUnit unit testing framework with a small amount of Moq for mocking etc. I use Resharper for running the testing inside Visual Studio. I have used this combination for some years.

Numerically I have less Unit Tests than JavaScript, currently C# has just under 500. However each test is often much more complicated and test many items per test. The reasons for this difference in approach is one of the reasons I wrote this blog, so read on to find out why.

My style of Unit Testing

I really believe in Unit Testing and I don’t think I have a project that doesn’t used Unit Testing. However I am not a fan of Test Driven Development (TDD) as I have found that way I don’t come up with coherent design. Writing Unit Tests is not an excuse for not doing some good designing of the system first.


The differences in how I Unit Test between the two languages

After maybe 10 months of JavaScript Unit Testing (and 5+ years of C# Unit Testing) I find it really interesting that I use different styles between the two. Here are the raw  differences and I’ll make some conclusions at the end.

1. In C# my first Unit Test in the compiler

Because C# is typed the compiler gives me lots of feedback. Therefore I my refactor code before I have even run it if I see opportunities to improve the code. This doesn’t happen with JavaScript, partly because JavaScript is not typed in the same way so tools like JSHint cannot give such comprehensive feedback.

(hint: In Visual Studio you can install Web Essentials which will then run JSHint and JSCS when a JavaScript file is closed. It may not be as good and the C# compiler at spotting things, but it can help spot silly mistakes.)

This means I am willing to write more C# code before I Unit Test than I would with JavaScript.

2. Large JavaScript is much easier to partition than C#

The ‘up’ side of not so tight checking in JavaScript is much easier to build in smaller chunks. The ‘interface’ between these parts is often an object which, by the nature of JavaScript, is not directly linked between modules. As well as making Mocking really easy it seems to help me think about each part separately.

In C# I use interfaces and layer to separate chunks of code, but for something complex  with five+ significant classes that I tend to think of them as a whole. The typed nature makes them more ‘linked’ than the JavaScript equivalent and mocking is harder.

3. The unit testing frameworks make a BIG difference

I have found that test frameworks like Mocha and Jasmine have some featured that encourage small tests. This is because these frameworks support ‘nested setups’, which other frameworks like NUnit don’t have. Let me explain what ‘nested setups’ are and why they encourage smaller, one item tests.

For most tests we need to setup some instances or data before we run the test. In NUnit we can run a setup method a) once at the start of the test class and/or b) once before each test is run. This is fine, but I find that I often need to run something specific before each test, which in NUnit you to at the beginning of the specific test. See example below where I run a ‘once at start’ method called ‘SetUpFixture’ (lines 1 to 6) and then an additional setup phase inside the specific test ‘CheckValidateTagError’ (line 14).

[TestFixtureSetUp]
public void SetUpFixture()
{
    using (var db = new SampleWebAppDb())
        DataLayerInitialise.ResetDatabaseToTestData(db, TestDataSelection.Simple);
}

[Test]
public void CheckValidateTagError()
{
    using (var db = new SampleWebAppDb())
    {
        //SETUP
        var existingTag = db.Tags.First();

        //ATTEMPT
        var dupTag = new Tag {Name = "duplicate slug", Slug = existingTag.Slug};
        db.Tags.Add(dupTag);
        var status = db.SaveChangesWithValidation();

        //VERIFY
        status.IsValid.ShouldEqual(false);
        status.Errors.Count.ShouldEqual(1);
        status.Errors[0].ErrorMessage.ShouldEqual("The Slug on tag 'duplicate slug' must be unique.");
    }
}

This fine, but I find that the individual setups can become quite long. I then have three choices if I have multiple things to check: a) duplicate the individual setup code in each test (Not good for DRY), b) write a small helper method which encapsulates the setup code (takes time and hides the setup), or c) test multiple items in the one test, which is what I have done above.

Both the Mocha and Jasmine JavaScript testing frameworks allow nesting of the setups, so in my last example I could have an outer group with the ‘SetUpFixture’ in it and then a small nested group, with a setup just for CheckValidateTagError, and then three separate tests for the three parts.

Here is an example from my actual code with some nested test groups, which Mocha does with the ‘describe’ command:

... various require code

describe('views/InfoPanel', function () {
    afterEach(function () {
        delete this.model;
        delete this.view;
    });

    describe('single item to display, no grades', function () {
        beforeEach(function () {
            this.layerInfo = helpers.loadFixture('/api/schemes/1').layerInfos[0];
            this.members = helpers.loadFixture('/api/layers/1/members');
            this.model = new InfoPanelModel({ selectedMembers: this.Members });
            this.view = new InfoPanelView({ model: this.model });
        });
        afterEach(function () {
            delete this.layerInfo;
            delete this.layer;
            delete this.members;
        });

        it('should be a Backbone.View', function () {
            expect(this.view).to.be.a(Backbone.View);
        });

        describe('after showing view', function () {
            beforeEach(function () {
                this.mainRegion.show(this.view);
            });

            it('should have something on view', function () {
                expect(this.view.$el).to.have.length(1);
            });

            it('title in window header should start with layer name', function () {
                ... etc.
            });

You can see the first setup, called ‘beforeEach’ on lines 10 to 15 and the the nested ‘beforeEach’ on lines 27 to 29. This works really well and is the reason why my JavaScript Unit Tests test almost always check one item per test. I really like Mocha’s approach and miss it in NUnit.

4. Mocking databases is hard!

One thing that definitely affects me is that the database side is hard to mock. I have used a number of ORMs, and direct T-SQL, and there isn’t a nice way to replace the real code with something for testing that will catch the same errors. Foreign keys, relational fixed etc. etc. is realy hard to mock (just have a look at my post Updating a many to many relationship in entity framework to see the complications that arise).

This means some of by Unit Tests in C# are more integration tests as I use the real database layer, but with a test database loaded with test data. There is a cost to that, but my experience is anything else lets errors through.

JavaScript using backbone.js does access data, but in a more controlled way and Gilles set up the classic ‘fixtures’ and filled it with test data created by the C# application. That makes JavaScript testing much easier, but only because the server is dealing with the complex parts.

5. PS: You must be able to debug inside Unit Tests

It goes without saying that Unit Tests are for finding bugs before you use the code in the real application. Therefore you MUST be able to debug code, with breakpoints and variable inspection etc., when you are running a unit test.

I mention this only because Resharper, which I find really useful, is very bad in this area when unit testing JavaScript. Resharper makes running unit tests really easy, especially on JavaScript using QUnit or Jasmine. However I have found it almost impossible to use it to debug Jasmine tests, and I assume QUnit tests, when running in the Resharper harness.

However the Mocha JavaScript test environment that Gilles set up is excellent on that score. It allows running with connected browser, use of debugger; statement to cause pauses when the debugger is open and the .only() statement to allow only certain tests to be run.

How this affects my Unit Testing

  1. My JavaScript Unit Tests are shorter and just test one item per test.
  2. I write new Unit Test groups more often in JavaScript than in C#.
  3. My C# Unit Tests are often more complex and often test multiple items per test. This is because:
    1. The setup code is complicated or time consuming.
    2. NUnit does not have nested setups like Mocha does
      (see The unit testing frameworks make a BIG difference above).
  4. I often write more C# code before writing Unit Tests as the the compiler catches a lot of ‘silly’ mistakes in C# which can get through in JavaScript.
  5. C# sees more ‘linked together’ than JavaScript, which affects mocking and designing.

I know that testing multiple items in one test, point 3 above, is often seen as ‘wrong’ by some, but cost of both code to set it up and time to run the test are all factors I have weighted up in coming to my current style.

I have more C# code than JavaScript code by maybe 3 to 1. My 500 C# Unit Tests take nearly 2 minutes to run while my 700 JavaScript Unit Tests take 30 seconds.

Conclusion

I like both C# and JavaScript. They each have their strengths and weaknesses. I also could not work a project without the feedback that Unit Tests give me on the quality of the software.

With all of this in mind it is worth thinking about how you Unit Test in each language. The language, the need, the project and the environment will change the way Unit Tests can and should be approached. Happy programming.