Reflections on Unit Testing in C# and JavaScript

It has been a while since I wrote my first blog post called ‘Why I’m not scared of JavaScript any more‘ and since then I have written a fair amount of JavaScript. One of the things that I have found interesting is how I use different approaches to Unit Testing in each language. I’m not saying I have the ‘best’ way of Unit Testing each language, in fact I think some areas are a bit weak, but maybe you will find my experiences and reflections useful.

If you want to get to the meat of the post skim the next three paragraphs that set the scene and jump to ‘The differences in how I Unit Test between the two languages’ section.


First, respect to Gilles Ruppert…

Before I start I should say that I was heavily helped on the JavaScript side by an excellent programmer called Gilles Ruppert. As I was new to JavaScript I sort help and my son recommended Gilles. Gilles worked for me for about 10 weeks and set up the architecture of the single page application that I needed. He also set up an excellent JavaScript development environment using Karma test runner, Mocha test framework, Expect assertions and finally Sinon stub/spy/mocks. I am building on the shoulders of a giant.

Setting the scene

have just finished the first release of Spatial Modeller™ , a medium sized ASP.NET MVC web application using ASP.NET MVC4 written in C# and a large single page application written in JavaScript using backbone with marionette.

When Gilles finished we had just under 300 JavaScript Unit Tests, a handful written by me. Now I have added a lot more code and written an additional 400+ Unit Tests, taking the JavaScript tests to over 700.

On the C# side I use NUnit unit testing framework with a small amount of Moq for mocking etc. I use Resharper for running the testing inside Visual Studio. I have used this combination for some years.

Numerically I have less Unit Tests than JavaScript, currently C# has just under 500. However each test is often much more complicated and test many items per test. The reasons for this difference in approach is one of the reasons I wrote this blog, so read on to find out why.

My style of Unit Testing

I really believe in Unit Testing and I don’t think I have a project that doesn’t used Unit Testing. However I am not a fan of Test Driven Development (TDD) as I have found that way I don’t come up with coherent design. Writing Unit Tests is not an excuse for not doing some good designing of the system first.


The differences in how I Unit Test between the two languages

After maybe 10 months of JavaScript Unit Testing (and 5+ years of C# Unit Testing) I find it really interesting that I use different styles between the two. Here are the raw  differences and I’ll make some conclusions at the end.

1. In C# my first Unit Test in the compiler

Because C# is typed the compiler gives me lots of feedback. Therefore I my refactor code before I have even run it if I see opportunities to improve the code. This doesn’t happen with JavaScript, partly because JavaScript is not typed in the same way so tools like JSHint cannot give such comprehensive feedback.

(hint: In Visual Studio you can install Web Essentials which will then run JSHint and JSCS when a JavaScript file is closed. It may not be as good and the C# compiler at spotting things, but it can help spot silly mistakes.)

This means I am willing to write more C# code before I Unit Test than I would with JavaScript.

2. Large JavaScript is much easier to partition than C#

The ‘up’ side of not so tight checking in JavaScript is much easier to build in smaller chunks. The ‘interface’ between these parts is often an object which, by the nature of JavaScript, is not directly linked between modules. As well as making Mocking really easy it seems to help me think about each part separately.

In C# I use interfaces and layer to separate chunks of code, but for something complex  with five+ significant classes that I tend to think of them as a whole. The typed nature makes them more ‘linked’ than the JavaScript equivalent and mocking is harder.

3. The unit testing frameworks make a BIG difference

I have found that test frameworks like Mocha and Jasmine have some featured that encourage small tests. This is because these frameworks support ‘nested setups’, which other frameworks like NUnit don’t have. Let me explain what ‘nested setups’ are and why they encourage smaller, one item tests.

For most tests we need to setup some instances or data before we run the test. In NUnit we can run a setup method a) once at the start of the test class and/or b) once before each test is run. This is fine, but I find that I often need to run something specific before each test, which in NUnit you to at the beginning of the specific test. See example below where I run a ‘once at start’ method called ‘SetUpFixture’ (lines 1 to 6) and then an additional setup phase inside the specific test ‘CheckValidateTagError’ (line 14).

[TestFixtureSetUp]
public void SetUpFixture()
{
    using (var db = new SampleWebAppDb())
        DataLayerInitialise.ResetDatabaseToTestData(db, TestDataSelection.Simple);
}

[Test]
public void CheckValidateTagError()
{
    using (var db = new SampleWebAppDb())
    {
        //SETUP
        var existingTag = db.Tags.First();

        //ATTEMPT
        var dupTag = new Tag {Name = "duplicate slug", Slug = existingTag.Slug};
        db.Tags.Add(dupTag);
        var status = db.SaveChangesWithValidation();

        //VERIFY
        status.IsValid.ShouldEqual(false);
        status.Errors.Count.ShouldEqual(1);
        status.Errors[0].ErrorMessage.ShouldEqual("The Slug on tag 'duplicate slug' must be unique.");
    }
}

This fine, but I find that the individual setups can become quite long. I then have three choices if I have multiple things to check: a) duplicate the individual setup code in each test (Not good for DRY), b) write a small helper method which encapsulates the setup code (takes time and hides the setup), or c) test multiple items in the one test, which is what I have done above.

Both the Mocha and Jasmine JavaScript testing frameworks allow nesting of the setups, so in my last example I could have an outer group with the ‘SetUpFixture’ in it and then a small nested group, with a setup just for CheckValidateTagError, and then three separate tests for the three parts.

Here is an example from my actual code with some nested test groups, which Mocha does with the ‘describe’ command:

... various require code

describe('views/InfoPanel', function () {
    afterEach(function () {
        delete this.model;
        delete this.view;
    });

    describe('single item to display, no grades', function () {
        beforeEach(function () {
            this.layerInfo = helpers.loadFixture('/api/schemes/1').layerInfos[0];
            this.members = helpers.loadFixture('/api/layers/1/members');
            this.model = new InfoPanelModel({ selectedMembers: this.Members });
            this.view = new InfoPanelView({ model: this.model });
        });
        afterEach(function () {
            delete this.layerInfo;
            delete this.layer;
            delete this.members;
        });

        it('should be a Backbone.View', function () {
            expect(this.view).to.be.a(Backbone.View);
        });

        describe('after showing view', function () {
            beforeEach(function () {
                this.mainRegion.show(this.view);
            });

            it('should have something on view', function () {
                expect(this.view.$el).to.have.length(1);
            });

            it('title in window header should start with layer name', function () {
                ... etc.
            });

You can see the first setup, called ‘beforeEach’ on lines 10 to 15 and the the nested ‘beforeEach’ on lines 27 to 29. This works really well and is the reason why my JavaScript Unit Tests test almost always check one item per test. I really like Mocha’s approach and miss it in NUnit.

4. Mocking databases is hard!

One thing that definitely affects me is that the database side is hard to mock. I have used a number of ORMs, and direct T-SQL, and there isn’t a nice way to replace the real code with something for testing that will catch the same errors. Foreign keys, relational fixed etc. etc. is realy hard to mock (just have a look at my post Updating a many to many relationship in entity framework to see the complications that arise).

This means some of by Unit Tests in C# are more integration tests as I use the real database layer, but with a test database loaded with test data. There is a cost to that, but my experience is anything else lets errors through.

JavaScript using backbone.js does access data, but in a more controlled way and Gilles set up the classic ‘fixtures’ and filled it with test data created by the C# application. That makes JavaScript testing much easier, but only because the server is dealing with the complex parts.

5. PS: You must be able to debug inside Unit Tests

It goes without saying that Unit Tests are for finding bugs before you use the code in the real application. Therefore you MUST be able to debug code, with breakpoints and variable inspection etc., when you are running a unit test.

I mention this only because Resharper, which I find really useful, is very bad in this area when unit testing JavaScript. Resharper makes running unit tests really easy, especially on JavaScript using QUnit or Jasmine. However I have found it almost impossible to use it to debug Jasmine tests, and I assume QUnit tests, when running in the Resharper harness.

However the Mocha JavaScript test environment that Gilles set up is excellent on that score. It allows running with connected browser, use of debugger; statement to cause pauses when the debugger is open and the .only() statement to allow only certain tests to be run.

How this affects my Unit Testing

  1. My JavaScript Unit Tests are shorter and just test one item per test.
  2. I write new Unit Test groups more often in JavaScript than in C#.
  3. My C# Unit Tests are often more complex and often test multiple items per test. This is because:
    1. The setup code is complicated or time consuming.
    2. NUnit does not have nested setups like Mocha does
      (see The unit testing frameworks make a BIG difference above).
  4. I often write more C# code before writing Unit Tests as the the compiler catches a lot of ‘silly’ mistakes in C# which can get through in JavaScript.
  5. C# sees more ‘linked together’ than JavaScript, which affects mocking and designing.

I know that testing multiple items in one test, point 3 above, is often seen as ‘wrong’ by some, but cost of both code to set it up and time to run the test are all factors I have weighted up in coming to my current style.

I have more C# code than JavaScript code by maybe 3 to 1. My 500 C# Unit Tests take nearly 2 minutes to run while my 700 JavaScript Unit Tests take 30 seconds.

Conclusion

I like both C# and JavaScript. They each have their strengths and weaknesses. I also could not work a project without the feedback that Unit Tests give me on the quality of the software.

With all of this in mind it is worth thinking about how you Unit Test in each language. The language, the need, the project and the environment will change the way Unit Tests can and should be approached. Happy programming.

  • Pingback: Mocking SignalR Client for better Unit Testing | The Reformed Programmer()

  • Leo Hinojosa

    Hi!,

    great post,

    i just had a similar experience and was introduced to the world of javascript testing in a project i had with he guys from Pivotal. We got trained in using jasmine and karma and it really changed my perspective on how to address the whole testing strategy.

    We would always have issues changing mindsets from jasmine to Unit Tests, so i decided to implement a minimal bdd testing framework that is similar to jasmine and mocha, https://github.com/leohinojosa/spec, it supports describe, it, beforeEach, beforeAll, etc. I would love if i could get your feedback on it.

    It’s implemented in plain C# and it includes a standalone console runner and a vstest.console runner for visual studio. ( haven’t started the resharper plugin )

    As for db we used EF and started using the effort library (https://effort.codeplex.com/) which uses Nmemory, which is a compltetly in memory database. It made our lives so much happier.

    Again, thanks for the great post

    • Hi Leo,

      Thanks for that. I had a quick look at you Spec project. I like the idea because, as I said in this post, the BeforeEach/AfterEach I found really helpful in Mocha and Jasmine as it cuts down the repetition of setup code. One thing I didn’t find in your code/documentation was any reference to async/await. Does it handle that? Async/await is now a very important part of the .NET framework so any Unit Tests need to cover that too. NUnit handles asycn well apart from catching expected exceptions inside async methods is a bit messy.

      I’m glad you provided something to run Spec inside VS, as that is very important. Making it run is Resharper is pretty important as many developers rely on Resharper(I do). In fact in some projects I put up with Resharper’s poor (non-existent!) debugging of JavaScript in Unit Tests just to set of standard tests for my JavaScript.

      My feeling having written an open-source project, GenericServices ( https://github.com/JonPSmith/GenericServices ), is that you will need more examples/documentation for your Spec project to take off. I produced two example web sites, a reasonable Wiki and numerous articles. I think Spec is simpler to understand than GenericServices but getting people to adopt something is hard. Good project though!

      Anders Baumann mentioned the Effort library on another of my posts http://www.thereformedprogrammer.net/architecture-of-business-layer-working-with-entity-framework/ . I’m in the middle of a fairly long project so I don’t change the Unit Test environment mid way, but its certainly a nice idea. If EF 7 comes out soon then that will have a Memory mode, which might be an even better option for new projects.

      I hope that feedback helps.

      • Leo Hinojosa

        Hi Jon,
        Thank you for having spent time reviewing my project, I will look forward in applying your suggestions as I agree on all the points you made

        I hope later the spec project can get a bit more mature so that it helps out in creating better software and providing tools that we al can use,

        Thanks again!

        • Hi Leo,

          Just to say I was writing more unit tests today and, again, I had to do the same set up for each test when I would like to have done one set up and then multiple tests to check the different aspects of the results, e.g. a) did it run successfully, b) did it return the right ‘answer’ and c) did it update the database as it should.

          So I do think you have something. Keep me posted!

        • Hi Leo,

          A quick note on async in setup/teardown. NUnit doesn’t (currently) support this – see https://github.com/nunit/nunit/issues/60 . It is a problem.

          • Leo Hinojosa

            i saw that, i was trying to learn how other testing frameworks were doing it, and found that most testing frameworks have small issues with the implementation. they either don’t support it or support them partially,

            so far what i have come up with is partial support but it seems a bit hacky, i’ll share it with you once i make some progress on it and have some decent tests to share

            cheers!

          • Hi Leo,

            Did you see the changes in NUnit 3? They support async in attributes like [Setup]/[TearDown]. There is some sort of hierarchy – see SetUp-Attribute->inheritanc. Also resharper 10 supports NUnit.

            Haven’t tried NUnit 3 in anger yet, but the async support is a very useful to me. Not sure how I would use inheritance though??

        • Hi Leo,

          I have written another article on Unit Testing, see http://www.thereformedprogrammer.net/when-is-it-not-worth-writing-a-unit-test/ , and mentioned you. I hope that is OK. Let me know if you want the comment changed/removed.

          • Leo Hinojosa

            Hi Jon,

            I appreciate you having mentioned the project, i feel that it will drive some of your readers to take a peek / collaborate into having a better testing framework, Thanks!.

            Side note, your new article is great, i really share the perspective and opinions you are writing about. I’ve found that people react good to the concept of Arrange, Act, Assert, http://c2.com/cgi/wiki?ArrangeActAssert so I tend to use it to ilustrate the unit test body. Please keep up the great work and of course i will let you know once i have made some progress on the missing features of the spec library. Cheers!

  • Pingback: When is it not worth writing a Unit Test? | The Reformed Programmer()