Evolving modular monoliths: 2. Breaking up your app into multiple solutions

This is the second article in a series about building Microsoft .NET applications using a modular monolith architecture. This article covers a way to extract parts of your application into separate solutions which you turn into NuGet packages that are installed in your main application. Each solution is physically separated from each other in a similar way to Microservice architecture, but without the performance and communication failure modes that Microservices can have.

My view is that the Microservice architecture is great for applications that need with large development teams and/or have to handle high levels of demand, like Netflix. But for smaller applications the Microservice architecture can be overkill. However, I do like the Microservice idea of having separate solutions, because that makes the application easier to understand, refactor and manage so I have defined a way to do extract parts of an application into their own solution. The result is an application that is easier to build because it’s using the simpler monolith architecture but has the benefits of having multiple separate solutions that are combined by using NuGet.

NOTE: If you are planning to build Microservice architecture, then the approach described is also a great starting point because it already creates separate solutions. Martin Fowler also suggests that starting with a monolith approach is the best way to build a Microservice application – see his article called “Monolith First”.

This article is part of the Evolving Modular Monoliths series, the articles are:

• Evolving modular monoliths: 1. An architecture for .NET
• Evolving modular monoliths: 2. Breaking up your app into multiple solutions (this article)
• Evolving modular monoliths: 3. Sharing data between bounded contexts

TL;DR – summary

• One of the best ways to structure your application is to break the business needs into what DDD calls bounded contexts (see this section in the first article for more on bounded contexts).
• In a modular monolith some bounded contexts can be large and/or complex. In these cases, giving a large/complex bounded context its own solution makes the code easier to understand, navigate, and manage.
• I have created a dotnet tool called MultiProjPack that automates the creation of the NuGet package for projects that follow the naming convention described in the first article.
• I describe a fast local build/test/debug cycle when adding/changing code in a separate solution. This uses a local NuGet package source and some features in the MultiProjPack tool.
• I describe the options for getting source code information while debugging a NuGet package inside your main application.
• I finish with a section on building the composite application for deployment to production and suggest ways to store your private NuGet packages.

Breaking up your app into multiple solutions

The idea is to get the separation that a Microservice pattern provides while keeping the performance and reliability of direct method or database access. To do this we extract a DDD bounded context (see the section on bounded contexts in the first article) section of your code into its own solution and then turn it into NuGet package to install in your main application.

Turning a DDD bounded context into a separate solution/NuGet package requires a bit more work, so I suggest you only apply this approach to large and/or complex parts of your application. The benefits are:

• The code is easier to understand/navigate because it’s isolated into its own solution.
• If the development team is large, it’s easier for one team to work on an isolated solution   and “publish” a NuGet package for other teams to use.
• On older application this approach means you can build new features using a more modern design without the structure of the old application hindering your new design.

I have created two example repos that show this approach in action I have created another version of the e-commence web app that sells books called BookApp. The main application is called BookApp.Main (see https://github.com/JonPSmith/BookApp.Main for the code example), which contains the FrontEnd and the Order processing parts (BookApp.Orders…). The part of the application deals with the querying and updating of the books in the database (referred to as BookApp.Books) is large enough to warrant turning into a solution (see https://github.com/JonPSmith/BookApp.Books for code example). The BookApp.Books solution is turned into a NuGet package and installed in the BookApp.Main. The figure below shows this in action.

This makes it much easier for a development team to work on a part of the application in a separate repo/solution. Also, the team can “publish” a new version via a private NuGet server, with fallback to the old version provided by simply changing back to the previous NuGet packages.

NOTE: One limitation around using a NuGet package is that all your projects in the solution must have the same framework, e.g. .net5.0. That’s because NuGet is designed to handle packages with work with multiple frameworks, for instance the Newtonsoft.Json package can work with seven types of frameworks (see its dependences). But in this usage you want all of your projects to be the same otherwise it won’t work.

Communication between the front-end and a NuGet package

Turning a bounded context into a NuGet package gives you the benefit of separation that a Microservice has. But while a Microservice architecture has one API front-end per Microservice, in our modular monolith design there is direct code link, via the NuGet package, to the BookApp.Books code.

This changes the communication channel we use between each bounded context and the user. In a Microservice design the front-end typically accesses a service via HTTP API, but in our modular monolith design it accesses a service via dependency injection to call method. So, in a modular monolith the API is defined mainly by interfaces and dependency injection, which a few key class definitions. And because we are using a monolith architecture there is one front-end in BookApp, that is a ASP.NET Core project.

NOTE: There are other communication paths between two bounded contexts, which I cover in part 3.

The API is mainly defined by the ServiceLayer, which contains many of the services linked to a bounded context (read this section about the ServiceLayer in one of my articles).  For instance, in my example BookApp.Main I want to display a list of books with sorting, filtering and paging. This uses a service referred to by the interface IListBooksService in the …ServiceLayer.GoodLinq project. The ServiceLayer will also contain some classes too, such as the BookListDto class to provide the data to display the books, and various other classes, interfaces, constants etc.

NOTE: For Separation of Concerns (SoC) reasons I also recommend adding a small project specifically to handle any startup code, e.g., registering services with the dependency injection provider. I also with application that loads data from a configuration file (e.g. appsettings.json) I pass the IConfiguration interface too in case the code needed that – see BookApp.Books.AppSetup as an example.  

How to create a NuGet packages

Now that we have defined how the bounded context with link to the main application, we now need to create a NuGet package. It turns out that creating a NuGet package containing multiple .NET projects is doable, but takes a lot of work building the .nuspec file properly.

To automate the creation the NuGet package I built a dotnet tool called MultiProgPack (repo found here) that builds the .nuspec file by scanning your solution for certain projects/namespaces and builds a NuGet package for you. This tool also contains features to make it much quicker to build and test your NuGet package on your development computer by using a local NuGet package server, which I describe later.

The MultiProgPack (which is a NuGet package) can be installed or updated on your computer the following three lines of command line commands:

dotnet tool install JonPSmith.MultiProjPack -global

dotnet tool update JonPSmith.MultiProjPack –global

Once you have installed the MultiProgPack you to call this dotnet tool via a command line in one of the projects in your solution (you can use any project, but I normally use the BookApp.Books.AppSetup project). Here are how you call it, with your selection of one of three of its options:

MultiProjPack <D|R|U>

The three options do the following

• D(ebug): This creates a NuGet package using the Debug configuration of the code.
• R(elease): This creates a NuGet package using the Release configuration of the code.
• U(pdate): This builds a NuGet package using the Debug configuration, but also updates the .dll’s in the NuGet cache (I explain this in this section).

The tool relied on a xml file called MultiProjPack.xml in the folder you run the tool from. This file defines the NuGet data (name, version and so on) and optional tool settings. Here is a typical setup, but there are a lot more settings if you need them (see this example file containing all the settings and the READMe file for more information)

<?xml version="1.0" encoding="utf-8"?>
<allsettings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<!-- this contains the typical information you should have in your settings -->
  <metadata>
    <id>BookApp.Books</id>
    <version>1.0.0-preview001</version>
    <authors>you must give a list of author(s)</authors>
    <description>you must provide a description of the NuGet</description>
    <releaseNotes>optional: what is changed in this release?</releaseNotes>
  </metadata>
  <toolSettings>
    <!-- This is used to find projects with names starting with this. If null, then uses NuGet id -->
    <NamespacePrefix></NamespacePrefix>
    <!-- excludes named projects (comma separated), e.g. "Test" would exclude a project starting with "BookApp.Books.Test" -->
    <ExcludeProjects>Test</ExcludeProjects>
    <!-- worth filling in with your local NuGet Package Source folder. See docs about using {USERPROFILE} in string -->
    <CopyNuGetTo>{USERPROFILE}\LocalNuGet</CopyNuGetTo> 
  </toolSettings>
</allsettings>

NOTE: Don’t worry, the tool has the command –CreateSettings that will create the MultiProjPack.xml with the typical settings for you to edit.

When you run the tool it:

1. Scans all the folder for all .NET projects starting with the toolSettings.NamespacePrefix value e.g., BookApp.Books, and create a .nuspec file containing all the .NET projects it found
2. It then calls dotnet pack to create the NuGet package using the .nuspec file created in step1.
3. If you set up the correct  <toolSettings> it will update your local NuGet package server. I explain this later. 
4. If you ran it in U(pdate) mode it will replace the data in the NuGet cache. I explain this later.  

But before I describe step 3 and 4 when running the MultiProgPack tool, lets consider how you might debug an application using a NuGet package.

The debugging an application that uses modular monolith NuGet package

There is a problem when developing an application using the modular monolith NuGet package approach, and that is the time it takes to upload a NuGet package to nuget.org (or any other web-based NuGet servers). That upload can take a few minutes, which doesn’t make a good development process. But be of good cheer – I have solved this problem and it takes less than a second to upload a NuGet package when testing locally. But before I describe the solution let’s look at the development process first.

For this example, you want to add a new feature, say adding a wish list where users can tell other people which books they would like for their birthday. And your application is called BookApp.Main, which contains the ASP.NET Core FrontEnd, and you decide to add the wish list feature to the code to BookApp.Books solution, which is in a separate solution and installed in the BookApp.Main application via a NuGet package.

The development process would require five things:

1. Add new pages and commands in the FrontEnd code in your BookApp.Main application.
2. Add new services in the part of the application that you have in separate BookApp.Books solution and write some unit tests to make sure they work.
3. Then create a new version of the NuGet package from the BookApp.Books solution.
4. Then you install the new version BookApp.Books NuGet package in the BookApp.Main application.
5. Then you can try out your new feature by running the BookApp.Main locally with dummy data.

If you are a genus, you might be able to write both parts and it all works first time, but most people like me would need to go around these five steps many times. And if you had to wait for a few minutes for each NuGet package upload, it would be a very painful development cycle. Thankfully there are ways around this – the first is using local NuGet server on your development computer.

1. Using a local NuGet server to reduce the package upload time to milliseconds

It turns out you can define a folder on your development computer (Windows, Mac or Linux) as a NuGet package source. This means the upload is now just a copy of your new NuGet package into a folder on your development computer, so it’s very fast. That means your development testing will also be fast.

To add a local NuGet package source via Visual Studio you need to get to its NuGet package sources page. The command to get to that page is Tools > NuGet Package Manager > Package Manager Settings > Package Sources. Then you can add a new package source that is linked to a folder on your computer. Below is a screenshot of my NuGet package sources page where I have added a local folder as a possible NuGet package source (see selected source).

Once you have done that any NuGet packages the folder you have defined will show up in the NuGet Package Manager display if the local NuGet package source (or All) is selected. See the example below where I have selected my Local NuGet in the package source, which is highlighted in yellow.

NOTE: If you are using VSCode and dotnet commands then see this article about setting up folder on your local computer.

To make this even easier/quicker the MultiProgPack tool has a feature that will copy the newly created NuGet package directly into your local folder. To turn on that feature you need to fill in the <toolSettings>.<CopyNuGetTo> setting with your local NuGet package folder. The CopyNuGetTo setting will support the string {USERPROFILE} to get your user’s account folder, which means it works for any developer. For instance, the string {USERPROFILE}\LocalNuGet on my computer would become C:\Users\JonPSmith\LocalNuGet on my computer. That means your MultiProjPack.xml doesn’t have to change for each developer on the team.

The end result of setting the CopyNuGetTo path is that the command MultiProgPack D (or R) is a new NuGet package will appear in your local NuGet package source and you can immediately add or update your main application with that NuGet package.

2. Directly updating the NuGet package cache

The local NuGet package source is great, but you need to in increment the NuGet version for each package, as once a package is in the NuGet package cache you can’t override it. I found that a pain, so I looked for a better solution. I found one by accessing the NuGet package cache.

The NuGet package cache speeds up the performance of builds using NuGet packages. When you install a new NuGet package it adds an unpacked version of the NuGet package in the NuGet package cache. The unpacked version of the NuGet package contains all the files in the NuGet package, e.g. the code (.dll), symbol files (.pdb), documentation files (.xml), and these are copied into your application on certain actions, e.g. Build > Rebuild Solution, Restore NuGet packages, etc.

I take advantage of this feature to speed up the whole write code, install, test cycle (steps 3 to 5 of the development process already described) by directly updating the NuGet package cache at the same time, which is triggered by the MultiProjPack U(pdate). This works if you have created and installed a NuGet package with new code, but it has a bug. Instead of you having to create a new version of the NuGet package the U(pdate) command update the existing NuGet package, both in the local NuGet package source and the NuGet package cache.

The end result is, after running the MultiProjPack U(pdate) command you use Visual Studio’s command Build > Rebuild Solution, which will rebuild your main application using the files in the NuGet package cache. That cuts out two manual steps (changing the NuGet package version and updating the NuGet package in the main application), but there is a limitation.

The limitation is if you add, remove or update the .csproj file (say, by adding the NuGet packages it need), then you can’t use the U(pdate) command. For these cases you have to create a new NuGet package with a higher version number. The MultiProjPack U(pdate) command is useful for testing or fixing a bug and need to go around multiple times.

NOTE: For users that don’t like with the idea of changing the cache another way to go is to add a new NuGet version using the -m: option e.g., MultiProjPack D -m: version=1.0.0.1-preview002. This overrides the version in the MultiProjPack.xml file, thus saving you having to edit the MultiProjPack.xml every time, but then you need to manually update the NuGet package via the NuGet Package Manager display.

These two features reduce the minutes of uploading a NuGet package to nuget.org into a one command that takes seconds to update a NuGet package in your main application.

Tips on how to debug your NuGet package

As part of the build/test/debug cycle you will install or update your NuGet package into the main application to check it works. If something goes fails in your main application you might want to debug code that is in your NuGet package. Here are some tips on how to do this.

The first thing you need to do is turn off the “Just My Code” setting in the Debugger (see figure below). This allows you to step into NuGet package, and set breakpoints, see the local data and so on inside your NuGet package.

The second thing you need to know about how Visual Studio accessed the code introduced by your NuGet package. There are various ways to add the source code, names of local variables, stack traces, and so on into your NuGet package. The ways to add source code information to a NuGet for debugging: embedding the code in the .dll file or using symbols files.

1. Embedding the source code in the .dll file

You can embed the source code in the code (.dll) file, which makes it easy for Visual Studio to find the source code related to the .dll file. This works well, but requires you to add some extra xml commands to the NET project’s .csproj files.

The downside of this approach is it makes the .dll file bigger, so I suggest that you only add the source code to the .dll file when working to a Debug configuration build. To do this need to add following xml commands in a .NET project’s .csproj files you want this feature.

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <DebugType>embedded</DebugType>
    <EmbedAllSources>true</EmbedAllSources>
  </PropertyGroup>

The next approach, using symbol files, is more automatic, but has some limitations.

2. Using the symbols files

By default, a .NET project will create a code (.dll) file and a symbols (.pdb) file in both the bin\{framework} and obj\{framework} folders when a project is build and the MultiProjPack tool will add then to the NuGet package. Visual Studio can use these when debugging, but if you change the code in a project in the NuGet package Visual Studio will stop you debugging that project. 

Visual Studio is very clever about finding debug information. If you open the Visual Studio’s Debuger > Windows > Modules window you will see all the .dll files in your application, and some of them will have a symbol status of loaded. By looking at this I can see that Visual Studio can find the folder holding the solution for the NuGet package, and it can access the symbols (.pdb) file of each project (I don’t know how Visual Studio does that, but it does).

This all sounds perfect, but as I said if you change the code in a project in your NuGet package, then Visual Studio will stop linking symbols file of the changed code. That means you can’t debug the change project anymore. The good news is the MultiProjPack U(pdate) does work, and you still debug the project via symbols.

NOTE: There are different formats for symbols files, which is a bit confusing. Also symbols files and NuGet symbol servers aren’t supported everywhere. The MultiProjPack tool will copy symbols (.pdb) files into the NuGet package if they are found in the project. It can also create a NuGet symbol package if you need it (see this setting).

What to do for production?

MultiProjPack’s D(ebug) and U(pdate) commands create NuGet packages using the Debug configuration of your code, and typically live in the local NuGet project source. But what should you do when you want to release your code for production?

Firstly, you should build a NuGet package using the release version of the code using the R(elease) command, i.e., MultiProjPack R. The created NuGet package will be in a folder called .nupkg in the folder where you ran the MultiProjPack tool. Now you need this release NuGet package so others can use it. How you do that depends on how you work and whether you want your NuGet packages to be private or not.

By default, the git ignore file will ignore NuGet packages so it you create a package you need to store your Release packages in some other form other than your Git repo. If your NuGet package can be public, then it’s easy – use https://www.nuget.org/. For holding NuGet packages privately, then here are some possibilities:

• GitHub: GitHub allows you to publish and consume NuGet packages and they can be private, even for free accounts. See the GitHub NuGet docs and this article by Bruno Hildenbrand.
• Azure DevOps: Azure provides a way to publish and consume NuGet packages within a DevOps pipeline. See the Azure NuGet docs and this article by Greg Margol.
• MyGet: MyGet has been around for years and used by a host of companies, but it costs.
• BaGet: Scott Hanselman’s article recommends BaGet, but at the time of writing it doesn’t have a private feed version.

Conclusion

In my quest to modularize a large monolith architecture I wanted to mimic the Microservice pattern approach of breaking the business problem into separate applications that communicate with each other. To do this I used NuGet packages to implement the “separate” solutions part of the Microservice architecture in a monolith application, using DDD bounded contexts to guide how to break my application into discrete parts.

It does take a big more work to use NuGet packages, but as a working developer I have strived to make this approach as automated and fast as possible. That’s why I build the MultiProjPack tool is a key part both creates the correct NuGet package and has features to make your build/test/debug cycle as quick as possible.

The first article described two ways to modularize a monolith so that its code is less likely to turn into “a ball of mud”. This article takes this another level of separation by allowing you build parts of your application as individual solutions and combine them into the main application using NuGet. The missing part is how you can communicate between bounded contexts, including bounded contexts build as NuGet packages, which I cover in the third article.