The (long) journey to a better sharding multi-tenant application

This article focuses on the sharding approach of building a .NET multi-tenant application using the AuthPermissions.AspNetCore library (shortened to AuthP). After working exclusively on a sharding-only multi-tenant application I found various issues that make building and using such an application is difficult. This article describes the issues I found and the changes I made to the AuthP’s 6 release to improve sharding multi-tenant application.

This article is part of the series that covers AuthP’s multi-tenant applications in general. The other articles in “Building ASP.NET Core and EF Core multi-tenant apps” series are:

1. The database: Using a DataKey to only show data for users in their tenant
2. Administration: different ways to add and control tenants and users
3. Versioning your app: Creating different versions to maximise your profits
4. Hierarchical multi-tenant: Handling tenants that have sub-tenants
5. Advanced techniques around ASP.NET Core Users and their claims
6. Using sharding to build multi-tenant apps using EF Core and ASP.NET Core
7. Three ways to securely add new users to an application using the AuthP library
8. How to take an ASP.NET Core web site “Down for maintenance”
9. Three ways to refresh the claims of a logged-in user
10. Use custom databases with the AuthP library – Part1: normal apps
11. Making .NET sharding multi-tenant apps easier to use with AuthP 5.1 (this article)
12. Use custom databases with the AuthP library – Part2: sharding / hybrid apps (coming soon)

TL;DR; – Summary of this article

• This article is about a sharding multi-tenant application. AuthP has two types of sharding:
  • Hybrid, which supports tenants that share a database with other users and tenants that have their own database, which I refer to Sharding-only.
  • Sharding-only, where every tenant have their own database.
• Part 1 covers two types of multi-tenant applications: one where admin is done by the app-maker and one where specific user(s) in a tenants handle the admin of their tenant.
• Part 2 describes the changes the AuthP’s sharding-only multi-tenant application, which are
  • Allow certain features to work, e.g. the “Sign up now” feature
  • To improve the administration of a sharding-only multi-tenant application.
  • Add a service to make creating / removing sharding-only tenants easier in code and human time.
• Part 3 covers how you can use the new features in AuthP version 6. They are
  • Where does each tenant’s data is stored when using AuthP’s sharding?
  • A service that makes it simpler to create / delete a sharding-only tenant.
  • How the AuthP’s 6 “Sign up now” feature handles sharding-only tenants.
• At the end of this article it provides links to relevant documentation, the file containing steps to update an sharding multi-tenant using an older version, and a new example app.

Part 1: Two types of multi-tenant applications

Recently I have been working on the multi-tenant features in the AuthP’s library, and I started to see that there are different types of multi-tenant applications. This insight came while building a new sharding-only app and it helped me to see areas where I need to improve the AuthP’s sharding futures. The two main types of multi-tenant are based on who is in administration of users and tenants – see the description of two types below.

1. Multi-tenant app controlled by the app-maker

This type of multi-tenant app is controlled by the organisation who owns the app. Normally there is some form of admin team (which I refer to as the app admin) in the organisation who manages the users and its data. Some examples of this type of multi-tenant apps are:

• A bank app allowing its users to manage their bank accounts.
• An in-house app that controls the inventory in the organisation.

The pro of this style is they have good control over who can access the app, but the downside is you need an admin team to manage the users, tenants etc., e.g. add / remove users, changing what they can do, etc.

2. Multi-tenant app controlled by tenant user

This type of multi-tenant app move much of the admin features out to a tenant user. The AuthP has the concept of a tenant admin allow a user to create their own tenant for their organisation (normally with a payment) and then manage the users in their tenant, such as adding new users, controlling what they can do, etc. Some examples of this type of multi-tenant apps are:

• GitHub, which allows users to create their own space and manage.
• Facebook / Meta,  where you post comments, share photos and have friends.

The pros of this style is that the tenant user can quickly makes changes and its reduces the size of the app admin team you need for to support your multi-tenant application. The downside is the multi-tenant administration code is more complex , but AuthP has built-in solutions to most of these admin features.

NOTE: I was asked to design a large multi-tenant application for a client, and I personally know that there is a LOT of admin code. Also, the app admin team were very busy when a large organisation joined their multi-tenant application. This is why I put a lot of work on the tenant admin features to reduce the amount of work the app admin team must deal with.

The AuthP library supports both multi-tenant types, but you will read how I needed to improve the AuthP’s features for sharding-only databases.

Part 2: The problems I found and what I did to improve sharding features

The AuthP’s implementation of sharding uses a “sharding entry” approach that links a tenant to a database. The diagram below the implementation of two types:  “controlled by the app-maker” and the “controlled by tenant users”.

What I found that both types could be improved, but the “controlled by tenants” had one feature didn’t work as it should, and admin code weren’t that easy to use, especially around creating / deleting sharding-only tenants. Part 2 (below) will cover the problems and how I solve them.

Part 2: The problems I found and what I did to solve them

The list below details the various problems and what I did about it 

1. The “Sign up for a new tenant” feature doesn’t handle sharding tenants
2. Solved a limitation in ASP.Net Core’s IOptions feature (breaking changes)
3. Add a service to make creating / removing sharding-only tenants.

2.1. The “Sign up for a new tenant” feature doesn’t handle sharding-only tenants

The AuthP’s “Sign up for a new tenant” feature allows to a user to create a new tenant, which can significantly reduce the amount of work for your admin people. However the version 5.0.0 of the “Sign up” feature doesn’t contain code to setup a new sharding entry in a sharding-only tenant.

But it was then that is hit a limitation of the ASP.Net Core’s IOptions features – see the next section below what the problem is and how I got around it.

2.2. Solved a limitation in ASP.Net Core’s IOptions feature (breaking changes)

The current sharding entries are held in a json file and accessed via ASP.NET Core’s IOptions feature (similar to how ASP.NET Core uses a appsetting.json file). What I found was that the I added a new sharding entry to a json file the IOptionsMonitor won’t see the changes until a new HTTP request starts – I assume it looks for updates at the end of every HTTP request. This causes me a problem as there are cases where a new sharding entry is added and another service needs to retrieve sharding entry within the same HTTP request.

There are a few ways to get around this, but in the end I changed to using my Net.DistributedFileStoreCache library (shortened to FileStoreCache)  that uses the FileSystemWatcher and updates are immediately updated. The upside of using the FileStoreCache is has a similar very fast (~25ns) read-time that the IOptions approach had. The downside is this change creates a breaking change to applications that have been created to older versions of AuthP.  

NOTE: I have created a detail list of the changes in the UpdateToVersion6 file and a console application to help with update your app’s sharding entries to the AuthP 6 format.

2.3. Add a service to make creating / removing sharding-only tenants

When you are adding a new sharding-only tenant there are four steps:

1. Add a sharding entry to define the new database that the tenant will use.
2. Create the tenant using AuthP’s AuthTenantAdminService which will…
3. Run your ITenantChangeService code to set up the database.

…And if you are using a cloud database service, then you need create a new database.

NOTE: If you need to create / delete cloud database, then you can do that withing your implementation of the ITenantChangeService – create your cloud database in the create tenant method and delete your cloud database in the delete tenant method.

You could do this yourself with the AuthP’s version 5 in two manual steps, but in AuthP version 6 there is a new service called IShardingOnlyTenantAddRemove which runs the three steps for creating and removing a tenant. This service makes it easier, and clearer, for an admin person to add or remove sharding-only tenants.

Part 3 – How you can use these new features

So, having told you what has changed in AuthP version 6 let’s see how you would use these updated features. I will cover the following topics, and I will point out the things that have changed

1. Where does each tenant’s data is stored when using AuthP’s sharding?
2. A service that makes it simpler to create / delete a sharding-only tenant
3. How the AuthP’s 6 “Sign up now” feature handles sharding-only tenants

3.1 Where does each tenant’s data is stored when using AuthP’s sharding?

A multi-tenant application provided a service for an organization / person finds useful, but instead of deploying a single web app / database for each tenant they have one web app and they segregate each tenant’s data. In AuthP provides three ways to segregate the tenant’s data, as shown in the table below.

Since AuthP version 3  supports all these ways to separate each tenant’s data, but when I ran a twitter poll on what type of approach they wanted the “Tenants have their own db” (4. In the diagram above) won by a large margin. That why this article (and the new release of the AuthP library) is all about.

AuthP version 3 defined a way that a tenant can be linked to a database. That database may have data from many tenants in it (shared) or a database has only one tenant’s data (shared-only). In this article we are mainly looking sharding, and in particular at the “Select the correct database” part, which applies to both hybrid and sharding-only approaches. The diagram shows the four stages that sets up the tenant’s data DbContext to the specific database for tenant that the logged-in user is linked to.

AuthP version 6 stills these four steps, but it had to change the service that implements stage 2 was changed due the limitation in the ASP.NET Core IOptionsMonitor features. This created **breaking changes** in the version 6 release and I created UpdateToVersion6.md file which shows how to update your existing sharding multi-tenant built using a previous release of the AuthP library.

NOTE: When fixing the ASP.NET Core IOptionsMonitor limitation I also found the existing code rather hard to understand, partly because of the code being complex and partly because the names of some of the classes and methods didn’t really say what they were doing. Therefore I wanted to make the code easier to understand, but this created more breaking changes. Thankfully these changes are obvious i.e. there will be a compile error as the name have changed.

3.2. A service that makes it simpler to create / delete a sharding-only tenant

Section 2.3 talks about a new service called IShardingOnlyTenantAddRemove that combines the creation of the tenant and its sharding entry at the same time. This service means that the admin user only has one page to fill in and the code sets up the rest. The service does the following:

• Gathers the information to create the tenant and sharding entry needed:
  • Tenant name – from the admin user
  • The connection string holding the database server (code or human – see diagram below)
  • Set the database provider used to create the tenant database (e.g. SqlServer) is set by your code
• Then, after various checks, it creates the sharding entry and Tenant in one go.

The diagram below shows two web pages that the app admin user (think “controlled by app-maker” type) to create a new tenant on one go. The two pages show the two types of app: left is where the application has a single database server while on the right is for applications needing multiple database servers.

NOTE: The Example7’s TenantController in the AuthP repo has an example of the multiple database servers option.

NOTE: AuthP’s sharding muti-tenant feature uses connections strings to define the database server, authentication, and other options, but AuthP will add the name of the database. That’s why the  multiple database servers option above uses the name “Servers” for the connection strings names.

The service has a companion class called ShardingOnlyTenantAddDto which makes setting up of the data very simple. In fact the code below will work for both create tenant options as it detects if you only have one valid connection string and sets the ConnectionStringName automatically.

public IActionResult Create([FromServices] 
    IGetSetShardingEntries shardingService)
{
    var dto = new ShardingOnlyTenantAddDto
    {
        DbProviderShortName = AuthPDatabaseTypes.SqlServer.ToString()
    };
    dto.SetConnectionStringNames(
        shardingService.GetConnectionStringNames());
    return View(dto);
}

NOTE: The name of the database and the sharding entry is created by the ShardingOnlyTenantAddDto’s FormDatabaseInformation method which uses time as the database name, e.g. 20230808101149-892. If you want to use the tenant name then the FormDatabaseInformation method can be overridden by you.

3.3 How the AuthP’s 6 “Sign up now” feature handles sharding-only tenants

Earlier I said that the IOptionsMonitor limitation made it impossible to use the “Sign up now” feature to create a new sharding-only tenant (think “controlled by tenant user” type). Once the IOptionsMonitor limitation was overcome it wasn’t that hard to create a service that handles the tenant and its sharding entry.

The AuthP repo contains a DemoShardOnlyGetDatabaseForNewTenant which handles sharding-only tenant and the required sharding entry. The Example7 app in the repo has the “Sign up now” and uses the DemoShardOnlyGetDatabaseForNewTenant. The screenshot below shows this working – note that allows the user to decide where they wanted their data located.

Lots of new / updated documentation for AuthP version 6

When I came back to programming after 21-year break (I went to the dark side, e.g. I was a manager) I picked Microsoft because their documentation was so good. Therefore I know that documentation is (nearly) as important as the code.

So, with the release of version 6 of the AuthP I have added / updated the documentation about sharding. Here is a list.

The documents give you the overview of what to do, but I also take the time to add comments on all the code. That’s because sometimes it easier read the code to really what it does.

Also, the AuthP’s repo now has new example called Example7 which implements a sharding-only multi-tenant application. The Example7 app uses all the new / updated features included in the new AuthP version 6 release.

Conclusion

When I started to look at improving Auth’s sharding-only tenant I kept finding things more and more issues and I wasn’t sure it was worth doing the update, so I did a twitter poll to find out what the users of the AuthP wanted. That poll clearly shows that developers preferred the sharding-only approach nearly twice more than a shared database approach. This feedback made my spend more time to make sharding-only really well.

I’m very happy about the AuthP version 6 update because it fixed the limitations in sharding-only multi-tenant applications and improved the admin of creating / deleting tenants. The only problem is that AuthP version 6 contains the breaking changes, which that means you have alter your the sharding multi-tenant application when update to AuthP version 6 – see UpdateToVersion6 for the details.

There was always going to be some breaking changes, but I found my previous sharding code was hard to understand so I tided up that code at the same time. This adds extra breaking changes, but it turned out to fairly simple. For instance the update of the Example6 (hybrid multi-tenant) from version 5 to version 6 wasn’t hard – usually it was just changing the names of the services and methods, with the complex changed only in the internal code.

I hope AuthP’s user’s will find version 6 helps them, or anyone building a sharding-only multi-tenant application can gets some ideas for their project. Do give me feedback on version 6 and I am happy to answer your questions via the AuthP ‘s GitHub issues pages.

Happy coding.