Home Analysis and system design CQRS and microservices in product development

CQRS and microservices in product development

by admin

How to design a product so as not to bury money in the ground

At what stage of creating a product or system to connect the architectural design of the system, so that then it was not painfully painful for the wasted money? How to decide whether to combine CQRS and microservices.

This article is for businesses with a request to develop an IT solution. We will tell how to launch the product and avoid unnecessary costs associated with the architecture. We’ll also look at how using CQRS will help in implementing functionality in different application clients, and whether microservices are the panacea.

Briefly about CQRS

CQRS (Command-Query Responsibility Segregation) – a pattern used in systems development which states that any system method can be either a query (which does not change the state of the system) or a command (which changes the state of the system). As practice shows, this is one of the most commonly used patterns in software development. It can be applied at different levels and in different situations. For example, the classic division of systems into OLTP/OLAP, where data is often written to the OLTP system and read from the OLAP system, is nothing more than an application of the CQRS pattern in the database architecture.

CQRS and microservices in product development

In the "ancient" times (early 2000s), popular systems pushed the use of CQRS. For example, with Interbase/FirebirdSQL, it was recommended to use different types of transactions for reading and writing. In today’s world, it is very common for two systems to coexist at different levels of architecture. For example, there can be a separation at the level of different systems, when the client’s personal account on the website implements only Query functionality, while all the changes take place in the CRM system inside the company through predefined Command interfaces. You can find examples of using CQRS at the level of JS application architecture. Who would have thought a few years ago that the words architecture and JS would be used in the same application… Though perhaps this is an unnecessary quip.

Two extremes in development

A typical situation : A Big Ball of Mud

CQRS and microservices in product development

An evolutionary approach to development is very common. As a rule, first the MVP is developed without elaboration of architecture and analysis of non-functional requirements, then a long stage of evolutionary refinement begins. As a result we end up with a poorly functioning system which is difficult to fine-tune. In critical cases we end up with a programming language that’s hard to find developers for. It is noteworthy that such consequences may occur regardless of the experience of the team or the quality of management. Everyone tries to do the best under the conditions available at the time, but over time the result is far from ideal.

In our experience, in the absence of regular practice in architectural design, there inevitably comes a point when the system has to be rewritten in its entirety. More effort and money is spent on this than on all previous development. And the worst part is that you have to retrain all the users to work with the new system. In some cases this is comparable in cost to the development of the system itself.

In such situations you may hear from business founders and stakeholders: "We’d like to get to the release and the first customers as soon as possible. And then let’s think about what to do with customers and how to improve our product. With this approach, when the real influx of the first customers comes, the system starts to fail, and the whole team starts to put out fires. In such a situation, the illusion of "the good life" is suddenly dispelled. Developers and all context holders gradually leave the team. The customers are not happy. As a result, the business collapses because it failed to scale.

Microservices won’t save the day

Microservices have been very popular lately. At the same time, some customers in a wave of hype demand that everything be implemented using microservices architecture, not knowing the real cost of its application. In order to benefit, you have to know how to properly prepare microservices. It’s not enough to just do a few independent projects and call it microservices.

When you partition a system into microservices, there are many questions about how the systems interact. For example, how to trace the chain of calls to the different services that led to a particular error. Or how to understand which service is the bottleneck in the system’s operation right now. These questions can be successfully resolved with different degrees of success either by using existing infrastructure solutions (Elastic for logging), or by developing custom infrastructure services. For example, a balancer that takes into account peculiarities of business logic when routing requests. These problems are typical not only for microservice systems, but for all distributed systems.

In most cases, large investments in infrastructure development, especially at the beginning of the system, are not justified. We don’t know if the system will work for any length of time, or if the business idea will fail and the project will be shut down. We do not know how many users will come to us. There are known cases when in a large project the cost of deploying the entire infrastructure for multiple services in multiple configurations, checking operability, scalability, etc. exceeded 1 million rubles. In an average project, the implementation of business logic costs much less. Still, it is an expense that is not reasonable at this stage.

In addition, correctly divide the system into independent microservices at the initial stage, as a rule, it is not possible due to the fact that neither the exact functionality of the designed system, nor the structure of the subject area is known yet. Consequently, an incorrectly selected division by services leads to difficulties and inevitably leads to additional costs in the implementation of the necessary scenarios of the system. In some cases – to the complete impossibility of the correct functioning of the system. A system based on microservices is a particular case of a distributed system and is subject to the CAP-theorem. If not to provide mechanisms of data integrity beforehand, that is often forgotten, then in real operation it is possible to get many unpleasant surprises in the form of data loss or desynchronization.

Golden Mean

Thoughtful architecture is the key to success

When developing traditional monolithic systems, the same issues arise in describing/partitioning the subject area intelligently. Incorrect partitioning of the system into loosely coupled contexts will lead to undesirable consequences: very difficult to make changes, impossibility to trace and verify all behavioral scenarios. Errors will occur and accumulate like a snowball in the most unexpected places.

Whether the customer/user is satisfied with the system depends on how well it is designed. Therefore, it is extremely important to think about the design of the system as early as possible. And, to do this, it is necessary to understand the operation of the system at all levels, who and how will use the system.

Example

Let’s say we are doing an online store, but the customer completely forgot about how shipping works and that in addition to selling you need to complete the order, and for this you need to make a handy order picker interface or make integration of the developed system with the warehouse logistics system. Simply speaking, if we do not have an idea of at least the basic business processes our system will take part in (i.e. business architecture) then we can miss a very large part of the needs and find ourselves in a situation where everything seems to be done right, but the system cannot be used and we need to look for additional funds for rework.

It is not uncommon in such situations to be out of funds because the budgetary resources remaining by this point have been spent on additional, not so critical functions. Without having thoroughly worked out all the interactions of the developed system with other systems (i.e. the solution architecture), we can start doing expensive and currently pointless things, such as developing a CRM system at a time when the customer does not need any particular functionality and can use a ready-made solution. And only after working out and understanding the environment of the developed system, you can reasonably decide on the choice of software architecture, division of the system into loosely coupled contexts, etc.

So where to start?

In practice, a one- or two-day workshop with the customer, where not only the solution architecture is elaborated, but also the low-fidelity system design and the main use cases are worked out, helps a lot to elaborate such architectural solutions. In the case of startups, an extremely important step is to work through Business Canvas together with the customer (if there is no customer yet) so that all parties understand the viability of the idea. It is quite possible that the result will be closing the project right after the workshop: it helps the stakeholders to see the failure of the business idea without wasting time and money on technical implementation. As strange as it may sound, even in such situations, trust between the project participants is greatly increased. One of the results of the workshop will be a document describing the non-functional requirements for the system to be developed. A reasonable question – should it be done, and is it expensive? We answer: it costs 2 days of hard work of an organized team. Most mistakes in product development, if not done, will be much more expensive.

It should be noted that even a conducted workshop does not guarantee the correctness and completeness of the data obtained. The business idea can change over time due to external circumstances. For example, there are very big changes in the way almost any business operates right now. We know for sure that the world will not be the same in a couple of months (hello, COVID-19 and quarantine!). In this case, good practices in the form of a CQRS template at the application architecture level can be used in system development and it will very likely allow you to reuse the functionality you’ve written by breaking it down into independent components.

What did we do

In one of the projects on management and automation of business processes of the company with a fleet of ships we faced the task to release the first version of the software in the shortest time and at the lowest cost. A logical architecture-level CQRS implementation approach was chosen.

The application itself had the following architecture scheme :

CQRS and microservices in product development

As you can see from the diagram, if you follow the basic principles of SOLID, namely Dependency Inversion and Dependency Injection, when you set up the control inversion containers well, all the commands and requests become small pieces that are very easy to start reusing in different parts of your system. That’s what happened in this case, which had its positive result.

In this case you can see 3 parts of the system that may well have similar operation to Model :

  1. Admin office Desktop App (Admin office Desktop App)
  2. Interaction between Ship Desktop Mechanic app and model (Ship Desktop Mechanic app)
  3. SyncService interaction with the model (SyncService)

It has been observed that people who use the app in the office are much more likely to read data from the ships than to fill it in. At the same time, people who are in the Offline area on ships, on the contrary, write more data into the app than read it. Why do we need to know this? Because so far it’s just a blueprint for scaling the architecture, which of course has already done its good, but can do much more good in the next stages of architecture development. If you want to allocate surrounding contexts from chunks of queries to some physical services with an optimized database for reading will be much easier, and contexts from commands to other services. All of this will already depend on the business objectives and the direction of the architecture.

Examples of how to implement such a breakdown in your project

Below I give a couple of basic C# classes that you can just re-use in your solutions.

/// <summary>/// A universal factory for creating queries./// Must be implemented in Composition Root(web api project, main site project, etc.)/// </summary>public interface IHandlersFactory{IQueryHandler<TQuery, TResult> CreateQueryHandler<TQuery, TResult> ();IAsyncQueryHandler<TQuery, TResult> CreateAsyncQueryHandler<TQuery, TResult> ();ICommandHandler<TCommand> CreateCommandHandler<TCommand> ();IAsyncCommandHandler<TCommand> CreateAsyncCommandHandler<TCommand> ();}///<summary>/// Basic interface for command execution/// </summary>public interface ICommandHandler<TCommand>{void Execute(TCommand command);}/// <summary>/// A basic interface for executing a query/// </summary>public interface IQueryHandler<TQuery, TResult>{TResult Execute(TQuery query);}///<summary>/// Factory for Ninject, creating typed commands and queries/// </summary>public class NinjectFactory : IHandlersFactory{private readonly IResolutionRoot _resolutionRoot;public NinjectFactory(IResolutionRoot resolutionRoot){_resolutionRoot = resolutionRoot;}public IAsyncCommandHandler<TCommand> CreateAsyncCommandHandler<TCommand> (){return _resolutionRoot.Get<IAsyncCommandHandler<TCommand> > ();}public IAsyncQueryHandler<TQuery, TResult> CreateAsyncQueryHandler<TQuery, TResult> (){return _resolutionRoot.Get<IAsyncQueryHandler<TQuery, TResult> > ();}public ICommandHandler<TCommand> CreateCommandHandler<TCommand> (){return _resolutionRoot.Get<ICommandHandler<TCommand> > ();}public IQueryHandler<TQuery, TResult> CreateQueryHandler<TQuery, TResult> (){return _resolutionRoot.Get<IQueryHandler<TQuery, TResult> > ();}}

Example of Binding requests via Ninject

public override void Load(){// queriesBind<IQueryHandler<GetCertificateByIdQuery, Certificate> > ().To<GetCertificateByIdQueryHandler> ();Bind<IQueryHandler<GetCertificatesQuery, List<Certificate> > > ().To<GetCertificatesQueryHandler> ();Bind<IQueryHandler<GetCertificateByShipQuery, List<Certificate> > > ().To<GetCertificateByShipQueryHandler> ();…………}

After injecting IHandlerFactory into your class, you get to use your commands and queries as follows :

Example query execution :

Ship ship = mHandlersFactory.CreateQueryHandler<GetShipByIdQuery, Ship> ().Execute(new GetShipByIdQuery(id));

Command execution example :

mHandlersFactory.CreateCommandHandler<DeleteReportCommand> ().Execute(new DeleteReportCommand(report));

When I first started using these developments, my thoughts were :

  1. Man, I used to write functions in the repository and now I have to make classes from almost every function.
  2. Turns out you can automate class creation, and it’s not a problem at all, and it only takes a couple of seconds longer.
  3. Wow, now it’s several times faster to find what I need in the code. I just know how folders are organized from contexts and commands/requests, and I don’t have to search for anything at all. I just open the class I want.
  4. Cool, this can also be overused!

But of course everything has to be applied situationally and wisely. That’s what the architecture is for. The other point is that you shouldn’t try to think it up 300 steps ahead. It has to evolve situationally along with the product, and be all the answer to at least 3 questions :

  1. Why is the structure of my IT product like this? Why should it be implemented this way at this point?
  2. If I come up with a cool implementation, how does it fit into the overall system picture?
  3. How will the non-functional requirements of the system as a whole be met?

Also a very important step is the moment of highlighting Bounded Context in the system. This practice does a good job of portraying the customer’s business structure, finding common ground with them, and managing regression in the product. But that’s the next article about it.

Benefits of CQRS at the logical architecture level

The basic architecture of the considered application was built following different design principles and patterns: MVVM, SOLID, CQRS, etc. This allowed to reuse the functionality of features for different clients of the application. The implementation did not take long and was quite inexpensive.

The implementation of this approach didn’t require any additional costs, since at the start of the development the team already had a working knowledge of classes and one level of understanding of the application’s architecture. In further revisions this approach justified itself completely: a large percentage of functionality could simply be reused flexibly. With this approach, the customer significantly reduced the cost of implementing the functions, some parts of which are duplicated for different clients of the application.

Lastly

Mistakenly, the Agile approach to development is interpreted as "we don’t need to design anything in advance, we need to get into the fight, and then the war will show." And if we lack the speed of work or speed of program changes, we charge a silver bullet – microservices. After all, at all conferences they tell us that microservices are easy and solve all problems at once. This optimism usually leads to a loss of money and a product that doesn’t work.

On the other hand, designing everything in advance in our fast-paced world is almost impossible. As always, you have to find a balance. Firstly, the elaboration of solution architecture allows you to consciously choose a suitable solution at the current stage and as a result save money for the customer. Secondly, documentation of architectural decisions will allow you in the future to understand why these decisions were made. Thirdly, periodic validation of changed conditions allows you to reduce the risk of failure of the solution and make an informed decision about the need for revision or change. Thus, following sound practices directly when writing code allows you to get a good foundation on which to scale the solution in the future.

You may also like