Home .NET Cooking ORM without leaving the stove

Cooking ORM without leaving the stove

by admin

Cooking ORM without leaving the stove
This article is not a call to extremism development of bicycles.The purpose of the post is to have a good understanding of the mechanism, often it needs to be built from scratch. This is especially true for a topic as shaky as an ORM.

What’s all this for?

Industrial products like MS Entity Framework, NHibernate are complex, have huge functionality, and are essentially a separate thing in themselves. Often to get the right behavior from such ORMs, you need a separate person well versed in the intricacies of such systems, which is not good in team development.
The main theoretical source for the post is Martin Fowler’s book Patterns of Enterprise Application Architecture (P of EAA)
There is a lot of information on what ORM, DDD, TTD actually are, DDS – I will try not to dwell on this, my goal is practical. This article is supposed to be two parts, at the very end there is a link to the full source code in the git hub.

What do we want to get?

Let’s define the terms at a glance: tracking will be understood as the tracking of changes of business objects within one transaction, to further synchronize the data stored by
in RAM and the contents of the database.
So what are the main types of ORMs?

  • By the use of tracking in the .NET world, there are two types of ORMs: with change tracking and without change tracking.
  • Regarding the sql-query generation approach : with explicit use of queries and based on query generators from object models.

For example, among known ORMs without tracking and with explicit use of queries, the most prominent example is Dapper ORM by Stack Overflow The main functionality of such ORMs is to map a relational database model to an object model, with the client explicitly defining what its database query will look like.
The basic concepts of MS Entity Framework and NHibernate in using tracking and query generators from object models. I won’t discuss advantages and disadvantages of these approaches here. All yogurts are equally useful and the truth is in combining approaches.
The customer (i.e. me) wanted to create an ORM using tracking (with the ability to turn off) and based on object model query generators. We will generate sql-queries from C# lambda-expressions, from scratch, without Linq2Sql, LinqToEntities (yes, only hardcore!).
Out of the box, MS Entity Framework has a problem with batch updates and deleting data: you need to first get all the objects in the database, then in the cycle to update/delete and then apply the changes to the database. As a result, we get more calls to the database, than necessary. The problem is described by here We solve this problem in the second part of the article.

Let’s define the main ingredients

Let’s define the way the client code will interact with the library being developed. Let’s define the main components of the ORM which will be available from the client code.
The concept of client code interaction with the ORM will be based on the idea of inheritance from the abstract repository. You also need to define a superclass to limit the set of business objects that the ORM will work with.
Each business object must be unique within its type, so the inheritor must explicitly override its identifier.
Empty repository and business object superclass

public abstract class Repository<T> : IRepository<T>where T : EntityBase, new(){//Later we will implement the methods}

//business object superclasspublic abstract class EntityBase : IEntity{//in the descendants it is necessary to explicitly override the identifierpublic abstract int Id { get; set; }public object Clone(){return MemberwiseClone();}}

The Clone() method we need to copy the object for tracking, we’ll talk about it below.
Let’s have a client business object that stores information about the user – the Profile class.
To use a business object in the ORM you need three steps :

  1. Bind a business object to a table in a database based on attributes
    Profile class
    //All business objects must be descendants of EntityBase superclass[TableName("profile")]public class Profile : EntityBase{//The attribute argument is the exact name of the field in the database[FieldName("profile_id")]public override int Id { get; }[FieldName("userinfo_id")]public int? UserInfoId { get; set; }[FieldName("role_id")]public int RoleId { get; set; }public string Info { get; set; }}

  2. Define a repository for each business object
    The profile repository will look like
    public class ProfileRepository : Repository<Profile>{// pass the name of the connection string as an argument to the base classpublic ProfileRepository(): base("PhotoGallery"){//implementation of the client CRUD methods}}

  3. Generate connection string to the database
    The connection string may look like this
    <connectionStrings><add name="PhotoGallery" providerName="System.Data.SqlClient" connectionString="server=PCSQLEXPRESS; database=db_PhotoGallery"/></connectionStrings>

To track changes, it is common to use the UnitOfWork pattern. The essence of UnitOfWork is to track actions performed on domain objects to further synchronize the data stored in RAM with the contents of the database. In this case, the changes are recorded at one point in time – all at once.
UnitOfWork interface part

public interface IUnitOfWork : IDisposable{void Commit();}

This would seem to be the end of it. But there are two things to consider:

  • Change tracking must serve the entire current business transaction and must be available for all business objects
  • The business transaction must be executed within a single thread, so you must associate the work unit with the currently running thread using the local thread store

If a UnitOfWork object is already associated with a business transaction thread, it should be placed in that object. In addition, from a logical point of view, the unit of work belongs to this session.
Let’s use the static class Session

public static class Session{//local stream storageprivate static readonly ThreadLocal<IUnitOfWork> CurrentThreadData =new ThreadLocal<IUnitOfWork> (true);public static IUnitOfWork Current{get { return CurrentThreadData.Value; }private set { CurrentThreadData.Value = value; }}public static IUnitOfWork Create(IUnitOfWork uow){return Current ?? (Current = uow);}}

If you want to do without tracking, you don’t need to create an instance of the UnitOfWork class, or call Session.Create.
So, after defining all the elements needed to interact with the ORM, here’s an example of how to work with the ORM.
Example of work with ORM

var uow = new UnitOfWork();using (Session.Create(uow)){var profileRepo = new ProfileRepository();//Calling repository methodsuow.Commit();}

Let’s get down to cooking

Everything we talked about earlier was about the public part. Now let’s look at the internal part. For the further development we need to define what the structure of the tracking object is.
You should not confuse a business facility with a tracking facility :

  • A business object has its own type, whereas a tracking object must be suitable for mass manipulation over multiple business objects, i.e. it must not depend on a specific type
  • A tracking object is an entity that exists within a particular business transaction, among the set of business objects, and its uniqueness should be defined within this transaction

It follows that such an object must have the properties :

  • Unique within its type
  • Immutable

In essence, the object for tracking is container for storing business objects. As noted earlier, all client business objects must be ancestors of the EntityBase superclass and the object identifier must be overridden for them. The identifier provides uniqueness within the type, that is, the table in the database.
Implementing an object container for tracking

internal struct EntityStruct{//object typeinternal Type Key { get; private set; }internal EntityBase Value { get; private set; }internal EntityStruct(Type key, EntityBase value): this(){Key = key;Value = value;}public override bool Equals(object obj){if (!(obj is EntityStruct)){throw new TypeAccessException("EntityStruct");}return Equals((EntityStruct)obj);}public bool Equals(EntityStruct obj){if (Value.Id != obj.Value.Id){return false;}return Key == obj.Key;}public override int GetHashCode(){//within the same database, the object type and identifier uniquely determine its//uniquenessreturn (unchecked(25 * Key.GetHashCode()) ^ Value.Id.GetHashCode()) 0x7FFFFFFF;}}

Tracking business objects

Object registration for tracking will take place at the stage of getting these objects from the repository.
After getting business objects from the database, you need to register them as tracking objects.
Such objects have two states: immediately after getting from the database and after getting, before committing changes.
We will call the former "clean" objects, the latter "dirty" objects.

var uow = new UnitOfWork();using (Session.Create(uow)){var profileRepo = new ProfileRepository();//register "clean" objects by copying obtained from the database, //Input objects are considered "dirty"var profiles = profileRepo.Get(x=> x.Info = "Good user");//change "dirty" objectsforeach (var profile in profiles){profile.Info = "Bad user";}//commit changesuow.Commit();}

The important point is that in order to save "clean" objects, copying operations are necessary, which can be detrimental to performance.
In general, the registration of tracking objects should be done for each type of operation, so there should be objects for update, delete, insert operations.
Note that it is necessary to register only really changed objects, which requires value comparison operations (above is the implementation of EntityStruct with the overridden Equals method). In the end the comparison operation will be reduced to comparing their hashes.
Tracking object registration events will be triggered from the functionality of the abstract repository class in its CRUD methods.
Implementation of Tracking Objects Registration Functionality

internal interface IObjectTracker{//for simplicity, here is registration code only for modified and new objectsICollection<EntityStruct> NewObjects { get; }ICollection<EntityStruct> ChangeObjects { get; }//registration methodsvoid RegInsertedNewObjects(object sender, AddObjInfoEventArgs e);void RegCleanObjects(object sender, DirtyObjsInfoEventArgs e);}internal class DefaultObjectTracker : IObjectTracker{//the dictionary key is "dirty" object, the value is "clean" objectprivate readonly Dictionary<EntityStruct, EntityStruct> _dirtyCreanPairs;public ICollection<EntityStruct> NewObjects { get; private set; }public ICollection<EntityStruct> ChangeObjects{get{// get the changed objectsreturn _dirtyCreanPairs.GetChangesObjects();}}internal DefaultObjectTracker(){NewObjects = new Collection<EntityStruct> ();//to avoid unnecessary boxing/unboxing operations, let's implement our own EqualityComparer_dirtyCreanPairs =new Dictionary<EntityStruct, EntityStruct> (new IdentityMapEqualityComparer());}public void RegInsertedNewObjects(object sender, AddObjInfoEventArgs e){NewObjects.Add(e.InsertedObj);}public void RegCleanObjects(object sender, DirtyObjsInfoEventArgs e){var objs = e.DirtyObjs;foreach (var obj in objs){if (!_dirtyCreanPairs.containsKey(obj)){//we get "clean" an object by cloning the original one using MemberwiseClone()var cloneObj = new EntityStruct(obj.Key, (EntityBase)obj.Value.Clone());_dirtyCreanPairs.Add(obj, cloneObj);}}}}

Functionality to detect objects changed by client

public static ICollection<EntityStruct> GetChangesObjects(this Dictionary<EntityStruct, EntityStruct> dirtyCleanPairs){var result = new List<EntityStruct> ();foreach (var cleanObj in dirtyCleanPairs.Keys){if (!(cleanObj.Key == dirtyCleanPairs[cleanObj].Key)){throw new Exception("incorrect types");}if (ChangeDirtyObjs(cleanObj.Value, dirtyCleanPairs[cleanObj].Value, cleanObj.Key)){result.Add(cleanObj);}}return result;}public static bool ChangeDirtyObjs(EntityBase cleanObj, EntityBase dirtyObj, Type type){var props = type.GetProperties();//loop for each object propertyforeach (var prop in props){var cleanValue = prop.GetValue(cleanObj, null);var dirtyValue = prop.GetValue(dirtyObj, null); //if at least one property is changed, believe the building is appropriate to register if (!cleanValue.Equals(dirtyValue)) { return true; } } return false; }

It should be taken into account that business objects of one transaction can be from different databases. It is logical to assume that for each database should be defined a different tracking instance (a class that implements IObjectTracker, for example DefaultObjectTracker).
The current transaction needs to "know" in advance from which databases the tracking will be performed. During the UnitOfWork instance creation phase, you need to initialize the tracking object instances (an instance of the DefaultObjectTracker class) according to the specified database connections in the configuration file.
Let’s change the UnitOfWork class

internal interface IDetector{//word key - base connection string, value - tracking objectDictionary<string, IObjectTracker> ObjectDetector { get; }}public sealed class UnitOfWork : IUnitOfWork, IDetector{private readonly Dictionary<string, IObjectTracker> _objectDetector;Dictionary<string, IObjectTracker> IDetector.ObjectDetector{get { return _objectDetector; }}public UnitOfWork(){_objectDetector = new Dictionary<string, IObjectTracker> ();foreach (ConnectionStringSettings conName in ConfigurationManager.ConnectionStrings){//each connection to the database has its own tracking instance_objectDetector.Add(conName.Name, new DefaultObjectTracker())}}}

Information about which tracking instance corresponds to which base should be available to all repository instances within a transaction. It’s convenient to create a single access point in a static Session class.
The Session class will look like

public static class Session{private static readonly ThreadLocal<IUnitOfWork> CurrentThreadData = new ThreadLocal<IUnitOfWork> (true);public static IUnitOfWork Current{get { return CurrentThreadData.Value; }private set { CurrentThreadData.Value = value; }}public static IUnitOfWork Create(IUnitOfWork uow){return Current ?? (Current = uow);}//The method returns the desired tracking instance for the current transaction// by the name of the connection stringinternal static IObjectTracker GetObjectTracker(string connectionName){var uow = Current;if (uow == null){throw new ApplicationException("Create unit of work context and using Session.");}var detector = uow as IDetector;if (detector == null){throw new ApplicationException("Create unit of work context and using Session.");}return detector.ObjectDetector[connectionName];}}}

Access to data

The data access functionality will directly call methods to access the database. This functionality will be used by the abstract repository class in its CRUD methods. In the simple case, the data access class includes CRUD methods to handle the data.
Implementation of DbProvider class

internal interface IDataSourceProvider : IDisposable{State State { get; }//for simplicity, we will fix changes in the database only for modified objectsvoid Commit(ICollection<EntityStruct> updObjs);ICollection<T> GetByFields<T> (BinaryExpression exp) where T : EntityBase, new();}internal class DbProvider : IDataSourceProvider{private IDbConnection _connection;internal DbProvider(IDbConnection connection){_connection = connection;State = State.Open;}public State State { get; private set; }public ICollection<T> GetByFields<T> (BinaryExpression exp) where T : EntityBase, new(){// delegate returning the text of the select request by the expression expFunc<IDbCommand, BinaryExpression, string> cmdBuilder = SelectCommandBulder.Create<T> ;ICollection<T> result;using (var conn = _connection){using (var command = conn.CreateCommand()){command.CommandText = cmdBuilder.Invoke(command, exp);command.CommandType = CommandType.Text;conn.Open();result = command.ExecuteListReader<T> ();}}State = State.Close;return result;}public void Commit(ICollection<EntityStruct> updObjs){if (updObjs.Count == 0){return;}// delegate key returning the text of the update request by the expression exp//value - modified objectsvar cmdBuilder = new Dictionary<Func<IDbCommand, ICollection<EntityStruct> , string> , ICollection<EntityStruct> > ();cmdBuilder.Add(UpdateCommandBuilder.Create, updObjs);ExecuteNonQuery(cmdBuilder, packUpdDict, packDeleteDict);}private void ExecuteNonQuery(Dictionary<Func<IDbCommand, ICollection<EntityStruct> , string> , ICollection<EntityStruct> > cmdBuilder){using (var conn = _connection){using (var command = conn.CreateCommand()){var cmdTxtBuilder = new StringBuilder();foreach (var builder in cmdBuilder){cmdTxtBuilder.Append(builder.Key.Invoke(command, builder.Value));}command.CommandText = cmdTxtBuilder.ToString();command.CommandType = CommandType.Text;conn.Open();if (command.ExecuteNonQuery() < 1)throw new ExecuteQueryException(command);}}State = State.Close;}private ICollection<T> ExecuteListReader<T> (EntityStruct objs)where T : EntityBase, IEntity, new(){Func<IDbCommand, EntityStruct, string> cmdBuilder = SelectCommandBulder.Create;ICollection<T> result;using (var conn = _connection){using (var command = conn.CreateCommand()){command.CommandText = cmdBuilder.Invoke(command, objs);command.CommandType = CommandType.Text;conn.Open();result = command.ExecuteListReader<T> ();}}State = State.Close;return result;}private void Dispose(){if (State == State.Open){_connection.Close();State = State.Close;}_connection = null;GC.SuppressFinalize(this);}void IDisposable.Dispose(){Dispose();}~DbProvider(){Dispose();}}

The DbProvider class needs an existing connection to the database. We delegate creation of connection and additional infrastructure to a separate class based on the factory method. Thus, create instances of the DbProvider class only through an auxiliary factory class.
DbProvider’s factory method

class DataSourceProviderFactory{static DbConnection CreateDbConnection(string connectionString, string providerName){if (string.IsNullOrWhiteSpace(connectionString)){throw new ArgumentException("connectionString is null or whitespace");}DbConnection connection;DbProviderFactory factory;try{factory = DbProviderFactories.GetFactory(providerName);connection = factory.CreateConnection();if (connection != null) connection.ConnectionString = connectionString;}catch (ArgumentException){try{factory = DbProviderFactories.GetFactory("System.Data.SqlClient");connection = factory.CreateConnection();if (connection != null){connection.ConnectionString = connectionString;}}catch (Exception){throw new Exception("DB connection has been failed.");}}return connection;}public static IDataSourceProvider Create(string connectionString){var settings = ConfigurationManager.ConnectionStrings[connectionString];var dbConn = CreateDbConnection(settings.ConnectionString, settings.ProviderName);return new DbProvider(dbConn);}public static IDataSourceProvider CreateByDefaultDataProvider(string connectionString){var dbConn = CreateDbConnection(connectionString, string.Empty);return new DbProvider(dbConn);}}

Tracking objects should be registered in the CRUD methods of the repository, which in turn delegates the functionality to the data access layer. So, we need to implement the IDataSourceProvider interface with tracking. We will register objects on the basis of the events mechanism, which will be triggered in this class. The supposed new implementation of IDataSourceProvider interface should be able both to initialize registration events for tracking and to refer to the database. In this case it is convenient to decorate the DbProvider class.
Implementation of the TrackerProvider class

internal class TrackerProvider : IDataSourceProvider{private event EventHandler<DirtyObjsInfoEventArgs> DirtyObjEvent;private event EventHandler<UpdateObjsInfoEventArgs> UpdateObjEvent;private readonly IDataSourceProvider _dataSourceProvider;private readonly string _connectionName;private readonly object _syncObj = new object();private IObjectTracker ObjectTracker{get{lock (_syncObj){// getting the necessary instance of trackingreturn Session.GetObjectTracker(_connectionName);}}}public TrackerProvider(string connectionName){_connectionName = connectionName;_dataSourceProvider = DataSourceProviderFactory.Create(_connectionName);// tracking event registrationRegisterEvents();}public State State{get{return _dataSourceProvider.State;}}private void RegisterEvents(){//The use of the class is correct only when using trackingif (Session.Current == null){throw new ApplicationException("Session has should be used. Create a session.");};//subscribing for the tracking eventsDirtyObjEvent += ObjectTracker.RegCleanObjects;UpdateObjEvent += ObjectTracker.RegUpdatedObjects;}public ICollection<T> GetByFields<T> (BinaryExpression exp)where T : EntityBase, IEntity, new(){//receive source objects from the database via an instance of the DbProvider classvar result = _dataSourceProvider.GetByFields<T> (exp);var registratedObjs = result.Select(r => new EntityStruct(typeof(T), r)).ToList();//Initiate the event of registering "dirty" objectsvar handler = DirtyObjEvent;if (handler == null)return result;handler(this, new DirtyObjsInfoEventArgs(registratedObjs));return result;}public void Commit(ICollection<EntityStruct> updObjs){//completely delegate execution to an instance of the DbProvider class_dataSourceProvider.Commit(updObjs, delObjs, addObjs, packUpdObjs, deleteUpdObjs);}public void Dispose(){_dataSourceProvider.Dispose();}}

Intermediate results

Let’s see what our public classes will look like now.
As noted above, the repository class must delegate its functionality to implementations of the IDataSourceProvider interface. When initializing the repository class, based on the connection string passed to the constructor, you need to create the right IDataSourceProvider implementation, depending on the use of tracking. Also, we must take into account that the data access class can "lose" the connection to the database at any time, for which we will monitor this connection with the help of the property.
The UnitOfWork class, as noted earlier, in its constructor must create a list of objects of DefaultObjectTracker class for all available databases in the connection string. It’s logical that committing changes should also take place on all databases: for each instance of the tracker, the method of committing its changes will be called.
Public – classes will take the form of

public abstract class Repository<T> : IRepository<T>where T : EntityBase, IEntity, new(){private readonly object _syncObj = new object();private IDataSourceProvider _dataSourceProvider;//using the property "monitor" the connection to the databaseprivate IDataSourceProvider DataSourceProvider{get{lock (_syncObj){if (_dataSourceProvider.State == State.Close){_dataSourceProvider = GetDataSourceProvider();}return _dataSourceProvider;}}}private readonly string _connectionName;protected Repository(string connectionName){if (string.IsNullOrWhiteSpace(connectionName)){throw new ArgumentNullException("connectionName");}_connectionName = connectionName;var dataSourceProvider = GetDataSourceProvider();if (dataSourceProvider == null){throw new ArgumentNullException("dataSourceProvider");} _dataSourceProvider= DataSourceProvider;}private IDataSourceProvider GetDataSourceProvider(){//if tracking is enabled, create an instance of the DbProvider class// otherwise create its ////decorated version - TrackerProviderreturn Session.Current == null ? DataSourceProviderFactory.Create(_connectionName): new TrackerProvider(_connectionName);}public ICollection<T> Get(Expression<Func<T, bool> > exp){return DataSourceProvider.GetByFields<T> (exp.Body as BinaryExpression);}}public sealed class UnitOfWork : IUnitOfWork, IDetector{private readonly Dictionary<string, IObjectTracker> _objectDetector;Dictionary<string, IObjectTracker> IDetector.ObjectDetector{get { return _objectDetector; }}public UnitOfWork(){_objectDetector = new Dictionary<string, IObjectTracker> ();foreach (ConnectionStringSettings conName in ConfigurationManager.ConnectionStrings){_objectDetector.Add(conName.Name, new DefaultObjectTracker());}}public void Commit(){SaveChanges();}private void SaveChanges(){foreach (var objectDetector in _objectDetector){//fixing changes in tracking instances for each of the databasesvar provider = new TrackerProvider(objectDetector.Key);provider.Commit(objectDetector.Value.ChangeObjects, objectDetector.Value.DeletedObjects, objectDetector.Value.NewObjects, objectDetector.Value.UpdatedObjects, objectDetector.Value.DeletedWhereExp);}}}

In this article I will continue working with related entities, generation of sql queries based on expression trees, methods of batch deletion/modification of data (like UpdateWhere, RemoveWhere).
The entire source code of the project, with no simplifications, is here

You may also like