When using messaging, in some scenarios many messages are generated as a result of an incoming message to process the initial request. Normally a fan-out pattern or competing-consumers can ensure these messages are processed inline with the designated SLA as we can scale out the processing side of it, but the queuing nature of messaging is also an important factor to remember.

Imagine an incoming request to process a batch of work like image processing, or text file cleaning, or downloading web pages from the internet. The initial incoming request could generate thousands of messages to deal with the request (e.g. a message per work item). Meanwhile a second message comes in and similarly generates thousands of related messages to deal with its work items. The work items for the second message will queue behind the first request’s messages and this could delay the processing for the second request beyond its SLA. This design is also wasteful when you consider cancellation scenarios. When the initial request is cancelled, you will still need to churn through all the generated messages that are in the queue, even though there is nothing to do. Can we do any better?

What is staggered-batch processing

An alternative approach is to not generate all the messages upfront and generate them in batches. We’ll need some sort of a persistence to remember how many of the messages are generated, and generate a new batch once a batch is fully processed. By not generating all the messages upfront, we’re not congesting the queue and in case of cancellations we simply stop generating more messages.

Here ‘batch’ synonymous to a group of messages generated to handle the work process.

While generally you can do this with vanilla messaging and a database, you will need to take consistencies between message queue and database and deal with concurrency issues. What happens when two messages try to update the database at the same time? These are already done by Saga abstractions for you, so you can focus on the business code, like in this case, batching the request and processing them. Let’s see how this can be done using NServiceBus saga implementations.

The saga to generate and track work orders has main events:

• Start a new instance when a work request comes in. At this point, we also start importing the first batch. (StartProcessing)
• Wait for acknowledgement of a batch being imported and import the next batch (WorkOrderCompleted)
• Finish up when all batches are imported and terminate (WorkAllDone)

When importing the next batch, either all work requests are already generated (no more batches to generate) or we need to generate the next one.

1
2
3
4
5
6
7
8
9
10
11

async Task ImportNextBatch(IMessageHandlerContext context)
{
    if (Data.Progress.AllWorkCompleted(Data.WorkCount))
    {
        await FinishWork(context);
    } 
    else if (Data.Progress.HasRemainingWork(Data.WorkCount))
    {
        await SendWorkRequest(MakeNextBatch(), context);
    }
}

The saga Data access allows concurrent and thread safe access to the progress information. The concurrency checks built into the saga means the Data is never updated by two concurrently running messages at the same time. This is all transparent to the users’ code.

The messages to process the work order (ProcessWorkOrder) is sent to a different queue, so it can be scaled out as it makes sense. The processing time of the messages can be monitored so that the number of consumers can be adjusted accordingly.

The output of this process is that the messages are generated in batches (here a batch consists of 100 messages):

1
2
3
4
5
6
7
8

Starting the process for '987' work orders.
Queueing next batch of work orders: (1 - 100).
Importing the next batch of work.
Queueing next batch of work orders: (101 - 200).
...
Importing the next batch of work.
Queueing next batch of work orders: (901 - 987).
All done.

Caveats

This process halts when a message errors out on the processing side, as the subsequent batches are no longer generated and processed. This may be desired in certain scenarios where the work as a whole makes sense (failing a request can fail the whole work), but if partial processing is acceptable this will need to change. There are a few strategies to handle this:

• The generation of the messages can be done via a time out. This will stagger the generation side so that the processing side can handle the messages as they come in.

• More elaborate work tracking can be designed so that errors from processing side are also reported. The errored tasks can be retried/discarded as appropriate.

Summary

The thread and concurrent safe programing model of Sagas allows easier redesigning of fan-out message processes in a way that is more efficient and results in better overall SLAs. The sample code can be found on my github repository.

Back in 2005, I released a set controls and date libraries to work with dates, in particular Solar Hijri Calendar (a.k.a Persian calendar).

To quote from Wikipedia:

The Solar Hijri calendar is one of the oldest calendars in the world, as well as the most accurate solar calendar in use today.

The calendar is in use in many countries such as Iran, Afghanistan, Tajikstan (with minor differences in month names).

The library called FarsiLibrary and was originally built on top of .NET Framework 1.1 and then ported to .NET Framework 2.0, which lacked official solar hijro calendar support. The library was published as an article on CodeProject and was a great hit. The Right-To-Left controls were not a common place and the lack of calendar and date calculation was an impediment to multi-cultural business application. In late 2006/early 2007 WPF was the new cool kid around the block, so as a way to learn, I ported my controls over and subsequently published a new set of controls. This was published as another article on Code Project.

Throughout this process, one thing was a common denominator: I leared a lot as I was working on it.

Now, after more than a decade, I have re-written the library, this time in GoLang as a way to help brush up on my Go. The first initial version can be found on my Github repository and will be officially in the following weeks.

Date Functions

Two way date conversions are now possible, using time.Time and PersianDate structs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import (
    "github.com/heskandari/farsilibrary.go/date"
    "time"
)

func Test() { 

    // convert time.Now
    pd := date.ToPersianDate(time.Now())

    // create directly
    pd2, err := date.NewPersianDate(1387, 7, 7) //7 Mehr 1387

    // same as above, but with parsed strings 
    pd3, err := date.Parse("1387/7/7") 

    // convert back
    gd := date.ToGregorianDate(pd)  // returns a time.Time instance
}

The PersianDate struct implements Stringer interface as well as json marshal/unmarshal interfaces, so you can use it accordingly:

1
2
3
4
5
6
7
8
9

func Serialize() {
    dob, _ := date.NewPersianDate(1400, 01, 01)
    p := Person{
        Name: "John",
        DoB:  &dob,
    }

    js, _ := json.Marshal(&p) // output is {"Name": "John", "DoB": "1400-01-01"}
}

Formatting

Other formatting functions are also available throught the Format() function.

1
2
3
4
5
6

pd, err := date.Parse("1388/02/30")
pd.Format(date.GenericFormat) // returns "۱۳۸۸/۰۲/۰۳"
pd.Format(date.MonthYearFormat) // returns "اردیبهشت ۱۳۸۸"
pd.Format(date.MonthDayFormat) // returns "۳ اردیبهشت"
pd.Format(date.GenericShortFormat) // returns "۱۳۸۸/۲/۳"
pd.Format(date.WrittenFormat) // returns "پنجشنبه ۳ اردیبهشت ۱۳۸۸"

Note that these string use the persian numerals. Other formats also exist that return arabic numerals:

1
2
3

pd.Format("S") // returns serializable format "1388-02-03"
pd.Format("") // returns "1388/02/03"
pd.String() // returns "1388/02/03"

Summary

The Go version of the FarsiLibrary is here! It is built on the trusted and matured .NET version of the library. Feel free to drop a comment if you have any comments/feedbacks.

Let me tell you what the problem was to begin with.

The controller has two actions with two different method names. Both of the methods accept a boolean (with different names). The caller would pass the parameter name and the value of the parameter. The controller code would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

[ApiController]
[Route("[controller]")]
public class MyController : ControllerBase
{
    [HttpPost]
    public ActionResult DoSomething(bool real)
    {
        return Content("Something Done");
    }
        
    [HttpPost]
    public ActionResult DoSomethingElse(bool unreal)
    {
        return Content("Something else done");
    }
}

And the caller would invoke the endpoints like /my?real=true or /my?unreal=true. While this code works fine in the old asp.net WebAPI, this would not work in the new asp.net core application.

But why? There is a history…

So although this was working in my case (asp.net mvc/api) as I was using ApiController class, if you used the other Controller class it would not work. Why the descrepancy? I don’t know for sure, but probably because MVC and WebAPI was developed by two different teams and had different timelines and design decisions.

The design for ApiController pipeline was that it would match the action parameter names against the Query String parameters. The Controller class does not, which means you have to make it non-ambiguous (e.g. have one action per HTTP verb in a controller).

Okay, so it worked in ApiController, so why is it not working in ASP.NET Core? In the .NET Core, they merged the two frameworks into one (yay) but they had to make a decision as to which model they wanted to support and they went the MVC controller model (boo).

When you try to read the controller via my?real=true, you get the following exception:

1
2
3
4

AmbiguousActionException: Multiple actions matched. The following actions matched route data and had all constraints satisfied:

Controllers.MyController.DoSomething
Controllers.MyController.DoSomethingElse

The Fix

The error message should be clear: you need to make your actions less ambiguous. A way to make it less-ambiguous is to map the parameters to querystring parameters (like how the old ApiController did).

The solution would be to create a new attribute for your actions and inherit from ActionMethodSelectorAttribute so that it gets invoke when the request comes in and you get a say if the action is a right match for the incoming request. The attribute would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class MapToQueryStingAttribute : ActionMethodSelectorAttribute
{
    public string QueryStingName { get; set; }
    
    public MapToQueryStingAttribute(string qname)
    {
        QueryStingName = qname;
    }

    public override bool IsValidForRequest(RouteContext routeContext, ActionDescriptor action)
    {
        StringValues value;

        routeContext.HttpContext.Request.Query.TryGetValue(QueryStingName, out value);

        return !StringValues.IsNullOrEmpty(value);
    }
}

and now you can decorate your actions like so.

1
2
3
4
5
6
7
8
9
10
11
12
13

[HttpPost]
[MapToQuerySting("real")]
public ActionResult DoSomething(bool real)
{
    return Content("Something Done");
}
    
[HttpPost]
[MapToQuerySting("unreal")]
public ActionResult DoSomethingElse(bool unreal)
{
    return Content("Something else done");
}

What we say here is that if the query string has that variable name (e.g. ‘real’ or ‘unreal’) match it against the action. Now is depending on your actions and the complexity (e.g. how many actions you have, if you have overloads that could cause overlaps and further ambiguity) this may not be enough. So you may need to further enhance the attribute to say: if the incoming request has this variable but doesn’t have that variable, then map it. In those cases you can use this version of the attribute:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public string QueryStingName { get; set; }
public bool ShouldInclude { get; set; }

public MapToQueryStingAttribute(string qname, bool shouldInclude=true)
{
    QueryStingName = qname;
    ShouldInclude = shouldInclude;
}

public override bool IsValidForRequest(RouteContext routeContext, ActionDescriptor action)
{
    StringValues value;

    routeContext.HttpContext.Request.Query.TryGetValue(QueryStingName, out value);

    if (ShouldInclude)
    {
        return !StringValues.IsNullOrEmpty(value);
    }

    return StringValues.IsNullOrEmpty(value);
}

and now you can say mix the attribute to say the request should include and should not include what parameters.

1
2
3
4
5
6
7

[HttpGet]
[MapToQuerySting("real")]
[MapToQuerySting("unreal", ShouldInclude = false)]
public ActionResult DoSomething(bool real)
{
    return Content("Something Done");
}

Happy porting!

It all boiled down to this: while each individual service capture all the detail of a successfull and failed job, the logs go into individual log files on those services. It is hard to connect these logs across different services. In a nutshell, distributed logging is hard. But does it have to be?

Let’s see how you can get a view of a distributed process by using a structured logging framework. In this example I’m using NServiceBus, Serilog and Seq.

Using Log Context

To understand the ‘context’ (pun intended) of a production error, you’re going to need a lot more information that just the exception and the stack trace. Imagine the error being thrown in a multi-tenant web application. Which customer request is causing this error? What are the request/header information for that header?

Typically the code that logs the message, would look something like this:

1
2
3

static ILog log = LogManager.GetLogger<SubmitOrderHandler>();
...
log.Info($"Received an order #{message.OrderNumber} for [{string.Join(", ", message.ProductIds)}] products(s).");

And the output would look something like this:

Even if you use a logging server like Seq, it is still hard to understand which logs are related to the same workflow.

But surely, we can do better.

Connecting your messages

To connect your workflow, you need to choose an identifier that would connect all messages in your workflow. With the identifier flowing through your services you can later connect your log entries using a LogContext. Usually things like an Order ID, JobID and identifiers like that can be picked but it depends on your code.

With Serilog, the code that’d use the OrderNumber as the added context would look like this:

1
2
3
4

using (LogContext.PushProperty(nameof(message.OrderNumber), message.OrderNumber))
{
    log.Info($"Received an order #{message.OrderNumber} for [{string.Join(", ", message.ProductIds)}] products(s).");
}

We can immediately see the benefit. In Seq, now you can filter for a specific OrderNumber and see all the logs across the services related to that order number.

Using Enrichers

Serilog supports ‘Enrichers’ which are independent modules of code adding context to the logs, similar to LogContexts.

Instead of the previous code-snipper, you can encapsulate that in an Enricher. With a bit of an extension method, you can tie that in to your NServiceBus message handler:

1
2
3
4
5
6
7
8
9
10
11
12

public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
{
    if (msg is IHaveCustomerInfo customerMsg)
    {
        AddProperty(logEvent, propertyFactory, nameof(customerMsg.ClientId), customerMsg.ClientId);
    }

    if (msg is IHaveOrderNumber orderMsg)
    {
        AddProperty(logEvent, propertyFactory, nameof(orderMsg.OrderNumber), orderMsg.OrderNumber);
    }
}

and the your message handler can now do this:

1
2

var logger = context.GetOrderWorkflowLogger(message);
logger.Information("Received an order {OrderNumber} for {ProductIds} products(s).", message.OrderNumber, message.ProductIds);     

Going beyond logging

Did you notice anything extra on the last screenshot?

If you don’t know, NServiceBus supports the concept of ‘ConversationID’ which has a similar purposes: connecting all messages in the same workflow. The way it works is that the first message being sent gets a new ConversationID and the subsequent messages sent in the (handler) context of that message automatically will get the same ID. This way you can identify a group of messages belong to the workflow.

When using Serilog in your NServiceBus endpoints, you should get a reference to the logger via the IMessageHandlerContext object. That way, all that information will also get logged which would be very useful for production use.

The library that does this for you out of the box is NServiceBus.Serilog. If you want to use NServiceBus and Serilog combination, you have to give it a try.

In fact, the conversation information is used in ServiceInsight to visualize the messages. If you are a NServiceBus platform, ServiceInsight can be used to visualize messages in a conversation.

In closing

If you are interesting in reading about distributed systems logging, Jimmy Bogard has an exellent series of posts that covers a lot more.

I have used the NServiceBus showcase codebase for this post but I have somewhat modified it to use Serilog. You can find the modified code on this repository.

Running On Startup

The code to start the endpoint would typically have two steps. The first step is to setup the endpoint using ‘EndpoingConfiguration’ object and the second step is to create, or potentially ‘start’ an endpoint. In NSB v6 and higher, the create and start are both an async method. Initialization of the endpoint during the startup of the WebHost / Generic Host would look like this:

1
2
3
4
5
6
7
8
9
10
11
12

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddOptions();
        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
      
        EndpointConfiguration endpointConfig = CreateEndpointConfiguration(); //<-- configure the endpoint here

        IEndpointInstance endpoint = Endpoint.Start(endpointConfig).GetAwaiter().GetResult();
    }
}

As it is obvious in the code above, you’d have to block the Start method call as it is async. It goes without saying that this is not a good practice and could cause issues like thread getting blocked or even deadlocks in certain situations.

More importantly, if you have message handlers in your process, when you start the endpoint it opens the flood gates and the handler will start receiving messages. This might be undesired as you are still initializing the application. The ConfigureServices stage is for preparing your services and not actually run anything.

Running async tasks

The proper way to run long running async tasks in aspnet core is to use IHostedService interface. The good thing about this interface is that it supports async already as both StartAsync and StopAsync methods are designed to support async method calls.

Starting NServiceBus as a background service would look something like this (this is a simplified version):

Note: there is a base class called BackgroundService implementing this interaface already, so you can inherit from that base class instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

public class NServiceBusServiceHost : IHostedService
{
    EndpointConfiguration _config;
    TaskCompletionSource<IEndpointInstance> _endpointTcs;
    
    public NServiceBusServiceHost(EndpointConfiguration config)
    {
        _config = config;
        _endpointTcs = new TaskCompletionSource<IEndpointInstance>(TaskCreationOptions.RunContinuationsAsynchronously);
    }
    
    public Task<IEndpointInstance> EndpointInstance => _endpointTcs.Task;
    
    public async Task StartAsync(CancellationToken cancellationToken)
    {
        try
        {
            var endpoint = await Endpoint.Start(_config).ConfigureAwait(false);
            _endpointTcs.TrySetResult(endpoint);
        }
        catch (Exception e)
        {
            _endpointTcs.TrySetException(e);
            throw;
        }
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        var endpoint = await EndpointInstance.ConfigureAwait(false);
        await endpoint.Stop().ConfigureAwait(false);
    }
}

And then we can easily plug in the hosted service in our Startup method like:

1

services.AddHostedService<NServiceBusServiceHost>();

And this, of course does not work. The problem is that the service expects the EndpointConfiguration object to be registered in the container. An alternative to that is to manually register the hosted service like so:

1

services.AddSingleton<IHostedService>(provider => new NServiceBusServiceHost(CreateEndpointConfiguration()));

There are still other things to figure out though. For example, how do you send messages in your controllers? With this approach, you’ll need to inject the hosted service into your controller to use the IEndpointInstance interface. If your controller has direct dependency on IEndpointInstance (or IMessageSession) to send messages, it won’t work.

Using the adapter

As you can see, making it work is a lot of chore and figuring out all the little details would add up. Luckily, there are two packages to use. Why two packages, you may ask? We’ll get to that in a bit. Using these packages allow seamless integration between Asp.net Core and NServiceBus. The package essentially does the same thing as here, but also registers IMessageSession in the container as well. Your code would look like this:

1

services.AddNServiceBus(CreateEndpointConfiguration());

There are two packages to choose from to achieve this. There is the Community.NServiceBus.WebHost package, designed for .NET Core 2.x and there is the official NServiceBus.Extensions.Hosting that works for .NET Core 3.x. Why two packages, again? The reason is that there is race-condition in Asp.net Core 2.x which would cause IMessageSession not being available when the application is being initialized. The community package has a workaround for this to ‘make it work’ but the better option is to use the official package but you’ll need to be on .NET Core 3.x to do so.

The official documentation can be found online.

If you have done database work before, you know that optimistic concurrency means if two things happen at the same time, one of them will fail. Armed with NServiceBus retries, we can easily recuperate from the ‘transient’ problems by merely retrying the offending message and eventually get it working since the second time - hopefully - no one has touched the state.

I was reviewing a code where the messages were being retired constantly during a batch import. Imagine importing data in batches via a remote HTTP service and the saga is completed when all the batches are imported successfully. The code looked something like this (skipping the saga signature):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

public async Task Handle(ImportData message, IMessageHandlerContext context)
{
    Data.PagesToImport = message.PagesToImport;

    foreach (var pageNo in Enumerable.Range(1, message.PagesToImport))
    {
        await context.Send(new ImportDataPage
        {
            ImportID = Data.ImportID,
            PageNo = pageNo
        });
    }
}

public async Task Handle(ImportDataPage message, IMessageHandlerContext context)
{
    await remote.ImportDataPage(message.PageNo); //Import the data from the remoteserver

    Data.DataPagesImported++; //Update our local state

    await CheckIfAllPagesImported(context);
}

private async Task CheckIfAllPagesImported(IMessageHandlerContext context)
{
    if (Data.DataPagesImported == Data.PagesToImport)
    {
        MarkAsComplete();
    }
}

Now the interesting thing that was happenning was because the calls to import the data were running concurrently. On one hand, this was a good thing as the call to get the data was a slow HTTP operation running slowly, but the saga was complaining about the concurrency, as 5 requests running at the same time, would clash if they finish around the same time when they are finished and need to update the saga’s state.

First stab: limiting the concurrency level

So if the saga is complaining about too many messages touching the state at the same time what’s the first thing you’d try to do? Maybe dialing down on the concurrency level? That might make sense, since if we have less messages running at the same time, we’ll have less clashing as well. Doing this is like ‘erasing the problem’ though since the concurreny was desired.

Furthermore, customizing concurrency levels per message type is only possible by customizing the message processing pipeline as NServiceBus does not allow you to do that out of the box - and for good reasons.

Second stab: understanding the real problem

Taking a step back, I had to ask myself a question: why is this a problem? Didn’t we just say NServiceBus retries take care of the concurrency issues? Didn’t all the messages eventually succeed? The answer to that question was key.

The real problem this was causing was not the concurrency or the number of retires. The problem was in the fact that the remote HTTP calls are made again, since the whole message had to get retried. Let’s have a closer look at the code:

1
2
3

await remote.ImportDataPage(message.PageNo); // <- Remote HTTP Operation

Data.DataPagesImported++; // <- Local Database Update

See the problem? The problem lies in the way the transactional and non-transactional resources are mixed. You should never do that.

Capturing the intent

How do we separate the two concerns then? By adding another level of indirection of course. Here it means sending a message to capture the intent rather than doing two things with the same message. This is a very important thing to remember.

Here’s how the final code would look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

public async Task Handle(ImportData message, IMessageHandlerContext context)
{
    Data.PagesToImport = message.PagesToImport;

    foreach (var pageNo in Enumerable.Range(1, message.PagesToImport))
    {
        await context.Send(new ImportDataPage
        {
            ImportID = Data.ImportID,
            PageNo = pageNo
        });
    }

    await CheckIfAllPagesImported(context); //If there's no data to import
}

public async Task Handle(ImportDataPage message, IMessageHandlerContext context)
{
    await agent.ImportDataPage(message.PageNo);

    await context.Publish(new DataPageImported
    {
        ImportID = Data.ImportID,
    });
}

public async Task Handle(DataPageImported message, IMessageHandlerContext context)
{
    Data.DataPagesImported++;
    await CheckIfAllPagesImported(context);
}

private async Task CheckIfAllPagesImported(IMessageHandlerContext context)
{
    if (Data.DataPagesImported == Data.PagesToImport)
    {
        MarkAsComplete();
    }
}

As you can see, the act to update the internal state is separated from the act of fetching the data. Does this means that there would not be any concurrency issues? No. There still will be, but the conflicting message that gets ‘retried’ is only to update the internal state and would not need to rerun the whole import process to do so!

New Releases

The following packages are all now available through Nuget:

• FarsiLibrary.Utils for date calculations and helper functions only.
• FarsiLibrary.Win for Windows Forms specific controls
• FarsiLibrary.WPF for WPF based controls and themes
• Farsi.Library.Win.DevExpress-v15.2 for extended controls that add Persian calendar support to DevExpress controls, targetting v15.2.
• Farsi.Library.Win.DevExpress-v16.1 for extended controls that add Persian calendar support to DevExpress controls, targetting v16.1.

But I want integration with the other frameworks…

This library took shape out of my personal need over time and through various projects. I’m no longer using this as extensively as before but I still have some time - although limited - to allocate to this framework. Feel free to send your bug reports or pull requests through Github, if you face any issues or want to give me a hand.

Fast-forward 13 years and things have changed quite a bit. Not only Microsoft added Persian Calendar on .NET Framework 2.0 but years after they finally changed its behavior in .NET Framework 4.6 so that the calculations are correct and consistent with the date calculations of Windows 10. This means more controls and libraries will start supporting the calendar just like libraries like NodaTime already has.

The Road Map

Farsi Library had its own date calculations based on the algorithm from day one. Even with proper support out of the box in .NET Framework, you still need things like helper functions and controls to make dates easier to work with.

On the control side of things, although I have had basic controls like Calendars and Date Pickers, component vendors like DevExpress provide vast array of controls, all of which now come with RightToLeft support. Now that doesn’t mean you can use them directly in your business application but it gives you a good foundation to build what you need.

I have decided to work on ‘extension’ controls off 3rd party components rather than developing new controls. This means that you will take a dependency on a 3rd party control like Telerik or DevExpress but if you are developing business application and need a sophisticated custom control like a Scheduler, changes are you already use a 3rd party control anyway.

New Releases

All the updates going forward are published through Nuget. Historically, only date calculation assembly was available on Nuget but from now on you can find the following packages:

• FarsiLibrary.Utils for date calculations and helper functions
• FarsiLibrary.Win for Windows Forms specific controls
• FarsiLibrary.WPF for WPF based controls and themes
• Farsi.Library.Win.DevExpress for extended controls that add Persian calendar support to DevExpress controls.

DevExpress controls package is updated to target v15.1 but you will need a DevExpress licence and accesss to use them with as they are not bundled together. More DevExpress controls will be available with proper Persian Calendar support.

Release 2.6 Now Available

This is for the most part a maintanance release and an update to target .NET Framework 4.6.

There is only one breaking change in this release. The name “Resources” turns out to be a undocumented reserved-word in WPF and that broke Visual Studio designer support for WPF projects. As such, I had to rename the FarsiLibrary.Resources.dll to FarsiLibrary.Localization.dll.

There’s also a new build system using FAKE (F# Make) scripts and old custom MSBuild scripts are removed from the solution. This means you can easily grab the source code from GitHub and run the Build.cmd from the command prompt to build everything from the source code.

Here’s how it started: I paved my machine and installed SQL 2012 Express for one of my projects. Once the installation went its smooth way and finished at the last step I got a warning that the Windows service can’t be started. Interestingly the installation was not rolled back and I was left with a SQL service that wouldn’t start. Event viewer wasn’t particularly helpful either. He’s how I’ve done my investigation which showed me the exact direction I needed to go.

To understand the issue, I had to run the SQL process somehow and see what’s happening. How do you do this? Pretty easy! You just check how Windows service is running:

So just copy that command into a command prompt, but before you run, we need to force the instance to start and put it in single user mode, so add the following switches to it:

C:\PathToYourSQLInstance\MSSQL\Binn\sqlservr.exe -sMSSQLSERVER /f /T3608

Note that I installed my database as the default instance. This will be a repeating assumption.

Once you run this, you get to see the logs which will tell you exactly what’s happening under the hood. Here are the interesting bits:

1
2
3
4
5
6
7

2015-09-01 19:37:19.92 spid12s     Database 'model' cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details.

    2015-09-01 19:37:19.89 spid12s     Starting up database 'model'.
    2015-09-01 19:37:19.89 spid12s     Error: 17204, Severity: 16, State: 1.
    2015-09-01 19:37:19.89 spid12s     FCB::Open failed: Could not open file E:\sql12_main_t.obj.x86Release\sql\mkmastr\databases\mkmastr.proj\model.mdf for file number 1.  OS error: 3(The system cannot find the path specified.).
    2015-09-01 19:37:19.89 spid12s     Error: 5120, Severity: 16, State: 101.
    2015-09-01 19:37:19.89 spid12s     Unable to open the physical file "E:\sql12_main_t.obj.x86Release\sql\mkmastr\databases\mkmastr.proj\model.mdf". Operating system error 3: "3(The system cannot find the path specified.)".

See that E:\ drive and system can not find the path for model database? That looks to be the issue. I don’t even have an E: drive to begin with!

How do I fix it though? At this stage since the service is up in forced mode, you can connect to it. So bring up another command prompt and run this:

1

SQLCMD -s .

Now from here, you can run the following statement:

1
2

SELECT name, physical_name, state_desc FROM sys.master_files ORDER BY database_id;
GO

This will show you paths to the data file and log file for your databases. Mine was something like the following, which clearly shows that my main databases (temp, model, msdb) all have wrong paths:

1
2

.E:\sql12_main_t.obj.x86Release\sql\mkmastr\databases\mkmastr.proj\tempdb.mdf  .ONLINE
.E:\sql12_main_t.obj.x86Release\sql\mkmastr\databases\mkmastr.proj\templog.ldf .ONLINE

So the fix would be to change that path that’s pointing to the non-existing E: drive to a proper location, which these statements would take care of:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

ALTER DATABASE model MODIFY FILE ( NAME = modeldev, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\model.mdf');
GO
ALTER DATABASE model MODIFY FILE ( NAME = modellog, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\modellog.ldf');
GO



ALTER DATABASE tempdb MODIFY FILE ( NAME = tempdev, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\tempdb.mdf');
GO
ALTER DATABASE tempdb MODIFY FILE ( NAME = templog, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\templog.ldf');
GO



ALTER DATABASE MSDB MODIFY FILE ( NAME = MSDBData, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\MSDBData.mdf');
GO
ALTER DATABASE MSDB MODIFY FILE ( NAME = MSDBLog, FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\MSDBLog.ldf');
GO

After doing this, I could start the service except I couldn’t login with my “sa” account. Fortunately you can use the same tecnique to reset the SA account’s password. Again run the service and connect to it like the above and run these:

1
2
3
4
5

ALTER LOGIN [sa] WITH DEFAULT_DATABASE=[master]
GO

ALTER LOGIN [sa] WITH PASSWORD=N'yoursapasswordgoeshere'
GO

And that’s that. One more thing to note is that even when I uninstalled SQL 2012 and installed SQL 2014 on the machine, exact same thing happened again. Hope this saves someone from a headache.

Nitpicker’s Corner: I understand that if you can live with Eventual Consistency, you can do pretty much everything in your domain with events and commands. In this post, I’m assuming Eventual Consistency isn’t what you are after and you want a transactional boundary.

If you ask what is exactly a Domain Event, here’s Marting Fowler’s definition:

The essence of a Domain Event is that you use it to capture things that can trigger a change to the state of the application you are developing. These event objects are then processed to cause changes to the system…

Depending on your domain, sometimes you can’t live with inconsistencies (or eventual consistency), so the events you raise should be handled in the same unit of work as the action on your Aggregate Root. Having said that, sometimes it conceptually makes sense to have an asynchronous event handler run after the main transaction is committed (and we’ll come back to this). Things like sending emails or communicating with a third party system which is inherently asynchronous and network bound can be done in this way. Let’s see how both can be done.

Since these events run in the same unit of work, they are very easy to manage due to lack of eventual consistency. Essentially, if you roll back the main transaction, the events that are raised are rolled back as well, as if nothing has happened. Let’s see an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

public class Order : AggregateRoot
{
    public Customer Customer { get; set; }

    public Approve() 
    {
        //Removed for brevity
        Raise(new OrderApproved(this));
    }
}

public class OrderApproved : IDomainEvent //Marker interface
{
    public Order ApprovedOrder { get; private set; }

    public OrderApproved(Order order) 
    {
        ApprovedOrder = order;
    }
}

public class PreferredCustomerCalculator : IHandle<OrderApproved>
{
    public void Handle(OrderApproved @event)
    {
        var order = @event.ApprovedOrder;
        var customer = order.Customer; //<-- Owner of the order, referencing Customer class.

        if(SatisfiesPreferredCustomerCriteria(customer))
        {
            customer.Preferred = true;
        }
    }
}

This is basically the gist of it, but there are a few details that you may miss at first glance. First, from performance aspects, since this runs within the same transaction (as I said the event handler is not asynchronous in this example) and we have a long running operation, there is higher chances of deadlocks and timeouts, but hey, the benefit is having consistency within the boundary of the transaction scope.

The main thing here, is that it is not clear ‘how’ the event is raised. What’s the magic to the ‘Raise’ method? The Raise method is simple really, it just adds the events you give to it to an in-memory collection of events. So who raises the event, I hear you ask? This is done somewhere in the bowls of our DbContext. Ideally, we want the event to be raised only when the object graph is persisted in the database, but the transaction is still not committed so that we can roll back, if need be. This way we can do our event processing and database interactions in one transaction to get consistency. Here’s the missing piece of the puzzle:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

public class DomainEvents
{
    private static readonly Dictionary<Type, IList<Type>> HandlerTypes = new Dictionary<Type, IList<Type>>();  //Subscription information

    public static void Subscribe(Type eventType, Type handlerType)
    {
        //On application startup, subscribe all the event
        //handlers to the respective events. You'll need some sort
        //Of subscription storage for them, but in-memory 
        //subscription e.g. a dictionary, may work fine for simple scenarios.
    }

    public void Publish<T>(T @event) where T : class, IDomainEvent
    {
        var handlerTypes = GetHandlersForEvent(@event.GetType()).ToList();
        if (!handlerTypes.Any()) return;

        foreach (var handlerType in handlerTypes)
            ExecuteHandler(@event, handlerType);
    }

    private void ExecuteHandler(IDomainEvent @event, Type handlerType)
    {
        var handler = (dynamic)Container.Resolve(handlerType); //Resolve the event handler from the container.
        handler.Handle((dynamic)@event); //Use of dynamics instead of casting and reflection makes it a bit easier.
    }

    private IEnumerable<Type> GetHandlersForEvent(Type eventType)
    {
        var implementedTypes = eventType.GetInterfaces().Union(new[] { eventType }); // Event type + implemented interfaces

        foreach (var implementedType in implementedTypes)
        {
            IList<Type> handlerTypes;
            if (!HandlerTypes.TryGetValue(implementedType, out handlerTypes)) continue;

            foreach (var handlerType in handlerTypes)
                yield return handlerType;
        }
    }
}

The other thing to note here is that since the event processing will be done in the same transaction, the more you add handlers for an event, the more time it is going to take to complete a transaction. What’s even worse is that it may not be obvious - due to decouple nature of event and event handler - that you’re adding more work to your transactions.

It is not all bad though. If you can live with Eventual Consistency, you can offload some your event handlers to work asynchronously. This means, even the main work is done within the transaction and the side effects are done in subsequent / separate transactions. On next post we’ll see how this works.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

public class Account
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Fullname { get { return FirstName + " " + LastName; } }     
}

public class AccountsController : ApiController
{
    public object Get()
    {
        return new Account {FirstName = "John", LastName = "Doe", AccountNo = 777};
    }
}

And when you issue a request with accept header of application/json, you’ll get this:

1
2
3
4
5

{
    FirstName: "John"
    LastName: "Doe"
    Fullname: "John Doe"
}

Which is all good and dandy. In our case, however, the object we were trying to work with was behaving differently. We didn’t have the source code for that object either, but the only difference we could pick was this:

1
2
3
4

{
    k__BackingField: "John"
    k__BackingField: "Doe"
}

What the….? Two issues here, if you’ve missed it. The name of the properties are totally messed up and the readonly property is not even serialized. The second issue was actually a default behavior when WebAPI was in beta versions, but it should work out of the box with the released / latest versions which we were using. And could really a SerializableAttribute make that big a difference?

Trying to serialize the object that had the SerializableAttribute with json, we could see that the outcome is as expected:

1

JsonConvert.SerializeObject(new Account {FirstName = "John", LastName = "Doe"}); // <- This worked fine

So what’s the issue? Isn’t Newtonsoft JSON library used in WebAPI out of the box?

The solution

After some digging, it turns out that WebAPI in fact is not using the DefaultContractResolver from Newtonsoft.Json library directly, but rather a customized one, although slightly. The one used by the WebAPI out of the box is System.Net.Http.Formatting.JsonContractResolver class. If you have a peek through the code, you’ll see something interesting.

1
2
3
4
5
6
7

public class JsonContractResolver : DefaultContractResolver
{
    public JsonContractResolver(MediaTypeFormatter formatter)
    {
        IgnoreSerializableAttribute = false;
    }
}

That’s suspiciously related to what seems to be our issue here, isn’t it? So the solution was to derive from JsonContractResolver and set that flag to true!

1
2
3
4
5
6
7

public class CustomContractResolver : System.Net.Http.Formatting.JsonContractResolver
{
    public CustomContractResolver() : base(new JsonMediaTypeFormatter())
    {
        IgnoreSerializableAttribute = true;
    }
}

The reasoning

This behavior apparently has introduced in JSON.NET version 4.5. From this version, the fields are serialized instead of properties but there are a couple of ways to change this behavior depending if you have access to the source code of not. You can read more about it here.

Now ORMs in general are not happy with these approach as they mostly rely on tables and Ids to store objects. To map these objects and store them in our database there are two scenarios: Storing an entity with a value object that’s not a collection and Storing it with a collection of value objects.

Taking a Customer and an Address as the example, from the domain perspective, it’d look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

public class Address 
{
    public Address(string streetType, string street, int number) 
    {
        StreetType = streetType;
        Street = street;
        Number = number;
    }
 
    public string Street { get; private set; }
    public string StreetType { get; private set; }
    public int Number { get; private set; }
}

public class Customer 
{
    public Customer(Address postal, Address mailing) 
    {
        Postal = postal;
        Mailing = mailing;
    }
    
    public Address Postal { get; private set; }
    public Address Mailing { get; private set; }
}

Reiterating that we don’t care about the Id, it’d make sense to store this somewhere where we don’t need an Id. If you’re using EntityFramework, it is not possible to store the ‘entity’ (as in an object that’s stored in the database table) without having an ID. If you try to store the address in its own table, you still need an Id and custom mapping (we’ll get to this in a minute), otherwise as in the example above, you can store it as a part of the Customer. To achieve this you’ll need to register your value object as a complex type either by using the [ComplexType] attribute or like this:

1
2
3
4
5
6
7
8
9

public class MyDataContext : DbContext 
{
    public IDbSet<Customer> Customers { get; set; }

    protected override void OnModelCreating(DbModelBuilder modelBuilder) 
    {
        modelBuilder.ComplexType<Address>();
    }
}

Now when you store a Customer, Address properties are stored as fields in the customer table, e.g. Customer table would have fields such as Postal_Street, Postal_StreetType and Mailing_Street, Mailing_StreetType, etc.

The ComplexTypes come with a caveat: They can not be null. You’ll have to new up all the complex type in your constructor. So if they can not be null - much like how an Struct would behave - how would you describe the lack of a value?

Lack of a Value

In value object world, what does it mean to not have a value? Specially if you use Structs, there’s no way to have a null value object. In fact when your value objects are not a collection, EntityFramework would not let you store them as null (where as all the value object fields are set to null). To work around this you can use Null Object pattern. A customer who doesn’t have a mailing address would be constructed like this:

1
2

var postalAddress = new Address(blah blah);
var customerWithJustPostalAddress = new Customer(postalAddress, Address.Null());

and under the hood, the Null static method would just return a new object, so to make EF happy. This is kind of the same as how ‘Empty’ or lack of value is express across .NET platform (remember String.Empty?), the complete implementation of the address class would be like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

public class Address 
{
    private Address() 
    {
    }
 
    public Address(string streetType, string street, int number) 
    {
        StreetType = streetType;
        Street = street;
        Number = number;
    }
     
    public static Address Null() 
    {
         return new Address();
    }
 
    public string Street { get; private set; }
    public string StreetType { get; private set; }
    public int Number { get; private set; }
    
    public static implicit operator string(Address address)
    {
        return string.Format("No. {0} {1} {2}", Number, Street, StreetType);
    }
}

If you need to check the null object being null or not, you can achieve it by checking all the fields being set to null. It may not be ideal, but the question you probably ask yourself if you are doing it is that if the thing you are modelling is a even a value object?

You might ask, what’s the private constructor for? Turns out, EntityFramework needs a parameterless constructor when it hydrates the object from the database. It does not have to be public though, so having it as private helps to encapsulate the entity/value objects.

Storing Collections

When you have a collection of value objects things are different. You can’t store them as a part of your entity and you have to store them in a joined table. Take a look at this example:

1
2
3
4
5
6
7
8
9

public class Customer 
{
    public Customer(IList<Address> addresses) 
    {
        Addresses = addresses;
    }
   
   public IList<Address> Addresses { get; private set; }
}

Here, the addresses should be stored in a separate joined table, while conceptually we don’t care about such a thing as PrimaryKey for our collection of value objects. Our lives would have been much easier if we were using a document database, or a more extensible ORM like NHibernate but in EntityFramework it is not possible to have a table with no PrimaryKey.

The dilemma here is that you want to have your Id for the value object private, since that’s just an implementation detail, but still for mapping and persistence purpose you need to expose it. So if we’re going to store our Address as a collection, this code snippet wouldn’t work:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

public class Address 
{
    private Address() 
    {
    }
 
    public Address(string streetType, string street, int number) 
    {
        StreetType = streetType;
        Street = street;
        Number = number;
    }
     
    private int Id { get; private set; } // <-- Note the private property
    
    //Rest of properties removed for brevity
}

public class AddressConfiguration : EntityTypeConfiguration<Address> 
{
    public ComponentConfiguration()
    {
        HasKey(a => a.Id).Property(a => a.Id).HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity); // <-- Compilation error because Id is private
        
        //Rest of the mapping
    }
}

public class MyDataContext : DbContext 
{
    public IDbSet<Customer> Customers { get; set; }

    protected override void OnModelCreating(DbModelBuilder modelBuilder) 
    {
        modelBuilder.Configurations.Add(new AddressConfiguration());
    }
}

Although you’ll get a compilation error with this code, there are couple of things to note. If you look close enough, we no longer register our address as a ComplexType, but rather as an entity (sigh) but since you will not directly access a value object, we’re not exposing it on the DataContext. The other thing is that we need a way of configuring that Id property to be auto-generated. Since the code does not compile, how do we make it work? There are two ways of fixing this. One way would be to moving that Id into a base class and exposing it so that we can do our mapping (yuck!) and the other way is to move the configuration/mapping code into our entity/value object so that they can access the private property (yuck!). Since there’s no proper solution to this Both workarounds left something to be desired but see which one works for you and pick one.

You still need to generate a unique key for that PrimaryKey but since you don’t care, you can use what your database gives you like Sequence or Identity. On next posts we’ll see which ID generation strategy works better with EntityFramework.

At the end of the day, you may say, these are all languages on top of .NET stack and compile into IL so there’s really not THAT much difference, is there?? Just show me one thing that I can do with F# and I can’t do with C#?

Turns out, there are a few things that you can’t do with C#. Among the things that is not possible to do - at least not at this point in time - is what Type Provider gives you. Let’s quickly see what they are.

What is a type provider

Essentially, type provider provide type information (wasn’t that obvious?) to the compiler as you write your code. If you’ve ever had a joy of working with WebServices (WSDLs), XML or CSV files and things of that nature you probably had to write or generate some sort of class hierarchy to represent that structure and the use that to do what you need. A type provider will do all that for you just by having a look at a schema - that being a sample XML / CSV file with the same structure or the actual WSDL over the internet - and it will generate all the types you need to work with it. Best of all, it is all statically typed, which means once your schema is changed your code automatically won’t compile. This is unlike the WSDL code generation approach that you’d have to go with, where as you’d have to regenerate the code manually to get compilation errors.

Where do I use it

As said, a simple usage would be generating a wrapper for your XML / CSV files to work with. But there are even more interesting things you can do. When it comes down to interconnected data structures like DBPedia, WorldBank and Freebase, generating and working with a huge API is probably not the best way, but with Type Providers for all of these rich information data sources, you can work with them as if there are statically type APIs for them.

Where do I go from here

This was barely even a definition of Type Providers. If you want to learn a bit more and see some samples, you can view my talk about F# Type Providers from at the Sydney F# User Group that is now online and you can watch it here.

Value Objects

So let’s start with something that might be basic. Value Objects, if you have read the Blue or Red book, is a well defined concept. Martin Fowler puts it really well:

I described Value Object as a small object such as a Money or date range object. Their key property is that they follow value semantics rather than reference semantics.

This is a very simple, yet powerful concept, even if you don’t do DDD, you probably will see the benefit of using this. Let me give you an example. Let’s suppose you have a Customer object that has an email address. You’d want to validate the email address to be in a correct format. A naive way of doing it would be validating the string on the UI (being a web or desktop type of application doesn’t really matter) by your validation framework, and then assuming that the string is in valid form. This approach leads to data corruption and still some other layer bypassing the UI is able to work around the validation. Some other would do that on the entity level, something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

public class Customer 
{
    public Customer(string name, string email) 
    {
        ValidateEmail(email);
        //Do the work
    }

    public void UpdateEmailAddress(string email) 
    {
        ValidateEmail(email)
        //Do the work
   }
}

But what if now you need to pass that customer to a service needs to send the actual email, do the service assumes then that the email is already validated? do you validate that again? If you think there must be a better way of doing this, well there is. Using the value objects, you put the validation inside a class for that concept and ensure that whenever you have an object that has got created, it is in a correct state. Not allowing the objects to be in an invalid state is a very powerful thing to have, so basically you wouldn’t need a validator framework or such, at least not on your domain level. When the object is created you can trust that it has satisfied the validity criteria.

With the Value Object approach, the above code looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

public class Email 
{
    public Email(string value) 
    {
        ValidateEmail(value);
        Value = value;
    }

    public string Value { get; private set; }

    private void ValidateEmail(string value) 
    {
        if(EmailIsNotValid) //Validate however makes sense
           throw new DomainRuleViolatedException("Provided value is not a valid email address.");
    }
}

One thing to note here is that the Email value object here is immutable and that is a general heuristic.

There’s still room for improvements. Chances are, you are going to persist this email address, or pass it down to a service that sends an email, one thing that changes your experience with value objects is implicit conversion. To think about this, the email address is in nature a string and can be stored and passed around as such, so we can implement a implicit type conversion to a string:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

public class Email
{
    public Email(string value) 
    {
        ValidateEmail(value);
        Value = value;
    }

    public string Value { get; private set; }

    private void ValidateEmail(string value) 
    {
        if(EmailIsNotValid) //Validate however makes sense
            throw new DomainViolatedException("Provided value is not a valid email address.");
    }

    public static implicit operator string(Email email) 
    {
        return email.Value;
    }
}

Note that Implicit conversion can be a double edged sword. Use it scarcely or you might end up in a situation where the conversion automagically happens and you’ll be clueless on why something is working the way it is.

Now you can pass it to your ORM as a string, or directly pass it to an API that’d require an string and the conversion from already valid email address to a string is automatically done for you.

There are numerous examples where the value object pattern is useful, things such as dates and date ranges where the date conceptually should always be in the past or future, or a concept of a numeric value where the value should always be a positive number or in a certain range.

Another thing to note here is that to be able to assign value objects by value (pardon the pun), you’ll need to use Structs and not Classes. The great example of a value object in struct is the System.DateTime. The reason for going with Struct is that inherently you can assign them by value and not by reference which aligns correctly with our concept of Value Objects. I’m not going to use Structs in these series since they don’t play well with EntityFramework.

Sometimes we just ignore this and expose the object’s internal, this is true specially when we come up with a limitation such as serialization and persistence in the frameworks that we use.

Entity Framework as of version six, has some limitations that can be worked around, in this post, I’ll give you some hints on how to apply encapsulation and information hiding on your domain objects, while letting EntityFramework do its work.

Object Construction

EntityFramework needs a parameterless constructor to construct an object when it needs to instantiate the object before hydrating the object. You might think that you need a public parameterless constructor (a.k.a nullary constructor) but that’s not true. You can get away with private constructor. This helps with encapsulation as the object can only be constructed by user of the code (e.g. your application) correctly.

1
2
3
4
5
6
7
8
9
10
11
12
13

public class Product 
{
    private Product() 
    {
    }

    public Product(string name)
    {
        Name = name;
    }

    public string Name { get; private set;}
}

As you can see from the code, there’s no way you can create an invalid Product object. This class works perfectly fine in EntityFramework as well.

Collection Mapping

Your domain entities might have collection that need persistence, but when you expose your collection as IList or ICollection, entries can be added directly from the outside world to the collection. Although you can enforce proper construction of the object by using the right constructors, this still doesn’t make it right.

Let’s take the above Product class as an example. If you want to keep history of the name changes, how would you do that? Here’s one way of doing that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

    public class Product 
    {
        private Product() 
        {
            NameHistory = new List<ProductNameHistory>();
        }
    
        public Product(string name)
        {
            Name = name;
        }
    
        public string Name { get; private set;}
    
        public void ChangeName(string newName)
        {
            NameHistory.Add(new ProductNameHistory(Name, newName));
            Name = newName;
        }
    
        private IList<ProductNameHistory> NameHistory { get; set; }
    }
    
    public class ProductNameHistory
    {
        private ProductNameHistory() 
        {
        }
    
        public ProductNameHistory(string oldName, string newName)
        {
            OldName = oldName;
            NewName = newName;
        }
    
        public string OldName { get; private set; }
        public string NewName { get; private set; }
    }
``` 

Now this looks fine, except it does not work well with EntityFramework as it won't store private collections/lists. A way to get around it is by creating [EntityTypeConfiguration][4] class for your entity, but because the HasMany method accepts an expression, it needs the collection to be public, which brings us back to where we were. As a workaround, you can move your EntityTypeConfiguration class to reside inside the entity, although not ideal, but that way you can access the private members to do the mapping.

Also if you still want to expose the collection for reading purposes, you can benefit from not-so-new readonly collection interfaces like `IReadOnlyList<T>` and expose your internal collection as readonly which makes it impossible to manipulate:

```cs
    public class Product
    {
        private Product()
        {
            NameHistory = new List<ProductNameHistory>();
        }
    
        public Product(string name)
        {
            Name = name;
        }
    
        public string Name { get; private set; }
    
        public void ChangeName(string newName)
        {
            NameHistory.Add(new ProductNameHistory(Name, newName));
            Name = newName;
        }
    
        private IList<ProductNameHistory> NameHistory { get; set; }
    
        public IReadOnlyList<ProductNameHistory> History
        {
            get { return new ReadOnlyCollection<ProductNameHistory>(NameHistory); }
        }
    
        public class ProductEntityTypeConfiguration : EntityTypeConfiguration<Product>
        {
            public ProductEntityTypeConfiguration()
            {
                HasMany(c => c.NameHistory);
            }
        }
    }

Although encapsulation and OO design practices in general are key, there’s more than just that when it comes to properly designing a domain. Since this is more a DDD topic, I’ll keep it for another post.

When building the invocation of ServiceInsight, I needed to read passed in parameters through the command line. So everyone with a bit of .NET knowledge knows that you can get the arguments like this:

1
2
3
4
5
6
7

public class Program
{
    public static void Main(string[] args)
    {
    //args has the arguments passed in.
    }
}

There’s another way of accessing passed in command line arguments, which is probably useful if you’re creating a WPF application or need to access to command line arguments later and not as soon as the app launches. You can access them through System.Environment object and it looks like this:

1
2
3
4
5
6
7
8

public class App : Application
{
protected override void OnStartup(StartupEventArgs e)
{
var args = System.Environment.GetCommandLineArgs();
//Almost the same as before
}
}

Now how would you test this? I’m aware of all the good Command Line parsing libraries out there, but it’d be an overkill, but you still need to parse the argument, and that’s going to need some unit testing. To make the story short, I came up with a parser and wrap that on an interface, so I had something like this:

1
2
3
4
5
6
7
8

public class CommandLineArgParser : ICommandLineArgParser
{
public ParseResults Parse()
{
var args = System.Environment.GetCommandLineArgs();
//do its thing and return the result
}
}

Now how would you test this thing? Because of the fact that it is using a static object (here System.Environment) and particularly it is a System object, you can’t just mock that, so how would you write any unit tests to ensure your parsing is doing what you intended?

First obvious way is to create an interface and a wrapper around the Environment object and then use the interface. That comes with the burden of creating the wrapper, which depending on the situation, is a lot of unnecessary code to write. But is there anther more generic solution? Turns out, there is one!

How about creating an interface, then on runtime, creating something similar to the wrapper: a class that would implement the interface and then delegate all the calls to the static class? Here’s how you can do it using a Autofac Module:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

public interface IEnvironment
{
    string[] GetCommandLineArgs();
}

public class InterfacesModule : Module
{
    private readonly ProxyGenerator _generator;

    public InterfacesModule()
    {
        _generator = new ProxyGenerator();
    }

    protected override void Load(ContainerBuilder builder)
    {
        builder.RegisterInstance(GenerateInterfaceProxyForType<IEnvironment>(typeof (Environment))).As<IEnvironment>();
    }

    private object GenerateInterfaceProxyForType<T>(Type implementation) where T : class
    {
        return _generator.CreateInterfaceProxyWithoutTarget<T>(new CallForwarderInterceptor(implementation));
    }
}

public class CallForwarderInterceptor : IInterceptor
{
    private readonly object _target;
    private readonly Type _targetType;

    public CallForwarderInterceptor(Type targetType)
    {
        _targetType = targetType;
        _target = null;
    }

    public CallForwarderInterceptor(object target)
    {
        _targetType = target.GetType();
        _target = target;
    }

    public void Intercept(IInvocation invocation)
    {
        var targetMethod = invocation.Method;
        var argumentTypes = invocation.Arguments.Select(a => a.GetType()).ToArray();
        var matchingTypeMethods = _targetType.GetMethods().Where(m =>
        {
            var parameterInfos = m.GetParameters();

            return m.Name == targetMethod.Name &&
                             argumentTypes.Length == parameterInfos.Length &&
                             parameterInfos.Select((p, i) => new { p.ParameterType, Index = i })
                                           .All(p => p.ParameterType.IsAssignableFrom(argumentTypes[p.Index]));
        }).ToList();

        if (matchingTypeMethods.Count == 0) return;

        object result;
        if (targetMethod.IsGenericMethod)
        {
            result = matchingTypeMethods.Single(m => m.IsGenericMethodDefinition)
                                        .MakeGenericMethod(targetMethod.GetGenericArguments())
                                        .Invoke(_target, invocation.Arguments);
        }
        else
        {
            result = matchingTypeMethods.Single(m => !m.IsGenericMethodDefinition).Invoke(_target, invocation.Arguments);
        }

        invocation.ReturnValue = result;
    }
}

The magic is in the CallForwarderInterceptor, which is a DynamicProxy interceptor. What it does is that it generates a proxy to the interface and when you make a method call at runtime on the proxy object (the interface), it looks for a method with exact matching arguments and output and then it’d call that method, get the result and return it as a part of the original method call. Pretty neat, if you ask me. The implementation posted here, works with either generic or non-generic classes alike and is not limited only to static classes.

Whether you use NServiceBus host or run in in-proc you can use the SqlTransport, but configuration is a bit different. To configure your endpoint when running using the host, you simple need an aditional interface on your EndpointConfig class.

1

public class EndpointConfig : IConfigureThisEndpoint, AsA_Publisher, UsingTransport<SqlServer> { }

To do the same thing when running the bus in-proc you need to use fluent configuration:

1
2
3
4
5
6
7
8
9

public void Init() 
{
    NServiceBus.Configure
               .With()
               .DefaultBuilder()
               .UseTransport<SqlServer>("YourOptionalConnectionStringName") 
               .UnicastBus()
               .CreateBus();
}

Once application is started, if you have auto installation of queues enabled, it will generate required tables for you, including Audit and Error queue/tables.

Now suppose something has gone wrong, you have fixed the problem, and you need to return the message to the original queue to process it again, what should you do? You probably have used “ReturnToSource” executable or the Powershell CmdLet, but as bad as it sounds, it is not going to help you here, but fret not! Since you use a database, it should be trivial to remove a record from one table and insert it in another table, right? If you don’t think so, this StoredProc should be good enough to get you started but if you are looking for better tooling, ServiceInsight (already pre-release) supports returning the message to the source queue functionality and is transport agnostic.

History

Allow me to give you some background on where ServiceInsight came from. If you have used NServiceBus - or any other tool relying on MSMQ transport - you must have used the limited built-in MSMQ tooling (if you can even call it that) that comes out of the box. It is so annoying even if you want to simply see a message in the queue. To top it off, if you have an error in your messaging system, like an exception in the message processing pipeline, you won’t even find out why by looking at the message/queue and probably have to check your logs to find out why something is not working, because in this case NServiceBus keeps that extra data as a part of the message header and it is not in the message body in the first place. So fed up with all the problems, I started creating a tool that could help develop my NServiceBus based solution easier.

The initial version which was more of a proof of concept, provided not only viewing of the messages in the queue, but more generic operations like creating queues, purging queues, and removing messages. After starting to building on top of just basic queue management functionality, the great minds behind NServiceBus came up with even a better solution! Why directly interface with the transports like MSMQ? It has its own limitations and you need to support multiple transports just like NServiceBus does. That’s how Management API came into existence. It is a RESTful API that’d allow you query the endpoints and messages in the system and ServiceInsight, among other tools, uses that to achieve the same functionality that I created for MSMQ, and even more. The good thing is that, going forward it’d support all other NServiceBus transports so you’d have seamless functionality regardless of the transport you choose, but there’s also a caveat: this is only available if you use NServiceBus v4.0 and higher.

What is ServiceInsight?

Current beta version contains features that a developer would use on a daily basis. Things like viewing a message body in a queue, the header information and even ability to search for a specific message based on the Id or a field inside the message body is also there and it all should make your life as a developer lot easier. There’s also a killer feature: Ever wondered what has caused message x to be generated? Every developer with his share of asynchronous messaging knows how hard it may be to find out what’s going on just by looking at the code or messages coming and going out of the system. The good thing is that now ServiceInsight would give you the big picture:

So you can see why a message was generated, how much time processing of the message has taken and even see that it has ended up with an error. This all should boost up your productivity.

Do I have to use the latest?

You’d still get some benefits from ServiceInsight even if you’re on previous versions of NServiceBus. The Queue Explorer, as we call it, would give you almost all MSMQ tooling that you need, but naturally since it uses MSMQ api under the hood, it comes with its limitation too, so you may not be able to connect to a remote queue if you don’t have proper permission on your domain setup, or if you don’t have MSMQ installed locally. Since the main audience of this tool at this stage are the developers, the limitations should not be surprising. Having said that, we strongly urge you to try the latest and greatest of what NServiceBus offers so that you can also use ServiceInsight at its best.

What’s vNext?

There are a lot in the works for ServiceInsight already. As I mentioned, next versions will support other transports like ActiveMQ, RabbitMQ and even the SQL Transport. More useful information for messages and endpoints will be surfaced through ServiceInsight as well as visualization of SLAs and message processing timeline so that you have all those information at your fingertips.

Should you happen to use ServiceInsight, we really appreciate your feedbacks and thoughts and the feature you’d like to see in vNext, so keep them coming!

If you’re interested in seeing a short introductory video on ServiceInsight to find out more about it and see major features of v1.0, you can check out this video.

A good example would be adding Hijri Shamsi calendar support to DevExpress scheduler controls. This now can be done by retemplating the whole Scheduler control and using date calculation support of my FarsiLibrary project. The best part is that unlike WinForms, the source code is not needed at all and you can apply this to the out of the box control directly.

Microsoft abandoning Silverlight platform in favour of HTML 5 makes my heart ache. Will there be that level of customization support in HTML 5 applications (and controls, maybe someday) is left to be seen.

Udi suggests that you shouldn’t actually use this, if possible. Instead send the message to the remote queue and let the other end worry about how and when to process it, but this may not work if you’re working in a locked down environment or your customer would simply play it by their rules.

Okay here’s the code snippet if you’re interested.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

public abstract class BaseCommandHandler<T> : IHandleMessages<T>
{
    private TimeSpan _wait = TimeSpan.FromSeconds(30);

    public TimeSpan DefaultWait
    {
        get { return _wait; }
        set { _wait = value; }
    }

    public virtual IBus Bus { get; set; }

    public virtual ITransport Transport { get; set; }

    public virtual void Handle(T message)
    {
        if (CanProcessMessage())
        {
            ProcessMessage(message);
        }
        else
        {
            PauseProcessing(DefaultWait);
        }
    }

    protected abstract void ProcessMessage(T message);

    protected virtual bool CanProcessMessage()
    {
        return true;
    }

    private void PauseProcessing(double interval)
    {
        Transport.PauseProcessing();
        Spin(interval);
    }

    private void Spin(double interval)
    {
        var timer = new Timer(interval) { Enabled = true };
        timer.Elapsed += OnSpinnigFinished;
        timer.Start();
    }

    private void OnSpinnigFinished(object sender, ElapsedEventArgs elapsedEventArgs)
    {
        var timer = (Timer) sender;
        timer.Enabled = false;
        timer.Dispose();
        Transport.ResumeProcessing();
    }
}

public static class TransportExtensions
{
    public static void PauseProcessing(this ITransport transport)
    {
        transport.AbortHandlingCurrentMessage();
        transport.ChangeNumberOfWorkerThreads(0);
    }

    public static void ResumeProcessing(this ITransport transport)
    {
        transport.ChangeNumberOfWorkerThreads(1);
    }
}