A single project Windows Service and Console

30 Aug 2015

I have found that when developing MicroServices, I often want to run them from within Visual Studio, or just as a console application, and not have to bother with the hassle of installing as windows services.

In the past I have seen this achieved by creating a Class Library project with all the actual implementation inside it, and then both a Console Application and Windows Service project referencing the library and doing nothing other than calling a .Start() method or similar.

While this works, it has always bugged me as there should be a straight forward way of achieving a single exe to do both roles. It turns out there is an easy way to do it too...

Creating the Project

First, create a WindowsService project in VisualStudio: New Windows Service

Then open the project properties, and change the project type to Console Application and set the startup object: Service Type

Next, open Service1.cs and add a new method (and rename it to Service if you feel the need!):

public void StartConsole()
{
    Console.WriteLine("Press any key to exit...");
    OnStart(new string[] { });

    Console.ReadKey();
    OnStop();
}

Finally open Program.cs and replace the Main method:

static void Main()
{
    var service = new Service();

    if (Environment.UserInteractive)
    {
        service.StartConsole();
    }
    else
    {
        ServiceBase.Run(new ServiceBase[] { service });
    }
}

Displaying Output

Calling Console.Write* and Console.Read* methods when running as a windows service will cause exceptions to be thrown, which suggest that you should redirect the console streams to use them under a windows service.

As a MicroService you shouldn't need to be reading keys from the console (other than the one in our StartConsole method), but writing output would be useful...

To do this I like to use my logging library of choice (Serilog), which I have setup to write to files and to a console:

private void InitializeLogging()
{
    var baseDirectory = AppDomain.CurrentDomain.BaseDirectory;
    var logs = Path.Combine(baseDirectory, "logs");

    Directory.CreateDirectory(logs);

    Log.Logger = new LoggerConfiguration()
        .MinimumLevel.Debug()
        .WriteTo.ColoredConsole()
        .WriteTo.RollingFile(Path.Combine(logs, "{Date}.log"))
        .CreateLogger();
}

And call this method inside the Service1 constructor:

public Service()
{
    InitializeComponent();
    InitializeLogging();
}

The Edge Case

There is one slight edge case which I am aware of, which is that the Environment.UserInteractive property can return true even when running as a windows service if when you install the service you tick Allow service to interact with desktop checkbox:

Service-Logon

My only solution to this is: Don't tick that box. I don't think I have ever used that option anyway!

Wrapping Up

Using this method means less code and projects to maintain, and a very easy path to go from running a service as a desktop application to service.

design, code, overseer, microservices, console, cli

---

Don't Let The Database Dictate Your Design

01 Apr 2015

I have been thinking recently about how the database can influence our design decisions, and perhaps makes them harder than they need to be in some cases. An example of this is the design of a system which stores data about people, specifically for this, their email addresses. A cut down version of the structure is this:

table people
id serial primary key
firstname varchar(50)
lastname varchar(50)

table emails
id serial primary key
person_id int => people.id
address varchar(100)
type int

Which is represented in code like so:

public class Person
{
    public int ID { get; private set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    public List<Email> Emails { get; private set; }

    public Person()
    {
        Emails = new List<Email>();
    }
}

public class Email
{
    public int ID { get; private set; }
    public int PersonID { get; set; }
    public string Address { get; set; }
    public EmailTypes Type { get; set; }
}

While this works, it is heavily influenced by the storage technology. Email addresses by definition are unique already, so why do we need a primary key column? They are also associated with exactly one person, so the person_id column is only here to facilitate that. Why not get rid of the emails table completely, and store the person's email addresses in a single column in the person table? This could be done with a simple csv, but it would be more fore-thinking to use json, so we can associate a little more data with each email address.

So before we get on to putting data in one column, what reasons we have to keep emails as a separate table?

  • So they can be queried by ID.
  • So we can put a constraint across [person_id, address] and [person_id, type].
  • So all emails of a given type can be found.
  • So a person can be looked up by email.
  • So we can attach rules to them.

The first three can be answered easily: you never query for an email address by its primary key, only by the address itself. The constraints are really just a safety net, and a nice to have at best - the collection which manages emails is well tested, implements all business rules properly, and everything which deals with emails uses it. Getting all emails of a particular type is a legitamate requirement, but can be gotten around in several ways: selecting the entire email column in a sql query, and doing additional filtering client side for the specific email types, or if you are using a database which supports json querying (such as postgres), using that to narrow the results down.

The final point is the most interesting, as it could be resolved with a few different designs. The current design has one additional table:

table rules
id serial primary key
person_id int => people.id
target_type int --e.g 1=email, 2=phone, 3=address etc
target_id int
active bool

And the Person object has a method like this:

public bool HasRuleInForce(Entity target)
{
    return Rules
        .Where(rule => rule.TargetType == target.Type)
        .Where(rule => rule.TargetID == target.ID)
        .Where(rule => rule.Active)
        .Any();
}

While this works, the design has a few problems:

  • There is no foreign keying of rules.target_id available
  • So you have to remember to delete rules when deleting any entity
  • You have to remember if an entity is valid for rules to be attached to
  • If normalisation was your argument for an emails table, explain this table relationship...

There are two solutions to this problem:

The first is to change the rules table to just have a target column, and put the unique data in there e.g. a rule for an email would have the email address in the target column, a rule for a phone number would have the actual phone number in the target column. While this works, it doesn't really improve the design of the system; we still have the existing joins and "remember to also" problems of before.

The second solution is to remove the rules table entirely and implement rules as small collections on each target entity, and make the person.Rules property a readonly aggregate. This has a few advantages: each entity explicitly has a rule collection if applicable, and we no longer need to remember to check another collection for updates/deletes.

The implementation of a .Rules property on each entity is trivial - just a standard list property:

public class Email
{
    public int ID { get; private set; }
    public int PersonID { get; set; }
    public string Address { get; set; }
    public EmailTypes Type { get; set; }
    public List<Rule> Rules { get; set; }
}

As we don't wish to repeat the logic on each collection of rules, we can add an extension method for checking if rules are in force:

public static class RulesExtensions
{
    public static bool HasRuleInForce(this IEnumerable<Rule> self)
    {
        return self.Any(rule => rule.Active);
    }
}

And finally on the Person object itself, we can make a simple aggregate property for all child entity's rules:

public IEnumerable<Rule> Rules
{
    get
    {
        var all = new[]
        {
            Emails.SelectMany(e => e.Rules),
            Phones.SelectMany(p => p.Rules),
        };

        return all.SelectMany(r => r);
    }
}

Personally I prefer the 2nd form of this, as it makes domain modelling a lot more straight forward - however like all things, you should consider all your requirements carefully - and don't let the database (sql or nosql variety) dictate your model.

design, code, postgres, sql, architecture

---

The problems with and solutions to Repositories

28 Mar 2015

Repositories are a design pattern which I have never been a huge fan of. I can see the use of them as a good layer boundary, but too often I see them being used all over the place instead of at an infrastructure level in a code base.

A particularly prevalent version of this misuse I see is self populating collections. These generally inherit List<TEntity> or Dictionary<TID, TEntity>, and provide a set of methods such as .LoadByParentID(TID id). The problem with this is that the collection still exposes methods such as .Add() and .Remove() - but these operations only run on the in-memory entities, and don't effect the data source itself.

The Alternative

The technique I prefer for reads are Query objects. These are simple classes which expose a single public method to return some data. For example:

public class GetDocumentsWaitingQuery : IDocumentsQuery
{
    private readonly IDataStore _dataStore;

    public GetDocumentsWaitingQuery(IDataStore datastore)
    {
        _dataStore = datastore;
    }

    public IEnumerable<Document> Execute()
    {
        using (var connection = _dataStore.Open())
        {
            return connection
                .Query<Document>(
                    "select * from documents where status == @status",
                    new { status = DocumentStatuses.Waiting})
                .ToList();
        }
    }
}

The code using this class might look something like this:

public class DocumentProcessor
{
    private readonly IDocumentsQuery _query;

    public DocumentProcessor(IDocumentsQuery waitingDocumentsQuery)
    {
        _query = waitingDocumentsQuery;
    }

    public void Run()
    {
        foreach (var document in _query.Execute())
        {
            //some operation on document...
        }
    }
}

This class is almost too simple, but resembles a system's processor which I wrote. They key here is that the DocumentProcessor only relies on an IDocumentsQuery, not a specific query.

Normal usage of the system looks like this:

public void ProcessAll()
{
    var query = new GetDocumentsWaitingQuery(_dataStore);
    var saveCommand = new SaveDocumentCommand(_dataStore);

    var processor = new DocumentProcessor(query, saveCommand);

    processor.Run();
}

When the user requests a single document get reprocessed, we just substitute in a different Query:

var query = new GetDocumentByIDQuery(_dataStore, id: 123123);
var saveCommand = new SaveDocumentCommand(_dataStore);

var processor = new DocumentProcessor(query, saveCommand);

processor.Run();

And finally, when the system is under test, we can pass in completely fake commands:

[Fact]
public void When_multiple_documents_for_the_same_user()
{
    var first = new Document { .UserID = 1234, .Name = "Document One" };
    var second = new Document { .UserID = 1234, .Name = "Document Two" };

    var query = Substitute.For<IDocumentsQuery>();
    query.Execute().Returns(new[] {first, second});

    var processor = new DocumentProcessor(query, Substitute.For<ISaveDocumentCommand>());
    processor.Run();

    first.Primary.ShouldBe(true);
    second.Primary.ShouldBe(false);
}

This means that in the standard usage, it gets passed an instance of GetDocumentsWaitingQuery, but when under test gets a Substitute.For<IDocumentsQuery>(), and for debugging a problem with a specific document, it gets given new GetSingleDocumentQuery(id: 234234) for example.

Commands

What about saving? Well it's pretty much the same story:

public class SaveDocumentCommand
{
    private readonly IDataStore datastore;

    public SaveDocumentCommand(IDataStore datastore)
    {
        _dataStore = datastore
    }

    public void Execute(Document document)
    {
        using (var connection = _dataStore.Open())
        {
            connection.Execute("update documents set status = @status where id = @id", document);
        }
    }
}

Obviously the sql in the save command would be a bit more complete...

But Repositories...

Well yes, you can create methods on your repositories to do all of this, like so:

public IDocumentRepository
{
    public void SaveDocument(Document document) { /* ... */ }
    public IEnumerable<Document> GetDocumentsWaiting() { /* ... */ }
}

But now your classes utilising this repository are tied to the methods it implements - you cannot just swap out the workings of .GetDocumentsWaiting for a single document query any more.

This is why I like to use Command and Query objects - the not only provide good encapsulation (all your sql is contained within), but they also provide a large level of flexibility in your system, and make it very easy to test to boot too!

design, code, cqrs, architecture

---

Communicating Intent in APIs

25 Mar 2015

Recently was trying to work out how to allow custom resources to be specified in Dashen. I already know what data is needed/defined for a resource: a name, a MIME type, and a Stream. We can make this required data known very easily:

public class Resource
{
    public string Name { get; private set; }
    public string MimeType { get; private set; }
    public Stream Content { get; private set; }

    public Resource(string name, string mimeType, Stream content)
    {
        Name = name;
        MimeType = mimeType;
        Content = content;
    }
}

As all the parameters can only be set through the constructor, you are communicating that they are all required.

However when it comes to adding this Resource into our configuration, we are met with 3 possible solutions:

Resource collection on the config

var dashboard = DashboardBuilder.Create(new DashboardConfiguration
{
    ListenOn = new Uri("http://localhost:3030"),
    Resources = new[] { new Resource("test.png", "image/png", new FileStrea(...))}
});

As the DashboardConfiguration object is only used in this one call, it implies that the contents of it only get read once. Nothing to stop you holding on to a reference to the Resources collection though.

AddResource method on the config

var config = new DashboardConfiguration
config.ListenOn = new Uri("http://localhost:3030");

config.AddResource(new Resource("test.png", "image/png", new FileStrea(...)));
//or
config.AddResource("test.png", "image/png", new FileStrea(...));

var dashboard = DashboardBuilder.Create(config);

Resources are still added to the DashboardConfiguration, but this time via a method. This hides the internal storage of resources. Second version also means we can hide the Resource class from the public too if we want. Also implies a level of uniqueness - could throw an exception on duplicate name being added, or rename the method to AddUniqueResource or similar.

AddResource method on the Dashboard

var dashboard = DashboardBuilder.Create(new DashboardConfiguration
{
    ListenOn = new Uri("http://localhost:3030"),
});

dashboard.AddResource(new Resource("test.png", "image/png", new FileStrea(...)));
//or
dashboard.AddResource("test.png", "image/png", new FileStrea(...));

Resource class is still hideable. Being able to add to the dashboard rather than the config implies that resources could be added at anytime, rather than just startup/config time.

Selected Solution

In the end I decided to expose the Resources as an IEnumerable<Resource> on the DashboardConfiguration object. I did this as I don't actually mind if the collection gets modified once the dashboard is started, and I can see some use-cases for dynamic resource resolution.

design, code, api

---

Encapsulation in Warcraft Addons - Inheritance

05 Dec 2014

Using Inheritance (sort of)

When we actually need inheritance, things get a little more complicated. We need to use two of lua's slightly harder features to get it to work: metatables and colon notation. A little background on these will help:

MetaTables

All "objects" in lua are tables, and tables can something called a metatable added to them. Metatables can have special methods on them which run under certain circumstances (called metamethods), such as keys being added. A full list of metamethods is available here.

The metamethod we are interested in is called called __index, which gets triggered when a key is not found in the table.

There are two ways of using __index. The first is to assign it a function, which gets passed two arguments: table, and key. This is useful if you want to provide a default value if a key in a table isn't found, which I use in the spellData example in the previous post.

The other way of using __index is to pass it another table of methods to call, like in this example:

local meta = {
    print = function()
        print("Hi from the metatable")
    end
}

local actual = {
    test = function()
        print("testing")
    end
}

--wont work:
-- actual.print()

setmetatable(actual, { __index = meta })

-- now it will!
-- actual.print()

By calling setmetatable on actual, we provide actual with all the methods on meta. A table can only have one meta table though, and you might break things by overwriting it (example, don't call setmetatable on a Frame or ActionButton...)

Colon Notation

All methods on a table can be called in two ways; with a colon, or with a period. The colon can be thought of as "fill in the first parameter with the table this is being called on". For example, these two statements are equivalent:

local x = string.gsub("hello world", "hello", "bye")
local x = "hello world":gsub("hello", "bye")

In the example above, the signature of gsub is something like this:

local string = {
    gsub = function(self, searchTerm, replacement)
        --self is the input string
    end,
}

The convention used is to call the first parameter self. We can now use this colon notation with metatables to make our version of inheritance.

Combining

local base = {
    increase = function(self)
        self.count = self.count + 1
    end,
    print = function(self)
        print("The count is " .. self.count .. ".")
    end
}

local first = {
    count = 0
}
setmetatable(first, { __index = base })

local second = {
    count = 100
}
setmetatable(second, { __index = base })

--usage
first:increase()
second:increase()

first:print()       -- prints 1
first:print()       -- prints 101

Due to the way the colon operator works, the self parameter is filled in with the table calling the method, not the table the method is defined on. So calling first:increase() is the same as base.increase(first)

Usage

We can now take these elements, and craft a set of classes designed for reuse. We start off with our root object (think System.Object if you are from a .net world.)

local class = {

    extend = function(self, this)
        return setmetatable(this, { __index = self })
    end,

    new = function(self, ...)

        local this = setmetatable({}, { __index = self })
        this:ctor(...)

        return this

    end,

    ctor = function(self, ...)
    end,
}

We have two methods here, extend and new. The new method is nice and straight forward - it creates a new table, assigns the meta to be class and calls the ctor method (which is the one you would want to replace in sub classes).

The extend method takes in a new table, and applies and sets the meta to class. This is what is used to inherit and add new functionality.

For example, in my control library, I have a base class with some common methods:

local control = class:extend({

    size = function(self, config)
        self.frame:SetSize(unpack(config))
    end,

    point = function(self, config)
        self.frame:SetPoint(unpack(config))
    end,

    parent = function(self, value)
        self.frame:SetParent(value)
    end,
})

And then many other classes which extend the base, cilling in the ctor method with how to actually create the frame:

local label = control:extend({

    ctor = function(self, name, parent)
        self.frame = CreateFrame("Frame", name, parent)
        self.label = self.frame:CreateFontString()
        self.label:SetAllPoints(self.frame)
        self.label:SetFont(fonts.normal, 12)
    end,
})

local textbox  = control:extend({

    ctor = function(self, name, parent)
        self.frame = CreateFrame("editbox", name, parent, "InputBoxTemplate")
        self.frame:SetAutoFocus(false)
        self.frame:SetFont(fonts.normal, 12)
    end,

    text = function(self, value)
        self.frame:SetText(value)
    end,
})

Some classes, such as the textbox provide other methods where they make sense.

Calling Base Class Methods

If we wish to start overriding a method and then call the original method within, things start to get a lot more complicated.

local class = {
    extend = function(self, this)
        this.base = self
        return setmetatable(this, { __index = self })
    end,
}

local child = class:extend({
    method = function(self)
        self.name = "child"
    end,
})

local grandchild = child:extend({
    method = function(self)
        self.base:method()
    end
})

While this looks like it will work, it will cause some strange and hard to debug problems (I know it will, it took me ages to figure out.)

The problem is that when you do self.base:method() you are effectively doing self.base.method(self.base), which means the base method is referencing the wrong table!

We can solve this, but it requires a certain level of voodoo. First we need to change our extend method:

extend = function(self, this)

    this.super = function(child)

        local parent = {
            __index = function(_, methodName)
                return function(_, ...)
                    self[methodName](child, ...)
                end
            end
        }

        return setmetatable({}, parent)
    end

    return setmetatable(this, { __index = self })
end

This took me far too long to come up with and get working. Essentially what it does is take all calls, and replace the self parameter with the correct table.

This method has some restrictions, in that you can only go 'up' one level in the class hierarchy, e.g. you cannot do item:super():super():super(). In practice though, I have never needed to do this.

The entirety of my class file can be found on my github.

Problems

There are two disadvantages to this method of creating objects. The first is using a table like this, you can no longer totally hide variables as you could do in the closure version. The other is the complexity added - especially if you wish to allow base method calling, however in balance, you only need to write the super() functionality once (or use mine!)

When writing addons, I use both methods of encapsulation where they fit best - as like everything else in development the answer to what to use is "it depends".

design, code, lua, warcraft

---