Inheritance Tree Serialization With Json

I'm currently implementing persistence in my side project and I wanted to use Json as it's supported by our likely caching technologies (MongoDB or Redis) and it doesn't have the verbosity of Xml. I've previously serialized to binary and didn't want the headaches of dealing with upgrades to objects. It's just easier dealing with text based serialization as the serialized objects can be modified without having to go down the whole SerializationSurrogate side of things. That will hopefully make product upgrades go smoother.

Having chosen Newtonsoft's Json library, I did a proof of concept program to see whether it would be up for the job. I don't want to serialize everthing in my objects, on deserialization I can get a fair amount of the components from the cache rather than having to serialize them. All I need to serialize is the unique identifier that allows me to find them again. That said, I didn't want to have to create classes to deal with each object's deserialization, I wanted each object to be able to know how to unpack itself.

Therefore, it was time to get to grips with how the OnDeserializing and OnDeserialized attributes work. I created the following classes:

[Serializable]
public class BaseClass
{
    public string StringProp { get; set; }

    [OnDeserialized]
    private void OnDeserializedMethod(StreamingContext context)
    {
        Console.WriteLine("BaseClass.OnDeserialized ({0}) - {1}", GetHashCode(), StringProp);
    }

    [OnDeserializing]
    private void OnDeserializingMethod(StreamingContext context)
    {
        Console.WriteLine("BaseClass.OnDeserializing ({0}) - {1}", GetHashCode(), StringProp);
    }
}

[Serializable]
public class Subclass : BaseClass
{
    public double DoubleProp { get; set; }

    [OnDeserialized]
    private void OnDeserialized(StreamingContext context)
    {
        Console.WriteLine("Subclass.OnDeserialized ({0}) - {1}", GetHashCode(), DoubleProp);
    }

    [OnDeserializing]
    private void OnDeserializing(StreamingContext context)
    {
        Console.WriteLine("Subclass.OnDeserializing ({0}) - {1}", GetHashCode(), DoubleProp);
    }
}

[Serializable]
public class TestContainer
{
    public IEnumerable<BaseClass> WrappedObjects { get; set; }

    [OnDeserialized]
    private void OnDeserializedMethod(StreamingContext context)
    {
        Console.WriteLine("TestContainer.OnDeserialized ({0})", GetHashCode());
    }

    [OnDeserializing]
    private void OnDeserializing(StreamingContext context)
    {
        Console.WriteLine("TestContainer.OnDeserializing ({0})", GetHashCode());
    }
}

Serialization was handled like so:

var first = new BaseClass { StringProp = "First", IntProp = 1 };
var second = new BaseClass { StringProp = "Second", IntProp = 2 };
var third = new Subclass { StringProp = "Third", IntProp = 3, DoubleProp = 3.0 };

var container = new TestContainer { WrappedObjects = new BaseClass[] { first, second, third } };

var serialized = JsonConvert.SerializeObject(container);
var deserialized = JsonConvert.DeserializeObject<TestContainer>(serialized);

Which resulted in the following output:

TestContainer.OnDeserializing (38104455)
BaseClass.OnDeserializing (578700) -
BaseClass.OnDeserialized (578700) - First
BaseClass.OnDeserializing (21411931) -
BaseClass.OnDeserialized (21411931) - Second
BaseClass.OnDeserializing (54043951) -
BaseClass.OnDeserialized (54043951) - Third
TestContainer.OnDeserialized (38104455)

We can see that the container and contained elements are deserializing as expected, however the third element isn't going into the subclass' OnDeserialized method, it only executes the base class' method. This isn't what we wanted. This is due to the fact that the Json doesn't include type information in the serialized object by default. We need to add a JsonSerializationSettings instance and specify that type info should be included:

var first = new BaseClass { StringProp = "First", IntProp = 1 };
var second = new BaseClass { StringProp = "Second", IntProp = 2 };
var third = new Subclass { StringProp = "Third", IntProp = 3, DoubleProp = 3.0 };

var container = new TestContainer { WrappedObjects = new BaseClass[] { first, second, third } };

var settings = new JsonSerializerSettings
{
    TypeNameHandling = TypeNameHandling.Auto
};

var serialized = JsonConvert.SerializeObject(container, settings);
var deserialized = JsonConvert.DeserializeObject<TestContainer>(serialized, settings);

This gives the following program output:

TestContainer.OnDeserializing (13476220)
BaseClass.OnDeserializing (15654122) -
BaseClass.OnDeserialized (15654122) - First
BaseClass.OnDeserializing (37839230) -
BaseClass.OnDeserialized (37839230) - Second
BaseClass.OnDeserializing (7904578) -
Subclass.OnDeserializing (7904578) - 0
BaseClass.OnDeserialized (7904578) - Third
Subclass.OnDeserialized (7904578) - 3
TestContainer.OnDeserialized (13476220)

That's more like it! There are a couple things we need to understand here:

1. The OnDeserialized and OnDeserializing methods are private in the BaseClass and Subclass definitions. If you try to change the methods to "protected virtual" in the base class and override them in the subclass you will get a runtime error. There's no need to do that though as they are firing in the expected order of Container class, Base class, Subclass. Just like how an instances of objects in an inheritance tree are constructed, you can rely on the fact that the base class methods will be called before subclass methods.
2. The member variables are deserialized by the framework sometime between the OnDeserializing and OnDeserialized method calls. This is handy if you want to pass state from the container class to the subclasses. That's not relevant to this example but I do need that functionality for my side project.

WiX

As mentioned in my first post, I've been working on a little project for most of this year. It's gotten to the point where people might actually want to install it now and for that we need an installer. I had originally chosen ClickOnce but I then found out that I needed to support both 32bit and 64bit versions of the product. We have embedded another technology which relies on a wrapped C++ dll so depending on the install, I either want the 32bit version or the 64bit version of that wrapped dll. Because a ClickOnce installer is created via the project properties, I didn't think that it would allow me to create both versions of the installer that I needed. I also wanted the application to be installed in the Program Files folder rather than ClickOnce's sandbox.

Knowing that support for Microsoft's installer project was discontinued after Visual Studio 2010, I looked into WiX. Getting it up and running is pretty easy, download the latest toolkit and install it.

According to StackOverflow, if you are using Visual Studio 2015, then you also need to copy the files from an earlier IDE to the latest:

C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\WiX to
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\WiX

After copying the files, start a command prompt as an Administrator and run the following:

"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv" /setup

Once installed, you can create a new Setup project.

I created a new setup project and was given Product.wxs to start with. Because my current project is an Excel Add-In, I searched for "WiX Excel Add In" which led me to the Add-In Express blog post. I already had an Add-in, so I skipped the first bit and went straight to the Product and Package elemet section.

Product element

I changed updated the Product tag so that the Name and Manufacturer were accurate:

<Product Id="CE2CEA93-9DD3-4724-8FE3-FCBF0A0915C1"
           Name="FastClose Excel Add-in"
           Language="1033"
           Version="1.0.0.0"
           Manufacturer="FastClose Ltd."
           UpgradeCode="7b3b630d-c617-419f-8272-95942cf21420">

These values are going to appear in the Add/Remove programs dialog.

ComponentGroup element

The ComponentGroup section defines the files to install and you need to specify each file individually. That is going to be a pain, each time we add a new assembly to our solution we are going to have to remember to update our installer to include the new dll in the install. There must be a better way, and we'll come to that a bit later. For now, I added the "AddInFiles" variable to the project's properties page:

then we add the list of files to be installed.

<Fragment>
    <ComponentGroup Id="ProductComponents" Directory="INSTALLFOLDER">     
      <Component Id="FastClose.ExcelAddIn_vsto_Component">
        <File Id="FastCloseExcelAddIn_vsto" KeyPath="yes"
              Name="FastClose.ExcelAddIn.vsto" Source="$(var.AddinFiles)\FastClose.ExcelAddIn.vsto"></File>
      </Component>
      <Component Id="FastClose.ExcelAddIn_dll_manifest_Component">
        <File Id="FastCloseExcelAddIn_dll_manifest" KeyPath="yes"
              Name="FastClose.ExcelAddIn.dll.manifest" Source="$(var.AddinFiles)\FastClose.ExcelAddIn.dll.manifest"></File>
      </Component>
      <Component Id="MSOfficeToolsCommon_dll_Component">
        <File Id="MSOfficeToolsCommon_dll" KeyPath="yes"
              Name="Microsoft.Office.Tools.Common.v4.0.Utilities.dll" Source="$(var.AddinFiles)\Microsoft.Office.Tools.Common.v4.0.Utilities.dll"></File>
      </Component>
      <!--  This dll isn't in the output folder
      <Component Id="MSOfficeToolsExcel_dll_Component">
        <File Id="MSOfficeToolsExcel_dll" KeyPath="yes"
              Name="Microsoft.Office.Tools.Excel.dll" Source="$(var.AddinFiles)\Microsoft.Office.Tools.Excel.dll"></File>
      </Component>-->
      <Component Id="FastClose.ExcelAddIn_dll_Component" >
        <File Id="FastCloseExcelAddIn_dll" KeyPath="yes"
              Name="FastClose.ExcelAddIn.dll" Source="$(var.AddinFiles)\FastClose.ExcelAddIn.dll" />
      </Component>
    </ComponentGroup>
  </Fragment>

Registry Entries

We need to let Excel know about our Add-in and that's done via the registry. The only difference between the example and my version is that I wanted to install to a subfolder in case we ever have multiple products. Adding directories is easy, it's just another level in the xml file and set the Name attribute to the name of the folder. In this example, we are installing to "[DriveLetter]:\Program Files (x86)\FastClose\ExcelAddIn.

<Fragment>
    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFilesFolder">
        <Directory Id="ManufacturerFolder" Name="FastClose">
          <Directory Id="INSTALLFOLDER" Name="ExcelAddIn" />
          <Component Id="Registry_FriendlyName">
            <RegistryValue Id="RegKey_FriendlyName" Root="HKCU"
                           Key="Software\Microsoft\Office\Excel\AddIns\FastClose"
                           Name="FriendlyName"
                           Value="FastClose Excel Add-In"
                           Type="string" KeyPath="yes" />
          </Component>
          <Component Id="Registry_Description">
            <RegistryValue Id="RegKey_Description" Root="HKCU"
                           Key="Software\Microsoft\Office\Excel\AddIns\FastClose"
                           Name="Description"
                           Value="FastClose reporting solution for Microsoft Excel."
                           Type="string" KeyPath="yes" />
          </Component>
          <Component Id="Registry_Manifest">
            <RegistryValue Id="RegKey_Manifest" Root="HKCU"
                           Key="Software\Microsoft\Office\Excel\AddIns\FastClose"
                           Name="Manifest" Value="[INSTALLFOLDER]FastClose.ExcelAddIn.vsto|vstolocal"
                           Type="string" KeyPath="yes" />
          </Component>
          <Component Id="Registry_LoadBehavior">
            <RegistryValue Id="RegKey_LoadBehavior" Root="HKCU"
                           Key="Software\Microsoft\Office\Excel\AddIns\FastClose"
                           Name="LoadBehavior" Value="3"
                           Type="integer" KeyPath="yes" />
          </Component>
        </Directory>
      </Directory>
    </Directory>
  </Fragment>

The folder which we want to install our files into is marked with the Id of "INSTALLFOLDER". I've seen other examples use "INSTALLLOCATION" so the name of the Id doens't matter as long as it's consistent. You can see in the 3rd registry key that we use this folder to specify where the vsto file is located.

Feature element

Now that we have defined the files we want to install, the location to install them to and the registry entries to use, we need to tell WiX to use those Xml fragments:

<Feature Id="ProductFeature" Title="FastClose Excel AddIn" Level="1">
      <ComponentGroupRef Id="ProductComponents" />
      <ComponentRef Id="Registry_FriendlyName" />
      <ComponentRef Id="Registry_Description" />
      <ComponentRef Id="Registry_Manifest" />
      <ComponentRef Id="Registry_LoadBehavior" />
</Feature>

The advantage to this block is that we can remove or comment out a line in this block to not use an either fragment. This makes it easy to remove compoents from the install without losing track of what the compoent was. The CompoentGroupRef has the same Id as the ComponentGroup that we defined our files in. Same for all the registry entries.

Other Elements

Add the Media element to embed the install files within the installer:

<Media Id="1" Cabinet="ExcelAddin1.cab" EmbedCab="yes"/>

Add the UIRef element to specify that we want a minimal UI:

<UIRef Id="WixUI_Minimal" />

Add a variable to specify the EULA. The EULA.rtf should be included in your install project.

<WixVariable Id="WixUILicenseRtf" Value="EULA.rtf" />

Testing

At this point we have done enough to test. Just build the solution and check the bin folder for a msi file. After installing, I found that I only had four files in my install folder. As mentioned above, we need to specify each file to be installed and so far I have only specified four of them in the ProductCompoents ComponentGroup. My project has a couple hundred dlls as it uses the excellent Humanizer NuGet package which contains resource assemblies for various languages. While we only support English at the moment, at some point we might need the other languages. We also have about 20 projects in our solution, each of which compile to a separate dll. So, what is the best way to do this? It can't be specifying them all by hand. The top search result on Google points to this StackOverflow question where the top answer is to write an application to read the files in your bin folder and generate the WiX xml.

Are you kidding me?

A bit more digging led me to the HeatDirectory.

HeatDirectory

Heat is a separate tool that basically does what the person on StackOverflow said to do, which is to create an xml file with all the files listed. It can be integrated into the installer's project file so that it runs on each build.

Here's how:

First define the pre-processor variable to tell Heat where to get the files from:

Next up, modify the project file to include the HeatDirectory in the BeforeBuild event. Rather than using Notepad++ or another editor, I prefer to edit the file directly in Visual Studio so that I don't get prompted to reload the project after the file is saved. To do so, right click the installer project, click "Unload Project". You can then right click on the installer project and edit it.

At the bottom of the file, there will be some BeforeBuild and AfterBuild events. They may be commented out. Add the HeatDirectory element.

  <Target Name="BeforeBuild">
    <HeatDirectory 
      NoLogo="$(HarvestDirectoryNoLogo)" 
      SuppressAllWarnings="$(HarvestDirectorySuppressAllWarnings)" 
      SuppressSpecificWarnings="$(HarvestDirectorySuppressSpecificWarnings)" 
      ToolPath="$(WixToolPath)" 
      TreatWarningsAsErrors="$(HarvestDirectoryTreatWarningsAsErrors)" 
      TreatSpecificWarningsAsErrors="$(HarvestDirectoryTreatSpecificWarningsAsErrors)" 
      VerboseOutput="$(HarvestDirectoryVerboseOutput)" 
      AutogenerateGuids="$(HarvestDirectoryAutogenerateGuids)" 
      GenerateGuidsNow="$(HarvestDirectoryGenerateGuidsNow)" 
      OutputFile="Components.wxs" 
      SuppressFragments="$(HarvestDirectorySuppressFragments)" 
      SuppressUniqueIds="$(HarvestDirectorySuppressUniqueIds)" 
      Transforms="%(HarvestDirectory.Transforms)" 
      Directory="$(SolutionDir)\bin\$(Configuration)" 
      ComponentGroupName="C_CommonAssemblies" 
      DirectoryRefId="INSTALLFOLDER" 
      KeepEmptyDirectories="false" 
      PreprocessorVariable="var.SourceDir" 
      SuppressCom="%(HarvestDirectory.SuppressCom)" 
      SuppressRootDirectory="true" 
      SuppressRegistry="%(HarvestDirectory.SuppressRegistry)" 
      RunAsSeparateProcess="true">
    </HeatDirectory>
  </Target>

The interesting attributes are:

• OutputFile - The name of the file that Heat will generate for you. It will be added to the root of your installer project.
• ComponentGroupName - The name of the component group that we will use in our main Product.wxs CompoentRef.
• DirectoryRefId - In Geoff Webber-Cross's example, he uses "INSTALLLOCATION" as this variable name. As mentioned earlier, it doesn't matter what the varaiable is called as long as it's consistent.
• PreprocessorVariable - This needs to match the variable you created in the project file.
• RunAsSeparateProcess - This wasn't in Geoff's example, but I found that the files would be locked during a build and I wouldn't be able to do another build unless I shut Visual Studio down. StackOverflow had the answer.

At this point, you can build and a file called "Components.wxs" will be created in your installer's project folder. You may need to click "Show All Files" in Solution Explorer to see it. Right click on the file and choose "Include in project". If you build again, you will be prompted to reload the file as it was changed by Heat outside of Visual Studio.

Using the new file

Back in the main installer xml, we need to reference this new file. To do so, just change the ProductFeature element to use the new ComponentGroup in the new file:

The old ComponentGroup is no longer required and can be removed.

Testing - Part 2

After resinstalling on my test machine all the files were installed and the add-in was appearing in Excel. But it didn't run. We store some settings in the App.Config file and the logging indicated that the values weren't being read. I checked the installed App.config file and it did have the correct elements. Back to StackOverflow to find the answer.

Next Steps

Now that I have a working installer, I need to investigate how to create 32bit and 64 bit versions. It's not enough to test the operating system as you can have a 32-bit version of Excel running on 64-bit Windows.

Back to Blogging

A number of years back I had a blog but it kind of fell by the wayside.  I set it up originally in order for people to be able to keep track of my journey around Australia and New Zealand.  Once I cut back on the travel, I cut back on the blog updates.

This time around, this blog is mostly for myself.  I was partly inspired by Troy Hunt's post on ghost coders leaving a trail about what they've done but my main motivation is for me wanting a resource for me to look up things I look up often.

Another motivation is the changable nature of the web itself.  When I've been trying to solve a programming problem and found an answer on StackOverflow, I have occasionally found that the linked references were no longer available.

The third motivation is that I've been working on a side project for most of this year and it's starting to look like it will be a viable product.  Without giving too many of the inner secrets away, I wanted to have some documentation about some of the technologies that I have used so that if/when we have other developers working on it then they will have a reference.

Lastly, to be a better writer you have to practice.

So, with all that said, here we go.  If anyone else gets any benefit from this then that's even better.