Then, I’ve looked at pricing of cloud storage providers, such as AWS, Azure and Google Cloud. Those offer storage around 1 cent ($0.01) per GB per month. That’s a quarter of the OneDrive cost! It’s even less if you consider their archive offerings (AWS Glacier, Archive in Azure, Coldline Storage for Google). The cheapest offering here is from Microsoft at 0.2 cents ($0.002) per GB per month, but with some usage caveats. Since the point of backups is to keep them for a long time, this quickly adds up though.

Now I’ve written a line or two of code before, so I figured I could as well write my own tool for this. So here is bart, the backup and restore tool. Note that at this point I do not offer bart as a ready-to-use executable, but only as MIT-licensed source code. In addition, bart currently works only with Azure Blob Storage – or with storage mounted into the machine’s file system. However, adding other cloud providers/archive destinations should be relatively easy, given the interfaces used in the tool.

Security

In terms of security, bart encrypts every file before storing it in the archive destination. A user-provided password is used together with a randomly generated salt to derive a key for encryption with AES. On first use of any archive destination, bart generates a random salt, and each archive has its own password and salt. To avoid anybody with access to the archive destination from even snooping the names of your files, the names are hashed (SHA1) and the hashes used to store the encrypted files. This has the disadvantage that renaming/moving a file results in another file in the destination archive, though.

Usage

Once you compiled bart, you can use it as follows.

./bart [-name string] [-path string] [-m noop|restore|delete] -acct string -key string
  -name string
        The name of the backup archive. (default "backup")
  -path string
        The path to the directory to backup and/or restore. (default ".")
  -m string
        A behavior for files missing locally: 'noop' to do nothing, 'restore' to restore them from the backup, 'delete' to delete them in the backup archive. (default "noop")
  -acct string
        The Azure Storage Account name.
  -key string
        The Azure Storage Account Key.

Sources

The sources are on GitHub @ https://github.com/rokeller/bart.

Conclusion

I’ve used bart for backup of some photos/videos for a while now. For the about 42GB I have uploaded so far my monthly bill from Microsoft is about 42 cents ($0.42). Those months where I upload new files the cost is a little higher (a few cents usually) because of the extra transactions. My backed up files are encrypted. If this isn’t cheap and secure cloud backups, what is?

The post Cheap and Secure Cloud Backups appeared first on Tales of a Code Monkey.

Updated Sources

To cut a long story short, the updated sources are now available on GitHub. I decided to track it in a separate branch for better isolation. This new branch is aptly called netstandard. I’ll try to stay up-to-date with the more recent releases of Lucene.Net, and also with .Net Standard 2.0. That is, provided that I find the time for it. You may notice that the project files have become quite a bit simpler. That’s certainly one change in .Net Standard and Core that I welcome. The other is the better integration of Nuget for package referencing and package creation/pushing.

Updated Unit Tests

As a side effect, I also figured that it was going to be easier to update NUnit to the latest version, since its toolset is also well integrated with the new dotnet toolset. Since I’m doing all changes through VSCode and with building/testing/packaging in Docker containers based on the microsoft/aspnetcore-build:2 images, I wanted to keep it simple. The good thing here is that the dotnet toolset seems to offer really everything I need for this, and is suprisingly easy to handle, especially when compared to the RC1 version.

Updated Nuget Package

As I’ve mentioned in the beginning, I primarily made this effort because I needed a newer version of Lucene.Net with compatibility for .Net Standard 2.0. As a result, I published a new RC build as a Nuget package too. It is built on the latest Lucene.Net 4.8 beta release and currently supports only .Net Standard 2.0. If there’s a great demand for it, I’ll see if I can add support for other targets – or accept pull requests accordingly.

Conclusions

Nothing much besides the obvious: .Net Standard seems to be in a good shape wrt libraries and toolset, as well as support on Linux. There are a few gotchas but overall nothing much of a problem. Lucene.Net is still somewhat badly documented itself, and the tracking of braking changes between major/minor versions (and in fact also revisions/beta releases of the same major/minor) could be greatly improved. An online documentation would be very useful – maybe it exists, and I just haven’t found it? In any case, skimming through the Lucene.Net sources on GitHub works too, though being much slower.

You can find more information about object mapping for Lucene.Net on the Lucene.Net.ObjectMapping page.

The post Lucene.Net.ObjectMapping for .Net Standard 2.0 appeared first on Tales of a Code Monkey.

All your JSON are belong to us!

Now as I wrote, I don’t particularly like the fact that my JSON data is sent over the wire for pretty printing. It may not be super secret or anything, but in these days, you cannot be careful enough. Besides, it’s completely unnecessary to do it. All you need is already in your browser! So I quickly built my own JSON pretty printer (and syntax highlighter). You can find it right here.

Offline JSON Pretty Printing to the Rescue

Actually, the design is very simple. All my JSON pretty printer is doing, is to take your JSON input and try to parse it as JSON in the browser.

JSON.parse(yourJsonInput)

If that fails, I’m showing the parsing error and it’s done. If it succeeds, I get back a JavaScript object/array/value, which then I’m inspecting. For objects, I’m using basic tree navigation to go through all the properties and nested objects/arrays/values for pretty printing. That’s it, really simple. No need to transmit the data anywhere — it stays right in your browser!

So like it, hate it, use it or don’t: cymbeline.ch JSON Pretty Printer

The post Offline JSON Pretty Printing appeared first on Tales of a Code Monkey.

using (Searcher searcher = new IndexSearcher(directory))
{
    IQueryable<BlogPost> posts =
        from post in searcher.AsQueryable<BlogPost>()
        where obj.Tag == "lucene"
        orderby obj.Timestamp descending
        select post;
}

Now granted, the above example is a very basic one. So here’s a short list of other methods on IQueryable<T> that are already supported at this point: Any ^*, Count ^*, First ^*, FirstOrDefault ^*, OrderBy, OrderByDescending, Single ^*, SingleOrDefault ^*, Skip, Take, ThenBy, ThenByDescending, and finally Where.

^* Method is supported both with and without a filter predicate.

With this, it becomes easy to build paging based on objects you get back as a result of a query on Lucene.Net. I’m still working on improving the supported filter expressions (most of all for Where, but all the other filterable methods naturally profit too). For instance, with the default JSON-based object mapping it is already possible to search for entries in a dictionary that maps a string to another property or object. Say you have a set of classes, defined as follows.

public class MyClass
{
    public int Id { get; set; }
    public Dictionary<string, MyOtherClass> Map { get; set; }
}

public class MyOtherClass
{
    public string Text { get; set; }
    public int Sequence { get; set; }
    public DateTime Timestamp { get; set; }
}

Now you can actually search for instances of MyClass that satisfy certain conditions in the Map dictionary, like this:

var query = from c in searcher.AsQueryable<MyClass>()
            where c.Map["MyKey"].Sequence == 123
            select c;

Since the items in the dictionary are mapped to analyzed fields in the Lucene.Net document, we can search on them!

Delete and Update By Query

Now since I have this query expression binder to create Lucene.Net queries based on LINQ filter expressions, I’ve added an extension method to update and one to delete documents that match a query. So it is now possible to do this:

indexWriter.Delete<MyClass>(x => x.Id == 1234);
indexWriter.Update(myObject, x => x.Id == myObject.Id);

Call to Action

Now with all this said, I’m looking for volunteers to help me get more coverage on the LINQ queries, because that’s definitely where the weak spot is right now. If you’re interested, leave a comment here or on GitHub.

The post LINQ with Lucene.Net.ObjectMapping appeared first on Tales of a Code Monkey.

CRUD Operations

The library now comes with support for all of the CRUD operations. Let’s look at them one by one, starting with Create.

Create / Add

In Lucene.Net terms, that would be AddDocument. Since the library does object to document mapping, this is simplified to an Add operation.

IndexWriter myIndexWriter = ...;
MyClass myObject = new MyClass(...);

myIndexWriter.Add(myObject);

Or, if you need a specific analyzer for the document the object gets mapped to, you can use the overload which accepts a second parameter of type Analyzer.

IndexWriter myIndexWriter = ...;
MyClass myObject = new MyClass(...);

myIndexWriter.Add(myObject, new MyOwnAnalyzer());

Retrieve / Query

The retrieve operation, or mapping of a document to an object hasn’t changed since v1.0.0. There are examples for how to query and retrieve in my previous post. Of course, if you happen to know the ID of the document without a query, then you can just map that document to your class without going through a query. But since the document IDs can change over time, it’s usually more practical to pivot off a query.

Update

Update is maybe the most interesting operation here. Since document IDs can change over time, there’s really no good way to reliably update a specific document, without making a query. That’s why the UpdateDocument method from the IndexReader asks you for a query/term to use to match the document to update. And that’s why it’s generally a good idea to bring your own unique identifier to the game. Suppose your class has a property of type Guid and name “Id”, which is used as your unique identifier for the objects of that type.

IndexWriter myIndexWriter = ...;
MyClass myObject = ...;

myObject.MyPropertyToUpdate = "new value";

myIndexWriter.Update(
    myObject,
    new TermQuery(new Term("Id", myObject.Id.ToString())));

Under the covers, this will find all the documents matching the query and matching the type (MyClass), delete them and then add a new document for the mapped myObject. If you need an analyzer, for the newly mapped document, you can use the second overload.

IndexWriter myIndexWriter = ...;
MyClass myObject = ...;

myObject.MyPropertyToUpdate = "new value";

myIndexWriter.Update(
    myObject,
    new TermQuery(new Term("Id", myObject.Id.ToString())),
    new MyOwnAnalyzer());

Delete

Just like the retrieve operation, the Delete operation is also supported since v1.0.0. I realize though that I haven’t given any examples yet. But really, it’s quite simple again. You give the type of objects you want to delete the mapped documents for, and you give a query to identify the objects to delete. No magic at all.

IndexWriter myIndexWriter = ...;
myIndexWriter.DeleteDocuments<MyClass>(
    new TermQuery(new Term("Tag", "deleted")));

Naturally, you can use any Query you want for the delete operation (as well as for updates). You can make them arbitrarily complex as long as they’re still supported by Lucene.Net.

Summary and Outlook

That’s it, CRUD with no magic, no tricks. Let me know if there’s functionality you’d like to see added, either by commenting here or by opening a bug/enhancement/whatever on GitHub. I’ve started working on LINQ support for the ObjectMapping library too, with the goal that you can write LINQ queries like the following.

var query = from myObject in mySearcher.AsQueryable<MyClass>()
            where myObject.Tag == "history"
            select myObject;

It will likely take a little longer to get that stable, but I’ll try to make a pre-release on NuGet in the next few weeks.

The post Improvements to Lucene.Net.ObjectMapping appeared first on Tales of a Code Monkey.

You already know Searcher

The Searcher class in Lucene.Net can be used to run queries on an index and retrieve documents matching that query. The Lucene.Net.ObjectMapping library comes with additional extensions to the Searcher class which help you search for Documents. There’s a variety of different extensions, some which just return a TopDocs object with the number of results you’ve specified, and some which allow sorting, but more powerful are the ones which require you to specify a Collector to gather the results. Using a Collector makes it very easy to support paging over all the results for a specific query, and after all that’s usually what you’d do today if you want to show search results. So let’s look at an example of searching for Documents that contain mapped .Net objects using a Collector. Let’s assume we’re building a blog engine, for which we want to index the posts.

public class BlogPost
{
    public Guid Id { get; set; }
    public DateTime Created { get; set; }
    public string Title { get; set; }
    public string Body { get; set; }
    public string[] Tags { get; set; }
}

// ... as before, you'd store your BlogPost objects like this:
luceneIndexWriter.AddDocument(thePost.ToDocument());

Use a Collector for Paging

Creating an paged index of all your blog posts is very easy, really. You’ll need a Searcher, a Collector (the TopFieldCollector will do for now) and that’s about it. Let’s look at some code.

private const int PageSize = 10;

public BlogPost[] GetPostsForPage(int page)
{
    // Sanitize the 'page' before doing anything with it.
    if (page < 0)
    {
        page = 0;
    }

    int start = page * PageSize;
    int end = start + PageSize;

    using (Searcher searcher = new IndexSearcher(myIndexReader))
    {
        TopFieldCollector collector = TopFieldCollector.Create(
            // Let's sort descending by create date.
            new Sort(new SortField("Created", SortField.LONG, true)),
            end, // Need to get the hits until 'end'.
            false,
            false,
            false,
            false);

        // Let's use the object mapping extensions for Search! This will
        // filter results to only those Documents which hold a BlogPost.
        searcher.Search<BlogPost>(new MatchAllDocsQuery(), collector);

        // At this point we know how many hits there are in total. So
        // let's check that the requested page is within range.
        if (start >= collector.TotalHits)
        {
            page = (collector.TotalHits - 1) / PageSize;
            start = page.Value * PageSize;
            end = start + PageSize;
        }

        TopDocs docs = collector.TopDocs(start, PageSize);
        List<BlogPost> posts = new List<BlogPost>();

        foreach (ScoreDoc scoreDoc in docs.ScoreDocs)
        {
            Document doc = searcher.Doc(scoreDoc.Doc);

            posts.Add(doc.ToObject<BlogPost>());
        }

        return posts.ToArray();
    }
}

That’s it, no magic, no tricks. One thing you could do, instead of just returning a plain array with the results is to return an object which holds some more meta information, like for instance the number of total hits, or the actual page you’re returning results for. But the core logic remains the same. You can play around with different ways to sort the results. Keep in mind though that tokenized/analyzed fields in Lucene.Net are sorted based on the tokens, not based on the actual string value. To help address this, I’m thinking about extending the object mappers to allow to specify not only to analyze a field (because you want to search it), but also to add a non-analyzed copy of the field for sorting purposes. That way, you have the advantage of being able to search and sort on the same logical field in the end. Keep in mind though that the index will grow since the data is indexed twice: once tokenized/analyzed, once as-is.

The post Search Mapped Objects in Lucene.Net appeared first on Tales of a Code Monkey.

How To Install

Since this is a NuGet package, installation is as simple as running the following command in the Package Manager Console

Install-Package Lucene.Net.ObjectMapping

Alternatively, you can just search for Lucene.Net.ObjectMapping in the package manager and you should find it.

How To Use It?

Using object mapping is as simple as calling two methods: ToDocument to convert an object into a document and ToObject to convert a Document (that was created with the ToDocument method) into the original object.

MyObject obj = ...;
Document doc = obj.ToDocument();
// Save the document to your Lucene.Net Index

// Later, load the document from the index again
Document docFromIndex = ...;
MyObject objFromDoc = docFromIndex.ToObject<MyObject>();

How does it work?

Under the covers, the library is JSON-serializing the object and stores the JSON in the actual Lucene.Net document. In addition, it stores some metadata like the actual and the static types of the object you stored, as well as the timestamp (ticks) of when the document was created. The type information is used when you search for documents that were created for a specific type. The static type is the type you pass in as the type parameter to ToDocument, whereas the actual type is the actual (dynamic) type of the object you’re passing in. Since all this information is stored in the document too, there are no issues re-creating objects from an class hierarchy too.
In addition to storing the object information itself, the library also indexes the individual properties of the object you’re storing, including nested properties. By default, it uses a mapper which works as follows.

• Public properties/fields of objects are mapped to Lucene.Net fields with the same name; e.g. a property called “Id” is mapped to a field called “Id”.
• Properties/fields that are arrays are mapped to multiple Lucene.Net fields, all with the same name (the name of the property that holds the array).
• Nested properties/fields, i.e. objects from properties/fields, use the name of the property as a prefix for the properties/fields of the object.

Each field is created with the following mapping of field types:

• Boolean properties are mapped to a numeric field (Int) with a value of 1 for true and 0 for false.
• DateTime properties are mapped to a numeric field (Long) with the value being the Ticks property of the DateTime.
• Float properties are mapped to a numeric field (Float) with the value being the float value.
• Double and Decimal properties are mapped to a numeric field (Double) with the value being the double value.
• Guid properties are mapped to string fields which are NOT_ANALYZED, i.e. you can search for the GUID as is.
• Integer (also Long, Short, and Byte as well as their unsigned/signed counterparts) properties are mapped to a numeric field (Long) with the value being the integer value.
• Null values are not mapped at all; thus, the absence of a field implies the corresponding property is null.
• String properties are mapped to string fields which are ANALYZED.
• TimeSpan properties are mapped to a numeric field (Long) with the value being the Ticks property of the TimeSpan.
• Uri properties are mapped to string fields which are ANALYZED.

Example Mapping

Let’s look at a simple example of an object and its mapping to a Lucene.Net Document. Consider the following object model.

public class MyObject
{
    public int Id { get; set; }
    public string Name { get; set; }
    public ObjectMeta Meta { get; set; }
}

public class ObjectMeta
{
    public DateTime LastModified { get; set; }
    public string ModifiedBy { get; set; }
    public string[] Modifications { get; set; }
}

// Create an instance of MyObject
MyObject obj = new MyObject()
{
    Id = 1234,
    Name = "My Lucene.Net mapped Object",
    Meta = new ObjectMeta()
    {
        LastModified = DateTime.UtcNow,
        ModifiedBy = "the dude",
        Modifications = new string[] { "changed a", "removed b", "added c" },
    },
};

Document doc = obj.ToDocument();

The mapping rules called out above will add the following fields for searching to the document. Please note that I’m not calling out the fields needed for the internal workings of the Lucene.Net.ObjectMapping library.

The mapper is by no means complete. Ideas to extend it in the future exist, including functionality to

• specify attributes on string properties (or properties mapped to string fields) to specify how to index the string (NO vs ANALYZED vs NOT_ANALYZED vs NOT_ANALYZED_NO_NORMS vs ANALYZED_NO_NORMS).
• specify attributes on any properties to define how to map the field, e.g. by specifying a class which can map the field

I’ll talk a little more on how to use this all when searching for documents in your Lucene.Net index. But as a sneak preview: the library also provides extension methods to the Searcher class from Lucene.Net that you can use to specify an object type to filter your documents on.

The post Lucene.Net Object Mapping appeared first on Tales of a Code Monkey.

Instrumentation Manifests to the Rescue!

For a while now this has actually been quite easy, using instrumentation manifests. You can read more about it here: http://msdn.microsoft.com/en-us/library/windows/desktop/dd996930(v=vs.85).aspx. These manifests allow you to define events, templates for events, messages for events, even your own event channels (so you wouldn’t need to log into that crowded “Application” channel anymore) and a lot more. But let’s look at a little example.

<?xml version="1.0" encoding="utf-8"?>
<instrumentationManifest xsi:schemaLocation="http://schemas.microsoft.com/win/2004/08/events eventman.xsd" xmlns="http://schemas.microsoft.com/win/2004/08/events" xmlns:win="http://manifests.microsoft.com/win/2004/08/windows/events" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:trace="http://schemas.microsoft.com/win/2004/08/events/trace">
    <instrumentation>
        <events>
            <provider name="MyService" guid="{DDB3FC6E-6CC4-4871-9F27-88C1B1F19BBA}" symbol="TheEventLog"
                      message="$(string.MyService.ProviderMessage)"
                      resourceFileName="MyService.Events.dll"
                      messageFileName="MyService.Events.dll"
                      parameterFileName="MyService.Events.dll">
                <events>
                    <event symbol="ServiceStarted" version="0" channel="Application"
                           value="1000" level="win:Informational"
                           message="$(string.MyService.event.1000.message)" />
                    <event symbol="ServiceStopped" version="0" channel="Application"
                           value="1001" level="win:Informational"
                           message="$(string.MyService.event.1001.message)"/>
                    <event symbol="ServiceConfigurationError" version="0" channel="Application"
                           value="1002" level="win:Error" template="ServiceException"
                           message="$(string.MyService.event.1002.message)"/>
                    <event symbol="ServiceUnhandledException" version="0" channel="Application"
                           value="1003" level="win:Error" template="ServiceException"
                           message="$(string.MyService.event.1003.message)"/>
                </events>
                <levels/>
                <channels>
                    <importChannel name="Application" chid="Application"/>
                </channels>
                <templates>
                    <template tid="ServiceException">
                        <data name="Exception" inType="win:UnicodeString" outType="xs:string"/>
                    </template>
                </templates>
            </provider>
        </events>
    </instrumentation>
    <localization>
        <resources culture="en-US">
            <stringTable>
                <string id="level.Informational" value="Information"/>
                <string id="level.Error" value="Error"/>
                <string id="channel.Application" value="Application"/>

                <string id="MyService.ProviderMessage"
                        value="My Windows Service"/>

                <string id="MyService.event.1000.message"
                        value="My Windows Service has started."/>
                <string id="MyService.event.1001.message"
                        value="My Windows Service has stopped."/>
                <string id="MyService.event.1002.message"
                        value="My Windows Service encountered a problem with its configuration. Please fix these issues and start the service again.:%n%n%1"/>
                <string id="MyService.event.1003.message"
                        value="My Windows Service encountered an unhandled exception:%n%n%1"/>
            </stringTable>
        </resources>
    </localization>
</instrumentationManifest>

Let’s start at the top. Lines 5-9 define some basic information about this instrumentation provider, like a name, a unique ID and a symbol (which will come in handy later). We can also define a friendly name for events logged this way (i.e. the event source). Let’s ignore the three xyzFileName attributes for now. On lines 11-22 we’re defining four events, some of them informational (like “the service started” or “the service stopped”), some are errors (e.g. configuration errors, or unhandled exceptions). If we wanted to define our own channel, we’d do so between lines 25 and 27. For now we’re just re-using (i.e. importing) the pre-defined “Application” channel.

Event Templates

Event templates are particularly handy if you want to write parameters with your events. Lines 29-31 define a template which has exactly one parameter, which happens to be a unicode string. We’ll use it to store exceptions. We can define more than one parameter and there’s a lot of types to use, but I’ll let you explore those on your own. This template, as you can see, is referred to by the two events with IDs 1002 and 1003.

Resources

The localization gods are with us to. Our event and template definitions so far were abstract, no actual UI strings were contained. We can define those per language, as you can see starting line 37. In the resources element and its sub-elements, we define the actual strings we want to show, including any parameters. Parameters are numbered (1-based) and are referred to with %1, %2, %3 and so on. As you can see on lines 51 and 53, we’re defining the strings for the two error events with one parameter each (“%1”), to contain the exception message. If you want line breaks, you’ll achieve those with “%n”.

Compile, with some Sugar added

So now we have a fancy manifest, but what can we do with it? Well, eventually we want to log events using the definitions from this manifest, so let’s get to it. The Windows SDK comes with two very handy tools, MC.exe (the message compiler) and RC.exe (the resource compiler). We’ll use the first to compile the manifest — and generate some c# code as a side effect — then use the second to compile the output of the first into a resource which can be linked into an executable. The commands are as follows.

mc.exe -css MyService.Events manifest.man -r obj\Debug
rc.exe obj\Debug\manifest.rc

MC.exe was nice enough to generate a file called manifest.cs for us. That file contains some code that you can use to log every event you defined in the manifest. This is why it was so handy to define the events (and templates): depending on how many parameters an event’s template has, the generated methods will ask you to provide just as many (typed) values for those parameters. Isn’t that great?! You’ll also find the compiled manifest.res file in obj\Debug. You can link that into its own executable (or your main assembly too, if you wanted), as follows:

csc.exe /out:MyService.Events.dll /target:library /win32res:obj\Debug\manifest.res

And you have a satellite assembly which holds the manifest you’ve built! CSC will log a warning about missing source files (because you didn’t add any .cs files to be compiled) but so far that doesn’t hurt anyone. We could probably also use link.exe but so far the C# compiler does a nice enough job.

Use that generated Code

Remember the code that was generated for us by MC.exe? Let’s go ahead and use it.

// ...
TheEventLog.EventWriteServiceStarted();
// ...
TheEventLog.EventWriteServiceConfigurationError(exception.Message); // ... or log the entire exception, including stack traces.
// ...

Wasn’t that very easy?

Install the Event Provider

There’s still something missing though: we’ll need to install our instrumentation/event provider with the system. It’s similar to creating the event source (which in fact will happen automatically when installing the manifest). This will typically happen in your application’s/service’s installer, using a command line as follows. But before that, remember the xyzFileName attributes we talked about? These need to be updated to point to the full path of the MyService.Events.dll assembly we generated. Otherwise the following command is going to fail.

wevtutil.exe im path\to\my\manifest.man

From now on, when your app or service starts and logs those events, they’ll show up in the event viewer. For the two events we defined with parameters, the values of the parameters are essentially the only thing that’s stored along with the ID of the event. Likewise, they’ll be the only thing that’s going to be exported with the event — so the files with the exported events you’re going to ask your customers to send you are going to be a lot smaller and won’t contain the static part of the events you already know anyway!

To uninstall the manifest, just run this command:

wevtutil.exe um path\to\my\manifest.man

Both commands need to run elevated (particularly important to remember when writing your installer).

Next Steps

As a next step, you’ll probably want to add the manual steps of compiling the manifest linking into the satellite assembly to the project file as automated targets. I’ll likely write another post about that in the future too.

Summary

As you can see, writing a manifest, compiling it and using the generated code to write to the event log is quite easy. So no more excuses to write each event as one big string (which is can be a lot harder to analyze when they come back from your customers because you first need to parse the strings).

The post Writing to Event Log — the right way appeared first on Tales of a Code Monkey.

Enter HttpMessageHandler

The HttpMessageHandler abstract base class and its derived classes are used by the HttpClient class to asynchronously send HTTP requests and receive the response from the server. But since we don’t actually want to send the message ourselves – just massage the body and headers a little bit before sending – we’ll derive a new class GzipCompressingHandler from DelegatingHandler so we can delegate sending (and receiving) to another handler and just focus on the transformation of the content. So here’s what that looks like.

public sealed class GzipCompressingHandler : DelegatingHandler
{
    public GzipCompressingHandler(HttpMessageHandler innerHandler)
    {
        if (null == innerHandler)
        {
            throw new ArgumentNullException("innerHandler");
        }

        InnerHandler = innerHandler;
    }

    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        HttpContent content = request.Content;

        if (request.Method == HttpMethod.Post)
        {
            // Wrap the original HttpContent in our custom GzipContent class.
            // If you want to compress only certain content, make the decision here!
            request.Content = new GzipContent(request.Content);
        }

        return base.SendAsync(request, cancellationToken);
    }
}

As you can see, all we’re doing is just wrapping the original HttpContent in our GzipContent class. So let’s get right to that.

Gzip-compressed HttpContent: GzipContent

We’re almost there, all we need to do is actually compressing the content and modify the request headers to indicate the new content encoding.

internal sealed class GzipContent : HttpContent
{
    private readonly HttpContent content;

    public GzipContent(HttpContent content)
    {
        this.content = content;

        // Keep the original content's headers ...
        foreach (KeyValuePair<string, IEnumerable<string>> header in content.Headers)
        {
            Headers.TryAddWithoutValidation(header.Key, header.Value);
        }

        // ... and let the server know we've Gzip-compressed the body of this request.
        Headers.ContentEncoding.Add("gzip");
    }

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        // Open a GZipStream that writes to the specified output stream.
        using (GZipStream gzip = new GZipStream(stream, CompressionMode.Compress, true))
        {
            // Copy all the input content to the GZip stream.
            await content.CopyToAsync(gzip);
        }
    }

    protected override bool TryComputeLength(out long length)
    {
        length = -1;
        return false;
    }
}

Easy, right? Of course you could add other supported compression algorithms, using more or less the same code (or even adding some abstraction for different compression algorithms), but this is basically all that’s required.

Summary

Using the HttpMessageHandler and its associated classes makes it extremely easy to apply transformations to all (or a well-defined subset) of HTTP requests you’re sending. In this case, we’re applying Gzip-compression to the bodies of all outgoing POST requests, but the logic to decide when to compress can be as customized as you want; you could even apply Gzip-compression only if the requested URI ends with “.gzip” or for certain content types.

The post Gzip Encoding an HTTP POST Request Body appeared first on Tales of a Code Monkey.

1. Have the server generate an RSA key pair and send the public key to the client.
2. Have the client generate an AES key, encrypt it with the received public key, and send the encrypted key to the server.
3. Let the server decrypt the AES key.
4. Both the client and the server are now in possession of the same AES key and can therefore communicate securely.

Of course, the generated AES key should only be used for the communication with the one client which sent it, so some sort of secure key management on the server (also regarding the RSA key pair) is vital. Also, the AES key could periodically be updated (i.e. a new key generated). At the very least, every message sent back and forth encrypted with AES will have to use a separate IV — but naturally that IV could be part of the transmitted message. So let’s get a very basic REST API-based implementation going.

Generate RSA key-pair on the Server

[...]

public sealed class SessionKey
{
    public Guid Id;
    public byte[] SymmetricKey;
    public RSAParameters PublicKey;
    public RSAParameters PrivateKey;
}

[...]

private Dictionary<Guid, SessionKey> sessionKeys;

[...]

public RSAParameters Generate(Guid sessionId)
{
    // NOTE: Make the key size configurable.
    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(2048))
    {
        SessionKey s = new SessionKey()
        {
            Id = sessionId,
            PublicKey = rsa.ExportParameters(false /* no private key info */),
            PrivateKey = rsa.ExportParameters(true /* with private key info */),
            SymmetricKey = null, // To be generated by the client.
        };

        sessionKeys.Add(id, s);

        return s.PublicKey;
    }
}

[...]

This key generation can then be used to generate a new RSA key pair whenever a new client connects and requests secure communication. Of course, make sure you send the public key back to the client, and not the private key — else there’s no point in encrypting in the first place.

Generate an AES key on the Client

[...]

// Get the Public Key from the Server
RSAParameters publicKey = GetFromServer(...);

// Holds the current session's key.
byte[] MySessionKey;

// Send encrypted session key to Server.
SendToServer(GenerateAndEncryptSessionKey(publicKey));

[...]

private byte[] GenerateAndEncryptSessionKey(RSAParameters publicKey)
{
    using (Aes aes = Aes.Create())
    {
        aes.KeySize = aes.LegalKeySizes[0].MaxSize;
        // Setting the KeySize generates a new key, but if you're paranoid, you can call aes.GenerateKey() again.

        MySessionKey = aes.Key;
    }

    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
    {
        rsa.ImportParameters(publicKey);

        return rsa.Encrypt(MySessionKey, true /* use OAEP padding */);
    }
}

[...]

As you can see, we just take the public key we got from the server to set up the RSA provider and then encrypt the generated AES key using that public key. Once the client sends the encrypted key to the server, they both share the same secret and can securely communicate with each other.

Decrypt AES Key on the Server

[...]

public void SetSymmetricKey(Guid id, byte[] encryptedKey)
{
    SessionKey session = sessionKeys[id];

    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
    {
        rsa.ImportParameters(session.PrivateKey);

        session.SymmetricKey = rsa.Decrypt(encryptedKey, true /* use use OAEP padding */);
    }
}

[...]

Since we already have the private key for this session, we can just use it to decrypt the AES key we got from the client. Again, making sure that the stored symmetric key is safe, is key to security.

Encrypt / Decrypt

Encrypting and decrypting can now be done the same way on both sides (since we’re using a symmetric-key algorithm). So here’s what that looks like.

[...]

public byte[] EncryptData(byte[] key, string data)
{
    using (Aes aes = Aes.Create())
    {
        byte[] result;

        aes.Key = key;
        aes.GenerateIV();

        using (ICryptoTransform encryptor = aes.CreateEncryptor())
        {
            using (MemoryStream ms = new MemoryStream())
            {
                using (CryptoStream cs = new CryptoStream(ms, encryptor, CryptoStreamMode.Write))
                {
                    using (StreamWriter writer = new StreamWriter(cs))
                    {
                        writer.Write(data);
                    }
                }

                byte[] encrypted = ms.ToArray();
                result = new byte[aes.BlockSize / 8 + encrypted.Length];

                // Result is built as: IV (plain text) + Encrypted(data)
                Array.Copy(aes.IV, result, aes.BlockSize / 8);
                Array.Copy(encrypted, 0, result, aes.BlockSize / 8, encrypted.Length);

                return result;
            }
        }
    }
}

public string Decrypt(byte[] key, byte[] data)
{
    using (Aes aes = Aes.Create())
    {
        aes.Key = key;

        // Extract the IV from the data first.
        byte[] iv = new byte[aes.BlockSize / 8];
        Array.Copy(data, iv, iv.Length);
        aes.IV = iv;

        // The remainder of the data is the encrypted data we care about.
        byte[] encryptedData = new byte[data.Length - iv.Length];
        Array.Copy(data, iv.Length, encryptedData, 0, encryptedData.Length);

        using (ICryptoTransform decryptor = aes.CreateDecryptor())
        {
            using (MemoryStream ms = new MemoryStream(encryptedData))
            {
                using (CryptoStream cs = new CryptoStream(ms, decryptor, CryptoStreamMode.Read))
                {
                    using (StreamReader reader = new StreamReader(cs))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
        }
    }
}

[...]

As you can see, each time we encrypt something we generate a new IV, which we send at the beginning of the data to the other side. The other side then extracts the IV first and uses it to initialize AES.

REST APIs?

Using all this through REST APIs is trivial: All you really need to make sure is that the client sends the session GUID (or whatever you use to identify a session) with every encrypted message, either through the URL, parameters or headers. Of course it is vital to guarantee that a client cannot get access to another client’s session (e.g. to provide a new session key), but through ordinary (secure) authentication that should easily be doable.

Next Steps

As far as encryption is concerned, this should already do the trick. You may want to add signatures to the encrypted messages too, to make sure that the encrypted blocks have not been tampered with. In addition, the AES key exchange could be repeated periodically (maybe even after every exchanged message).

The post Dynamic AES Key Exchange Through RSA Encryption appeared first on Tales of a Code Monkey.