Category: Software Engineering

Dynamic AES Key Exchange Through RSA Encryption

I wanted to prototype encrypted communication channel between a client and a server. Now of course there are HTTPS and other TLS channels that work quite well, but what I have in mind is supposed to be used to transfer rather sensitive data. So how can I establish a secure channel through an HTTP/HTTPS channel?

  1. Have the server generate an RSA key pair and send the public key to the client.
  2. Have the client generate an AES key, encrypt it with the received public key, and send the encrypted key to the server.
  3. Let the server decrypt the AES key.
  4. Both the client and the server are now in possession of the same AES key and can therefore communicate securely.

Of course, the generated AES key should only be used for the communication with the one client which sent it, so some sort of secure key management on the server (also regarding the RSA key pair) is vital. Also, the AES key could periodically be updated (i.e. a new key generated). At the very least, every message sent back and forth encrypted with AES will have to use a separate IV — but naturally that IV could be part of the transmitted message. So let’s get a very basic REST API-based implementation going.

Generate RSA key-pair on the Server

[...]

public sealed class SessionKey
{
    public Guid Id;
    public byte[] SymmetricKey;
    public RSAParameters PublicKey;
    public RSAParameters PrivateKey;
}

[...]

private Dictionary<Guid, SessionKey> sessionKeys;

[...]

public RSAParameters Generate(Guid sessionId)
{
    // NOTE: Make the key size configurable.
    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider(2048))
    {
        SessionKey s = new SessionKey()
        {
            Id = sessionId,
            PublicKey = rsa.ExportParameters(false /* no private key info */),
            PrivateKey = rsa.ExportParameters(true /* with private key info */),
            SymmetricKey = null, // To be generated by the client.
        };

        sessionKeys.Add(id, s);

        return s.PublicKey;
    }
}

[...]

This key generation can then be used to generate a new RSA key pair whenever a new client connects and requests secure communication. Of course, make sure you send the public key back to the client, and not the private key — else there’s no point in encrypting in the first place.

Generate an AES key on the Client

[...]

// Get the Public Key from the Server
RSAParameters publicKey = GetFromServer(...);

// Holds the current session's key.
byte[] MySessionKey;

// Send encrypted session key to Server.
SendToServer(GenerateAndEncryptSessionKey(publicKey));

[...]

private byte[] GenerateAndEncryptSessionKey(RSAParameters publicKey)
{
    using (Aes aes = Aes.Create())
    {
        aes.KeySize = aes.LegalKeySizes[0].MaxSize;
        // Setting the KeySize generates a new key, but if you're paranoid, you can call aes.GenerateKey() again.

        MySessionKey = aes.Key;
    }

    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
    {
        rsa.ImportParameters(publicKey);

        return rsa.Encrypt(MySessionKey, true /* use OAEP padding */);
    }
}

[...]

As you can see, we just take the public key we got from the server to set up the RSA provider and then encrypt the generated AES key using that public key. Once the client sends the encrypted key to the server, they both share the same secret and can securely communicate with each other.

Decrypt AES Key on the Server

[...]

public void SetSymmetricKey(Guid id, byte[] encryptedKey)
{
    SessionKey session = sessionKeys[id];

    using (RSACryptoServiceProvider rsa = new RSACryptoServiceProvider())
    {
        rsa.ImportParameters(session.PrivateKey);

        session.SymmetricKey = rsa.Decrypt(encryptedKey, true /* use use OAEP padding */);
    }
}

[...]

Since we already have the private key for this session, we can just use it to decrypt the AES key we got from the client. Again, making sure that the stored symmetric key is safe, is key to security.

Encrypt / Decrypt

Encrypting and decrypting can now be done the same way on both sides (since we’re using a symmetric-key algorithm). So here’s what that looks like.

[...]

public byte[] EncryptData(byte[] key, string data)
{
    using (Aes aes = Aes.Create())
    {
        byte[] result;

        aes.Key = key;
        aes.GenerateIV();

        using (ICryptoTransform encryptor = aes.CreateEncryptor())
        {
            using (MemoryStream ms = new MemoryStream())
            {
                using (CryptoStream cs = new CryptoStream(ms, encryptor, CryptoStreamMode.Write))
                {
                    using (StreamWriter writer = new StreamWriter(cs))
                    {
                        writer.Write(data);
                    }
                }

                byte[] encrypted = ms.ToArray();
                result = new byte[aes.BlockSize / 8 + encrypted.Length];

                // Result is built as: IV (plain text) + Encrypted(data)
                Array.Copy(aes.IV, result, aes.BlockSize / 8);
                Array.Copy(encrypted, 0, result, aes.BlockSize / 8, encrypted.Length);

                return result;
            }
        }
    }
}

public string Decrypt(byte[] key, byte[] data)
{
    using (Aes aes = Aes.Create())
    {
        aes.Key = key;

        // Extract the IV from the data first.
        byte[] iv = new byte[aes.BlockSize / 8];
        Array.Copy(data, iv, iv.Length);
        aes.IV = iv;

        // The remainder of the data is the encrypted data we care about.
        byte[] encryptedData = new byte[data.Length - iv.Length];
        Array.Copy(data, iv.Length, encryptedData, 0, encryptedData.Length);

        using (ICryptoTransform decryptor = aes.CreateDecryptor())
        {
            using (MemoryStream ms = new MemoryStream(encryptedData))
            {
                using (CryptoStream cs = new CryptoStream(ms, decryptor, CryptoStreamMode.Read))
                {
                    using (StreamReader reader = new StreamReader(cs))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
        }
    }
}

[...]

As you can see, each time we encrypt something we generate a new IV, which we send at the beginning of the data to the other side. The other side then extracts the IV first and uses it to initialize AES.

REST APIs?

Using all this through REST APIs is trivial: All you really need to make sure is that the client sends the session GUID (or whatever you use to identify a session) with every encrypted message, either through the URL, parameters or headers. Of course it is vital to guarantee that a client cannot get access to another client’s session (e.g. to provide a new session key), but through ordinary (secure) authentication that should easily be doable.

Next Steps

As far as encryption is concerned, this should already do the trick. You may want to add signatures to the encrypted messages too, to make sure that the encrypted blocks have not been tampered with. In addition, the AES key exchange could be repeated periodically (maybe even after every exchanged message).

Using Cloud-Init to Auto-Deploy New EC2 Instances

So you have a machine role that you’re auto-scaling in AWS and you want new EC2 instances to initialize automatically after their creation? Cloud-init is here to help; it let’s you manage and configure many aspects of the machine (like installing the latest updates or missing packages). It also let’s you run any commands you want. So what else would you need to get your own bits installed? That’s right, nothing!

#cloud-config
# Update existing packages
apt_update: true

# Install more packages
packages:
- tomcat7
- jq

# Download the latest bits from S3, extract and setup.
runcmd:
- aws s3api get-object --bucket my-bits --key latest/mybits.tar.gz mybits.tar.gz
- mkdir mybits
- tar xvzf mybits.tar.gz -C mybits
- bash mybits/my-own-setup-script.sh

In this case, the script my-own-setup-script.sh is run at the end of initialization and can make sure to deploy everything as required. All you need to do, is push a package with the latest bits onto S3 and all new instances for this role will automatically get the latest bits installed when they are created. Of course you’ll need to make sure that the machine’s IAM role is allowed access to the corresponding S3 bucket and file.

Side-by-Side Tracing

Whenever you run into problems with SxS, e.g. because an assembly cannot be loaded because of missing dependencies or similar, using the tracing tool which SxS offers can help you a lot. Here’s how to use it.

Start Tracing

You start tracing by calling the following command:

sxstrace Trace –logfile:sxs.etl

This will keep the tracer running until you actually press enter. If you want to stop it through issuing another command, you should call

sxstrace Trace –logfile:sxs.etl -nostop

This will exit as soon as the tracing session was created.

Stop Tracing

In case you were using the nostop option, you’ll have to stop tracing through the following command.

sxstrace StopTrace

Transforming the Traces

As with most traces, you’ll need to transform the traces from the raw format in the ETL file to a human readable format. You’ll do this with the following command.

sxstrace Parse –logfile:sxs.etl –outfile:sxs.log

In the out file you should then be able to easily figure out if your problem is related to SxS or not.

A Basic Framework for Using Performance Counters with .Net Applications

In my last post I promised to tell you more about the performance counters. So here we are: today I’m going to tell you about how to use the performance counter infrastructure offered by Windows in a .net application. On top of this I’m providing you with a basic framework that you can use to author performance counters through XML and then get the code to read/write the counters generated for you at build time. So let’s get started.

Some Basics

Before I go off to the code, here’s a little overview on performance counters in Windows. I’m sure if you’re reading this, you already know about Perfmon.exe, a nice little tool to look at various performance aspects of Windows machines. When plottig performance counters, you’re typically adding the counters you’re interested in. These counters are grouped in categories, for instance Processor or PhsyicalDisk. Each category can contain multiple counters, for instance Disk Read Bytes/sec or Disk Write Bytes/sec. And finally, each counter can have multiple instances, for instance in multi-processor machines, you’ll find one instance of the % Idle Time counter per processor.

What tools like Perfmon.exe do is to grab the values of the counters you selected every second (by default, but that can typically be changed) and record/plot the values. Your job here is to build performance counters that measure certain aspects of your application so these tools can help you in analyzing your performance. And that’s why you’re here, right?

Step 1: The XML Schema

In order to know what we talk about in the XML I announced, let me start with the XML schema. I use this mainly to make sure that what I have in the XML declaring the perf counters makes some sense and is legal input. And of course because Visual Studio tells you that something’s wrong when the schema is not adhered to.

<?xml version="1.0" encoding="utf-8"?>
<xs:schema targetNamespace="urn:Cymbeline.Diagnostics.PerfCounters"
           xmlns="urn:Cymbeline.Diagnostics.PerfCounters"
           xmlns:tns="urn:Cymbeline.Diagnostics.PerfCounters"
           xmlns:xs="http://www.w3.org/2001/XMLSchema"
           attributeFormDefault="unqualified"
           elementFormDefault="qualified">
  <xs:element name="PerfCounters">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Category" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="Counter" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:attribute name="Name" type="xs:string" use="required" />
                  <xs:attribute name="Symbol" type="tns:Symbol" use="required" />
                  <xs:attribute name="Type" use="required">
                    <xs:simpleType>
                      <xs:restriction base="xs:string">
                        <xs:enumeration value="NumberOfItems32" />
                        <xs:enumeration value="RateOfCountsPerSecond32" />
                        <xs:enumeration value="RawFraction" />
                      </xs:restriction>
                    </xs:simpleType>
                  </xs:attribute>
                  <xs:attribute name="Help" type="xs:string" use="required" />
                </xs:complexType>
              </xs:element>
            </xs:sequence>
            <xs:attribute name="Name" type="xs:string" use="required" />
            <xs:attribute name="Symbol" type="tns:Symbol" use="required" />
            <xs:attribute name="Help" type="xs:string" use="required" />
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:simpleType name="Symbol">
    <xs:restriction base="xs:token">
      <xs:pattern value="[a-zA-Z_][\w_]*" />
    </xs:restriction>
  </xs:simpleType>
</xs:schema>

On lines 21 to 23 you find some enumeration values. These map to the values defined in the PerformanceCounterType Enumeration. This is also where you can add support for more performance counter types when you need it. The other elements and attributes are used to describe the category for the performance counters and the performance counters themselves, including the help text that will show up in Perfmon.exe and also including the symbol that you can use in the code to access the counter.

Step 2: Declaring the Performance Counters

Now let’s use the schema we built above. To give you an idea of the context, I’m providing here some of the XML I use to build the perf counters for my SMTP server.

<?xml version="1.0" encoding="utf-8" ?>
<PerfCounters xmlns="urn:Cymbeline.Diagnostics.PerfCounters">
  <Category Name="CymbeMail" Symbol="CymbeMail" Help="CymbeMail SMTP Server v1">
    <Counter Name="# Total Connections"
         Symbol="TotalConnections"
         Type="NumberOfItems32"
         Help="The total number of connections since the server was started." />
    <Counter Name="# Total Connections Refused"
         Symbol="TotalRefusedConnections"
         Type="NumberOfItems32"
         Help="The total number of connections refused since the server was started." />
    <Counter Name="# Total Connections with Errors"
         Symbol="TotalErroneousConnections"
         Type="NumberOfItems32"
         Help="The total number of connections which reported errors since the server was started." />
    <Counter Name="# Active Connections"
         Symbol="ActiveConnections"
         Type="NumberOfItems32"
         Help="The number of currently active connections." />
    <Counter Name="# Authorization Records"
         Symbol="AuthorizationRecords"
         Type="NumberOfItems32"
         Help="The number of authorization records currently kept in the server." />
    <Counter Name="# Connections/sec"
         Symbol="ConnectionsPerSec"
         Type="RateOfCountsPerSecond32"
         Help="The number of connections per second." />
    <Counter Name="# Refused Connections/sec"
         Symbol="RefusedConnectionsPerSec"
         Type="RateOfCountsPerSecond32"
         Help="The number of connections refused per second." />
  </Category>
</PerfCounters>

As you can see, I create seven perf counters, most of them ordinary counters that count the number of occurrences of a certain event (like when a client makes a connection to the server). I actually also use the NumberOfItems32 type to count the number of active connections — I basically increment the counter when a connection was established and decrement it when the connection is terminated. And I have a couple of counters which count the number of occurrences per second (the RateOfCountsPerSecond32 counter type). The good thing about these counters is that you don’t need to provide your own counter base (check out MSDN for more info on this, starting with the above mentioned PerformanceCounterType Enumeration).

Step 3: Building the XSLT to Generate Code

We already have our perf counters declared, so let’s generate some code from that XML. Using XSL stylesheets makes this very easy: We’ll use the XML as input and get c# code as output that we can simply include into our project. I won’t show the full XSLT here (it’s about 240 lines).

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
        xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
        xmlns:msxsl="urn:schemas-microsoft-com:xslt"
        xmlns:tns="urn:Cymbeline.Diagnostics.PerfCounters"
        exclude-result-prefixes="msxsl">
  <xsl:output method="text" indent="no"/>

  <xsl:param name="TargetNamespace" />
  <xsl:param name="TargetClassName" select="'PerfCounters'" />
  <xsl:param name="AccessModifier" select="'public'" />

  <xsl:template match="tns:PerfCounters">
    <xsl:if test="$TargetNamespace=''">
      <xsl:message terminate="yes">
        The Parameter 'TargetNamespace' is undefined.
      </xsl:message>
    </xsl:if>
    <!-- ... -->
  </xsl:template>

  <xsl:template match="tns:Counter" mode="Counter">
    <xsl:text>
      private static PerformanceCounter _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>;

      </xsl:text><xsl:value-of select="$AccessModifier"/><xsl:text> static PerformanceCounter </xsl:text><xsl:value-of select="@Symbol"/><xsl:text>
      {
        get
        {
          MakeSureCountersAreInitialized();
          return _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>;
        }
      }
</xsl:text>
  </xsl:template>

  <xsl:template match="tns:Counter[@Type='RawFraction']" mode="Counter">
    <xsl:text>
      private static PerformanceCounter _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>;

      </xsl:text><xsl:value-of select="$AccessModifier"/><xsl:text> static PerformanceCounter </xsl:text><xsl:value-of select="@Symbol"/><xsl:text>
      {
        get
        {
          MakeSureCountersAreInitialized();
          return _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>;
        }
      }

      private static PerformanceCounter _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>Base;

      </xsl:text><xsl:value-of select="$AccessModifier"/><xsl:text> static PerformanceCounter </xsl:text><xsl:value-of select="@Symbol"/><xsl:text>Base
      {
        get
        {
          MakeSureCountersAreInitialized();
          return _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>Base;
        }
      }
</xsl:text>
  </xsl:template>

  <xsl:template match="tns:Counter" mode="InitCounter">
    <xsl:text>
        _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text> = new PerformanceCounter(
          CategoryName,
          "</xsl:text><xsl:value-of select="@Name"/><xsl:text>",
          false);
</xsl:text>
  </xsl:template>

  <xsl:template match="tns:Counter[@Type='RawFraction']" mode="InitCounter">
    <xsl:text>
        _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text> = new PerformanceCounter(
          CategoryName,
          "</xsl:text><xsl:value-of select="@Name"/><xsl:text>",
          false);

        _</xsl:text><xsl:value-of select="@Symbol"/><xsl:text>Base = new PerformanceCounter(
          CategoryName,
          "</xsl:text><xsl:value-of select="@Name"/><xsl:text> Base",
          false);
</xsl:text>
  </xsl:template>

  <xsl:template match="tns:Counter" mode="CreateCounter">
      <xsl:text>
      {
        CounterCreationData counter = new CounterCreationData();

        counter.CounterName = "</xsl:text><xsl:value-of select="@Name" /><xsl:text>";
        counter.CounterHelp = "</xsl:text><xsl:value-of select="@Help" /><xsl:text>";
        counter.CounterType = PerformanceCounterType.</xsl:text><xsl:value-of select="@Type" /><xsl:text>;

        counters.Add(counter);
      }
</xsl:text>
  </xsl:template>

  <xsl:template match="tns:Counter[@Type='RawFraction']" mode="CreateCounter">
      <xsl:text>
      {
        CounterCreationData counter = new CounterCreationData();

        counter.CounterName = "</xsl:text><xsl:value-of select="@Name" /><xsl:text>";
        counter.CounterHelp = "</xsl:text><xsl:value-of select="@Help" /><xsl:text>";
        counter.CounterType = PerformanceCounterType.</xsl:text><xsl:value-of select="@Type" /><xsl:text>;

        CounterCreationData counterBase = new CounterCreationData();

        counterBase.CounterName = "</xsl:text><xsl:value-of select="@Name" /><xsl:text> Base";
        counterBase.CounterHelp = "</xsl:text><xsl:value-of select="@Help" /><xsl:text>";
        counterBase.CounterType = PerformanceCounterType.RawBase;

        counters.AddRange(new CounterCreationData[]{counter, counterBase});
      }
</xsl:text>
  </xsl:template>

  <!-- ... -->
</xsl:stylesheet>

On lines 9 to 11 you’ll find the declaration for the parameters which you can use to change the namespace, the class name and the access modifiers (public vs internal really) for the generated classes.

Starting on lines 22 and 37 respectively, you find different templates for general counters and the RawFraction type counters. I use this to automatically generate the base counter so you won’t have to worry about it. You can do similar things for other counters which need a base counter. Then, starting on lines 63 and 72 respecctively, I initialize the counters, with specialization again for the RawFraction counter types. And starting on lines 86 and 100 respectively you’ll find the code to create and set the CounterCreationData which is used to register perf counters.

You can find the full code in the ZIP file linked to at the end of this post. With this we basically have all the tools we need to actually generate the code as part of the build. We just need to bring the pieces together.

Step 4: Bringing it Together aka Updating the Project

Well yes, I have silently assumed that you have a c# project (.csproj or any other project that is supported by MSBuild.exe) that you’re working on to update. All we need to do in the .csproj is to add the files we authored, then run the XSLT transformation and add the output .cs file to the list of source code files as well. And here’s how you can do that.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">
  <!-- ... -->
  <ItemGroup>
    <PerfCounterXml Include="Diag\PerfCounters.xml">
      <OutputCs>Diag\PerfCounters.Generated.cs</OutputCs>
      <Parameters>
        <Parameter Name="TargetNamespace" Value="$(RootNamespace).Diag" />
      </Parameters>
      <SubType>Designer</SubType>
    </PerfCounterXml>
    <PerfCounterXslt Include="Diag\PerfCounters.xslt" />
    <Compile Include="Diag\PerfCounters.Generated.cs">
      <PerfCounters>true</PerfCounters>
      <AutoGen>true</AutoGen>
      <DependentUpon>PerfCounters.xml</DependentUpon>
    </Compile>
  </ItemGroup>
  <ItemGroup>
    <None Include="Diag\PerfCounters.xsd">
      <SubType>Designer</SubType>
    </None>
  </ItemGroup>
  <!-- XSL Transform for Perf Counters -->
  <Target Name="GeneratePerfCounters" BeforeTargets="BeforeBuild"
      Inputs="@(PerfCounterXml);@(PerfCounterXslt)"
      Outputs="@(PerfCounterXml->'%(OutputCs)')">
    <XslTransformation XmlInputPaths="@(PerfCounterXml)"
      XslInputPath="@(PerfCounterXslt)"
      OutputPaths="@(PerfCounterXml->'%(OutputCs)')"
      Parameters="@(PerfCounterXml->'%(Parameters)')" />
  </Target>
</Project>

Add lines 4 to 32 to the end of the project file and make sure the paths you’re using point to the places where you actually stored the files. On line 8 you can see that I’m passing the target namespace parameter (called out in step 3) by concatenating the assemblies default root namespace and “.Diag”. This way you can actually put this stuff into a .targets file that you can include in all of the projects which take advantage of this framework.

Starting on line 25 I defined the actual target which runs the XslTransformation task that comes with .net 4.0. I also declared the inputs as being the XML file and the XSLT file — this way, while experimenting with the XSLT, the code gets regenerated also if the XML file hasn’t changed, but the XSLT has.

Using the Generated Classes

So now you have pretty much all the code you need to use these performance counters of yours. All that’s left is actually writing the code which

  1. Installs the Performance Counters
  2. Updates the Performance Counters
  3. Uninstalls the Performance Counters

#1 and #3 would probably be used by your installer and #2 by your application at runtime. Incrementing and decrementing the counters is as easy as

PerfCounters.CymbeMail.ActiveConnections.Increment();
//...
PerfCounters.CymbeMail.ActiveConnections.Decrement();

Where PerfCounters is the name of the generated class which contains all the generated perf counters, CymbeMail is the name of the class for the category (the Symbol attribute, remember?) and ActiveConnections is of course the symbol name for the counter we’re modifying. Isn’t this simple?

As for setting things up (or removing them), the relevant piece of code gets generated for you, too. So all you need to do is actually call it, potentially from a small application that you run — as I mentioned before — from your installer.

// Setup the Performace Counters
PerfCounters.Setup();

// And for the Uninstaller:
// Remove the Performance Counters
PerfCounters.Remove();

Of course you’ll have to make sure that the version of the generated code that’s run by the uninstaller is the same version as the installer did run — else you may end up with stale categories / counters on the machine. But then again, if you’re installing through an MSI package, you get that for free … when done properly.

Summary

In just a few steps I have shown how you can build a framework to use performance counters in .Net applications. On top of it’s simplicity, it’s also quite easy to modify the framework and adapt it to your own needs or extend it to allow more performance counter types.

And finally, as promised, here’s the ZIP file which contains the relevant pieces. Please forgive me for not adding the .csproj file — that actually contains other relevant data that I didn’t want to share. Instead I added the generated c# code file for reference.

PerfCounters.zip (4.08kb)