Help! Github is ignoring my .gitignore file!

It’s been a while since my last post, but now I’m fully emersed in learning .NET Core I’m sure I’ll start to post more regularly again. For my first trick, an aides-memoires on getting the .gitignore file to work. Here goes…

I normally use GitHub (or other git-based systems) as my source code repository when writing code in Visual Studio. I really like the fact that I can use the .gitignore file to prevent all the chaff that Visual Studio creates from being sent to the repository. For example, the ‘bin’ and ‘obj’ directories, and all the temporary solution files Visual Studio creates that have no business being in a source code reposity. A good rule of thumb is not to allow any generated files to be commited. They can be re-generated at any point so committing them just bloats commits and slows things down.

However, I often find that if I make changes to the .gitignore file in an established project these changes are not honoured in subsequent commits. Executing the the following commands in Git bash solves this:

git rm -r --cached .
git add .
git commit -m "fixed untracked files"  

Now, back to my .NET Core learning…

Basic compression in .NET

Following on from my previous post on encryption, a similar technique often used in conjunction with encryption is compression.

As an aside, if you are going to use encryption and compression together it’s important that you compress and then encrypt rather than the other way round. This is due to the way compression engines work. They reduce data size by looking for repeating patterns in data and then storing the pattern once along with the locations in the data in which it appears. It is often the case that unencrypted data is far more repetitive than the seemingly random output you get when using encryption. This is especially true for human readable text. Consider how many times words are repeated in a piece of prose versus how many repeating patterns there are likely to be in encrypted version of the prose.

As with encryption, we’ll start with a basic interface so that compression can be injected in other objects using dependency injection:

namespace Compression
{
    public interface ICompressor
    {
        string Compress(string text);
        string Decompress(string compressedText);
    }
}

The implementation for this is as follows:

using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Text;

namespace Compression
{
    public class Compressor : ICompressor
    {
        public string Compress(string value)
        {
            if (string.IsNullOrEmpty(value))
            {
                return value;
            }

            var inputArray = StringToByteArray(value);

            using (var outputStream = new MemoryStream())
            {
                using (var compressionStream = new GZipStream(outputStream, CompressionMode.Compress))
                {
                    // Compress:
                    compressionStream.Write(inputArray, 0, inputArray.Length);
                    // Close, but DO NOT FLUSH as this could result in data loss:
                    compressionStream.Close();

                    // Get a byte array from the output stream:
                    var outputArray = outputStream.ToArray();
                    outputStream.Close();

                    return ByteArrayToString(outputArray);
                }
            }
        }

        public string Decompress(string value)
        {
            if (string.IsNullOrEmpty(value))
            {
                return value;
            }

            var inputArray = StringToByteArray(value);

            using (var inputStream = new MemoryStream(inputArray))
            {
                using (var compressionStream = new GZipStream(inputStream, CompressionMode.Decompress))
                {
                    var outputList = new List<byte>();
                    int nextByte;
                    while ((nextByte = compressionStream.ReadByte()) != -1)
                    {
                        outputList.Add((byte)nextByte);
                    }

                    inputStream.Close();
                    compressionStream.Close();

                    return ByteArrayToString(outputList.ToArray());
                }
            }
        }

        private static byte[] StringToByteArray(string value)
        {
            var array = new byte[value.Length];
            for (var i = 0; i < array.Length; i++)
            {
                array[i] = (byte)value[i];
            }

            return array;
        }

        private static string ByteArrayToString(byte[] array)
        {
            var stringBuilder = new StringBuilder(array.Length);
            foreach (var b in array)
            {
                stringBuilder.Append((char)b);
            }

            return stringBuilder.ToString();
        }
    }
}

This can be demonstrated via the following simple test harness:

using System;
using Compression;

namespace TestHarness
{
    public static class Program
    {
        public static void Main()
        {
            const string text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.";
            var compressor = new Compressor();
            var compressedText = compressor.Compress(text);
            var decompressedText = compressor.Decompress(compressedText);

            ShowText("Text", text);
            ShowText("CompressedText", compressedText);
            ShowText("DecompressedText", decompressedText);
            
            Console.ReadLine();
        }

        private static void ShowText(string label, string text)
        {
            Console.WriteLine(label + ":");
            Console.WriteLine("Length: " + text.Length);
            Console.WriteLine(text);
            Console.WriteLine();
        }
    }
}

I’m not feeling particularly creative this morning so I opted for “Lorem ipsum” rather than anything witty as my input text.

The output of the program is as follows:

compression

Note that the compressed text is 282 characters in length while the original text is 445 characters.

Extending the Compressor class to be able to compress arrays and other data would be pretty simple too, given that the input strings are converted to arrays before the compression is performed.

Basic two-way encryption in .NET

There are lots cases when we need to encrypt and then decrypt data using .NET. A great example is if we want to store something secure in a cookie. We write the encrypted data to the web-response and decrypt it again again later from the next web-request.

I’ve been using the following algorithm for two-way encryption for a number of years now. I found it on the web a long time ago, so unfortunately don’t have a link to the original source any more. If anyone can help me here that would be appreciated.

First, let’s start with an interface as the chances are we’re going to need to inject encryption capabilities into other objects using dependency injection:

namespace Encryption
{
    public interface IEncryptor
    {
        string Encrypt(string text);
        string Decrypt(string encryptedText);
    }
}

The implementation for this is as follows:

using System;
using System.Security;
using System.Security.Cryptography;
using System.Text;
using System.IO;
using System.Globalization;
using System.Runtime.InteropServices;

namespace Encryption
{
    public class Encryptor : IEncryptor
    {
        private readonly SecureString _password;

        public Encryptor(SecureString password)
        {
            _password = password;
        }

        public string Encrypt(string text)
        {
            return Encrypt(text, GetDefaultSalt());
        }

        public string Encrypt(string text, string salt)
        {
            if (text == null)
            {
                return null;
            }

            RijndaelManaged rijndaelCipher;
            byte[] textData;
            ICryptoTransform encryptor;

            using (rijndaelCipher = new RijndaelManaged())
            {
                var secretKey = GetSecretKey(salt);

                // First we need to turn the input strings into a byte array.
                textData = Encoding.Unicode.GetBytes(text);

                // Create a encryptor from the existing secretKey bytes.
                // We use 32 bytes for the secret key. The default Rijndael 
                // key length is 256 bit (32 bytes) and then 16 bytes for the 
                // Initialization Vector (IV). The default Rijndael IV length is 
                // 128 bit (16 bytes).
                encryptor = rijndaelCipher.CreateEncryptor(secretKey.GetBytes(32), secretKey.GetBytes(16));
            }

            MemoryStream memoryStream;
            byte[] encryptedData;

            // Create a MemoryStream that is going to hold the encrypted bytes:
            using (memoryStream = new MemoryStream())
            {
                // Create a CryptoStream through which we are going to be processing 
                // our data. CryptoStreamMode.Write means that we are going to be 
                // writing data to the stream and the output will be written in the 
                // MemoryStream we have provided.
                using (var cryptoStream = new CryptoStream(memoryStream, encryptor, CryptoStreamMode.Write))
                {
                    // Start the encryption process.
                    cryptoStream.Write(textData, 0, textData.Length);

                    // Finish encrypting.
                    cryptoStream.FlushFinalBlock();

                    // Convert our encrypted data from a memoryStream into a byte array.
                    encryptedData = memoryStream.ToArray();

                    // Close both streams.
                    memoryStream.Close();
                    cryptoStream.Close();
                }
            }

            // Convert encrypted data into a base64-encoded string.
            // A common mistake would be to use an Encoding class for that.
            // It does not work, because not all byte values can be
            // represented by characters. We are going to be using Base64 encoding.
            // That is designed exactly for what we are trying to do.
            var encryptedText = Convert.ToBase64String(encryptedData);

            // Return encrypted string.
            return encryptedText;
        }

        public string Decrypt(string encryptedText)
        {
            return Decrypt(encryptedText, GetDefaultSalt());
        }

        public string Decrypt(string encryptedText, string salt)
        {
            if (encryptedText == null)
            {
                return null;
            }

            RijndaelManaged rijndaelCipher;
            byte[] encryptedData;
            ICryptoTransform decryptor;

            using (rijndaelCipher = new RijndaelManaged())
            {
                var secretKey = GetSecretKey(salt);

                // First we need to turn the input strings into a byte array.
                encryptedData = Convert.FromBase64String(encryptedText);

                // Create a decryptor from the existing SecretKey bytes.
                decryptor = rijndaelCipher.CreateDecryptor(secretKey.GetBytes(32), secretKey.GetBytes(16));
            }

            MemoryStream memoryStream;
            byte[] unencryptedData;
            int decryptedDataLength;

            using (memoryStream = new MemoryStream(encryptedData))
            {
                // Create a CryptoStream. Always use Read mode for decryption.
                using (var cryptoStream = new CryptoStream(memoryStream, decryptor, CryptoStreamMode.Read))
                {
                    // Since at this point we don't know what the size of decrypted data
                    // will be, allocate the buffer long enough to hold EncryptedData;
                    // DecryptedData is never longer than EncryptedData.
                    unencryptedData = new byte[encryptedData.Length];

                    // Start decrypting.
                    try
                    {
                        decryptedDataLength = cryptoStream.Read(unencryptedData, 0, unencryptedData.Length);
                    }
                    catch
                    {
                        throw new CryptographicException("Unable to decrypt string");
                    }                    

                    cryptoStream.Close();
                    memoryStream.Close();
                }
            }

            // Convert decrypted data into a string.
            var decryptedText = Encoding.Unicode.GetString(unencryptedData, 0, decryptedDataLength);

            // Return decrypted string.  
            return decryptedText;
        }

        private PasswordDeriveBytes GetSecretKey(string salt)
        {
            // We are using salt to make it harder to guess our key
            // using a dictionary attack.
            var encodedSalt = Encoding.ASCII.GetBytes(salt);

            var valuePointer = IntPtr.Zero;
            try
            {
                // The Secret Key will be generated from the specified
                // password and salt.
                valuePointer = Marshal.SecureStringToGlobalAllocUnicode(_password);
                return new PasswordDeriveBytes(Marshal.PtrToStringUni(valuePointer), encodedSalt);
            }
            finally
            {
                Marshal.ZeroFreeGlobalAllocUnicode(valuePointer);
            }            
        }

        private string GetDefaultSalt()
        {
            return _password.Length.ToString(CultureInfo.InvariantCulture);
        }
    }
}

This can then be used as follows:

using System;
using System.Security;

namespace Encryption
{
    public static class Program
    {
        public static void Main()
        {
            using (var password = GetPassword())
            {
                var encryptor = new Encryptor(password);

                const string text = "Hello World!";

                var encryptedText = encryptor.Encrypt(text);
                var decryptedText = encryptor.Decrypt(encryptedText);

                Console.WriteLine(encryptedText);
                Console.WriteLine(decryptedText);

                Console.ReadLine();
            }
        }

        private static SecureString GetPassword()
        {
            var password = new SecureString();

            password.AppendChar((char)80); // P
            password.AppendChar((char)97); // a
            password.AppendChar((char)115); // s
            password.AppendChar((char)115); // s
            password.AppendChar((char)119); // w
            password.AppendChar((char)111); // o
            password.AppendChar((char)114); // r
            password.AppendChar((char)100); // d
            password.AppendChar((char)49); // 1

            return password;
        }
    }
}

Note that we store the password using the SecureString class. This makes it harder to read the string from the memory while the program is running in the event that the host machine is hacked. Also, as I’m super-paranoid, I write to the SecureString using ASCII values converted to chars to prevent any readable values from appearing in string tables in the compiled code.

When using the Encryptor class you also have the option to add “salt”. If no salt is supplied a default based on the length of the password is used. Salt is effectively an additional piece of information you must supply in order to decrypt information, which becomes part of the decryption password. It is recommended that you use salt wherever possible as it makes it much harder for a hacker to decrypt sets of data using brute-force attacks.

Happy encrypting!

Serialising constructor-less .NET types (e.g. MailAddress) using Newtonsoft.Json.JsonConvert in the Json.NET library

Newtonsoft’s Json.NET library is brilliant for serialising .NET objects into JSON and back again. However, there are some things it can’t do. For instance, if we try serialising an object based on the following class:

using System.Net.Mail;

namespace JsonConvertors
{
    public class Person
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public MailAddress EmailAddress { get; set; }
    }
}

…using the following code:

var person = new Person
{
    FirstName = "Joe",
    LastName = "Bloggs",
    EmailAddress = new MailAddress("joe@bloggs.com"),
};

var json = JsonConvert.SerializeObject(person);
var personCopy = JsonConvert.DeserializeObject<Person>(json);

We get the following nasty error:

Unable to find a constructor to use for type System.Net.Mail.MailAddress. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'EmailAddress.DisplayName', line 1, position 69.

Why Microsoft didn’t add a constructor to the MailAddress type is beyond me. Of course, we could just store the EmailAddress as a string and bypass the issue that way, but I am a firm believer that everything should be strongly typed. Fortunately, we can use the JsonConverter class to tell JsonConvert how to handle constructor-less types like MailAddress.

Here’s how to write a converter for the MailAddress type:

using System;
using System.Net.Mail;
using Newtonsoft.Json;

namespace JsonConvertors
{
    public class MailAddressConverter : JsonConverter
    {
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var mailAddress = value as MailAddress;
            writer.WriteValue(mailAddress == null? string.Empty : mailAddress.ToString());
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            var text = reader.Value as string;
            MailAddress mailAddress;

            return IsValidMailAddress(text, out mailAddress) ? mailAddress : null;
        }

        public override bool CanConvert(Type objectType)
        {
            return objectType == typeof(MailAddress);
        }

        private static bool IsValidMailAddress(string text, out MailAddress value)
        {
            try
            {
                value = new MailAddress(text);
                return true;
            }
            catch
            {
                value = null;
                return false;
            }
        }
    }
}

Note that care has been taken to handle null and non-convertible values.

Then simply add the following attribute to your Person class:

using System.Net.Mail;
using Newtonsoft.Json;

namespace JsonConvertors
{
    public class Person
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }

        [JsonConverter(typeof(MailAddressConverter))]
        public MailAddress EmailAddress { get; set; }
    }
}

…and the conversion will work like a charm.

Error: Failed to compile both on .NET 4.5 and on .NET 4.0. 4.5 Exception: System.IO.IOException: The file exists.

Today I had a weird issue with a Microsoft IIS web-server. The latest .NET Framework was correctly installed and registered on the server, but when I put my .NET code in the web directory and browsed to it from a remote browser I got the standard .NET error page. I would get the error regardless of whether I was trying to access a page served by .NET or a basic HTML page hosted on the server. I confirmed that IIS was working correctly by removing all files (including the web.config file) from the web-directory and putting in a basic test HTML page on its own. The test page rendered correctly from a remote browser.

Having put the .NET code back in, I browsed to a .NET URL locally on the web-server in order to get a full view of the error message. The rather strange error was as follows:

File Exists Error

In order to help people find this page in search engines, the full text was as follows:

Server Error in '/' Application.

Failed to compile both on .NET 4.5 and on .NET 4.0. 4.5 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder). 4.0 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder) 
  Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

 Exception Details: System.InvalidOperationException: Failed to compile both on .NET 4.5 and on .NET 4.0. 4.5 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder). 4.0 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder)

Source Error: 


 An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  

Stack Trace: 



[InvalidOperationException: Failed to compile both on .NET 4.5 and on .NET 4.0. 4.5 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder). 4.0 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder)]
   HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder) +712

[InvalidOperationException: System.InvalidOperationException: Failed to compile both on .NET 4.5 and on .NET 4.0. 4.5 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder). 4.0 Exception: System.IO.IOException: The file exists.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileInternally(String fileName, List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String outputFolder, String compilerVersion, Int32 tryCount)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder)
   at HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder)
ImageRuntimeVersion: v4.0.30319]
   HibernatingRhinos.Profiler.Appender.Util.GenerateAssembly.CompileAssembly(List`1 sources, HashSet`1 assemblies, HashSet`1 defineSymbols, String assemblyFileName, String outputFolder) +1042
   HibernatingRhinos.Profiler.Appender.NHibernate.NHibernateProfiler.RegisterAppenderUsingNHibernateLogger() +314
   HibernatingRhinos.Profiler.Appender.NHibernate.NHibernateProfiler.Initialize(NHibernateAppenderConfiguration configuration) +53
   XXXXX.App_Start.NHibernateProfilerBootstrapper.PreStart() in c:\TeamCity\buildAgent\work\XXXXX\App_Start\NHibernateProfilerBootstrapper.cs:18

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
   System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
   System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +260
   System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +146
   System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) +34
   WebActivator.BaseActivationMethodAttribute.InvokeMethod() +288
   WebActivator.ActivationManager.RunActivationMethods() +698
   WebActivator.ActivationManager.RunPreStartMethods() +46
   WebActivator.ActivationManager.Run() +68

[InvalidOperationException: The pre-application start initialization method Run on type WebActivator.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..]
   System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection`1 methods, Func`1 setHostingEnvironmentCultures) +874
   System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection`1 methods) +169
   System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +166
   System.Web.Compilation.BuildManager.ExecutePreAppStart() +176
   System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +882

[HttpException (0x80004005): The pre-application start initialization method Run on type WebActivator.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..]
   System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +579
   System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +112
   System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +716

  


Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1055.0  

I found this article on stack overflow which helped me to solve the problem. The answer was simply to clear down the %TEMP% folder, which for me was C:\windows\temp. Apparently, if there are more than 65,535 files in there .NET is not able to serve web-pages through IIS. There were hundreds of thousands of files in the folder so I couldn’t even browse to it using Windows Explorer. It ended up being a command prompt job:

File Exists Error Command

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>cd c:\Windows\Temp

c:\Windows\Temp>del c:\Windows\Temp\*.*
c:\Windows\Temp\*.*, Are you sure (Y/N)? y

That fixed the problem.

Trigger TeamCity to build and test your project on commits to multiple branches

A while ago I learnt a great trick to get TeamCity to perform the same action for commits to multiple branches, rather than just a specific branch. To do this, simply enter +:* into the Branch specification setting, as follows:

TeamCity Multiple Branches

This option can be found in the Edit VCS Root screen.

Once this is set, you’ll see the branch name next to each build run in the results screen:

TeamCity Multiple Branches Results

Simples.

What’s my DNS?

If you’re working on globally distributed web architectures, users in different locations can sometimes have very different experiences of your site, particularly when pages are being served from different data centres. Some users may get awesome performance while others have to wait ages while pages download agonisingly slowly.

Often it’s not even clear which data centres user are being directed to, particularly if you work in a big organisation where there is a dedicated team responsible for DNS/routing. If you have access to servers in different countries, you may be able to remote in and do DNS lookups from those boxes in order to work out where users are likely to be directed, but if not, you’re going to need another way to work out how the DNS is set up.

A great way to get a view of DNS settings right across the world is to use What’s My DNS. Simply enter a URL and you will find out which IP addresses it resolves to at different locations right across the globe.

It’s interesting to see how Google and Facebook do DNS.

Facebook What's My DNS

Using Microsoft Log Parser to run queries across multiple IIS log files

It’s often useful to be able to perform queries across multiple log files. A great example is when dealing with IIS log files. I recently had a situation in which I needed to know the average response times and number of requests received for a couple of IIS web-servers in order to troubleshoot an issue we were experiencing. I needed to know the results over a large time period, so this meant collating data across a large number of IIS log files. Fortunately I had a call open with Microsoft and the support engineer I was dealing with told me about the Microsoft Log Parser, which was perfect for this job.

The tool allows you to group together a set of log files in a single location and run queries across them using a SQL-like language. It can be used with any type of file, but comes into its own when used with IIS log files.

Here are some examples of queries I performed using the tool:

Gets the number of hits per hour, grouped by hour, and writes them to a CSV file:

LogParser.exe "SELECT TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date, time),3600)) AS Hours, COUNT(*) AS Hits INTO ReqPerhour.csv FROM u_ex*.log GROUP BY Hours ORDER BY Hours " -i:W3C -o:csv

Gets information about response times for the 20 slowest URLs and writes them to a CSV file:

LogParser.exe "SELECT TOP 20 cs-uri-stem, COUNT(*) AS TotalRequest, MAX(time-taken) AS MaxTime, AVG(time-taken) AS AvgTime INTO avg.csv FROM u_ex*.log GROUP BY cs-uri-stem ORDER BY TotalRequest DESC" -i: IISW3C -o:csv

Gets information about response times for the 25 slowest URLs, filtered between a start and end time, and writes them to a CSV file:

LogParser.exe "SELECT TOP 25 cs-uri-stem, COUNT(*) AS TotalRequest, MAX(time-taken) AS MaxTime, AVG(time-taken) AS AvgTime INTO BeforeavgProb.csv FROM u_ex*.log WHERE TO_TIME(time) BETWEEN TIMESTAMP('00:00:00','hh:mm:ss') AND TIMESTAMP('14:00:00','hh:mm:ss') GROUP BY cs-uri-stem ORDER BY TotalRequest DESC" -i: IISW3C -o:csv

Gets all request URLs that resulted in a 500 error:

LogParser.exe "SELECT cs-uri-stem AS FileName FROM u_ex*.log WHERE sc-Status = 500" -i:W3C -rtp:-1

Gets all request URLs that took more than a second to return:

LogParser.exe "SELECT cs-uri-stem AS FileName, time-taken AS Time from u_ex*.log WHERE time-taken > 1000" -i:W3C -rtp:-1

This beast (which I’ve not tested!) appears to bring back a summary of status codes for requests:

LogParser.exe "SELECT to_lowercase(cs-uri-stem) AS URI, SUM([_200]) AS [200s], SUM([_304]) AS [304s], SUM([_302]) AS [302s], SUM([_404]) AS [404s], SUM([_301]) AS [301s], SUM([_500]) AS [500s],SUM([_501]) AS [501s],SUM([_403]) AS [403s],SUM([_206]) AS [206s],SUM([_406]) AS [406s],SUM([_400]) AS [400s], sub(count(*),add([200s],[206s])) as Failures USING CASE TO_STRING(scstatus) WHEN '200' THEN 1 ELSE 0 END AS [_200], CASE TO_STRING(sc-status) WHEN '304' THEN 1 ELSE 0 END AS [_304], CASE TO_STRING(sc-status) WHEN '302' THEN 1 ELSE 0 END AS [_302], CASE TO_STRING(sc-status) WHEN '404' THEN 1 ELSE 0 END AS [_404], CASE TO_STRING(sc-status) WHEN '301' THEN 1 ELSE 0 END AS [_301], CASE TO_STRING(sc-status) WHEN '500' THEN 1 ELSE 0 END AS [_500], CASE TO_STRING(sc-status) WHEN '501' THEN 1 ELSE 0 END AS [_501], CASE TO_STRING(sc-status) WHEN '403' THEN 1 ELSE 0 END AS [_403], CASE TO_STRING(sc-status) WHEN '206' THEN 1 ELSE 0 END AS [_206], CASE TO_STRING(sc-status) WHEN '406' THEN 1 ELSE 0 END AS [_406], CASE TO_STRING(scstatus) WHEN '400' THEN 1 ELSE 0 END AS [_400] FROM ex*.log GROUP BY URI ORDER BY Failures DESC"

The possibilities are endless. Here are some articles I have been recommended which demonstrate alternative queries, starting with two that deal specifically with IIS logs:

IIS Log Parser examples 1
IIS Log Parser examples 2

Here is an example involving more generic text files:

Reading large text files with Log Parser

…and finally, here is some generic information about the Log Parser from Microsoft:

Generic information on Log Parser

Happy parsing!

Help! My SQL Server 2008 ‘sa’ account got locked out and I can’t do anything with my databases!

I recently had an issue where the sa account for one of my SQL Servers got locked out. I assume the reason for the lock out was that the account was marked with Enforce password policy and someone had got the password wrong too many times. Regardless, I was no longer able to log in as sa.

Unfortunately there was a database on my server that only the sa account had access to, and it was starting to look like there was no way to get access back. To make matters worse, none of the other accounts had the correct access level to alter the sa account, and without the sa account, I was unable to elevate the access level of existing accounts. My server has mixed-mode authentication enabled but even domain administrator accounts couldn’t touch the sa account. It seemed like the perfect catch-22.

I Googled a lot of pages, many of which weren’t helpful, but then I found this:

Connect to SQL Server When System Administrators Are Locked Out

Good old Microsoft for once came up with the goods with an easy to follow, step-by-step guide to gaining access again.

The solution involved temporarily setting up single-user mode for the server, and then logging in to both the server and SQL Server Management Studio using an account with local machine administrator access. I was then able to remove the lock on the sa account, reset the password and gain access once more. Phew!

For all but production database servers, I would definitely recommend removing the Enforce password policy on all sa accounts in future. In the meantime, I hope this post has been useful.

Help! My .NET application refuses to open more than 2 concurrent internet connections!

A while ago some colleagues and I were writing a .NET application that fired off web-requests to a third party web-API. The details of the application are unimportant, other than that it was a migration tool which sent data from one third party store to another. There was a lot of data to shift (~9TB) and it needed to be done very quickly. So, we ended up writing a multi-threaded application which could read in many pieces of the stored data concurrently, and fire them off to the third party web-API concurrently.

We wrote a beautiful, elegant and totally thread-safe application and all watched in anticipation as we fired it off for the first time, expecting to see data throughputs the likes of which had never been seen before.

Instead the performance was only slightly better than the first version of our application, which just sent data through one piece at a time in a single thread. After a lot of digging, it turned out that all .NET applications have a restriction on the number of network connections that can be opened. By default this number is two. Fortunately, this number can be changed using the configuration file.

Simply add the following to override this value:

<system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "100" />
    </connectionManagement>
</system.net>

To demonstrate this in action, try running the following .NET console application without overriding the configuration file:

using System;
using System.Net;
using System.Threading;

namespace ConnectionTestApplication
{
    public static class Program
    {
        private const int ThreadCount = 100;
        private static volatile int _timesPageGot;
        
        public static void Main()
        {
            var timerThread = new Thread(TimerThreadMethod);

            var getPageThreads = new Thread[ThreadCount];
            for (var i = 0; i < ThreadCount; i++)
            {
                getPageThreads[i] = new Thread(GetPageMethod);
            }

            timerThread.Start();
            for (var i = 0; i < ThreadCount; i++)
            {
                getPageThreads[i].Start();
            }

            Console.ReadLine();
        }

        private static void TimerThreadMethod()
        {
            var seconds = 0;

            while (true)
            {
                Thread.Sleep(1000);
                seconds++;
                Console.WriteLine("Seconds: {0}", seconds);
            }
        }

        private static void GetPageMethod()
        {
            var request = WebRequest.Create("https://en.wikipedia.org/wiki/List_of_law_clerks_of_the_Supreme_Court_of_the_United_States");
            using (var response = request.GetResponse())
            {
                _timesPageGot++;
                Console.WriteLine("Page got: {0}", _timesPageGot);
            }
        }
    }
}

Note that this will download what is supposed to be the longest page on Wikipedia (at the time of writing) a whole bunch of times concurrently using threads:

List of law clerks of the Supreme Court of the United States

You’ll get something like this on the screen depending on the speed of your internet connection:

Concurrent Internet Connections Before

Note that it took under 12 seconds to download the page 100 times on my internet connection.

However, with the configuration file override set to allow 100 concurrent connections I get:

Concurrent Internet Connections After

This time it took under 3 seconds.

Result!