Serialising constructor-less .NET types (e.g. MailAddress) using Newtonsoft.Json.JsonConvert in the Json.NET library

Newtonsoft’s Json.NET library is brilliant for serialising .NET objects into JSON and back again. However, there are some things it can’t do. For instance, if we try serialising an object based on the following class:

using System.Net.Mail;

namespace JsonConvertors
{
    public class Person
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public MailAddress EmailAddress { get; set; }
    }
}

…using the following code:

var person = new Person
{
    FirstName = "Joe",
    LastName = "Bloggs",
    EmailAddress = new MailAddress("joe@bloggs.com"),
};

var json = JsonConvert.SerializeObject(person);
var personCopy = JsonConvert.DeserializeObject<Person>(json);

We get the following nasty error:

Unable to find a constructor to use for type System.Net.Mail.MailAddress. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'EmailAddress.DisplayName', line 1, position 69.

Why Microsoft didn’t add a constructor to the MailAddress type is beyond me. Of course, we could just store the EmailAddress as a string and bypass the issue that way, but I am a firm believer that everything should be strongly typed. Fortunately, we can use the JsonConverter class to tell JsonConvert how to handle constructor-less types like MailAddress.

Here’s how to write a converter for the MailAddress type:

using System;
using System.Net.Mail;
using Newtonsoft.Json;

namespace JsonConvertors
{
    public class MailAddressConverter : JsonConverter
    {
        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
        {
            var mailAddress = value as MailAddress;
            writer.WriteValue(mailAddress == null? string.Empty : mailAddress.ToString());
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        {
            var text = reader.Value as string;
            MailAddress mailAddress;

            return IsValidMailAddress(text, out mailAddress) ? mailAddress : null;
        }

        public override bool CanConvert(Type objectType)
        {
            return objectType == typeof(MailAddress);
        }

        private static bool IsValidMailAddress(string text, out MailAddress value)
        {
            try
            {
                value = new MailAddress(text);
                return true;
            }
            catch
            {
                value = null;
                return false;
            }
        }
    }
}

Note that care has been taken to handle null and non-convertible values.

Then simply add the following attribute to your Person class:

using System.Net.Mail;
using Newtonsoft.Json;

namespace JsonConvertors
{
    public class Person
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }

        [JsonConverter(typeof(MailAddressConverter))]
        public MailAddress EmailAddress { get; set; }
    }
}

…and the conversion will work like a charm.

Help! My .NET application refuses to open more than 2 concurrent internet connections!

A while ago some colleagues and I were writing a .NET application that fired off web-requests to a third party web-API. The details of the application are unimportant, other than that it was a migration tool which sent data from one third party store to another. There was a lot of data to shift (~9TB) and it needed to be done very quickly. So, we ended up writing a multi-threaded application which could read in many pieces of the stored data concurrently, and fire them off to the third party web-API concurrently.

We wrote a beautiful, elegant and totally thread-safe application and all watched in anticipation as we fired it off for the first time, expecting to see data throughputs the likes of which had never been seen before.

Instead the performance was only slightly better than the first version of our application, which just sent data through one piece at a time in a single thread. After a lot of digging, it turned out that all .NET applications have a restriction on the number of network connections that can be opened. By default this number is two. Fortunately, this number can be changed using the configuration file.

Simply add the following to override this value:

<system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "100" />
    </connectionManagement>
</system.net>

To demonstrate this in action, try running the following .NET console application without overriding the configuration file:

using System;
using System.Net;
using System.Threading;

namespace ConnectionTestApplication
{
    public static class Program
    {
        private const int ThreadCount = 100;
        private static volatile int _timesPageGot;
        
        public static void Main()
        {
            var timerThread = new Thread(TimerThreadMethod);

            var getPageThreads = new Thread[ThreadCount];
            for (var i = 0; i < ThreadCount; i++)
            {
                getPageThreads[i] = new Thread(GetPageMethod);
            }

            timerThread.Start();
            for (var i = 0; i < ThreadCount; i++)
            {
                getPageThreads[i].Start();
            }

            Console.ReadLine();
        }

        private static void TimerThreadMethod()
        {
            var seconds = 0;

            while (true)
            {
                Thread.Sleep(1000);
                seconds++;
                Console.WriteLine("Seconds: {0}", seconds);
            }
        }

        private static void GetPageMethod()
        {
            var request = WebRequest.Create("https://en.wikipedia.org/wiki/List_of_law_clerks_of_the_Supreme_Court_of_the_United_States");
            using (var response = request.GetResponse())
            {
                _timesPageGot++;
                Console.WriteLine("Page got: {0}", _timesPageGot);
            }
        }
    }
}

Note that this will download what is supposed to be the longest page on Wikipedia (at the time of writing) a whole bunch of times concurrently using threads:

List of law clerks of the Supreme Court of the United States

You’ll get something like this on the screen depending on the speed of your internet connection:

Concurrent Internet Connections Before

Note that it took under 12 seconds to download the page 100 times on my internet connection.

However, with the configuration file override set to allow 100 concurrent connections I get:

Concurrent Internet Connections After

This time it took under 3 seconds.

Result!

Text truncation at a word boundary using C#

Following on from my last post on stripping HTML from text using C#, once I had removed all signs of HTML from the incoming text, I was also required to show a short preview of the text. I originally went with a truncation method, as follows:

namespace ExtensionMethods
{
    public static class StringExtensionMethods
    {
        public static string Truncate(this string text, int maximumLength)
        {
            if (string.IsNullOrEmpty(text))
            {
                return text;
            }

            return text.Length <= maximumLength ? text : text.Substring(0, maximumLength);
        }
    }
}

This works, but the results look a little odd if the truncate happens half-way through a word.

Instead, I came up this method to truncate at the first word break within the allowed number of characters:

using System.Linq;

namespace ExtensionMethods
{
    public static class StringExtensionMethods
    {
        private static readonly char[] Punctuation = {'.', ',', ';', ':'};

        public static string TruncateAtWordBoundary(this string text, int maximumLength)
        {
            if (string.IsNullOrEmpty(text))
            {
                return text;
            }

            if (text.Length <= maximumLength)
            {
                return text;
            }

            // If the character after the cut off is white space or punctuation 
            // then return what we've got using substring:
            var isCutOffWhiteSpaceOrPunctuation = char.IsWhiteSpace(text[maximumLength]) || Punctuation.Contains(text[maximumLength]);
            text = text.Substring(0, maximumLength);

            if (isCutOffWhiteSpaceOrPunctuation)
            {
                return text;
            }

            // Find the last white-space or punctuation and chop off there:
            var lastWhiteSpaceOrPunctuationPosition = 0;
            for (var i = text.Length - 1; i >= 0; i--)
            {
                if (char.IsWhiteSpace(text[i]) || Punctuation.Contains(text[i]))
                {
                    lastWhiteSpaceOrPunctuationPosition = i;
                    break;
                }
            }

            text = text.Substring(0, lastWhiteSpaceOrPunctuationPosition).Trim();

            return text;
        }
    }
}

While not perfect, this approach works a lot better. Please feel free to suggest improvements.

Stripping HTML from text using C#

I recently had a situation where I needed to show some text received in HTML format as plain text. This is the method I now use for this purpose, implemented as an extension method:

using System.Linq;
using System.Text.RegularExpressions;

namespace ExtensionMethods
{
    public static class StringExtensionMethods
    {
        public static string StripHtml(this string text)
        {
            if (string.IsNullOrEmpty(text))
            {
                return text;
            }

            var tagRegex = new Regex(@"(?></?\w+)(?>(?:[^>'""]+|'[^']*'|""[^""]*"")*)>");
            var tagMatches = tagRegex.Matches(text);

            var commentRegex = new Regex(@"\<![ \r\n\t]*(--([^\-]|[\r\n]|-[^\-])*--[ \r\n\t]*)\>");
            var commentMatches = commentRegex.Matches(text);

            // Replace each tag match with an empty space:
            text = tagMatches.Cast<object>().Aggregate(text, (current, match) => current.Replace(match.ToString(), " "));

            // Replace each comment with an empty string:
            text = commentMatches.Cast<object>()
                .Aggregate(text, (current, match) => current.Replace(match.ToString(), string.Empty));

            // We also need to replace &nbsp; as this can mess up the system:
            text = text.Replace("&nbsp;", " ");

            // Trim and remove all double spaces:
            text = text.Trim().RemoveDoubleSpaces();

            return text;
        }

        public static string RemoveDoubleSpaces(this string text)
        {
            if (string.IsNullOrEmpty(text))
            {
                return text;
            }

            // Condense all double spaces to a single space:
            while (text.Contains("  "))
            {
                text = text.Replace("  ", " ");
            }

            return text;
        }
    }
}

The method RemoveDoubleSpaces was also needed, since after replacing HTML elements with empty space it is possible to end up with multiple empty spaces where a single space would do. This is quite a useful method in its own right, hence separating it out.

If you find any inputs which trip this method up, please let me know.

Exotic .NET XML configuration file sections

The .NET framework makes application configuration really simple by supplying an out-of-the-box XML configuration file structure and associated classes to access the information contained within it.

By default, it allows configuration key-value pairs to be set and retrieved easily at run time. For example, the following configuration settings:

  <appSettings>
    <add key="ServerName" value="WinDB1"/>
    <add key="DatabaseName" value="MyData"/>
    <add key="EnableFeature" value="true"/>
  </appSettings>

…can be retrieved using:

var appSettingsReader = new AppSettingsReader();

var serverName = (string)appSettingsReader.GetValue("ServerName", typeof(string));
var databaseName = (string)appSettingsReader.GetValue("DatabaseName", typeof(string));
var enableFeature = (bool)appSettingsReader.GetValue("EnableFeature", typeof(bool));

…or (if you add a reference to System.Configuration):

var serverName = ConfigurationManager.AppSettings["ServerName"];
var databaseName = ConfigurationManager.AppSettings["DatabaseName"];
var enableFeature = bool.Parse(ConfigurationManager.AppSettings["EnableFeature"]);

…although as per a past post of mine, Encapsulated and strongly-typed access to .NET configuration files I don’t recommend that you litter your code with references to AppSettingsReader (or ConfigurationManager), but instead that you wrap all configuration up in a class so that settings are encapsulated away.

It is often necessary to hold more complex configurations though. Many developers I’ve worked with have tried to use the key-value model to store such settings, using a single string with various delimiters to hold separate different values. For instance, given a person construct with an ID, title, first name, last name and birth date attributes, some developers would be tempted to do something like this:

  <appSettings>
    <add key="Person1" value="I:1|T:Mr|F:Joe|L:Bloggs|D:1980-01-01"/>
  </appSettings>

…and then use string split functions to populate a Person class when reading information from the configuration file. While this approach works, it results in brittle, hard to read configuration files and complex string logic to take strings apart.

Instead, it is better to use the exotic configuration capabilities included in the framework. Here is an example demonstrating how you might store configuration information to populate the following Person class:

using System;

namespace ConfigurationExample
{
    public class Person
    {
        public int Id { get; set; }
        public string Title { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public DateTime BirthDate { get; set; }
    }
}

The configuration file to store such settings might look like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  
  <configSections>
    <section name="people" type="ConfigurationExample.PeopleConfigurationSectionHandler, ConfigurationExample" />
  </configSections>
  
  <people>
    
    <person id="1">
      <title>Mr</title>
      <firstName>Joe</firstName>
      <lastName>Bloggs</lastName>
      <birthDate>
        <year>1980</year>
        <month>1</month>
        <day>1</day>
      </birthDate>
    </person>

    <person id="2">
      <title>Miss</title>
      <firstName>Jane</firstName>
      <lastName>Black</lastName>
      <birthDate>
        <year>1984</year>
        <month>2</month>
        <day>3</day>
      </birthDate>
    </person>
    
  </people>
  
</configuration>

A class implementing IConfigurationSectionHandler must be written to read this exotic configuration file structure; you will need a reference to System.Configuration in order to use this interface. The location of your class is given in the ‘configSections’ declaration at the top of the file. In this case the framework is being instructed to look for a class with name and namespace ‘ConfigurationExample.PeopleConfigurationSectionHandler’ in the ‘ConfigurationExample’ DLL/project in order to read the configuration section named ‘people’.

That class could look something like this:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Xml;

namespace ConfigurationExample
{
    public class PeopleConfigurationSectionHandler : IConfigurationSectionHandler
    {
        public object Create(object parent, object configContext, XmlNode section)
        {
            var people = new List<Person>();

            foreach (XmlNode node in section.ChildNodes)
            {
                if (node.NodeType == XmlNodeType.Comment)
                {
                    continue;
                }

                var id = GetAttributeInt(node, "id");
                var title = GetChildNodeString(node, "title");
                var firstName = GetChildNodeString(node, "firstName");
                var lastName = GetChildNodeString(node, "lastName");

                var birthDateNode = GetChildNode(node, "birthDate");

                var birthDateYear = GetChildNodeInt(birthDateNode, "year");
                var birthDateMonth = GetChildNodeInt(birthDateNode, "month");
                var birthDateDay = GetChildNodeInt(birthDateNode, "day");

                people.Add(new Person
                {
                    Id = id,
                    Title = title,
                    FirstName = firstName,
                    LastName = lastName,
                    BirthDate = new DateTime(birthDateYear, birthDateMonth, birthDateDay),
                });
            }

            return people;
        }

        private static string GetAttributeString(XmlNode node, string attributeName)
        {
            try
            {
                return node.Attributes[attributeName].Value;
            }
            catch
            {
                var message = string.Format("Could not read attribute named '{0}' in people section of configuration file", attributeName);
                throw new ConfigurationErrorsException(message);
            }
        }

        private static int GetAttributeInt(XmlNode node, string attributeName)
        {
            try
            {
                var value = GetAttributeString(node, attributeName);
                return int.Parse(value);
            }
            catch
            {
                var message = string.Format("Could not convert value stored in attribute named '{0}' to an integer", attributeName);
                throw new ConfigurationErrorsException(message);
            }
        }

        private static XmlNode GetChildNode(XmlNode parentNode, string nodeName)
        {
            try
            {
                return parentNode[nodeName];
            }
            catch
            {
                var message = string.Format("Could not find node named '{0}' in people section of configuration file", nodeName);
                throw new ConfigurationErrorsException(message);
            }
        }

        private static string GetChildNodeString(XmlNode parentNode, string nodeName)
        {
            try
            {
                return parentNode[nodeName].InnerText;
            }
            catch
            {
                var message = string.Format("Could not read node named '{0}' in people section of configuration file", nodeName);
                throw new ConfigurationErrorsException(message);
            }
        }

        private static int GetChildNodeInt(XmlNode parentNode, string nodeName)
        {
            try
            {
                var value = GetChildNodeString(parentNode, nodeName);
                return int.Parse(value);
            }
            catch
            {
                var message = string.Format("Could not convert value stored in node named '{0}' to an integer", nodeName);
                throw new ConfigurationErrorsException(message);
            }
        }
    }
}

Although there is quite a lot of code there, it’s all pretty simple. More importantly, it’s not too brittle and very simple to extend should you wish to add more attributes to the Person class.

The method required by the IConfigurationSectionHandler interface is Create:

public object Create(object parent, object configContext, XmlNode section)

This is what gets called when the framework asks for the contents of the ‘people’ section. The ‘section’ parameter contains the people node from the XML. In essence, the code creates a list in which Person objects are stored, and loops over the child nodes in order to extract the correct information.

There are two important points to note about the code:

1) The code that detects for the presence of XML comments:

if (node.NodeType == XmlNodeType.Comment)
{
    continue;
}

…ensures that developers don’t break the code by adding comments into the XML.

2) All the helper methods in the class (e.g. GetAttributeString, GetChildNodeString, etc…) must expect there to be issues with the XML and throw meaningful, self-explanatory exceptions when things do go wrong. When there are problems with these types of sections it is not always obvious what the problem is from the default exceptions .NET throws (particularly if you’re using this technique in the start up code of a Windows Service) so you will save yourself a lot of head scratching further down the line by writing good error messages. The chances are that you will have more than one section handler in your project so it’s a good idea to encapsulate such helper methods away in an abstract ConfigurationSectionHandler base class to cut down on duplication and to promote reuse.

Assuming you have registered your section handlers correctly in the ‘configSections’ declaration, you can retrieve the Person objects as follows:

var people = (IList<Person>)ConfigurationManager.GetSection("people");

Since the IConfigurationSectionHandler interface uses the ‘object’ type as its return value you need to cast the result to the correct type. It would have been nice if the interface was made generic, but I guess you can’t have everything!

If you’re following the advice in my previous post on configuration encapsulation, you’ll probably want to wrap this call up in your ConfigurationFile class, as follows:

using System.Collections.Generic;
using System.Configuration;

namespace ConfigurationExample
{
    public class ConfigurationFile
    {
        private readonly IList<Person> people;

        public ConfigurationFile()
        {
            people = (IList<Person>)ConfigurationManager.GetSection("people");
        }

        public IList<Person> People
        {
            get
            {
                return people;
            }
        }
    }
}

Note that a reference to the Person list is stored when the class constructs to save the running the cast each time you want to retrieve the contents of the configuration file section.

In summary, it’s a little more work to write exotic configuration file sections, but the advantages over brittle, difficult to extend key-value implementations is certainly worth the extra effort.

NHibernate and log4net

Here’s a quick trick for people using NHibernate and log4net within their .NET applications. (…and if you’re not using NHibernate or log4net, you should seriously think about it!)

Add the following to your log4net configuration to find out what’s going on inside NHibernate:

  <logger name="NHibernate">
    <level value="DEBUG" />
  </logger>
  <logger name="NHibernate.SQL">
    <level value="DEBUG" />
  </logger>

As those kind folks at NHibernate have already set up the loggers behind the scenes, everything just works. The SQL logger is especially useful as it’ll show you explicitly which SQL statements are being generated by NHibernate. You can adjust the levels of NHibernate logging independently of your normal application logging, which is useful as the NHibernate logs tend to be pretty verbose.

The full configuration will look something like this:

<log4net>
  <appender name="TraceAppender" type="log4net.Appender.TraceAppender" >
    <layout type="log4net.Layout.PatternLayout">
      <param name="ConversionPattern" value="%d %-5p- %m%n" />
    </layout>
  </appender>
  <appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender">
    <file value="Application.log"/>
    <appendToFile value="true"/>
    <rollingStyle value="Date"/>
    <datePattern value="yyyy-MM-dd'.log'"/>
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline"/>
    </layout>
  </appender>
  <root>
    <level value="DEBUG"/>
    <appender-ref ref="TraceAppender"/>
    <appender-ref ref="RollingFileAppender"/>
  </root>
  <logger name="NHibernate">
    <level value="ERROR" />
  </logger>
  <logger name="NHibernate.SQL">
    <level value="ERROR" />
  </logger>
</log4net>

Taking databases offline in SQL Server 2012

Whenever I try and use the Take Offline feature in SQL Server 2012 via SQL Server Management Studio, the progress window hangs, and no matter how long I wait, nothing seems to happens.

I found an alternative way to do this today, which solves the problem nicely. Simply run the following T-SQL:

ALTER DATABASE {DatabaseName} SET OFFLINE WITH ROLLBACK IMMEDIATE

…and the database goes offline, ready for other operations such as database restores to be performed.

Basic Log4Net step-up

The following post explains how to get Log4Net working in a .NET console application.

The first step is create your console application and add references to Log4Net. The easiest way to add the correct references is via NuGet, where it is simply listed as log4net.

You’ll need to add a configuration file to tell Log4Net how you want it to log. A basic example would be:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" />
  </configSections>

  <log4net>
    <appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
      <file value="Test.log" />
      <appendToFile value="true" />
      <maximumFileSize value="5000KB" />
      <maxSizeRollBackups value="50" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%utcdate{yyyy-MM-dd HH:mm:ss.fffffff} - %level - %logger - %message%newline" />
      </layout>
    </appender>
    <root>
      <level value="DEBUG" />
      <appender-ref ref="RollingFile" />
    </root>
  </log4net>
  
</configuration>

This example simply adds a rolling log file. It’s called rolling because it starts a new log file at triggered intervals, allowing you to delete historical log files when you’ve finished with them. In this case, a new log file is started each time the current log file gets to 5000KB. It’s worth taking a look at the Log4Net documentation to find out about the different types of logging available. It is also possible to write your own appender if you want to do something not already built it to Log4Net.

In order to tell Log4Net to pick your configuration, you need to add the following to your application:

[assembly: XmlConfigurator]

The most obvious place for this is the AssemblyInfo.cs file.

This tells Log4Net to look in the default configuration file for the Log4Net configuration settings, but you can also tell Log4Net to look in another custom configuration file using the overloads available to the XmlConfigurator, although I prefer to keep all configuration in a single file.

Then it’s just a case of using the logger:

using System;
using log4net;

namespace Test
{
    public static class Program
    {
        public static void Main()
        {
            var log = LogManager.GetLogger(typeof(Program));
            log.Debug("Test console application started");

            Console.ReadLine();
        }
    }
}

Note that the LogManager returns an object implementing the ILog interface.

If you’re using dependency injection, you should wire up the dependency injector to return your ILog instance. The following shows how to do this if you’re using Castle Windsor:

using Castle.MicroKernel.Registration;
using Castle.Windsor;
using Castle.Windsor.Installer;
using log4net;

namespace Test
{
    public static class Injector
    {
        private static readonly object InstanceLock = new object();

        private static IWindsorContainer instance;

        public static IWindsorContainer Instance
        {
            get
            {
                lock (InstanceLock)
                {
                    return instance ?? (instance = GetInjector());
                }
            }
        }

        private static IWindsorContainer GetInjector()
        {
            var container = new WindsorContainer();

            container.Install(FromAssembly.This());

            RegisterInjector(container);
            RegisterLog(container);

            return container;
        }

        private static void RegisterInjector(WindsorContainer container)
        {
            container.Register(
                Component.For<IWindsorContainer>()
                    .Instance(container));
        }

        private static void RegisterLog(WindsorContainer container)
        {
            container.Register(
                Component.For<ILog>()
                    .Instance(LogManager.GetLogger(typeof(Injector)))
                    .LifestyleSingleton());
        }
    }
}

The following statement will return an ILog instance as expected:

var log = Injector.Instance.Resolve();

There you have it!

Changing the default naming convention for foriegn keys when using Fluent NHibernate

By default, NHibernate will assume that foreign keys should be named using the convention ReferencedTableName_id (e.g. Person_id). I personally don’t like this conversion and would prefer ReferencedTableNameId (e.g. PersonId).

Fortunately, this is pretty easy to achieve by extending the ForeignKeyConvention class in Fluent NHibernate:

using System;
using FluentNHibernate;
using FluentNHibernate.Conventions;

namespace NHibernate.Helpers.Conventions
{
    public class CustomForeignKeyConvention : ForeignKeyConvention
    {
        protected override string GetKeyName(Member property, System.Type type)
        {
            if (property != null)
            {
                return property.Name + "Id";
            }

            if (type != null)
            {
                return type.Name + "Id";
            }

            // If both are null, we throw an exception:
            throw new ArgumentNullException("property", "The property and type parameters cannot both be null");
        }
    }
}

This needs to be wired into your SessionFactory creation code as follows:

sessionFactory = Fluently.Configure()
    .Database(DatabaseConfiguration)
    .Mappings(x => x.FluentMappings.AddFromAssembly(MappingsAssembly))
    .Mappings(x => x.FluentMappings.Conventions.Add(new CustomForeignKeyConvention()))
    .ExposeConfiguration(x => new SchemaUpdate(x).Execute(false, true))
    .BuildSessionFactory();

Note that the example above will also update the database schema.

Once you’ve done that, all foreign keys will use the desired naming convention.

Removing duplicates with LINQ in C#

Here are a couple of tricks to remove duplication using LINQ.

Firstly, for simple lists:

var numbersWithDuplicates = new[] { "One", "Two", "Two", "Three", "Four" };
var numbers = numbersWithDuplicates.GroupBy(x => x).Select(group => group.First());

…and secondly for more complex objects:

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
    public int Age { get; set; }
}

…you can de-duplicate using any property you like:

var peopleWithDuplicates = new[]
{
    new Person { Id = 2, Name = "Alice", Age = 29 },
    new Person { Id = 1, Name = "Bob", Age = 30 },
    new Person { Id = 3, Name = "Claire", Age = 31 },
    new Person { Id = 4, Name = "Bob", Age = 50 },
};
var people = peopleWithDuplicates.GroupBy(x => x.Name).Select(group => group.First());