.NET String Object and Invalid Unicode Code Points

While the answer given by @DPenner is excellent (and I used it as a starting point), I want to give some other details. Apart from orphan surrogates, which in my opinion are a clear sign of an invalid string, there is always the possibility that the string contains unassigned code points, and this case cannot be considered a .NET Framework bug as new characters are always added to the Unicode standard, see ., for example, the Unicode versions http://en.wikipedia.org/wiki/Unicode#Versions . And to make things clearer, this call Char.GetUnicodeCategory(Char.ConvertFromUtf32(0x1F01C), 0);

returns UnicodeCategory.OtherNotAssigned

when using .NET 2.0, but when using .NET 4.0, it will return UnicodeCategory.OtherSymbol

.

In addition, there is another interesting point: even the methods of the .NET class library are not consistent with how to handle Unicode non-characters and unpaired surrogate characters. For example:

• unpaired surrogate char
  • System.Text.Encoding.Unicode.GetBytes("\uDDDD");
    
    - returns the { 0xfd, 0xff}
    
    encoding for the Substitute Character , that is, the data is considered invalid.
  • "\uDDDD".Normalize();
    
    - generates an exception with the message "Invalid Unicode code point found at index 0.", that is, the data is considered invalid.
• uncharacteristic codes
  • System.Text.Encoding.Unicode.GetBytes("\uFFFF");
    
    - returns {0xff, 0xff}
    
    , that is, the data is considered valid.
  • "\uFFFF".Normalize();
    
    - generates an exception with the message "Invalid Unicode code point found at index 0.", that is, the data is considered invalid.

Below is a method that will search for invalid characters in a string:

/// <summary>
/// Searches invalid charachters (non-chars defined in Unicode standard and invalid surrogate pairs) in a string
/// </summary>
/// <param name="aString"> the string to search for invalid chars </param>
/// <returns>the index of the first bad char or -1 if no bad char is found</returns>
static int FindInvalidCharIndex(string aString)
{
    int ch;
    int chlow;

    for (int i = 0; i < aString.Length; i++)
    {
        ch = aString[i];
        if (ch < 0xD800) // char is up to first high surrogate
        {
            continue;
        }
        if (ch >= 0xD800 && ch <= 0xDBFF)
        {
            // found high surrogate -> check surrogate pair
            i++;
            if (i == aString.Length)
            {
                // last char is high surrogate, so it is missing its pair
                return i - 1;
            }

            chlow = aString[i];
            if (!(chlow >= 0xDC00 && chlow <= 0xDFFF))
            {
                // did not found a low surrogate after the high surrogate
                return i - 1;
            }

            // convert to UTF32 - like in Char.ConvertToUtf32(highSurrogate, lowSurrogate)
            ch = (ch - 0xD800) * 0x400 + (chlow - 0xDC00) + 0x10000;
            if (ch > 0x10FFFF)
            {
                // invalid Unicode code point - maximum excedeed
                return i;
            }
            if ((ch & 0xFFFE) == 0xFFFE)
            {
                // other non-char found
                return i;
            }
            // found a good surrogate pair
            continue;
        }

        if (ch >= 0xDC00 && ch <= 0xDFFF)
        {
            // unexpected low surrogate
            return i;
        }

        if (ch >= 0xFDD0 && ch <= 0xFDEF)
        {
            // non-chars are considered invalid by System.Text.Encoding.GetBytes() and String.Normalize()
            return i;
        }

        if ((ch & 0xFFFE) == 0xFFFE)
        {
            // other non-char found
            return i;
        }
    }

    return -1;
}