Improper Handling of Unicode Encoding
Draft Variant
Structure: Simple
Description
This vulnerability occurs when software fails to correctly process or interpret Unicode-encoded input, leading to security bypasses, data corruption, or unexpected behavior.
Unicode allows characters from many languages to be represented, but this complexity can confuse security checks. For example, an application might filter dangerous characters like '../' in ASCII but miss equivalent Unicode representations, allowing attackers to bypass path traversal defenses. Similarly, visual lookalike characters (homoglyphs) can be used in phishing or to impersonate trusted data. To prevent this, developers must normalize and validate all Unicode input consistently. Use established libraries for canonicalization to ensure characters are compared in their standard form, and apply security checks after normalization—not before. Always treat input handling and comparison logic as encoding-aware, never assuming text is only in basic ASCII.
Common Consequences 1
Scope: Integrity
Impact: Unexpected State
Potential Mitigations 3
Phase: Architecture and Design
Strategy: Input Validation
Avoid making decisions based on names of resources (e.g. files) if those resources can have alternate names.
Phase: Implementation
Strategy: Input Validation
Assume all input is malicious. Use an "accept known good" input validation strategy, i.e., use a list of acceptable inputs that strictly conform to specifications. Reject any input that does not strictly conform to specifications, or transform it into something that does.
When performing input validation, consider all potentially relevant properties, including length, type of input, the full range of acceptable values, missing or extra inputs, syntax, consistency across related fields, and conformance to business rules. As an example of business rule logic, "boat" may be syntactically valid because it only contains alphanumeric characters, but it is not valid if the input is only expected to contain colors such as "red" or "blue."
Do not rely exclusively on looking for malicious or malformed inputs. This is likely to miss at least one undesirable input, especially if the code's environment changes. This can give attackers enough room to bypass the intended validation. However, denylists can be useful for detecting potential attacks or determining which inputs are so malformed that they should be rejected outright.
Phase: Implementation
Strategy: Input Validation
Inputs should be decoded and canonicalized to the application's current internal representation before being validated (Incorrect Behavior Order: Validate Before Canonicalize). Make sure that the application does not decode the same input twice (Double Decoding of the Same Data). Such errors could be used to bypass allowlist validation schemes by introducing dangerous inputs after they have been checked.
Demonstrative Examples 1
ID : DX-189
Windows provides the MultiByteToWideChar(), WideCharToMultiByte(), UnicodeToBytes(), and BytesToUnicode() functions to convert between arbitrary multibyte (usually ANSI) character strings and Unicode (wide character) strings. The size arguments to these functions are specified in different units, (one in bytes, the other in characters) making their use prone to error.In a multibyte character string, each character occupies a varying number of bytes, and therefore the size of such strings is most easily specified as a total number of bytes. In Unicode, however, characters are always a fixed size, and string lengths are typically given by the number of characters they contain. Mistakenly specifying the wrong units in a size argument can lead to a buffer overflow.
The following function takes a username specified as a multibyte string and a pointer to a structure for user information and populates the structure with information about the specified user. Since Windows authentication uses Unicode for usernames, the username argument is first converted from a multibyte string to a Unicode string.
Code Example:
Bad
C
cThis function incorrectly passes the size of unicodeUser in bytes instead of characters. The call to MultiByteToWideChar() can therefore write up to (UNLEN+1)*sizeof(WCHAR) wide characters, or (UNLEN+1)*sizeof(WCHAR)*sizeof(WCHAR) bytes, to the unicodeUser array, which has only (UNLEN+1)*sizeof(WCHAR) bytes allocated.
If the username string contains more than UNLEN characters, the call to MultiByteToWideChar() will overflow the buffer unicodeUser.
Observed Examples 3
CVE-2000-0884Server allows remote attackers to read documents outside of the web root, and possibly execute arbitrary commands, via malformed URLs that contain Unicode encoded characters.
CVE-2001-0709Server allows a remote attacker to obtain source code of ASP files via a URL encoded with Unicode.
CVE-2001-0669Overlaps interaction error.
References 1
The Art of Software Security Assessment
Mark Dowd, John McDonald, and Justin Schuh
Addison Wesley
2006
ID: REF-62
Applicable Platforms
Languages:
Not Language-Specific : Undetermined
Modes of Introduction
Implementation
Related Attack Patterns
Related Weaknesses
ChildOf:
Encoding Error (CWE-172)
Taxonomy Mapping
- PLOVER
- CERT C Secure Coding