Understanding Unicode and Character Sets: A Guide for Developers

Every developer should understand Unicode and character sets, no matter how much experience they have. Many don’t, which is more common than you might expect. This article gives you a short overview so you have the basics covered.

The mystery of character sets: Have you ever wondered about the ‘Content-Type’ tag in HTML? Or received an email with garbled characters in the subject line? A lot of developers have never worked out how character sets, encodings, and Unicode fit together. Since most software now crosses borders and languages, it pays to know these concepts.

A brief history: It helps to look back a bit. Ancient character sets like EBCDIC may not matter to you anymore, but the path from ASCII to today’s encodings is worth following. ASCII worked fine for English speakers at first. Once computing went global, its limits showed. Different regions started using the 128-255 byte range for their own characters, and that caused confusion and miscommunication. Out of that mess came the ANSI standard and the idea of code pages.

The advent of Unicode: Unicode was the answer to all this growing complexity. Despite what many assume, Unicode isn’t just a 16-bit code. It assigns a unique number, or ‘code point’, to every character in every writing system. That gives you room for a huge number of characters, well beyond the 65,536 that two bytes can hold.

Encoding matters: Knowing a character’s Unicode code point is only half the story. How those code points are stored or transmitted comes down to their encoding. The first idea was to use two bytes per character, but that ran into problems, which is why methods like UTF-8, UTF-16, and others came along.

Why you should specify encodings: A string means nothing without knowledge of its encoding. Strings with no specified encoding can display wrong or corrupt your data. For web content, set the encoding with the ‘Content-Type’ header or meta tag. Browsers may try to guess it, but you can’t rely on that.

Top 5 most used Unicode charsets

1. UTF-8: the dominant charset

• Overview: UTF-8 (8-bit Unicode Transformation Format) is a variable-width character encoding that can represent every character in the Unicode standard. It has become the dominant charset for the web and many other applications.
• Advantages:
  • Backward compatibility: UTF-8 is compatible with ASCII, so any ASCII text is also valid UTF-8 text.
  • Flexibility: It can encode any character in the Unicode standard yet stays compact for ASCII characters.
  • Widespread adoption: From websites to databases, UTF-8 is the preferred encoding because it works almost everywhere.

2. UTF-16: bridging the gap

• Overview: UTF-16 (16-bit Unicode Transformation Format) uses 16 bits as its basic unit. It represents the most common characters with a single 16-bit code unit and the rest with a pair of them.
• Usage: You’ll find it in many programming environments, like Java and Windows, and it suits texts that mix common and uncommon characters.
• Considerations: It balances size and range, but it isn’t as compact as UTF-8 for ASCII text.

3. UTF-32: direct mapping to Unicode code points

• Overview: UTF-32 uses a single 32-bit code unit per character, which maps directly to Unicode code points.
• Advantages:
  • Simplicity: Each character is 4 bytes, which makes certain operations simpler.
  • Full coverage: It can represent every Unicode character without multi-unit sequences.
• Drawbacks: Because it’s fixed-width, it isn’t as space-efficient as UTF-8 or UTF-16 for texts mostly in ASCII or the BMP (Basic Multilingual Plane).

4. UCS-2: the precursor to UTF-16

• Overview: UCS-2 (2-byte Universal Character Set) came before UTF-16 and uses two bytes per character. It can only represent characters in the BMP.
• Limitations: UCS-2 can’t handle characters outside the BMP, which makes it less versatile than UTF-16. Its use has dropped off in favor of UTF-16.

5. GB18030: bridging Unicode and China

• Overview: GB18030 is a Chinese government standard that covers all Unicode characters and keeps China’s character sets compatible with Unicode.
• Significance: It’s mandatory for all software products in China, so international developers need to support it.

How to specify Unicode charsets in htaccess, html pages, WordPress and more?

Setting character sets correctly is what makes web content display the way you meant it to. Here’s how you specify Unicode charsets across various platforms and technologies:

1. .htaccess (for Apache servers):

To set the character set for your entire website using the .htaccess file, use the AddDefaultCharset directive:

AddDefaultCharset UTF-8

2. HTML pages:

In HTML documents, specify the character set with the <meta> tag inside the <head> section:

<meta charset="UTF-8">

For older HTML (HTML 4), the specification looks a bit different:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

3. WordPress:

In WordPress, the charset usually sits in the theme’s header file (header.php), though WordPress sets it to UTF-8 automatically. If you want to check or change it:

1. Go to your theme’s directory and open header.php.
2. Look for the <meta charset> tag. It should look like this:

<meta charset="<?php bloginfo( 'charset' ); ?>">

This PHP function outputs the charset from your WordPress settings, which is UTF-8 by default.

4. PHP:

To send a raw HTTP header with the charset in a PHP script, use the header() function:

header('Content-Type: text/html; charset=UTF-8');

5. MySQL:

When you create or modify a database or table in MySQL, you can specify the character set:

CREATE DATABASE mydatabase DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Or for a table:

CREATE TABLE mytable (
...
) DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

6. CSS:

It’s not common, but you can specify a charset in a CSS file:

@charset "UTF-8";

This should be the very first line of your CSS file.

7. JavaScript:

For external JavaScript files, you can set the charset when you link to them in your HTML:

<script src="myscript.js" charset="UTF-8"></script>

8. XML:

In XML documents, the charset goes in the XML declaration:

<?xml version="1.0" encoding="UTF-8"?>

mysqldump -u username -p database_name > backup.sql

2. Check current character set and collation:

First, check the current character set and collation of your database and its tables.

SHOW CREATE DATABASE database_name;
SHOW CREATE TABLE table_name;

3. Alter database and tables:

If the database or tables aren’t on the character set you want (say, utf8mb4) and collation (say, utf8mb4_unicode_ci), you can alter them.

Alter database:

ALTER DATABASE database_name CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Alter tables and columns:

For each table:

ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

4. Fixing data:

If the database holds wrongly encoded data, you may need to convert it. A common case is double-encoded UTF-8 data. Here’s how you fix it:

UPDATE table_name SET column_name = CONVERT(CAST(CONVERT(column_name USING latin1) AS BINARY) USING utf8mb4) WHERE some_condition;

Replace table_name, column_name, and some_condition with the right values. This snippet assumes the data started as UTF-8, was mistakenly treated as Latin1, and then stored as UTF-8 again.

5. Update MySQL client:

Make sure your MySQL client connection uses the correct character set:

SET NAMES 'utf8mb4';

Or, if you use a configuration file (like my.cnf or my.ini), you can add:

[client]
default-character-set = utf8mb4

6. Check application code:

Check that your application’s database connection string names the correct character set. For example, in PHP with PDO:

Conclusion:

Fixing Unicode issues in MySQL means adjusting the database schema and making sure the data itself is encoded correctly. Test your changes somewhere safe before you run them on a production database.