NCL 209: Unicode

In NCL 208: Character Codes, we learned that characters can be identified using numbers.

For ASCII characters, this looked suspiciously simple:

Property	A
Character code	65
Character code in hexadecimal	41
UTF-8 byte	41

The hexadecimal character code and the byte used to represent it happened to be the same.

That stops being true once we move beyond ASCII.

Consider the copyright symbol:

Property	©
Character code	169
Character code in hexadecimal	00A9
UTF-8 bytes	C2 A9

© is one character, but UTF-8 uses two bytes to represent it.

Using the hexadecimal byte escapes from the previous lesson, we could write:

D.TXT "\xC2\xA9"
D.BLT

This displays:

©

It works, but there is an obvious inconvenience.

We wanted the character ©. Why did we have to know how that character is encoded into bytes?

Unicode code points

Unicode assigns a number to each character.

That number is called a code point.

Unicode code points are conventionally written in hexadecimal using U+ followed by the code:

Character	Unicode code point
A	U+0041
©	U+00A9
é	U+00E9
☀	U+2600
中	U+4E2D
🚀	U+1F680

The character codes we worked with in the previous lesson are more specifically Unicode code points.

For example, the copyright symbol is Unicode code point U+00A9.

You are not expected to memorize these numbers.

Unicode escapes

NCL can identify a Unicode character directly using a Unicode escape:

D.TXT "\u00A9"
D.BLT

This also displays:

©

Compare that with the byte-oriented version:

D.TXT "\xC2\xA9"

These two strings produce the same character, but they describe it at different levels.

NCL source	What it specifies
"\xC2\xA9"	UTF-8 bytes C2 A9
"\u00A9"	Unicode code point U+00A9

The first specifies the bytes used to encode the character. The second identifies the character itself by its Unicode code point.

For another example:

D.TXT "\u2600"
D.BLT

displays:

☀

because ☀ is Unicode code point U+2600.

You can also just type the character

A Unicode escape is not required when the character can conveniently be written directly in the source file.

These both represent the same character:

D.TXT "☀"
D.TXT "\u2600"

And, using its UTF-8 representation, so does:

D.TXT "\xE2\x98\x80"

We now have three ways to describe the same resulting text:

NCL source	What it specifies
"☀"	The character itself
"\u2600"	Unicode code point U+2600
"\xE2\x98\x80"	UTF-8 bytes E2 98 80

For ordinary text, writing or pasting the character directly is usually easiest.

Unicode escapes are useful when the exact code point matters, when a character is difficult to type, or when writing the code point explicitly makes the source easier to understand.

Hexadecimal byte escapes are useful when the encoded bytes themselves matter.

Unicode is not UTF-8

Unicode tells us which character we mean.

A computer still needs some way to represent that character as data.

An encoding provides the rules for doing that.

NCL uses UTF-8, an encoding that represents Unicode code points using one or more bytes.

Character	Unicode code point	UTF-8 bytes
A	U+0041	41
©	U+00A9	C2 A9
☀	U+2600	E2 98 80
🚀	U+1F680	F0 9F 9A 80

ASCII characters use only one byte in UTF-8.

For those characters, the Unicode code point and UTF-8 byte have the same value:

Property	A
Unicode code point	U+0041
UTF-8 bytes	41

That convenient correspondence is why character codes and encoded bytes looked almost interchangeable when we first encountered them.

Outside ASCII, the distinction becomes visible:

Property	☀
Unicode code point	U+2600
UTF-8 bytes	E2 98 80

☀ is one Unicode code point, encoded using three UTF-8 bytes.

The important distinction is:

Unicode identifies characters. UTF-8 encodes them.

Other Unicode encodings

UTF-8 is not the only way to encode Unicode.

You may also encounter encodings such as UTF-16 and UTF-32. They represent the same Unicode code points using different arrangements of data.

For example, a rocket remains Unicode code point U+1F680 regardless of which Unicode encoding is used to store or transmit it.

NCL uses UTF-8, so that is the encoding we will normally encounter here.

Code points beyond four digits

The \u escape contains four hexadecimal digits:

\uXXXX

That is enough for characters such as:

Character	Code point
©	U+00A9
☀	U+2600
中	U+4E2D

But Unicode contains code points larger than four hexadecimal digits.

For example, 🚀 is U+1F680.

For these, NCL provides the larger \U form:

\UXXXXXXXX

The rocket can therefore be written as:

D.TXT "\U0001F680"
D.BLT

or simply:

D.TXT "🚀"
D.BLT

The leading zeroes in \U0001F680 fill the eight hexadecimal digits required by the escape.

NCL source	Code point	Character
"\u2600"	U+2600	☀
"\U0001F680"	U+1F680	🚀

Characters are still characters

The number of bytes used by UTF-8 does not change how NCL normally indexes a string.

Consider:

#text "A©☀中🚀"
#length r0

SLEN #length #text

The string contains five Unicode code points, so #length is 5.

NCL sees five positions in the string:

Index	Character	UTF-8 bytes
0	A	41
1	©	C2 A9
2	☀	E2 98 80
3	中	E4 B8 AD
4	🚀	F0 9F 9A 80

The UTF-8 representation varies from one to four bytes, but those encoded byte lengths do not change the string indices.

That means the string operations we learned earlier remain pleasantly cooperative.

For example:

#text "A©☀中🚀"
#character s0

SSUB #character #text 4 4

stores 🚀 in #character.

Although UTF-8 requires four bytes to encode the rocket, it occupies one position in the string.

Inspecting Unicode characters

ORD also works with the full Unicode code point.

For example:

#text "🚀"
#length r0
#code r1

SLEN #length #text
ORD #code #text 0

After these instructions:

Property	Value
Character	🚀
String length	1
String position	0
Unicode code point	U+1F680
Code point in decimal	128640
UTF-8 encoding	F0 9F 9A 80
UTF-8 byte count	4

The character is still one item in the string. The four bytes are simply how UTF-8 represents that character underneath.

We can also walk through a mixed string using exactly the same indexed ORD operation we used in the previous lesson:

#text "A©☀🚀"

#index r0
#code r1

MOVE #index 0

$next
ORD #code #text #index

D.TXT #index
D.TXT ": "
D.TXT #code
D.TXT "\r\n"

INC #index
BLT $next #index 4

D.BLT

This produces:

0: 65
1: 169
2: 9728
3: 128640

Each ORD reads one Unicode code point, regardless of how many UTF-8 bytes are used to encode it.

Characters and glyphs

There is one more distinction worth making.

NCL can store a Unicode character without requiring the Display to know how to draw it.

A character is part of the text.

A glyph is the shape the Display uses to represent that character visually.

For example:

#symbol s0

SMOVE #symbol "\U0001F680"

stores the Unicode character 🚀 in #symbol.

Whether:

D.TXT #symbol
D.BLT

can draw the expected rocket depends on whether the attached Display has a glyph for U+1F680.

Unicode support in strings and glyph support in a Display are separate things.

We'll return to how characters physically occupy the Display when we begin working with the Display itself.

Try it

Start with a short string containing characters from different parts of Unicode:

#text "A©☀🚀"

#length r0
#character s0
#code r1

SLEN #length #text
D.TXT "Length: "
D.TXT #length
D.TXT "\r\n"

SSUB #character #text 1 1
D.TXT "Index 1: "
D.TXT #character
D.TXT "\r\n"

SSUB #character #text 2 2
D.TXT "Index 2: "
D.TXT #character
D.TXT "\r\n"

ORD #code #text 3
D.TXT "Index 3 code: "
D.TXT #code

D.BLT

It should report a length of 4, extract one character at a time, and return the full Unicode code point for the final character.

Then try changing the string. Use characters from the Unicode charts, enter some directly, and try replacing some with \uXXXX or \UXXXXXXXX escapes.

You can also compare different ways of creating the same character:

D.TXT "☀"
D.TXT "\r\n"

D.TXT "\u2600"
D.TXT "\r\n"

D.TXT "\xE2\x98\x80"

D.BLT

All three lines produce the same resulting character. The first writes it directly, the second identifies its Unicode code point, and the third specifies its UTF-8 representation as raw bytes.

---

In NCL 210: Styling the Display, we'll use another kind of escape sequence to control how text is displayed.