NCL 208: Character Codes

In the previous lesson, we learned that an escape sequence such as:

\n

represents one actual character.

For line feed, that character isn't normally visible. The Display recognizes it and moves its cursor down one row.

But this raises another question:

If line feed is one character, how does the computer identify which character it is?

Characters are identified using character codes.

These codes are numbers.

Characters as numbers

Consider the letter:

A

Its character code is:

65

The next few letters have the next few codes:

Character	Code
A	65
B	66
C	67
D	68

The character "A" and the integer 65 are not the same kind of value.

One is a character in a string.

The other is a number that identifies that character.

NCL can work in both directions: it can find the numeric code for a character, or find the character identified by a numeric code.

Finding a character's code

ORD reads a character from a string and gives us its character code.

Because a string can contain more than one character, we also tell ORD which character we want by giving it an index.

Like the other string operations we've used, indexes start at 0.

For example:

#code r0

ORD #code "A" 0

D.TXT #code
D.BLT

The Display shows:

65

Our string contains only one character, at index 0.

ORD becomes more useful when a string contains several characters:

#code r0

ORD #code "ABC" 1

D.TXT #code
D.BLT

The string contains:

Index	Character	Code
0	A	65
1	B	66
2	C	67

We asked for the character at index 1, so ORD gives us:

66

This is the same kind of indexing we used with SFIND and SSUB.

ORD doesn't require us to extract a one-character string first. It can inspect the character at the position we want.

For example, we can store a longer string and inspect it:

#message s0
#code r0

SMOVE #message "Hello"

ORD #code #message 0

D.TXT #code
D.BLT

This gives us the code for the first character of #message.

Turning a code into a character

CHAR performs the opposite operation.

Instead of asking:

What is the code for the character at this position?

we can ask:

What character has this code?

For example:

#character s0

CHAR #character 65

D.TXT #character
D.BLT

The Display shows:

A

ORD and CHAR therefore work in opposite directions:

Operation	Input	Result
ORD	character "A" at index 0	integer 65
CHAR	integer 65	string "A"

For ORD, we specify which character in the string we want to inspect:

ORD r0 "A" 0

We can also use a register containing the code:

#code r0
#character s0

MOVE #code 66
CHAR #character #code

D.TXT #character
D.BLT

which displays:

B

The character code is just an integer. It can be stored, calculated, compared, and changed like the other integers we've worked with.

Invisible characters have codes too

Character codes aren't only used for letters, numbers, and punctuation.

The control characters from the previous lesson have codes as well.

Remember line feed:

\n

Its character code is:

10

We can create it using CHAR:

#linefeed s0

CHAR #linefeed 10

#linefeed now contains one character: LF.

For example:

#linefeed s0

CHAR #linefeed 10

D.TXT "ONE"
D.TXT #linefeed
D.TXT "TWO"
D.BLT

has the same cursor movement as using \n directly.

We can also ask ORD to inspect that character:

#code r0

ORD #code "\n" 0

D.TXT #code
D.BLT

The result is:

10

So \n and character code 10 refer to the same character.

One describes the character conveniently in NCL source.

The other identifies it with a number.

Another way to write numbers

Character codes are often written using hexadecimal.

So far, we've written numbers using ten different digits:

0 1 2 3 4 5 6 7 8 9

This is decimal.

Hexadecimal uses sixteen digits.

Since we run out of ordinary digits after 9, the letters A through F are used for the remaining six:

Decimal	Hexadecimal
0	0
1	1
2	2
3	3
4	4
5	5
6	6
7	7
8	8
9	9
10	A
11	B
12	C
13	D
14	E
15	F

After F, hexadecimal rolls over just as decimal rolls over after 9.

In decimal:

8
9
10
11

In hexadecimal:

E
F
10
11

This means:

10 hexadecimal = 16 decimal

The number itself hasn't changed.

We're only writing it differently.

For example:

41 hexadecimal = 65 decimal

And we already know what character has code 65:

A

We won't need to do hexadecimal arithmetic in these lessons. For now, it's enough to recognize that hexadecimal is another way of writing a number, using the digits 0 through F.

Hexadecimal byte escapes

NCL lets us put a byte into a string using its hexadecimal value.

The escape sequence is:

\xXX

where XX is the value of one byte written in hexadecimal.

For ASCII characters, the character code and the byte used to encode the character have the same value.

We already know that A has character code:

65 decimal

which is:

41 hexadecimal

The ASCII character A is also encoded using the byte:

41

So we can write:

D.TXT "\x41"
D.BLT

The Display shows:

A

\x41 does not put the characters \, x, 4, and 1 into the string.

The escape is interpreted while the NCL source is being prepared, producing the byte represented by 41.

For ASCII, that byte directly represents one character.

This distinction doesn't make much difference while we're working with ASCII characters.

It becomes much more important once we start working with Unicode.

Revisiting line feed

Line feed has character code:

10 decimal

In hexadecimal, 10 is:

0A

Line feed is an ASCII control character, so it is encoded using the byte:

0A

That means we can also write it as:

\x0A

Compare:

D.TXT "ONE\nTWO"
D.BLT

with:

D.TXT "ONE\x0ATWO"
D.BLT

Both strings contain the same control character between ONE and TWO.

The source representations \n and \x0A produce the same resulting character.

\n gives a convenient name to a commonly used character.

\x0A represents that character using its encoded byte.

We can verify that with ORD:

#first r0
#second r1

ORD #first "\n" 0
ORD #second "\x0A" 0

D.TXT #first
D.TXT " "
D.TXT #second
D.BLT

The Display shows:

10 10

Different source notation, same resulting character.

Hexadecimal numbers in strings

In NCL 202: Changing Text, we learned that MOVE can convert a string containing a number into an integer.

For example:

#value r0

MOVE #value "42"

stores the integer:

42

NCL can also recognize hexadecimal numbers written with a 0x prefix.

For example:

#value r0

MOVE #value "0x41"

D.TXT #value
D.BLT

displays:

65

The string "0x41" describes the hexadecimal number 41.

When MOVE converts it to an integer, the resulting integer value is 65.

Likewise:

MOVE #value "0x0A"

stores:

10

Notice that this uses 0x rather than \x.

They look similar, but they do very different things.

\x and 0x

Compare these instructions:

SMOVE s0 "\x41"
SMOVE s0 "0x41"
MOVE r0 "0x41"
CHAR s0 65
ORD r0 "A" 0

They contain related values and similar-looking notation, but produce different kinds of results.

NCL source	Result
SMOVE s0 "\x41"	string "A"
SMOVE s0 "0x41"	string "0x41"
MOVE r0 "0x41"	integer 65
CHAR s0 65	string "A"
ORD r0 "A" 0	integer 65

The important distinction is what the notation represents.

\x41 is a byte escape inside an NCL string literal:

SMOVE s0 "\x41"

It represents one byte with hexadecimal value 41.

Because A is an ASCII character, that byte produces:

A

By contrast:

SMOVE s0 "0x41"

contains ordinary text.

The resulting string contains four characters:

0x41

Finally:

MOVE r0 "0x41"

asks MOVE to convert that textual hexadecimal number into an integer.

The result is:

65

The same distinction works with our line feed example.

SMOVE s0 "\x0A"

stores the byte used to encode line feed, resulting in one LF character.

SMOVE s0 "0x0A"

stores the four characters:

0x0A

And:

MOVE r0 "0x0A"

stores the integer:

10

Similar notation, but three different results.

Inspecting a string

Because ORD accepts an index, we can use it to inspect the characters of a string one at a time.

For example:

#message "NCL"
#index r0
#code r1

MOVE #index 0

$next
ORD #code #message #index

D.TXT #message
D.TXT " "
D.TXT #index
D.TXT " "
D.TXT #code
D.TXT "\r\n"

INC #index
BLT $next #index 3

D.BLT

The program examines indexes 0, 1, and 2.

It displays:

NCL 0 78
NCL 1 67
NCL 2 76

There's nothing fundamentally different about these character codes and the other numbers we've used.

ORD simply gives our program a way to ask:

What character is stored at this position, as a number?

Once we have that number, all the integer operations and comparisons we learned in the 100-level lessons can work with it.

Character codes and encoded bytes

For an ASCII character such as A, several values happen to line up neatly:

Property	Value
Character	A
Character code	65
Character code in hexadecimal	41
UTF-8 byte	41
Hexadecimal byte escape	\x41

This makes the different representations look almost interchangeable.

For ASCII, they often are.

But they describe different things.

A character code identifies a character.

An encoded byte is part of the representation used to store that character.

For A, one UTF-8 byte is enough, and its value happens to match the character's code:

Property	Value
Character	A
Character code	65
UTF-8 byte	41

As we saw earlier, that isn't always the case. © is one character, but UTF-8 represents it using the two bytes C2 A9.

One character can require multiple encoded bytes.

We'll explore why in the next lesson.

The \x and 0x examples above also show why it's useful to keep these ideas separate. ORD, CHAR, hexadecimal byte escapes, and hexadecimal numeric strings may involve the same numeric values when we're working with ASCII, but they operate on different kinds of data.

Not every possible conversion is built directly into these instructions.

For example, we haven't learned an operation that takes:

65

and produces the hexadecimal text:

"41"

Nor can we treat the bare string:

"41"

as hexadecimal without indicating that it is hexadecimal.

More general hexadecimal parsing and formatting are available through the String module. We don't need them to work with character codes.

Try it

Let's investigate a string using ORD.

Start with:

#text "ABCD"
#code r0

ORD #code #text 0

D.TXT #code
D.BLT

The result is:

65

Change the index from 0 to 1, then 2, then 3.

Before running each version, predict the code you'll get.

Then try changing the string:

#text "abcd"

Are the codes the same as the uppercase letters?

You can also use CHAR to go in the other direction:

#character s0

CHAR #character 65

D.TXT #character
D.BLT

Try changing 65 to:

66
67
68

What pattern do you notice?

Finally, prove that these two source representations result in the same character:

#first r0
#second r1

ORD #first "\n" 0
ORD #second "\x0A" 0

D.TXT #first
D.TXT " "
D.TXT #second
D.BLT

Both values should be:

10

Different ways of representing the character.

Same numeric code.

---

Character codes let us identify and inspect characters using numbers. Hexadecimal also gives us a convenient way to describe the bytes used to encode those characters.

In NCL 209: Unicode, we'll see what happens when one byte is no longer enough to represent a character.