A. Programming Concepts and Skills · Programming Basics: Data, Decisions and LoopsHow Computers Store Text: ASCII and Unicode
A1.2About 65 minutes
Learning goals- I can explain that every character is stored as a number called a character code or code point.
- I can use
ord() and chr() to convert between characters and their codes. - I can describe the difference between ASCII and Unicode, and why Unicode is needed.
- I can explain that UTF-8 stores a character in 1 to 4 bytes.
Words to know
character
A single letter, digit, punctuation mark, space, symbol or emoji.
character set
A list that gives every character a number (its code).
ASCII
American Standard Code for Information Interchange: a 7-bit character set with 128 codes (0 to 127) for English letters, digits, punctuation and control codes.
Unicode
The worldwide character set used by modern computers. It gives a code point to every character of almost every writing system, plus symbols and emoji.
code point
The number Unicode assigns to a character, written like U+0041 (hexadecimal) for A.
UTF-8
The most common way to store Unicode code points as bytes. It uses 1 byte for ASCII characters and 2 to 4 bytes for others.
Letters are numbers too
In the last lesson you saw that a computer stores only bits. So how does it store the word Bonjour, a French é, or a 🍁 emoji? The answer: every character is given a number, and the number is stored in binary.
The list that matches characters to numbers is called a character set. If two computers use the same character set, a message typed on one appears correctly on the other.
ASCII
ASCII was published in the 1960s. It uses 7 bits, so it has 2⁷ = 128 codes (0 to 127):
- codes 0 to 31 (and 127) are invisible control codes, such as new line (10) and tab (9);
- 32 is the space;
- 48 to 57 are the digits
0 to 9; - 65 to 90 are the capitals
A to Z; - 97 to 122 are the lower-case letters
a to z.
Two patterns are worth remembering. The letters and digits are in order, and each lower-case letter is exactly 32 more than its capital (32 is 2⁵, a single bit). Also notice that the character "7" (code 55) is not the number 7.
ord("C") is 2 more than ord("A"), and every lower-case letter is exactly 32 more than its capital.ord() and chr() in Python
ord("A") gives the code of a character: 65.chr(65) gives the character with that code: "A".
They are opposites: chr(ord(c)) is always c. Because the codes are numbers, you can do arithmetic with them, then turn the result back into a character.
print(ord("A"), ord("a"), ord("0"), ord(" "))
print(chr(72) + chr(105))
print(chr(ord("C") + 3))
print(chr(ord("g") - 32))
Line 3 moves 3 letters forward from C. Line 4 turns a lower-case letter into a capital by subtracting 32. Output, line by line: 65 97 48 32 | Hi | F | G
Why "Zebra" comes before "apple"
When Python compares strings (Lesson 8), it compares character codes, one character at a time. "Z" is 90 and "a" is 97, so "Zebra" < "apple" is True: in code order, every capital comes before every lower-case letter. To sort words the way a dictionary does, programs convert them to the same case first.
Unicode: every writing system
ASCII has no é, no ç, no Chinese or Arabic characters, no syllabics and no emoji. In Canada alone, French needs accented letters, and First Nations and Inuit languages such as Cree and Inuktitut are often written with syllabics. Before Unicode, different countries used different, incompatible character sets, and text often turned into nonsense when it moved between computers.
Unicode fixes this with one huge character set for the whole world. It has well over 100 000 characters and is still growing. Each character has a code point, written U+ followed by hexadecimal digits:
A is U+0041 (65, the same as ASCII: Unicode's first 128 code points are ASCII);é is U+00E9 (233);- the syllabics character
ᐃ is U+1403 (5123). The Inuktitut word for the language, ᐃᓄᒃᑎᑐᑦ (Inuktitut), is 6 characters from Unicode's Unified Canadian Aboriginal Syllabics block, the official name of that block; 🍁 (MAPLE LEAF) is U+1F341 (127 809).
Python 3 strings are Unicode, so ord("é") is 233 and len("ᐃᓄᒃᑎᑐᑦ") is 6.
Storing code points: UTF-8
A code point can be as large as about 1.1 million, which needs 21 bits. Storing every character in 4 bytes would waste space for English text. UTF-8, the encoding used by most web pages and files, uses a variable number of bytes:
- 1 byte for code points 0 to 127 (so plain ASCII text is unchanged);
- 2 bytes for most accented Latin letters, Greek, Cyrillic, Arabic and Hebrew;
- 3 bytes for most other characters still in common use, including syllabics and Chinese;
- 4 bytes for emoji and rarer characters.
In Python, "é".encode("utf-8") shows the actual bytes.
word = "café"
print(len(word))
print(len(word.encode("utf-8")))
print(len("🍁".encode("utf-8")))
café is 4 characters but 5 bytes in UTF-8, because é needs 2 bytes. The maple leaf needs 4 bytes. Output, line by line: 4 | 5 | 4
Worked exampleExample 1: Encoding a word
Show how the string "Hi!" is stored in ASCII, in decimal and in 8-bit binary.
- Look up each code (or use
ord): H = 72, i = 105, ! = 33. - 72 = 64 + 8 → 01001000.
- 105 = 64 + 32 + 8 + 1 → 01101001.
- 33 = 32 + 1 → 00100001.
- The string takes 3 bytes, stored one after another.
Hi! is stored as 72, 105, 33, which is 01001000 01101001 00100001.
Worked exampleExample 2: A one-letter Caesar shift
A Caesar cipher replaces each letter with the letter a fixed number of places later in the alphabet, wrapping from Z back to A. Write code that shifts the capital letter in letter by shift places.
- The capitals have codes 65 to 90. First turn the letter into a position 0 to 25:
ord(letter) - 65. - Add the shift and use
% 26 so that 26 wraps to 0, 27 to 1, and so on. - Turn the position back into a code by adding 65, then use
chr. - One line does it all:
new_letter = chr((ord(letter) - 65 + shift) % 26 + 65). - Test with
letter = "X" and shift = 5: ord is 88, 88 − 65 = 23, 23 + 5 = 28, 28 % 26 = 2, 2 + 65 = 67, which is C.
chr((ord(letter) - 65 + shift) % 26 + 65); X shifted by 5 is C. (To encode a whole message you need a loop, which you will learn in Lesson 12.)
letter = "X"
shift = 5
new_letter = chr((ord(letter) - 65 + shift) % 26 + 65)
print(new_letter)
Example 2 as a program. Try other letters and shifts. Output: C
Just for funUnicode really does have a character named SNOWMAN (☃, U+2603), and a separate one named SNOWMAN WITHOUT SNOW (⛄, U+26C4). Canada approves.
Hands-on activitySecret codes at the kitchen table
You need: paper, a pencil, a family member or friend, and Python.
- Write a short message (3 to 6 letters, capitals only) as a list of ASCII codes, for example
76 69 65 70 83. Use 65 for A and count up. - Give the codes to your partner and have them decode it with the alphabet. Then swap roles.
- In Python, check your codes with
ord() and decode your partner's message with chr(). - Now write the same message in Caesar code with a shift of 3 (A → D, X → A). Use the one-line formula from Example 2 to check two of the letters.
- Find two French words with accents (for example été and français). Use
len(word) and len(word.encode("utf-8")). Why are the numbers different?
What to look for: 76 69 65 70 83 decodes to LEAFS. été is 3 characters but 5 bytes (each é takes 2 bytes); français is 8 characters but 9 bytes (ç takes 2 bytes).
Your turn
1What is ord("B")? What is chr(100)?
2Decode the ASCII codes 79 78 84 65 82 73 79.
3What is ord("m") - ord("M")? Why?
32. Every lower-case letter's code is 32 more than its capital's code.
4Is "5" the same as 5 inside the computer? Explain using codes.
No. The character "5" is stored as its code, 53, while the integer 5 is stored as the number 5 (101 in binary).
5How many characters can ASCII represent, and why that number?
128, because ASCII uses 7 bits and 2⁷ = 128.
6Give two reasons why Unicode replaced ASCII.
Sample: ASCII has no accented letters needed for French, no syllabics, no characters for most of the world's languages and no emoji. Unicode gives every character one agreed code, so text displays correctly on any computer.
7What does print(chr(ord("a") + 25)) display?
z. 97 + 25 = 122, the code for z.
More practice: warm-up
1What function gives the code of a character?
2What function gives the character for a code?
3What is the ASCII code for a space?
4How is the code point of A written in Unicode notation?
5How many bytes does UTF-8 use for the letter k?
1 byte (it is an ASCII character).
More practice: on your own
1Write ASCII codes in decimal for Go 2.
G = 71, o = 111, space = 32, 2 = 50: 71 111 32 50.
2Write the binary for the character C (8 bits).
C = 67 = 64 + 2 + 1 = 01000011.
3Write one line of Python that turns the capital letter in c into lower case using ord and chr.
lower = chr(ord(c) + 32). (Python also has a built-in way, which you will meet in Arrays and Subprograms.)
4What does print("Zoo" < "apple") display? Explain.
True. Strings are compared by character codes; Z is 90 and a is 97, so "Zoo" comes first.
5Shift the letter W forward by 6 places in a Caesar cipher (wrap around). Show the calculation.
ord("W") − 65 = 22; 22 + 6 = 28; 28 % 26 = 2; 2 + 65 = 67 → C.
Challenge
1A text file contains 1000 characters of plain English plus 10 emoji, saved as UTF-8. How many bytes is it (ignore anything else)?
1000 ASCII characters × 1 byte = 1000 bytes, plus 10 emoji × 4 bytes = 40 bytes. 1040 bytes.
2The digit characters "0" to "9" have codes 48 to 57. Write one expression that turns a digit character d into its integer value without using int().
ord(d) - 48 (or ord(d) - ord("0")). For d = "7": 55 − 48 = 7.
3Why is UTF-8 designed so that ASCII characters take only 1 byte, instead of giving every character 4 bytes?
Much of the world's stored text, including program code and web page markup, is mostly ASCII. With 1 byte for those characters, files are up to 4 times smaller than with a fixed 4 bytes, and old ASCII files are already valid UTF-8, so nothing had to be converted.
Quiz
3. How many codes does ASCII have?
4. Which statement about Unicode is true?
5. How many bytes does the string "été" take in UTF-8?
6. What does print(chr(ord("b") - 32)) show?
Big idea: this lesson completes expectation A1.2: how a computer uses ASCII and Unicode (with UTF-8) to represent text internally.
How to help: play the secret-code game together. If your family speaks or reads a language other than English, ask your teen to look up the code point of a character from it with ord().
Common difficulties:
- Confusing the character
"5" (code 53) with the number 5. - Forgetting that capitals and lower-case letters have different codes.
- Mixing up characters and bytes:
len() of a string counts characters, while the UTF-8 encoding may need more bytes. - When doing a Caesar shift by hand, forgetting to wrap from Z back to A.