{ "cells": [ { "cell_type": "markdown", "id": "f679c306-33d5-4a2d-b32f-8f959fedaff8", "metadata": {}, "source": [ "(sec-encodings-and-unicode)=\n", "# Encodings and Unicode" ] }, { "cell_type": "markdown", "id": "3c6a21fe-c5ac-4930-9d24-8b805de25a4b", "metadata": {}, "source": [ "All data, whether numerical, text, images, or some other data type must be stored as ones and zeroes, and encodings are the conventions to make sure everyone agrees how to store and how to interpret strings of bytes. \n", "\n", "**Unicode** is the most broadly applicable encoding for texts in all human languages. Even if you don't use text as data, metadata usually includes text, and you may need to debug issues with getting text data into Python. " ] }, { "cell_type": "markdown", "id": "2d78505e-c7f8-4811-93be-7935697a8316", "metadata": {}, "source": [ ":::{admonition} Note\n", "***bits*** are the smallest units of information storage; single devices that can be in one of two states, conventionally called 0 and 1.\n", "\n", "A ***byte*** is a collection of 8 bits that can be interpreted as a number between 0 and 255.\n", "\n", "***hexadecimal notation*** is a way of writing numbers stored in bytes using the symbols 0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,and F to stand for the numbers 0-15. Bytes are written as two-character symbols, where the first character counts 16s and the final character counts ones:\n", "\n", " 100 = 6*16 + 4 = 0x64\n", " 127 = 7*16 + 25 = 0x7F\n", " 128 = 8*16 + 0 = 0x80\n", " 250 = 15*16 + 10 = 0xFA\n", "\n", "The \"0x\" is a convention to remind you that 0x64 is written in hexadecimal." ] }, { "cell_type": "markdown", "id": "b6a9edd4-fcae-4900-a00f-0bbd27b68182", "metadata": {}, "source": [ "## ASCII - the first 128 code points" ] }, { "cell_type": "markdown", "id": "88f2904a-036e-4ea9-8696-d1708a2d6aec", "metadata": {}, "source": [ "ASCII is a long-standing standard (published by the American Standards Association in 1963) that assigns bytes with number values between 0 and 127 to letters in the basic Latin alphabet, punctuation marks, and a handful of control characters, including TAB (escaped in Python as `\\t`), carriage return CR (escaped in Python as `\\r`) and line feed LF (escaped in Python as `\\n`). \n", "\n", "These 128 symbols have the widest support and will usually cause no problems. " ] }, { "cell_type": "raw", "id": "23ffe398-6a28-4745-87e4-65f9063751c0", "metadata": {}, "source": [ "Dec Char Dec Char Dec Char Dec Char\n", "--------- --------- --------- ----------\n", " 0 NUL (null) 32 SPACE 64 @ 96 `\n", " 1 SOH (start of heading) 33 ! 65 A 97 a\n", " 2 STX (start of text) 34 \" 66 B 98 b\n", " 3 ETX (end of text) 35 # 67 C 99 c\n", " 4 EOT (end of transmission) 36 $ 68 D 100 d\n", " 5 ENQ (enquiry) 37 % 69 E 101 e\n", " 6 ACK (acknowledge) 38 & 70 F 102 f\n", " 7 BEL (bell) 39 ' 71 G 103 g\n", " 8 BS (backspace) 40 ( 72 H 104 h\n", " 9 TAB (horizontal tab) 41 ) 73 I 105 i\n", " 10 LF (NL line feed, new line) 42 * 74 J 106 j\n", " 11 VT (vertical tab) 43 + 75 K 107 k\n", " 12 FF (NP form feed, new page) 44 , 76 L 108 l\n", " 13 CR (carriage return) 45 - 77 M 109 m\n", " 14 SO (shift out) 46 . 78 N 110 n\n", " 15 SI (shift in) 47 / 79 O 111 o\n", " 16 DLE (data link escape) 48 0 80 P 112 p\n", " 17 DC1 (device control 1) 49 1 81 Q 113 q\n", " 18 DC2 (device control 2) 50 2 82 R 114 r\n", " 19 DC3 (device control 3) 51 3 83 S 115 s\n", " 20 DC4 (device control 4) 52 4 84 T 116 t\n", " 21 NAK (negative acknowledge) 53 5 85 U 117 u\n", " 22 SYN (synchronous idle) 54 6 86 V 118 v\n", " 23 ETB (end of trans. block) 55 7 87 W 119 w\n", " 24 CAN (cancel) 56 8 88 X 120 x\n", " 25 EM (end of medium) 57 9 89 Y 121 y\n", " 26 SUB (substitute) 58 : 90 Z 122 z\n", " 27 ESC (escape) 59 ; 91 [ 123 {\n", " 28 FS (file separator) 60 < 92 \\ 124 |\n", " 29 GS (group separator) 61 = 93 ] 125 }\n", " 30 RS (record separator) 62 > 94 ^ 126 ~\n", " 31 US (unit separator) 63 ? 95 _ 127 DEL" ] }, { "cell_type": "markdown", "id": "8245a42c-9ddd-4d60-a22b-84efd1bb651a", "metadata": {}, "source": [ "The symbols come in a conventional order:\n", "\n", " * control characters like newline and tab,\n", " * some punctuation marks, including \" ' + and - \n", " * Arabic numerals,\n", " * more punctuation,including < = ? and @, \n", " * the upper-case Latin letters,\n", " * more punctuation, including [ \\ and _, \n", " * lower-case Latin letters, and finally, \n", " * final four punctuation marks { | } and ~ and a non-printing character DEL. \n", " \n", "This order controls string comparisons and string sorting operations by default. If you have ever sorted a list (such as a list of filenames) and found all the files beginning with capital letters appear before all of the files beginning with lower case letters, this sorting order is the reason. " ] }, { "cell_type": "markdown", "id": "b7c176b6-813d-42ec-a7ca-c6b0c6070d17", "metadata": {}, "source": [ "This gives us about a hundred printable characters, corresponding to the keys on an English-language keyboard. This is somewhat limiting. Bytes have 256 possible values, why not use values 128-255 for something?" ] }, { "cell_type": "markdown", "id": "c0c548aa-b91e-43d9-9b9b-6d39908d3a87", "metadata": {}, "source": [ "## Latin-1 encoding - the rest of the characters?" ] }, { "cell_type": "markdown", "id": "1d6aa776-2dd5-4dbb-988f-3922c115cf11", "metadata": {}, "source": [ "There are a handful of (now obsolete) encodings that assign the remaining byte values (128-255) to characters. The **Latin-1 character encoding**, also known as **ISO-8859**, is one such, developed for European languages using Latin alphabets. It provides another hundred or so characters: " ] }, { "cell_type": "raw", "id": "be6725a7-3646-48e9-b35c-9cef93982229", "metadata": {}, "source": [ "Latin-1 (ISO-8859) encodings\n", "Dec Char Name Dec Char Name \n", "--- ---- -------------------------------------------- --- ---- ----------------------------- \n", "128 PAD padding character 192 À capital a with grave accent \n", "129 HOP high octet preset 193 Á capital a with acute accent \n", "130 BPH break permitted here 194  capital a with circumflex \n", "131 NBH no break here 195 à capital a with tilde \n", "132 IND index 196 Ä capital a with diaeresis \n", "133 NEL next line 197 Å capital a with ring \n", "134 SSA start of selected area 198 Æ capital ae ligature \n", "135 ESA end of selected area 199 Ç capital c with cedilla \n", "136 HTS character tabulation set 200 È capital e with grave accent \n", "137 HTJ character tabulation with justification 201 É capital e with acute accent \n", "138 VTS line tabulation set 202 Ê capital e with circumflex \n", "139 PLD partial line forward 203 Ë capital e with diaeresis \n", "140 PLU partial line backward 204 Ì capital i with grave accent \n", "141 RI reverse line feed 205 Í capital i with acute accent \n", "142 SS2 single shift two 206 Î capital i with circumflex \n", "143 SS3 single shift three 207 Ï capital i with diaeresis \n", "144 DCS device control string 208 Ð capital eth \n", "145 PU1 private use one 209 Ñ capital n with tilde \n", "146 PU2 private use two 210 Ò capital o with grave accent \n", "147 STS set transmit state 211 Ó capital o with acute accent \n", "148 CCH cancel character 212 Ô capital o with circumflex \n", "149 MW message waiting 213 { capital o with tilde \n", "150 SPA start of guarded area 214 Ö capital o with diaeresis \n", "151 EPA end of guarded area 215 × multiplication sign \n", "152 SOS start of string 216 Ø capital o with slash \n", "153 SGCI single graphic character introducer 217 Ù capital u with grave accent \n", "154 SCI single character introducer 218 Ú capital u with acute accent \n", "155 CSI control sequence introducer 219 Û capital u with circumflex \n", "156 ST string terminator 220 Ü capital u with diaeresis \n", "157 OSC operating system command 221 Ý capital y with acute accent \n", "158 PM privacy message 222 Þ capital thorn \n", "159 APC application program command 223 ß small sharp s \n", "160 NBS non-breaking space 224 à small a with grave accent \n", "161 ¡ inverted exclamation mark 225 á small a with acute accent \n", "162 ¢ cent sign 226 â small a with circumflex \n", "163 £ pound sterling sign 227 ã small a with tilde \n", "164 ¤ currency sign 228 ä small a with diaeresis \n", "165 ¥ yen sign 229 å small a with ring \n", "166 ¦ broken bar 230 æ small ae ligature \n", "167 § section sign 231 ç small c with cedilla \n", "168 ¨ diaeresis (umlaut) 232 è small e with grave accent \n", "169 © copyright sign 233 é small e with acute accent \n", "170 ª feminine ordinal 234 ê small e with circumflex \n", "171 « left angle quote 235 ë small e with diaeresis \n", "172 ¬ not sign 236 ì small i with grave accent \n", "173 ­ soft hyphen 237 í small i with acute accent \n", "174 ® registered sign 238 î small i with circumflex \n", "175 ¯ macron 239 ï small i with diaeresis \n", "176 ° degree sign 240 ð small eth \n", "177 ± plus-minus sign 241 ñ small n with tilde \n", "178 ² superscript two 242 ò small o with grave accent \n", "179 ³ superscript three 243 ó small o with acute accent \n", "180 ´ acute accent 244 ô small o with circumflex \n", "181 µ micro sign 245 õ small o with tilde \n", "182 ¶ paragraph sign (pilcrow) 246 ö small o with diaeresis \n", "183 · middle dot 247 ÷ division sign \n", "184 ¸ cedilla 248 ø small o with slash \n", "185 ¹ superscript one 249 ù small u with grave accent \n", "186 º masculine ordinal 250 ú small u with acute accent \n", "187 » right angle quote 251 û small u with circumflex \n", "188 ¼ one-fourth fraction 252 ü small u with diaeresis \n", "189 ½ one-half fraction 253 ý small y with acute accent \n", "190 ¾ three-quarter fraction 254 þ small thorn \n", "191 ¿ inverted question mark 255 ÿ small y with diaeresis " ] }, { "cell_type": "markdown", "id": "43183f95-d0cc-4278-beda-3a9538107b9e", "metadata": {}, "source": [ "Can I just make a string with four characters, with values `\\x43`, `\\x61`, `\\x66`, and `\\xE9` (corresponding to C, a, f, and é) ? (`\\xNN` is a Python escaping syntax for bytes by hexadecimal value NN). \n", "\n", "Yes. This works, and gives me a four-character Python bytestring that I need to remember is encoded using Latin-1 encoding. " ] }, { "cell_type": "code", "execution_count": 1, "id": "addda68f-7ea5-4e1d-adc9-427b48312913", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "4\n" ] } ], "source": [ "cafe_string_latin1 = b'\\x43\\x61\\x66\\xe9'\n", "print(type(cafe_string_latin1))\n", "print(len(cafe_string_latin1))" ] }, { "cell_type": "code", "execution_count": 2, "id": "8934db9b-660d-4bdb-8192-661cfd70ea51", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "b'Caf\\xe9'\n" ] } ], "source": [ "print(cafe_string_latin1)" ] }, { "cell_type": "raw", "id": "febbd998-0618-42d0-bd23-99c080743370", "metadata": {}, "source": [ " C a f é \n", "0x43 61 66 E9 \n", " 0 1 2 3 " ] }, { "cell_type": "markdown", "id": "76a170c1-217a-46b3-a727-75c8d0bc02af", "metadata": {}, "source": [ "This string doesn't print nicely; I have to decode the latin1 encoding to print it nicely:" ] }, { "cell_type": "code", "execution_count": 3, "id": "e23208ba-614e-4f44-b462-273c2a74daa4", "metadata": {}, "outputs": [], "source": [ "cafe_string = cafe_string_latin1.decode(\"latin-1\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "de05a0df-07f7-435b-bb18-0e3287748e65", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Café\n" ] } ], "source": [ "print(cafe_string)" ] }, { "cell_type": "markdown", "id": "ace11afc-bbbe-4096-8535-b0ff3438a48e", "metadata": {}, "source": [ "This has converted a \"bytes\" datatype into a [unicode] \"string\" datatype." ] }, { "cell_type": "code", "execution_count": 5, "id": "7d05eff7-791a-413a-a15a-99485664fcd2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "4\n" ] } ], "source": [ "print(type(cafe_string ))\n", "print(len(cafe_string ))" ] }, { "cell_type": "markdown", "id": "9e5f8007-78dd-4f36-a533-ce0c668e8037", "metadata": {}, "source": [ "## Unicode!" ] }, { "cell_type": "markdown", "id": "be52ccef-ae75-4895-ada9-415fbc4a858b", "metadata": {}, "source": [ "Unicode is a system for encoding every human language, unifying computer representations of disparate languages. \n", "The unicode standard defines **code points** or **characters** that include every letter or symbol you could want in essentially every alphabet. [Unicode version 15](https://unicode.org/versions/Unicode15.0.0/) contains definitions for 149,000 characters, including 11,000 Hangul syllables, 98,000 Chinese characters, and a few thousand emojis. Unicode code points are written by convention as U + four-digit hexadecimal numbers.\n", "\n", "Modern operating systems, keyboards, and browsers can render and use unicode in a wide variety of contexts; in the 21st century, you can create a file called 论文最终版.tex and expect it not to kill your thesis-building tools or your hard drive. " ] }, { "cell_type": "markdown", "id": "40b3b007-e6a9-400b-847c-8c7edf83d101", "metadata": {}, "source": [ "Python uses unicode to store strings, and the word \"string\" without qualification means a string that is permitted to contain any unicode sequence." ] }, { "cell_type": "markdown", "id": "ed5e0ce7-f27a-4cec-ad31-e178bfd067fe", "metadata": {}, "source": [ "The word Café has four letters, and can be written as the sequence of four unicode symbols:\n", "\n", " C U+0043 LATIN CAPITAL LETTER C\n", " a U+0061 LATIN SMALL LETTER A\n", " f U+0066 LATIN SMALL LETTER F\n", " é U+00E9 LATIN SMALL LETTER E WITH ACUTE" ] }, { "cell_type": "markdown", "id": "7913240b-d18e-44bc-8eb2-cb40873c6d0d", "metadata": {}, "source": [ "Can I just make a string with four characters, with values \\u0043, \\u0061, \\u0066, and \\u00E9? Yes. This works, and gives me a four-character Python (unicode) string:" ] }, { "cell_type": "code", "execution_count": 6, "id": "222db58c-0963-4f00-a41b-eb4f02769634", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Café\n" ] } ], "source": [ "cafe_string = '\\u0043\\u0061\\u0066\\u00E9'\n", "print(cafe_string)" ] }, { "cell_type": "markdown", "id": "91c11a03-022a-41ef-b31d-84a6298f0afa", "metadata": {}, "source": [ "But I can just write the accented character in my source code and it creates the same four-character string, so why am I worrying?" ] }, { "cell_type": "code", "execution_count": 7, "id": "160cb437-27fc-4136-8aa9-61902ac398a3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "print(cafe_string == \"Café\")" ] }, { "cell_type": "markdown", "id": "a86a04aa-2647-404e-9f13-6658796d978f", "metadata": {}, "source": [ "## Encoding/decoding" ] }, { "cell_type": "markdown", "id": "6b75a287-ea5a-4729-b8fb-4ae38200b4d0", "metadata": {}, "source": [ "When it is time to save this string in a datafile (or save my source code) the unicode symbols are stored as bytes. \n", "\n", "Although any content can be written as a sequence of unicode symbols, the way that unicode symbols are encoded in digital storage is not entirely trivial. The unicode symbols, called ***codepoints*** are the truth; the sequence of bytes that indicates a particular unicode symbol is the **encoding**. The most popular encoding is UTF-8, an encoding which uses between 1 and 4 bytes per codepoint, depending on the codepoint.\n", "\n", "Python enforces a difference in data types between data that are interpreted as a sequence of bytes (`bytes`) **without an encoding** and sequences of characters **where the encoding has been specified and decoded** (called `string`s). \n", "\n", "This shows us the five bytes that would be written in the utf-8 encoding:" ] }, { "cell_type": "code", "execution_count": 8, "id": "dc9cd8ec-8328-4734-97fc-d8a834c788a6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "b'Caf\\xc3\\xa9'\n" ] } ], "source": [ "print(bytes(cafe_string.encode(\"utf8\")))" ] }, { "cell_type": "raw", "id": "770fa0b4-7a4c-47e6-b6bd-46101f3584a0", "metadata": {}, "source": [ " C a f é - \n", "0x43 61 66 C3 A9\n", " 0 1 2 3 4" ] }, { "cell_type": "markdown", "id": "e840a284-6e94-466d-be1d-6cbbaf34204d", "metadata": {}, "source": [ "UTF-8 encodes the unicode codepoint \\u00E9 as the two bytes \\xC3 and \\xA9. This makes our encoded bytestring 5 characters long. (If you ever discover that the Google Translate API fails for submissions that are only 4950 unicode characters but more than 5000 bytes you will find that `len()` has a different behavior on string datatypes (where it counts characters) and bytes datatypes (where it counts bytes)." ] }, { "cell_type": "markdown", "id": "a4d9bf04-8e02-4ee6-85bf-4681b735f07c", "metadata": {}, "source": [ "\"UTF-8 uses the following rules:\n", "* If the code point is < 128, it’s represented by the corresponding byte value.\n", "* If the code point is >= 128, it’s turned into a sequence of two, three, or four bytes, where each byte of the sequence is between 128 and 255.\"\n", "[Python Unicode howto](https://docs.python.org/3/howto/unicode.html)" ] }, { "cell_type": "markdown", "id": "54b8fbf4-d04e-4f68-a8ea-f0abe15bdf0f", "metadata": {}, "source": [ "\"The value of each individual byte indicates its UTF-8 function, as follows:\n", "\n", " 00 to 7F hex (0 to 127): first and only byte of a sequence.\n", " 80 to BF hex (128 to 191): continuing byte in a multi-byte sequence.\n", " C2 to DF hex (194 to 223): first byte of a two-byte sequence.\n", " E0 to EF hex (224 to 239): first byte of a three-byte sequence.\n", " F0 to FF hex (240 to 255): first byte of a four-byte sequence.\" \n", " \n", "(https://www.fileformat.info/info/unicode/utf8.htm)" ] }, { "cell_type": "markdown", "id": "c825c38a-46ff-4185-aef4-1a41fe7a4900", "metadata": {}, "source": [ "A consequence of this variable-length encoding is that there are sequences of bytes that are invalid unicode encodings: sequences of random bytes will often have bytes that are not valid as the first byte of a multi-byte sequence, or have continuing bytes that do not have the right values and cannot be interpreted as valid code points. If Python is asked to decode bytes into unicode (common when reading files or downloading content from the internet) and finds a problem, you get a `UnicodeDecodeError`. This is the most common kind of difficulty you will have with unicode: converting to and from utf8 when needed, and occasionally converting from formats other than utf-8." ] }, { "cell_type": "markdown", "id": "dddd93d7-6026-42cc-a780-4281554be250", "metadata": {}, "source": [ "The Latin-1 encoding has the virtue that it will interpret every byte (and every sequence of bytes) as a valid string; there are no \"rules\" that can be violated when every byte is a valid symbol. If you encounter a `UnicodeDecodeError` on some data, as a first attempt at debugging, try decoding the data using latin-1. If the data is not obviously corrupted, it is likely the input data was not encoded in utf-8 to start with." ] }, { "cell_type": "markdown", "id": "0e6f5844-ee6d-4dc0-9724-f1c6b0e8bc23", "metadata": {}, "source": [ "If we try to decode latin-1 data as utf-8, we get a `UnicodeDecodeError`: " ] }, { "cell_type": "code", "execution_count": 9, "id": "b103d308-5155-46cd-beb2-e3822eb33a1a", "metadata": { "tags": [ "raises-exception" ] }, "outputs": [ { "ename": "UnicodeDecodeError", "evalue": "'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mUnicodeDecodeError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[9], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mcafe_string_latin1\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mutf8\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n", "\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data" ] } ], "source": [ "cafe_string_latin1.decode(\"utf8\")" ] }, { "cell_type": "markdown", "id": "ef979ac4-c650-41bc-ad8d-b7fb5812ad02", "metadata": {}, "source": [ "Decoding latin-1 bytestrings correctly as latin-1 turns them back into Python strings:" ] }, { "cell_type": "code", "execution_count": 10, "id": "632e6276-b245-42ca-b8a3-cdf36d0af481", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Café'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cafe_string_latin1.decode(\"latin-1\")" ] }, { "cell_type": "markdown", "id": "5717052d-9974-49c8-b871-1f6dc1ff133d", "metadata": {}, "source": [ "And this matches our earlier Python string that was born in unicode." ] }, { "cell_type": "code", "execution_count": 11, "id": "a3b12fde-a9c0-480e-bb29-91766ffbc9ab", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cafe_string_latin1.decode(\"latin-1\") == cafe_string" ] }, { "cell_type": "markdown", "id": "63005b03-5e43-4284-b285-903f64c8a398", "metadata": {}, "source": [ "Making the converse mistake, decoding UTF-8 strings with the latin-1 decoder **does not generate an error, but instead corrupts the data**:" ] }, { "cell_type": "code", "execution_count": 12, "id": "2a06545b-bc06-4afb-822c-7a8189005cde", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Café\n" ] } ], "source": [ "print ( cafe_string.encode(\"utf-8\").decode(\"latin-1\")) " ] }, { "cell_type": "markdown", "id": "9c7d746e-338f-4a17-921f-2729f022c976", "metadata": {}, "source": [ "This is not what we want. The symbols é is a technical detail of how the unicode symbol U+003E9 is stored; it is not intended to be displayed without being decoded.\n", "\n", "If you have ever seen content where accented Latin characters or punctuation marks are replaced with multiple, nonsensical accented \n", "characters, the cause is usually failing to decode utf-8-encoded text as utf-8. " ] }, { "cell_type": "markdown", "id": "ad211d56-e43e-4180-be07-8b99833ab8b5", "metadata": {}, "source": [ "Another place where you are likely to encounter unicode decoding is ingesting data from the web. The results from HTTP queries will often require explicit `.decode(\"utf-8\")`. \n", "\n", "The Python documentation for when encoding / decoding is required, should you need it, is here:\n", "[Python Unicode Howto](https://docs.python.org/3/howto/unicode.html)\n" ] }, { "cell_type": "code", "execution_count": 14, "id": "01775316-866d-4fca-9eae-b171766f3c9f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "b'\\xe5\\x8d\\x81\\xe5\\x85\\xad\\xe8\\xbf\\x9b\\xe5\\x88\\xb6'" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Chinese characters usually require three bytes each in utf-8:\n", "content = \"十六进制\"\n", "content.encode(\"utf-8\")" ] }, { "cell_type": "code", "execution_count": 15, "id": "6010bf17-8cdb-4818-a00f-ef7e8e1c2f44", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "十六进制 让我头疼\n" ] } ], "source": [ "# But there is no reason to worry too much.\n", "print (content, \"让我头疼\")" ] }, { "cell_type": "code", "execution_count": null, "id": "0bcfed32-f694-46b1-a1f8-fa0bfb4863dd", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }