{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "f36707ce-48d4-4b4b-9a27-1d7be5658de3", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "markdown", "id": "155bd5ae-94a7-4fa9-8b84-c037b728a662", "metadata": {}, "source": [ "# Reduced alphabet similarity" ] }, { "cell_type": "markdown", "id": "d9f1e5a3-393a-4753-a3dc-ad29befdbcd6", "metadata": {}, "source": [ "Since exact matches are cheap, one might wonder if there is a way to use a lossy encoding on the data and test for exact matches of an inexact encoding. This is the way of restricted alphabets." ] }, { "cell_type": "markdown", "id": "9a31a2ae-9218-47ef-b928-25a8889ffdfc", "metadata": {}, "source": [ "## Soundex\n", "\n", "The Soundex algorithm is a procedure for encoding family names developed by the U.S. Census bureau. It is described in patents 1,261,167 [^*] and 1,435,663[^**]. The Works Progress Administration hired thousands of people to create indexes to the 1920, 1900, and 1880 U.S. Decennial censuses during the 1930s, copying some of the information onto cards that were sorted by Soundex code and transferred to microfilm.[^***]\n", "\n", "\n", "Soundex constructs a four-character symbol (one initial letter and three numbers) for each name. This symbol represents only some of the essential letters and sounds in the name. This dramatically reduces the number of (distinct) objects in the index, giving names with similar sounds (or misspellings) the same symbol." ] }, { "cell_type": "markdown", "id": "706f396f-19e6-4e34-b7f4-00e64ff46f03", "metadata": {}, "source": [ "### The algorithm" ] }, { "cell_type": "markdown", "id": "007331e1-9ef2-48bf-8571-fdb07ffd5cd5", "metadata": {}, "source": [ "The algorithm is as follows:\n", "* The first letter of the name is retained as the first letter of the code.\n", "* Double letters and letters with identical numeric codes are combined.\n", "* If two consonants are separated by a vowel, both consonants are encoded; if two consonants are separated by a H or W and have the same Soundex number, they are combined.\n", "* The first three consonants in the name are replaced by numbers, excepting final S and Z; if there are fewer than three, add zeros." ] }, { "cell_type": "raw", "id": "85c044fb-26b6-4607-aafe-c091fbf805a9", "metadata": {}, "source": [ "ignore A,E,I,O,U,Y\n", "1 B,F,P,V\n", "2 C,G,K,Q,X,S,Z\n", "3 D,T\n", "4 L\n", "5 M,N\n", "6 R" ] }, { "cell_type": "markdown", "id": "8b88edf0-089e-4447-aae7-88f907a66f1b", "metadata": {}, "source": [ "If we work through the algorithm, we can encode a handful of names:\n", "\n", "CAMPBELL: The first letter is C, \n", "* second consonant is M which encodes as 5\n", "* the next consonants P and B both encode to the same code (1), so are combined.\n", "* the third consonant to encode is L (4)\n", "* So CAMPBELL encodes as C514.\n", "\n", "NICOLAE: The first letter is N,\n", "* the second consonant is C which codes as 2,\n", "* the next consonant is L which codes as 4\n", "* and we have run out of consonants, so we code NICOLAE as N24 or N240." ] }, { "cell_type": "markdown", "id": "10fa7eb3-bd9b-4cfe-a5f3-4da47e7f0e47", "metadata": {}, "source": [ "### Installing a library to calculate Soundex" ] }, { "cell_type": "markdown", "id": "f50e8f84-48cf-4914-beed-140b4ab0002f", "metadata": {}, "source": [ "Santhosh Thottingal has written an implementation of Soundex that we can use since it's licensed under the GPL: https://github.com/libindic/soundex" ] }, { "cell_type": "code", "execution_count": 2, "id": "11c6fb3c-d901-4c66-8cb0-11196ad84b3b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: soundex in /opt/conda/lib/python3.11/site-packages (1.1.3)\n", "Requirement already satisfied: silpa-common>=0.3 in /opt/conda/lib/python3.11/site-packages (from soundex) (0.3)\n" ] } ], "source": [ "%pip install soundex;" ] }, { "cell_type": "code", "execution_count": 3, "id": "9285812d-57c4-46e0-bfc3-c7d505a16504", "metadata": {}, "outputs": [], "source": [ "from soundex import Soundex\n", "soundex = Soundex().soundex" ] }, { "cell_type": "markdown", "id": "fdd8ca6a-35bb-410c-80cd-1798c956f150", "metadata": {}, "source": [ "And we can check that our answers above match Thottingal's implementation:" ] }, { "cell_type": "code", "execution_count": 4, "id": "1ed8eb49-fa5d-427f-b693-10f70378c3b0", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('C514', 'N24')" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soundex(\"CAMPBELL\") , soundex(\"NICOLAE\") " ] }, { "cell_type": "markdown", "id": "9d142b02-2bc2-4fb5-abb9-7a62eb94cd63", "metadata": {}, "source": [ "We get the same answer for these two names." ] }, { "cell_type": "markdown", "id": "fed780ff-fd3f-45b4-84e8-a83ab3540a33", "metadata": {}, "source": [ "### Applying Soundex to a list of names" ] }, { "cell_type": "markdown", "id": "5a646d00-e340-48c1-9ce9-a38cce68ba4d", "metadata": {}, "source": [ "To get a sense of what Soundex does to names, let us calculate the Soundex codes for the 162k most common surnames from the 2010 US Decennial Census[^****]. Each of these names occurred more than 100 times in the 2010 US Decennial census, and together these names represent about 90% of the people enumerated." ] }, { "cell_type": "code", "execution_count": 5, "id": "17e11da2-82eb-49cc-9aba-eeacb99b851d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namerankcountprop100kcum_prop100kpctwhitepctblackpctapipctaianpct2pracepcthispanic
0SMITH12442977828.19828.1970.923.110.50.892.192.4
1JOHNSON21932812655.241483.4258.9734.630.540.942.562.36
2WILLIAMS31625252550.972034.3945.7547.680.460.822.812.49
3BROWN41437026487.162521.5657.9535.60.510.872.552.52
4JONES51425470483.243004.8055.1938.480.4412.612.29
....................................
162249DIETZMANN1609751000.0390062.939600(S)0(S)
162250DOKAS1609751000.0390062.9694(S)00(S)(S)
162251DONLEA1609751000.0390062.999400006
162252DORIOTT1609751000.0390063.03890(S)05(S)
162253ALL OTHER NAMES0293120019936.979936.9766.658.537.970.862.3213.67
\n", "

162254 rows × 11 columns

\n", "
" ], "text/plain": [ " name rank count prop100k cum_prop100k pctwhite \\\n", "0 SMITH 1 2442977 828.19 828.19 70.9 \n", "1 JOHNSON 2 1932812 655.24 1483.42 58.97 \n", "2 WILLIAMS 3 1625252 550.97 2034.39 45.75 \n", "3 BROWN 4 1437026 487.16 2521.56 57.95 \n", "4 JONES 5 1425470 483.24 3004.80 55.19 \n", "... ... ... ... ... ... ... \n", "162249 DIETZMANN 160975 100 0.03 90062.93 96 \n", "162250 DOKAS 160975 100 0.03 90062.96 94 \n", "162251 DONLEA 160975 100 0.03 90062.99 94 \n", "162252 DORIOTT 160975 100 0.03 90063.03 89 \n", "162253 ALL OTHER NAMES 0 29312001 9936.97 9936.97 66.65 \n", "\n", " pctblack pctapi pctaian pct2prace pcthispanic \n", "0 23.11 0.5 0.89 2.19 2.4 \n", "1 34.63 0.54 0.94 2.56 2.36 \n", "2 47.68 0.46 0.82 2.81 2.49 \n", "3 35.6 0.51 0.87 2.55 2.52 \n", "4 38.48 0.44 1 2.61 2.29 \n", "... ... ... ... ... ... \n", "162249 0 0 (S) 0 (S) \n", "162250 (S) 0 0 (S) (S) \n", "162251 0 0 0 0 6 \n", "162252 0 (S) 0 5 (S) \n", "162253 8.53 7.97 0.86 2.32 13.67 \n", "\n", "[162254 rows x 11 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "surnames2010 = pd.read_csv(\"../../data/Names_2010Census.csv\", na_filter=False)\n", "surnames2010" ] }, { "cell_type": "markdown", "id": "c02c213b-1161-42af-a78b-69af707936f1", "metadata": {}, "source": [ "Now that we have a list of 160k names, we can apply Soundex to each one.\n", "The Soundex codes will group the names even more broadly; we will be able to see the collections of identically-coded names once we calculate them." ] }, { "cell_type": "code", "execution_count": 6, "id": "6e0c4b15-6ead-4d50-b881-f9bca0acb405", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 S53\n", "1 J525\n", "2 W452\n", "3 B65\n", "4 J52\n", " ... \n", "162249 D325\n", "162250 D2\n", "162251 D54\n", "162252 D63\n", "162253 A43652\n", "Name: name, Length: 162254, dtype: object" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "surnames2010.name.apply(soundex)" ] }, { "cell_type": "markdown", "id": "7c1f2f82-f7c5-43cd-802e-ce4ef0899835", "metadata": {}, "source": [ "These codes are of variable length, and I was expecting codes no longer than four characters. I can define a function to apply Soundex but return a code that only has the first four characters:" ] }, { "cell_type": "code", "execution_count": 7, "id": "f3e6e23b-6a32-48cc-aa58-b9dc010cf5ac", "metadata": {}, "outputs": [], "source": [ "def soundex4(name):\n", " return soundex(name, length=4)[0:4]" ] }, { "cell_type": "code", "execution_count": 8, "id": "3df8d85a-8405-4d22-a1a6-e93f3c7295e0", "metadata": {}, "outputs": [], "source": [ "codes = surnames2010.name.apply(soundex4)" ] }, { "cell_type": "code", "execution_count": 9, "id": "e1907127-6acc-4f45-bf99-6c9c9116b7b0", "metadata": {}, "outputs": [], "source": [ "# Put Soundex codes into a new column\n", "surnames2010[\"soundex\"] = codes" ] }, { "cell_type": "code", "execution_count": 10, "id": "edc82b07-09c9-481c-80e5-e56273813a9f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namesoundexcount
0SMITHS532442977
1JOHNSONJ5251932812
2WILLIAMSW4521625252
3BROWNB651437026
4JONESJ521425470
............
162249DIETZMANND325100
162250DOKASD2100
162251DONLEAD54100
162252DORIOTTD63100
162253ALL OTHER NAMESA43629312001
\n", "

162254 rows × 3 columns

\n", "
" ], "text/plain": [ " name soundex count\n", "0 SMITH S53 2442977\n", "1 JOHNSON J525 1932812\n", "2 WILLIAMS W452 1625252\n", "3 BROWN B65 1437026\n", "4 JONES J52 1425470\n", "... ... ... ...\n", "162249 DIETZMANN D325 100\n", "162250 DOKAS D2 100\n", "162251 DONLEA D54 100\n", "162252 DORIOTT D63 100\n", "162253 ALL OTHER NAMES A436 29312001\n", "\n", "[162254 rows x 3 columns]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "surnames2010[[\"name\", \"soundex\", \"count\"]]" ] }, { "cell_type": "markdown", "id": "7e1bf698-231e-42f5-bd99-fc0a82d30e09", "metadata": {}, "source": [ "Now I have a datastructure that allows me to look up the codes for 162k names." ] }, { "cell_type": "markdown", "id": "4befa064-a784-4d77-bd48-5ddca6272171", "metadata": {}, "source": [ "It seems likely that all the names are distinct, but we can check:\n", " " ] }, { "cell_type": "code", "execution_count": 11, "id": "d8752ad2-e90e-4dce-bea0-294ff4c10cef", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(162254, 162254)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(surnames2010.name.value_counts()), len(surnames2010)" ] }, { "cell_type": "markdown", "id": "cc80ace2-482f-48a5-8a22-d80bc2458f87", "metadata": {}, "source": [ "Yes, they are all distinct. " ] }, { "cell_type": "markdown", "id": "458a1f82-c908-4f47-8b0e-012c7e20aa5e", "metadata": {}, "source": [ "So, how many Soundex codes are there for 162k names?" ] }, { "cell_type": "code", "execution_count": 12, "id": "a49bc7e1-0d7d-44ba-aadc-e974c3bb2ecd", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "4162" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(surnames2010.soundex.value_counts())" ] }, { "cell_type": "markdown", "id": "8e52411a-5d7b-4f4b-a3fe-fcee41487167", "metadata": {}, "source": [ "There are only 4,000 codes. This was the purpose of the indexing system; to put similarly-sounding names in the same file despite differences in spelling. \n", "\n", "What are the most common codes?" ] }, { "cell_type": "code", "execution_count": 13, "id": "271adcc6-134c-49a8-a24d-85ada858eaf8", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "soundex\n", "M2 921\n", "B2 818\n", "B62 777\n", "R2 768\n", "L2 733\n", " ... \n", "U312 1\n", "I436 1\n", "Q341 1\n", "T136 1\n", "C464 1\n", "Name: count, Length: 4162, dtype: int64" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "surnames2010.soundex.value_counts()" ] }, { "cell_type": "markdown", "id": "75eb5a6a-dcce-49d4-a97d-aa22a7097724", "metadata": {}, "source": [ "Note there are more than 900 names on this list with the code M2. What are these names?" ] }, { "cell_type": "code", "execution_count": 14, "id": "ac4d156c-d097-4297-96dc-08aba6eaaadd", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namerankcountprop100kcum_prop100kpctwhitepctblackpctapipctaianpct2pracepcthispanicsoundex
277MCCOY27811074437.5426593.5766.8826.710.460.892.432.64M2
299MEJIA30010405735.2827391.543.370.451.330.140.294.51M2
422MOSS4237690826.0731052.6269.7324.710.480.682.122.28M2
438MCGEE4397454225.2731462.0762.2531.530.450.732.722.32M2
467MACK4687105624.0932176.6744.7649.060.70.532.592.37M2
.......................................
161463MIGOYA1609751000.0390036.2815000085M2
161467MIHAS1609751000.0390036.42910(S)0(S)(S)M2
161736MAKAS1609751000.0390045.54950(S)0(S)(S)M2
161785MAZAIKA1609751000.0390047.2010000000M2
161786MAZEWSKI1609751000.0390047.2310000000M2
\n", "

921 rows × 12 columns

\n", "
" ], "text/plain": [ " name rank count prop100k cum_prop100k pctwhite pctblack \\\n", "277 MCCOY 278 110744 37.54 26593.57 66.88 26.71 \n", "299 MEJIA 300 104057 35.28 27391.54 3.37 0.45 \n", "422 MOSS 423 76908 26.07 31052.62 69.73 24.71 \n", "438 MCGEE 439 74542 25.27 31462.07 62.25 31.53 \n", "467 MACK 468 71056 24.09 32176.67 44.76 49.06 \n", "... ... ... ... ... ... ... ... \n", "161463 MIGOYA 160975 100 0.03 90036.28 15 0 \n", "161467 MIHAS 160975 100 0.03 90036.42 91 0 \n", "161736 MAKAS 160975 100 0.03 90045.54 95 0 \n", "161785 MAZAIKA 160975 100 0.03 90047.20 100 0 \n", "161786 MAZEWSKI 160975 100 0.03 90047.23 100 0 \n", "\n", " pctapi pctaian pct2prace pcthispanic soundex \n", "277 0.46 0.89 2.43 2.64 M2 \n", "299 1.33 0.14 0.2 94.51 M2 \n", "422 0.48 0.68 2.12 2.28 M2 \n", "438 0.45 0.73 2.72 2.32 M2 \n", "467 0.7 0.53 2.59 2.37 M2 \n", "... ... ... ... ... ... \n", "161463 0 0 0 85 M2 \n", "161467 (S) 0 (S) (S) M2 \n", "161736 (S) 0 (S) (S) M2 \n", "161785 0 0 0 0 M2 \n", "161786 0 0 0 0 M2 \n", "\n", "[921 rows x 12 columns]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "surnames2010.loc[surnames2010.soundex=='M2']" ] }, { "cell_type": "markdown", "id": "aaabfa5d-03ee-4188-ae4b-90737b447574", "metadata": {}, "source": [ "It now may be clear that while it is easy to calculate Soundex from the name, the reverse procedure, determining the name from the Soundex code, is not possible.\n", "\n", "Here is the table of Soundex codes and the number of names (out of our list of 162k):" ] }, { "cell_type": "code", "execution_count": 15, "id": "812e53de-84a1-4007-9947-df4d28245db6", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "soundex\n", "M2 921\n", "B2 818\n", "B62 777\n", "R2 768\n", "L2 733\n", " ... \n", "U312 1\n", "I436 1\n", "Q341 1\n", "T136 1\n", "C464 1\n", "Name: count, Length: 4162, dtype: int64" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "surnames2010.soundex.value_counts()" ] }, { "cell_type": "markdown", "id": "59664e69-2c95-4450-b1ac-7b9c3438ba35", "metadata": {}, "source": [ "From this, we can examine a few of the names corresponding to Soundex codes that are unique in our list of 162k names: " ] }, { "cell_type": "code", "execution_count": 16, "id": "0ab1f223-a5bf-4d51-a84b-12eb308512f1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "( name soundex count\n", " 85498 QUANSTROM Q523 219,\n", " name soundex count\n", " 85280 IBARROLA I164 220,\n", " name soundex count\n", " 85205 PAPAPIETRO P136 220,\n", " name soundex count\n", " 162125 COLARELLI C464 100)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(surnames2010.loc[surnames2010.soundex=='Q523'][[\"name\", \"soundex\", \"count\"]], \n", "surnames2010.loc[surnames2010.soundex=='I164'][[\"name\", \"soundex\", \"count\"]],\n", "surnames2010.loc[surnames2010.soundex=='P136'][[\"name\", \"soundex\", \"count\"]],\n", "surnames2010.loc[surnames2010.soundex=='C464'][[\"name\", \"soundex\", \"count\"]])" ] }, { "cell_type": "markdown", "id": "a421262b-fa97-49d3-9578-9cc9be7ae1e0", "metadata": {}, "source": [ "But we still can't recover names from Soundex codes, even apparently unique ones, because our list of names was not comprehensive; the 2010 surnames dataset did not include 6 million distinct names that occurred fewer than 100 times. Many of our codes with \"unique\" names will lose their status when we add more names. " ] }, { "cell_type": "markdown", "id": "39ff1950-c909-4b31-b266-0fda3a36141c", "metadata": {}, "source": [ "### Soundex isn't the end" ] }, { "cell_type": "markdown", "id": "6c71c295-358a-4cf7-ac23-ab426b949b92", "metadata": {}, "source": [ "There are other phonetic encoding systems, including Double Metaphone and New York State Identification and Intelligence System Phonetic Code (NYIIS) that aim to identify similar-sounding names and words, and were designed with deficiencies in the Soundex encoding in mind. " ] }, { "cell_type": "markdown", "id": "64a5b7b2-9f51-43f3-9647-70f84f633df1", "metadata": {}, "source": [ "A phonetic name-matching system is used by the US Transportation Security Administration for enforcing the U.S. No-Fly List.[^*****]. Matching on lossy phonetic encodings increases the number of travelers identified by the system as meriting closer scrutiny by requiring only similar-sounding names to names on the list. " ] }, { "cell_type": "markdown", "id": "87c0d7ad-c823-4a3d-a9c2-391a37437629", "metadata": {}, "source": [ "## Murphy alphabets\n", "\n", "(Murphy 2000 10.1093/protein/13.3.149)\n", "\n", "In biological sequence analysis, alignments between biological molecules inferred to have common ancestry are a central problem. There are technologies that can turn information containing polymers into data representing the genetic information they contain. " ] }, { "cell_type": "markdown", "id": "f6590ac8-e5f3-4a47-b310-423ed8ca7478", "metadata": {}, "source": [ "[^*]: Russell, Robert C. \"Improvements in Indexes.\" US Patent 1,261,167, granted 1918 \n", "[^**]: Russell, Robert C. \"Improvements in Indexes.\" US Patent 1,435,663, granted 1922\n", "[^***]: Prechtel-Kluskens, Claire. The WPA Census Soundexing Projects Prologue Magazine. Spring 2002, Vol. 34, No. 1 \n", "https://www.archives.gov/publications/prologue/2002/spring/soundex-projects.html\n", "[^****]: Comentz, Joshua. \"Frequently Occurring Surnames in the 2010 Census\" U.S. Census bureau technical report. October 2016 \n", "https://www.census.gov/topics/population/genealogy/data/2010_surnames.html\n", "[^*****]: Goo, Sara Kehaulani. \"Faulty No-Fly System Detailed.\" Washington Post. 2004-10-09 " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }