| Author |
Message |
![[Post New]](/gforum/templates/default/images/icon_minipost_new.gif) 18/12/2010 16:28:52
|
crontab
Joined: 18/12/2010 15:46:04
Messages: 2
Offline
|
I'm trying to import all class 'P' entries in allCountries.txt to a MySQL database. To optimize my search indexes I decided to change the charset setting for the asciiname column to use the 'ascii' charset. Looks like overall the entries conform to 7-bit ASCII with some exceptions. The extract below is a result of a query that selects empty asciiname entries plus those that contain question marks (?) -- presumably Unicode chars that can't be represented in 7-bit ASCII. There are total of 305 of them and they clearly can be fixed.
Please note that, again, these are only from the class 'P' subset. Naturally, there may be many more in non-P features.
The selected columns are: geonameid, name, asciiname, tab-delimited.
Code:
7501451 مینو ????
146970 Xankǝndi Xank?ndi
147622 Lǝnkǝran L?nk?ran
6618546 Anta Q´hawa
6618571 Luyuluyu Ck´asa
6618574 K´allpa
7535694 Métabetchouan–Lac-à-la-Croix Metabetchouan?Lac-a-la-Croix
7397284 Da?tun Da?tun
7397780 Zhong?tun Zhong?tun
7399112 Guo?zi Guo?zi
7402101 ?zigou ?zigou
7402385 ??wo ??wo
7506579 雄鸡埭 ???
6545245 Saint John´s woods Saint John?s woods
6545246 St. John´s wood St. John?s wood
7530607 ღვაშიგვგრდი ???????????
6735908 Jo’ong
7290222 Periyapattinam (Tamil:பெரியபட்டினம்) Periyapattinam (Tamil:periya?pa?ttina?m)
98576 آشورن
6647876 Ḩayy al Bayḑāʼ
6647881 Ḩayy al Janāʼin
6647896 Ḩayy al Muʻtaşam
6647926 Qaryat Khaḑīr ʻAbbās
7501320 Qaryat as Saʻdūn Qaryat as Sa?dun
7501321 Qaryat ʻAbd Allāh as Sarāy Qaryat ?Abd Allah as Saray
7501323 Ibrāhīm ʻAţīyyah Ibrahim ?At?iyyah
7501326 Ḩamīd Ḩasan H?amid H?asan
7501328 Qaryat ʻAlīy Suhayl Qaryat ?Aliy Suhayl
7501463 Qaryat al Muʼāḑīd Qaryat al Mu?ad?id
7501468 ʻUbayd Dafār ?Ubayd Dafar
7501476 Qaryat Ālbū ʻĪsá Qaryat Albu ?Isa
7501477 Aḩmad Daḩām Ah?mad Dah?am
7501505 Qaryat Salmān aḑ Ḑārī Qaryat Salman ad? D?ari
7501558 Qaryat Kāz̧im ʻIdhāb Qaryat Kazim ?Idhab
43864 صادق آباد
118876 Raʼīs
120523 Qal‘eh Ḩasan
123110 Mūʼīneh
132988 Gomānj-e Pāʼīn
139798 Būʼīn-e Soflá
404980 Qal‘eh Solţān Bājī
6648088 Sho‘eyţeh
6648638 Khānehhā-ye Sāzemānī-ye Şanāye‘-e Fūlād
6677119 Tāzeh Qalˈeh
6677476 ‘Alī Şafar
6677687 Kalāteh-ye Ḩājjī ‘Abdol
6685385 ‘Abdol Ḩoseyn Kandī
6686968 Shahrak-e Maskūnī-ye Valī-ye ‘Aşr
6687741 Ḩamūdī-ye Saʻdūn
6709097 Gerūsheh-ye ‘Olyā
6709100 Qal‘eh Tūt-e ‘Olyā
6709112 Banjū-ye ‘Olyā
6709471 ‘Aşrābād-e Kohneh
6711433 La‘lābād-e Ḩoseyn Qolīkhānī
6711465 Ḩājat ‘Alī
6719833 Morād ʽAlī
6719926 Shahrak-e Valīye‘aşr
6720817 Shahrak-e Beʻs̄at
6727552 Shahrak-e Şan‘atī-ye Jānbāzān
6731125 Ḩājjī Khān ‘Alī
6731126 Ḩājjī ‘Avaẕ
6731137 Qeshlāq-e Ḩājj ‘Īsá (2)
6731138 Qeshlāq-e Ḩājj ‘Īsá (1)
6758151 ‘Alī Maḩalleh
6758159 Sa‘ādat Maḩalleh
6776677 Noşratābād-e ‘Olyá
6776710 ‘Ālamābād-e Soflá
7011502 قلعه سنگ
7347231 ‘Avaẕ Ḩājjī `Avaz H?ajji
7351313 Ḩoseynābād H?oseynabad
7351314 Moḩammadābād Moh?ammadabad
7380791 Dashtak-e Vosţá Dashtak-e Vost?a
7388749 Ḩodyyā H?odyya
7389077 Maḩal-e Eḩdās̄-e Shahrak-e Qodūs Mah?al-e Eh?das-e Shahrak-e Qodus
7389078 Chūlak-e Şādeqābād Chulak-e S?adeqabad
7389111 Shahrak-e Shahīd Ḩamīdī Shahrak-e Shahid H?amidi
7406972 Mīrfattāḩ Mirfattah?
7407007 Zanganeh-e Soflá Zanganeh-e Sofla?
7407011 Magasān-e Soflá Magasan-e Sofla?
7407020 Shahrak-e Shahīd Moḩammad-e Borūjerdī Shahrak-e Shahid Moh?ammad-e Borujerdi
7414671 Tall-e Vostá Tall-e Vosta?
7416567 Faraḩānī-ye Moqaddam Farah?ani-ye Moqaddam
7416607 Arbīḩāt Arbih?at
7416609 Ālbū Mūsá Albu Musa?
7416611 Dowraqī-ye Ālbū Mūsá Dowraqi-ye Albu Musa?
7416622 Şabbāḩīyeh S?abbah?iyeh
7416623 Şabbāḩīyeh S?abbah?iyeh
7416624 Şafā’ī S?afa'i
7416642 Nahr-e Raḩmeh Nahr-e Rah?meh
7416643 Mendovān-e ‘Aţīyeh Mendovan-e `At?iyeh
7425535 Vasaţ Maḩalleh-ye Kharā Rūd Vasat Mah?alleh-ye Khara Rud
7425536 Bālā Maḩalleh-ye Kharā Rūd Bala Mah?alleh-ye Khara Rud
7457210 Moḩammad Ḩoseynābād Moh?ammad H?oseynabad
7457216 Bāgh-e Movaḩed Bagh-e Movah?ed
7490317 Nehẕatābād Nehz?atabad
7490374 Ẕarghāmābād Z?arghamabad
7529548 Sar Por-e Nūr Moḩammadī Sar Por-e Nur Moh?ammadi
7529871 Deh-e ‘Avaẕ Deh-e `Avaz?
7596964 ΄Alī Kandī ?Ali Kandi
7616389 Faẕlābād Faz?labad
7625081 Ḩasanābād H?asanabad
7625486 Ţālesh Maḩalleh-ye Ketālem T?alesh Mahalleh-ye Ketalem
7625512 Ţālesh Sarā T?alesh Sara
6281260 Angol
6281261 Ansŏnggol
6281270 Chingni
6281279 Chungsangni
6281286 Ipsŏngni
6281287 Kaltaedong
6281294 Kogaegol
6281303 Naedong
6281313 Sŏngdong
6281322 Pŏltŏkkol
6281323 Pugyŏlli
6281327 Saekkol
6281340 Tongdonggŏmŭngol
6281341 Ŭmyangdong
6281344 Yongamni
6281347 Yongdŏngni
6302777 Chŭngsalli
6305267 Segoktong
6305268 Tomaktong
6305269 Paegamdong
6305271 Mangdandong
6305277 Chosindong
6363922 Pŏlp´an
6363967 Sangdaech´ang
6364142 Ugŏch´on
6364260 T´apkol
6799199 T’ǒ-gol
6799218 Chech’ǒng-mal
6802104 Korant’ǒ
2378335 El ‛Erg
7370519 Goth Hāji Sohrāb Khān Jakhrāده Goth Haji Sohrab Khan Jakhra??
7389281 Sar Būlāk Sar B?l?k
7389430 Lālo Qila L?lo Qila
7389541 Nawāb Kārez Naw?b K?rez
7418436 Vazīri Gund Vazi?ri Gund
474109 Vesёloye
583509 Ak”yar Ak?yar
797381 Ozёrnoye
816188 Kovalёvshchina
816729 Shmelёvo
817448 Berёzovka
818241 Zelёnyy Klin
818401 Zhёltaya Akatsiya
818778 Belёvaya
819321 Vorob’yёvka
819705 Morёvo
819709 Zelёnyy Gay
819756 Korolёvka
7411316 Ponоmarevo Pon?marevo
7434419 Grachёvka Grach?vka
7434454 Vesёlyy Ves?lyy
7434526 Krasnyy Posёlok Krasnyy Pos?lok
7434539 Kovalёvka Koval?vka
7434541 Semёnovodcheskiy Sem?novodcheskiy
7434576 Vishnёvka Vishn?vka
7436295 Belozёrnyy Beloz?rnyy
7436344 Zelёnaya Balka Zel?naya Balka
7436353 Opёnki Op?nki
7436411 Klёny Kl?ny
7437892 Bakharёvskiy Bakhar?vskiy
7442300 Malyye Ozёra Malyye Oz?ra
7450090 Berёzovka Ber?zovka
7450101 Yerёmin Yer?min
7450109 Kulёvo Kul?vo
7450135 Zelёnyy Gay Zel?nyy Gay
7450136 Vishnёvskiy Vishn?vskiy
7450187 Alёshkin Al?shkin
7451366 Berёzovaya Ber?zovaya
7451382 Fёdorovskiy F?dorovskiy
7451435 Zelёnyy Rog Zel?nyy Rog
7451437 Zelёnaya Roshcha Zel?naya Roshcha
7451438 Vesёlyy Ves?lyy
7451439 Zelёnyy Kust Zel?nyy Kust
7454571 Zaozёr’ye Zaoz?r'ye
7454763 Berёzovskaya Ber?zovskaya
7454813 Zaozёr’ye Zaoz?r'ye
7493616 Konёvka Kon?vka
7493620 Khmelёvitsa Khmel?vitsa
7493634 Konёvka Kon?vka
7496459 Berёziny Ber?ziny
7496482 Gremyachёvskaya Gremyach?vskaya
7496494 Ponomarёvskaya Ponomar?vskaya
7496555 Vesёlaya Griva Ves?laya Griva
7590869 Zalivnoе Zalivno?
7595007 Sтаroye Baltachëvo S??roye Baltachevo
7604674 Trёkhrech’ye Tr?khrech'ye
7604741 Kotёnovskiy Kot?novskiy
7353780 Al Ḩabīl Al H?abil
7357833 Al Faţīḩah Al Fat?ih?ah
7366853 Qarn Aḩad Yūbah Qarn Ah?ad Yubah
7367673 Al Makhlūţah Al Makhlut?ah
7370930 Āl Ḩalāmī Al H?alami
7371000 Āl Ḩadīlah Al H?adilah
7439930 At Tibāḩ At Tibah?
6943439 كاف الجاع
6855641 Dalakéna Ⅱ
6690124 สี่แยก อสมท
6691800 บ้านตลาดใหม่
7503210 Gϋzelyalı G?zelyali
73282 Maḩallat al ̒Uways
406587 Al Ḩamrā̛
406689 Āl ‘Ammār
406933 Az̧ Zā̧li‘
407070 Qaryat as Sabī ̔
7341021 Al Maḩaţţah Al Mahat?t?ah
7341723 Al Mahādimah Maḩāll aţ Ţaraf Al Mahadimah Mahall at T?araf
7341827 Ḩaḑan a̧s Şabl Hadan a?s Sabl
7342177 Şāfiyat Ţāmish Safiyat T?amish
7342477 Hi̧şn Bayt Na‘am Hi?sn Bayt Na`am
7345331 Khuḑā‘ah Khud?a`ah
7345333 Āl Jālī A?l Jali
7345339 Ar Rajaḩ Ar Rajah?
7346043 Al Kadā’ Al Kada?'
7346048 Thāhir Dijāj Thahir Dija?j
7346070 Jāḑi‘ Muşabbiḩ Jad?i` Musabbih
7346081 Aş Şarw As? Sarw
7346094 Ḩilf Banī al Qamm H?ilf Bani al Qamm
7346098 Dhī al Bawāqil Dhi al Bawa?qil
7346108 Ţaybān Tayba?n
7346114 Ahl Şa‘dah Ahl S?a`dah
7346115 Şarw Ḩayyān Sarw Hayya?n
7346140 Gharbī Sha‘bān Gharbi Sha`ba?n
7346146 Suwār Suwa?r
7346177 Uqrun Āl Ja‘dān Uqrun A?l Ja`da?n
7346185 Duwāḩ Duwah?
7346219 Āl Abū Ghubārah A?l Abu Ghubarah
7346221 Alt ‘Ayshān Alt `Aysha?n
7346222 Maşḩan Mas?h?an
7346247 Ibn Lakwān Ibn Lakwa?n
7346266 Alt ‘Āmir Alt `A?mir
7346313 Adh Dhārī Adh Dha?ri?
7346316 Al ‘Awārij Al `Awa?rij
7346318 Ad Dumāh Ad Duma?h
7346319 Aţ Ţāh At Ta?h
7346326 Tī Yarkham Ti? Yarkham
7346334 ‘Āsī al Qarn `A?si al Qarn
7346335 Aş Şa‘rūf As Sa`ru?f
7346344 Adh Dhirā‘ Adh Dhira?`
7346346 Ḩablān Habla?n
7346347 Al Baqāmah Al Baqa?mah
7346379 Al Ḩaḑīrah Al Hadi?rah
7346400 Qullat al Ḩamrī Qullat al Hamri?
7346401 Şāyid S?ayid
7346416 Al Maşna‘ah Al Mas?na`ah
7346458 Āl aţ Ţawq Al at T?awq
7346496 Banī Sa‘d Bani? Sa`d
7346509 Adh Dhirā‘ Adh Dhira?`
7346516 Alt al Muţayrī Alt al Mut?ayri
7346537 Tikinnāyah Tikinna?yah
7346539 An Nab‘ī An Nab`i?
7346548 Bilād Qudam Bila?d Qudam
7346553 Aqārin Aqa?rin
7346560 Ad Dārah Ad Da?rah
7346676 Alt ‘Uţayf Alt `Ut?ayf
7354857 Raha̧bah Raha?bah
7355347 Al Mudayrīş Al Mudayris?
7356955 ‘Arḑ al A‘wir `Ard? al A`wir
7357208 Z̧ābi ̓ Zabi ?
7358714 ̒̒Adayrah ??Adayrah
7359992 Aţ Ţabaqahb ̒Ard al Jabūb At? T?abaqahb ?Ard al Jabub
7360133 Nuqţat Banī Shihāb Nuqt?at Bani Shihab
7360954 Bayḩān al ‘Uly̧ā Bayhan al `Uly?a
7361264 Aş Şayyāţ As Sayyat?
7361723 Bayt ash Shaḩţirah Bayt ash Shaht?irah
7361738 Quţayn al Qashāmī Qut?ayn al Qashami
7361917 Bayt al Khubāţ Bayt al Khubat?
7361930 Bayt Ḩūmī Bayt H?umi
7362051 Ash Shijān Ash Shija?n
7363127 Al Ḩamrā’ Al H?amra'
7363141 Al Lāţī Al Lat?i
7363142 Al Ma‘faţ Al Ma`fat?
7363143 Al Madārah Al Mada?rah
7363187 Mirāḩ Quzzān Mirah? Quzzan
7364429 ̧Jabal aş Şafrā’ ?Jabal as Safra'
7364440 Maḩaţţat al ‘Ubaydī Mahat?t?at al `Ubaydi
7364699 Qaryat al ‘Aţf Qaryat al `At?f
7364724 Al Wāsiţah Al Wasit?ah
7364784 Al ̒Aqabah Al ?Aqabah
7364898 Al Ḩaţab Al Hat?ab
7365070 Maḩall Bulghat Fīd Mahall Bulghat Fi?d
7365266 Maḩall al Ḩisņ Mahall al Hisn?
7365312 Adh Dhībīnīyah Adh Dhibi?niyah
7365334 Al Kadḩīyah Al Kadh?iyah
7365381 Dayr Bāshā Dayr Basha?
7365392 Ḑubāyah Duba?yah
7365464 Maḩall al ‘Aḑābīyah Mahall al `Ad?abiyah
7365503 Yaḩyá aş Şaghīr Yahya as? Saghir
7365772 ‘Abs Thawāb `Abs Thawa?b
7365915 Al Quţa‘ Al Qut?a`
7366321 Maḩall ad Dimnah aţ Ţawīlah Mahall ad Dimnah at T?awilah
7366339 Dayr Zābiţ Dayr Zabit?
7366368 Maḩall al Qurayţī Mahall al Qurayt?i
7366438 Al Muqdhūlah Al Muqdhu?lah
7366488 Al Maţ‘an Al Mat?`an
7366504 Maḩall Banī Āmir al Ḩawātim Mahall Bani Amir al H?awa?tim
7366565 Al Kassās Al Kassa?s
7366591 Bayt Ţaḩāmah Bayt T?ahamah
7366797 Az̧ Z̧ahār Az Zaha?r
7366802 Abū al Khadārīsh Abu al Khada?rish
7366833 Al Balāyiţ Al Balayit?
7366864 ̒̒Aqabāt ??Aqabat
7367271 Ḩişn aḑ Ḑabyatayn His?n ad Dabyatayn
|
|
|
 |
![[Post New]](/gforum/templates/default/images/icon_minipost_new.gif) 20/12/2010 20:30:14
|
marc
Joined: 08/12/2005 07:39:47
Messages: 4501
Offline
|
Thanks, I will run a check and set the asciiname to null for the problematic entries.
Cheers
Marc
|
 |
|
|
 |
![[Post New]](/gforum/templates/default/images/icon_minipost_new.gif) 21/12/2010 00:58:47
|
crontab
Joined: 18/12/2010 15:46:04
Messages: 2
Offline
|
Hi Marc
(BTW notifications from the forum end up in Spam at least on GMail, just to let you know...)
A quick glance at the non-ASCII names shows that: in many of these names the non-ASCII character could have been omitted, e.g.
Moḩammad Ḩoseynābād -> Moh?ammad H?oseynabad -> Mohammad Hoseynabad
Also, there are some Russian names where the letter "ё" is used (pronounced "yo")
Vishnёvka -> Vishn?vka
I'm not sure what this letter is translated to in Geographical names traditionally, could be either "yo" or "e". Again, can be replaced easily in either case.
|
|
|
 |
![[Post New]](/gforum/templates/default/images/icon_minipost_new.gif) 21/12/2010 07:07:53
|
marc
Joined: 08/12/2005 07:39:47
Messages: 4501
Offline
|
I have run an update:
7566171 rows checked, 621 updated, 348 failed.
If you want the 348 failed ones to be not null you are welcome to update them.
Best
Marc
|
 |
|
|
 |
![[Post New]](/gforum/templates/default/images/icon_minipost_new.gif) 12/06/2011 02:11:54
|
Pit
Joined: 08/06/2011 09:02:59
Messages: 2
Offline
|
Edit: forget about it, wanted to commit some fixes but I just noticed that my ascii converter messed up
|
|
|
 |
|
|