Skip to main content

Phonetic comparison of strings

I was working on data clean up to some schools database. Data can be in any language.
One of the thing that I have to as part of this, is to find possible duplicates. First we tried out soundex algorithm, which has build in sql server, it is a good algorithm but is most suitable names and surnames (as it was developed for use in census ). Next algorithm we tried is Levenshtien distance, this takes 2 strings and gives no of character different in both strings. For a table with 10k rows for checking one field for possible duplicates it was taking about 15 min, which is quite long. Levenshtien does not well for string with spaces.
Which make reduces its usefulness very much.
So I searched around for few better algorithms. I found a soundex derivative 'Daitch–Mokotoff Soundex' , but it is also for surname with Slavic and Germanic support. Another one I found is Metaphone, which is suitable for most English words, but it is somewhat suitable for english related language (like spanish), but for language like japanese,korean etc.. it doesn't work at all. For strings with spaces it is not much accurate.
There is an improved version of Metaphone named Double-Metaphone. On language wise it is same as Metaphone. For strings with spaces it is much better, it identifies both "bank of india" and "bankofindia" as
same, while non of other algorithm is able to find it. Here are the queries and there output 


select soundex('bank of india'),soundex('bankofindia')
-- B520, B521
select dbo.Levenshtein('bank of india','bankofindia')
--NULL
select dbo.Metaphone('bank of india'),dbo.Metaphone('bankofindia')
--bnk of int, bnkfnt
select dbo.DoubleMetaPhone('bank of india'),dbo.DoubleMetaPhone('bankofindia')
--PNKFNPNKFN,PNKFNPNKFN

Comments

Popular posts from this blog

Installing Ralink 3090 wifi driver

I installed Kubuntu 10.10 on my new laptop, my laptop is Compaq CQ62- 308AX. It has Ralink 3090 wifi, which is not automatically detected by Kubuntu. Even restricted driver manager does not show any option for this device. So I downloaded the driver from Ralink site (http://www.ralinktech.com/support.php?s=2).  It is in the source format. I thought I might have to patch the kernel source. I went through the readme file and I found that I don’t have to compile kernel ( yippee an hr saved). Only device driver source is needed to be compile. These days things are much more simple. So I compiled the driver by running command “make” and installed it by “make install”.  It has compiled other ralink wifi modules along with 3090 module. After installation (using ‘make install’)  reboot the machine, reboot is not necessary in Linux after installing driver.  After reboot it loaded incorrect  rt2086 module. So I tried manually unloading and loading module using insmode...

html encode/decode in drupal

Like all major CMS drupal stores text fields in encoded html format, to prevent cross site scripting attacks. Some times drupal does not process custom fields and encoded text is displayed as it is. I come across similar issue in my current project. I also post question on StackOverflow. While search for solution I come to know about decode_entities function. decode_entities converts html encoded text to more friendly text. Similary to encode special charters in text you can use check_plain. decode_entities function http://api.drupal.org/api/drupal/includes!unicode.inc/function/decode_entities/7 check_plain function http://api.drupal.org/api/drupal/includes!bootstrap.inc/function/check_plain/7 My question on StackOverflow http://stackoverflow.com/questions/14229641/rss-feeds-view-showing-encoded-special-character-in-title

Adding additional class to a button in drupal

Recently we were building a multi-domain site, we are using same theme for all sites but we want to have different colors for buttons for each domain. We achieved this using my overriding theme_button in template.php of theme, here is the snippet. /** * Overwrite theme_button() * @file template.php */ function mytheme_button($variables) { $element = $variables['element']; // Add some extra conditions to make sure we're only adding // the classto the right submit button Now add our custom class if (isset($element['#attributes'])) { $element['#attributes']['class'] [] = 'button'; $domain = domain_get_domain(); $element['#attributes']['class'] [] = $domain['machine_name']; } $variables['element'] = $element; return theme_button($variables); }