Script Normalization for Unconventional Writing

[Paper (ACL 2023)] [Slides] [GitHub] [Presentation]

  • "mar7aba!"
  • "هاو ئار یوو؟"
  • "Μπιάνβενου α σετ ντεμό!"

What do all these sentences have in common? Being greeted in Arabic with "mar7aba" written in the Latin script, then asked how you are ("هاو ئار یوو؟") in English using the Perso-Arabic script of Kurdish and then, welcomed to this demo in French ("Μπιάνβενου α σετ ντεμό!") written in Greek script. All these sentences are written in an unconventional script.

Although you may find these sentences risible, unconventional writing is a common practice among millions of speakers in bilingual communities. In our paper entitled "Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities", we shed light on this problem and propose an approach to normalize noisy text written in unconventional writing.

This demo deploys a few models that are trained for the normalization of unconventional writing. Please note that this tool is not a spell-checker and cannot correct errors beyond character normalization. For better performance, you can apply hard-coded rules on the input and then pass it to the models, hence a hybrid system.

For more information, you can check out the project on GitHub too: https://github.com/sinaahmadi/ScriptNormalization

Language in unconventional script
Examples
Noisy Text 🥴 Language in unconventional script

Created and deployed by Sina Ahmadi (https://sinaahmadi.github.io/).