Language Detection Plugin for Apache Nutch

I developed a Language Detection plugin for Apache Nutch with our LangDetect library.

  • Download (bundled in the LangDetect library)
  • Setup manual
  • Compatible to the Standard language identification plugin of Nutch
  • 99% over accuracy
  • Supports 49 languages
    • Afrikaans, Albanian, Arabic, Bengali, Bulgarian, Chinese(Simplified/Traditional)
    • Croatian, Czech, Dannish, Dutch, English, Finnish, French, German, Greek
    • Gujarati, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Kannada
    • Korean, Macedonian, Malayalam, Marathi, Nepali, Norwegian, Persian, Polish
    • Portuguese, Punjabi, Romanian, Russian, Slovak, Somali, Spanish, Swahili
    • Swedish, Tagalog, Tamil, Telugu, Thai, Turkish, Ukrainian, Urdu, Vietnamese

language index

The above is sample of detected indexs which are crawled from several sites.
If you confirm as well, display the Nutch’s index using luke after crawl.

I also wrote the articles how to develop Nutch’s plugin. Read them if you are interested.

Advertisements
This entry was posted in i18n, NLP, Nutch, Plugin, Search Engine, text analysis. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s