(Teh+ 2006) introduced updates of Collapsed Gibbs sampling for a general framework of HDP, but not for HDP-LDA.
To obtain updates of HDP-LDA, it is necessary to apply the base measure H and the emission F(phi) on HDP-LDA’s setting into the below equation:

, (eq. 30 on [Teh+ 2006])

where h is a probabilistic density function of H and f is one of F.
In the case of HDP-LDA, H is a Dirichlet distribution over vocabulary and F is a topic-word multinominal distribution, that is

where ,
.

To substitute these for equation (30), we obtain

,

where

We also need f_k^new when t takes a new table. It is obtained as the following:

And it is necessary to write down f_k(x_jt) also for sampling k.

For

(it means “term count of word w with topic k”)
(excluding ),

When implementation in Python, it is faster not to unfold Gamma functions than another. It is necessary to use these logarithms in either case, or f_k(x_jt) must overflow float range.

Finally,

DP-MRM [Kim+ ICML12] is a supervised topic model like sLDA [Blei+ 2007], DiscLDA [Lacoste-Julien+ 2008] and MedLDA [Zhu+ 2009], and is regarded as a nonparametric version of Labeled LDA [Ramage+ 2009] in particular.

Although Labeled LDA is easy to implement (my implementation is here), it has a disadvantage that you must specify label-topic correspondings explicitly and manually.
On the other hand, DP-MRM can automatically decide label-topic correspondings as distributions. I am very interested in it.
But it is hard to implement because it is a nonparametric bayesian modal.
Hence I don’t want infinite topics but hierarchical label-topic correspondings, I guess that it will become very useful and handy and fast to replace DPs into normal Dirichlet distributions in this model… I am going to try it!

Tweets are too short to detect their languages precisely. I guess that one reason is because features extracted from a short text are not enough to detect.
Another reason is because tweets have some unique representations, for example, u as you, 4 as for, LOL, F4F, various face marks and so on.

I developed ldig, a prototype of short text language detection, that solved those problems.
ldig can detect langages of tweets with over 99% accuracy for 19 languages.

• https://github.com/shuyo/ldig

The above slide explains how ldig solves those problems.

This paper targets Amazon Mechanical Turk(AMT) which separates a large task into microtasks. Each worker in AMT may be a spammer (who answers randomly to earn fee) or a hammer (who answers correctly).
This paper’s model simply assumes that each microtask has a coherent binary answer, each worker has probability to answer correctly which is independent on tasks. On the assumption, it estimates an average error rate when task size is enough large.
I don’t mind it needs simple strong assumption, but I’m sorry the model parameter q can’t be known its true value so that a practical problem can’t fit the model if the assumption was accepted.

This paper is to construct a concept and relation extraction system rapidly.
I’m afraid that there are no definite construction methods (they might be confidential…) but length of time to do each task in it.
Hence I have not yet learned this field very much yet, I knew what tasks the information extraction consists of generally.

This shows that the accuracies of Norwegian and Danish are lower than others.
It is because Norwegian and Danish are very similar.

Here is top 25 of the word distribution of Norwegian and Danish.

Most of high frequency words, ‘er’(it nearly corresponds to ‘is’ in English, following is the same), ‘det’(‘it’) and’i'(‘in’) are common function words between 2 languages, so it is very difficult to identify them.
Useful words for the identification are a pretty few. For example, ‘of’ in English corresponds to ‘af’ in Danish or ‘av’ in Norwegian, ‘me’ corresponds to ‘mig’(da) or ‘meg’(no), ‘now’ corresponds to ‘nu’(da) or ‘nå’(no), ‘just’ correspond to ‘lige’(da) or ‘like’(no), ‘what’ corresponds to ‘hvad’(da) or ‘hva’(no), ‘no’ corresponds to ‘nej’(da) or ‘nei’(no), and so on.

The most serious problem is my language identification skill for Danish and Norwegian…
I collected tens of thousands of tweets in them, but I’m afraid that they contains several percent errors.
Of cource, that causes detection errors.

If you know native Norwegian or Danish who are interested in language detection and corpus annotation, I’m glad you to introduce me!

At first, it needs to comvert their dataset into ldig-available format.
Here is a ruby script to convert it.

#!/usr/bin/env ruby
# -*- coding: utf-8 -*-

open("liga_dataset.txt", "wb:UTF-8") do |f|
  ["de_DE","en_UK","es_ES","fr_FR","it_IT","nl_NL"].each do |dir|
    lang = dir[0,2]
    Dir.glob("#{dir}/*.txt") do |file|
      line = open(file, "rb:UTF-8") {|f| f.read}.gsub(/[\u0000-\u001f]/, " ").strip
      f.puts "#{lang}\t#{line}"
    end
  end
end

Here is a estimation of ldig for the generated dataset, liga_dataset.txt.

It shows that ldig can detect over 99% accuracy for their dataset.

Reference

• [Erik Tromp and Mykola Pechenizkiy 11] “Graph-Based N-gram Language Identification on Short Texts”

The sum of data size is not equal to the amount of detected languages because ldig outputs “” as language when the max probability is lower than 0.6.
And the data size is not equal to one in the previous article because the dataset is updated.

I reckoned it doesn’t make sense over 99% accuracy, then what’s about?

• https://github.com/shuyo/ldig

It detects tweets in 17 languages with 99.1% accuracy (Czech, Dannish, German, English, Spanish, Finnish, French, Indonesian, Italian, Dutch, Norwegian, Polish, Portuguese, Romanian, Swedish, Turkish and Vietnamese).
ldig specialized with noisy short text (more than 3 words) and is limited to Latin alphabet language because input text can separate into character type blocks and Latin alphabet detection is most difficult.

My language-detection (langdetect) is not good at short text detection, so that most users seem troubled in language detection for twitter.
langdetect uses character 3-grams as feature so it is insufficient for short text detection.
I supposed that maximal substrings [Okanohara+ 09] makes sufficient features for short text detection and prepared twitter corpus with 17 languages.
Then training/test corpus size and estimation of ldig prototype is the below.

I’m preparing Catalan corpus with some helps (THANKS!), so supported languages will increase sooner.

ldig write out model as numpy binary format now, but I will modify it into more portable format, MessagePack like, then detector of ldig can probably port in other platforms easily.

The presentation of ldig is here (but this is written in Japanese! )

• Language Detection with Infinity-Gram (in Japanese)

And I’ll read its paper at the annual conference of The Association for Natural Language Processing in Japan (NLP2012).
I’ll publish the paper on this blog after the conference (but it’s in Japanese! ).

P.S.

I opened a slide about twitter language detection in English.

Reference

• [Daisuke Okanohara and Jun'ichi Tsujii 09] “Text Categorization with All Substring Features”

Google Code Project supports subversion, git and Mercurial as Version Control System. Each repository is independent and exclusive, and it is necessary to migrate between them by manual.
So I wrote migration process from subversion into git as step-bt-step.

1. Prepare git and git-svn

I used them on Cygwin. But I reckon it is easier on Linux

2. Migrate into local git repositories

At first, migrate the subversion repository into local git repositories (because of their exclusiveness).
The subversion repository of Google Code Project has histories of not only codes(trunk/branch/tag) but also wikis. In the case of git, codes and wikis are stored in each repository.
So you need to migrate 2 repositories if you didn’t use branches/tagges

Execute the below commands on an appropriate directory.

$ cd your-working-directory
$ git svn clone -s http://your-project-name.googlecode.com/svn/ 
$ git svn clone http://your-project-name.googlecode.com/svn/wiki/

“your-working-directory” and “your-project-name” must be replaced into your suitable names. It matters that the second command does not has -s option.
If you use branches and tagges, then you might need several steps for tagging on the git repository, but I didn’t research its methods because I didn’t use them

If you run into an error like the below with git-svn on Cygwin (it may often occur at Windows 7 x64???),

10192702 [main] perl 9904 C:\cygwin\bin\perl.exe: *** fatal error - unable to re 
map C:\cygwin\bin\cygsvn_client-1-0.dll to same address as parent: 0xAB0000 != 0xAE0000

then execute rebaseall on cmd.exe as the below.

$ cd c:\cygwin\bin
$ ash rebaseall -v

3. Switch the project’s VCS into git

Switching into git is available at the source tab on the administrator’s console of the project.
The operation does not affect no inference to the subversion repository, but limit the only selected system to access. So you can restore the subversion at any time.

4. Push the local repositories into new ones

Execute the below commands.

$ cd svn
$ git push https://code.google.com/p/your-project-name/ master
$ cd ..
$ cd wiki 
$ git push https://code.google.com/p/your-project-name.wiki/ master

Then the git repositories will be available. You shoud check whether the project’s wiki is as before.
I like that the repositories of codes and wikis are separeted.

Reference:
- http://d.hatena.ne.jp/kkobayashi_a/20090615/p1 (in Japanese)