Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

thanks for the reply but I don't think I understood what you said in the context of my question.

I'm stuck on why we care about surprisal as `log 1/p`.



I think "information" is a better name than surprisal.

If your distribution has N equally-likely values, `p(x) = 1/N`, and information/surprisal `I(x) = log(N)`. In base 2, this is how many bits are required to specify exactly WHICH of the N values you're talking about.

If `x` is not a single one of the N states but an event consisting of `n(x)` states, then `I(x) = log(N) - log(n(x))`, suggesting it takes _somewhat less information_ to specify this particular state, and it correctly gives 0 if `n(x) = N`, i.e. there's only one state.

Exactly what this "less information" means is vague, but you might think of it in terms of compression: if you compress some stream of data which is sampled from `X` with probability `p(x)`, you could use use the shortest codes (0, 10, 11, etc) for the most common values with some "stop word" to say when the end of a datum is reached. `I(x)` captures this sense in general, but it might only become literally true in the limit of a very large stream of data with a very large dictionary.


log 1/p is just -log p




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: