Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Nope, because that wouldn't work with stemming and pluralizing.

Think finding "weisswürste" when people search for wurst. Unless you correctly build a tvector of (weiss, wurst) from "weisswürste", even with wildcarding hacks, you'd have to search for "würste" to find it.

Also: Not all words are compoundable. Doing a blind wildcard search would lead to false-positives which is somewhere between annoying and very annoying for the user.



Midly OT: I don't really know German at all but I've wondered in the past how easy it is to split the compound words in the language. Is it something that can be easily automated? How many words can be in a single compound?


As evidenced by the (somewhat humorous) Donaudampfschiffahrtsgesellschaftskapitänsmütze there isn't really a limit. Usually, if compounds get longer hyphens are used in select places to aid readability (they have sort of a semantic meaning as well, depending on the compound, so simply saying that you could write the above one as Donaudampfschiffahrts-Gesellschaftskapitänsmütze would be misleading (and in this case really mutilates the compound)).


In Swedish there is no theoretical limit, but more than 4 is uncommon. To split words you will need a dictionary and stemming/compounding rules for Swedish and process the word from left to right. Note also that in rare cases there can be multiple legal ways to split a given compound word.

As far as I can tell it is the same for German other than that they have a slightly different set of compounding rules, e.g. hyphen can be used in more cases in German.


It can be done quickly and accurately. Check out Peter Norvig's implementation of Viterbi algorithm for text segmentation.

Segmentation is typically used for scripts of languages like Thai and Khmer that don't feature word boundaries. I don't know the ins and outs of German word compounding, but it should work for breaking apart compounds too given a fully conjugated/declined word list as input.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: