Does the photo show one mountain, or six? Does each peak on a mountain also have its own name? Where, exactly, is the bottom of the mountain? Lexicographers have similar problems framing the boundaries of words. What merits its own entry in a dictionary? What should be shown as additional information for an entry? What is so obvious, or cumbersome, that it should be left out? The conundrum has three faces.
First is meaning
. Is swing
, the dance music, a different word from swing
, the playground equipment? At Kamusi, we say they are different things, so they get different entries. From a conceptual perspective, separating senses is straightforward, although it means that one spelling, such as t-a-k-e, can have dozens of different entries.
For cataloguing, shape
is the more difficult face. We can define a sense of big
as "being of a substantial size" - but do we include bigger
as forms of that idea within that entry, or are they separate words? English verbs have five forms that we can easily list in an entry, and hope all our readers will know what we mean by past (took
), third person present (takes
), past participle (taken
), and present participle (taking
). Listing such inflections is much harder for a language like French, with 96 verb forms
, many of which actually require conjugating a second verb and contracting with part of a pronoun ("I have read" = j'ai lu
[je+avoir + lire]). Should each sense entry list all the possible forms, or can they refer to a common table that may not apply in all cases (e.g., swings
for children have a plural form, but swing
music does not)?
And then there are the agglutinative languages. German
can bind several independent words together to make a brand new compound word that everyone understands, sometimes with letters changing internally according to various rules - think of the way English creates words like supermarket
, but do that on the fly, with any idea you can make up, such as Autobahnmarkierungsentfernungskomitee
(a committee that is responsible for removing the road marks on highways). Hundreds of African languages can have an entire sentence in a single unit, such as "We are squeezing each other" in Swahili
=we + na
=now + bana
=squeeze + na
=each other). Each Kinyarwanda
verb can take 900,000,000 different forms. Obviously, a dictionary cannot list nearly a billion forms within an entry. Our strategy is computational, to find the rules that people use to agglutinate words in their language, and build parsers that locate the entries for their component parts.
The third face is the most difficult for computer processing: multiword expressions. We call these party terms
because they are composed of items that dance together. That is, the words in the expression each have their own meaning, but together they form a new idea - a head case
is a disturbed person, for example, and a northern right whale dolphin
is a type of dolphin that is neither a compass direction nor correct nor a whale. In Vietnamese
, every syllable is separated by a space, so every multisyllabic word looks like a party term. Are party terms "words"? Kamusi treats them as entries, if the meaning cannot be gleaned from its parts. Party terms present additional challenges because they can be separated, can change shape, and can have multiple meanings. Fortunately, our unique architecture can handle all of that, and, using GOLDbox
, a variety of games
, and other techniques, we can build from the basic canonical lemmas that are the usual initial dataset or first-round user contributions for a language, toward a full set of forms associated with each concept.
Just don't ask us to tell you what exactly we mean by the word "word".