Preprint

The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions

This article is a preprint and has not been certified by peer review [What does this mean?].

Author(s) / Creator(s)

Lavi-Rotbain, Ori
Arnon, Inbal

Abstract / Description

One of the striking commonalities between languages is the way word frequencies are distributed. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank (Zipf, 1949). Intuitively, this means that languages have relatively few high-frequency words and many low-frequency ones. While studied extensively, little work has explored the learnability consequences of the greater predictability of words in such distributions. Here, we propose such distributions confer a learnability advantage for word segmentation, a foundational aspect of language acquisition. We capture the greater predictability of words using the information-theoretic notion of efficiency, which tells us how predictable a distribution is relative to a uniform one. We first use corpus analyses to show that child-directed speech is similarly predictable across fifteen different languages. We then experimentally investigate the impact of distribution predictability on children and adults. We show that word segmentation is uniquely facilitated at the predictability levels found in language, compared both with uniform distributions and with skewed distributions that are less predictable than those of natural language. We further show that distribution predictability impacts learning more than distribution shape, and that learning is not improved further in distributions more predictable than natural language. These novel findings illustrate learners' sensitivity to the overall predictability of the linguistic environment; suggest that the predictability levels found in language provide an optimal environment for learning; and point to the possible role of cognitive pressures in the emergence and propensity of such distributions in language.
Preprint of: Lavi-Rotbain, O. & Arnon, I. (2022). The learnability consequences of Zipfian distributions in language. Cognition, 223. https://doi.org/10.1016/j.cognition.2022.105038

Keyword(s)

Language acquisition Statistical learning Information theory Zipf's law Word segmentation

Persistent Identifier

Date of first publication

2020-06

Publisher

PsychArchives

Is version of

Citation

Lavi-Rotbain, O., & Arnon, I. (2020). The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions. PsychArchives. https://doi.org/10.23668/PSYCHARCHIVES.3079
  • 2
    2020-06-16
    The new version was formed in order to fix mistakes that appeared in the graphs of the original version.
  • 1
    2020-06-09
  • Author(s) / Creator(s)
    Lavi-Rotbain, Ori
  • Author(s) / Creator(s)
    Arnon, Inbal
  • PsychArchives acquisition timestamp
    2020-06-16T12:36:51Z
  • Made available on
    2020-06-09T14:29:57Z
  • Made available on
    2020-06-16T12:36:51Z
  • Date of first publication
    2020-06
  • Submission date
    2019-11
  • Abstract / Description
    One of the striking commonalities between languages is the way word frequencies are distributed. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank (Zipf, 1949). Intuitively, this means that languages have relatively few high-frequency words and many low-frequency ones. While studied extensively, little work has explored the learnability consequences of the greater predictability of words in such distributions. Here, we propose such distributions confer a learnability advantage for word segmentation, a foundational aspect of language acquisition. We capture the greater predictability of words using the information-theoretic notion of efficiency, which tells us how predictable a distribution is relative to a uniform one. We first use corpus analyses to show that child-directed speech is similarly predictable across fifteen different languages. We then experimentally investigate the impact of distribution predictability on children and adults. We show that word segmentation is uniquely facilitated at the predictability levels found in language, compared both with uniform distributions and with skewed distributions that are less predictable than those of natural language. We further show that distribution predictability impacts learning more than distribution shape, and that learning is not improved further in distributions more predictable than natural language. These novel findings illustrate learners' sensitivity to the overall predictability of the linguistic environment; suggest that the predictability levels found in language provide an optimal environment for learning; and point to the possible role of cognitive pressures in the emergence and propensity of such distributions in language.
    en
  • Abstract / Description
    Preprint of: Lavi-Rotbain, O. & Arnon, I. (2022). The learnability consequences of Zipfian distributions in language. Cognition, 223. https://doi.org/10.1016/j.cognition.2022.105038
    en
  • Publication status
    other
    en
  • Review status
    notReviewed
    en
  • Sponsorship
    The research was funded by the Israeli Science Foundation grant number 584/16 awarded to the second author.
    en
  • Citation
    Lavi-Rotbain, O., & Arnon, I. (2020). The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions. PsychArchives. https://doi.org/10.23668/PSYCHARCHIVES.3079
    en
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/2693.2
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.3079
  • Language of content
    eng
  • Publisher
    PsychArchives
    en
  • Is version of
    https://doi.org/10.1016/j.cognition.2022.105038
  • Is related to
    https://doi.org/10.23668/psycharchives.3009
  • Keyword(s)
    Language acquisition
    en
  • Keyword(s)
    Statistical learning
    en
  • Keyword(s)
    Information theory
    en
  • Keyword(s)
    Zipf's law
    en
  • Keyword(s)
    Word segmentation
    en
  • Dewey Decimal Classification number(s)
    150
  • Title
    The learnability consequences of Zipfian distributions: Word Segmentation is Facilitated in More Predictable Distributions
    en
  • DRO type
    preprint
    en
  • Visible tag(s)
    Language acquisition
    en
  • Visible tag(s)
    Statistical learning
    en
  • Visible tag(s)
    Information theory
    en
  • Visible tag(s)
    Zipf's law
    en
  • Visible tag(s)
    Word segmentation
    en