Free Access
Uncertainty About the Rest of the Sentence
John Hale,
Corresponding Author
John Hale
Department of Linguistics and Germanic, Slavic, Asian and African Languages, Michigan State University
Department of Linguistics and Germanic, Slavic, Asian and African Languages, Michigan State University, East Lansing, MI 48824-1027Search for more papers by this authorJohn Hale,
Corresponding Author
John Hale
Department of Linguistics and Germanic, Slavic, Asian and African Languages, Michigan State University
Department of Linguistics and Germanic, Slavic, Asian and African Languages, Michigan State University, East Lansing, MI 48824-1027Search for more papers by this authorFirst published: 01 July 2006
Citations: 137
Abstract
A word-by-word human sentence processing complexity metric is presented. This metric formalizes the intuition that comprehenders have more trouble on words contributing larger amounts of information about the syntactic structure of the sentence as a whole. The formalization is in terms of the conditional entropy of grammatical continuations, given the words that have been heard so far. To calculate the predictions of this metric, Wilson and Carroll's (1954) original entropy reduction idea is extended to infinite languages. This is demonstrated with a mildly context-sensitive language that includes relative clauses formed on a variety of grammatical relations across the Accessibility Hierarchy of Keenan and Comrie (1977). Predictions are derived that correlate significantly with repetition accuracy results obtained in a sentence-memory experiment (Keenan & Hawkins, 1987).
References
-
Alexiadou, A.,
Law, P.,
Meinunger, A. &
Wilder, C. (Eds.). (2000). The syntax of relative clauses.
Philadelphia
: John Benjamins.
10.1075/la.32 Google Scholar
-
Aoun, J., &
Li, Y. (2003). Essays on the representational and derivational nature of grammar: The diversity of wh-constructions.
Cambridge
,
MA
: MIT Press.
10.7551/mitpress/2832.001.0001 Google Scholar
- Bar-Hillel, Y., Gaifman, C., & Shamir, E. (1960, June). On categorial and phrase-structure grammars. Bulletin of the Research Council of Israel, 9F, 1–16.
- Bever, T. G. (1970). The cognitive basis for linguistic structures. In J. Hayes (Ed.), Cognition and the development of language (pp. 279–362). New York : Wiley.
- Bianchi, V. (2000). The raising analysis of relative clauses: A reply to Borsley. Linguistic Inquiry, 31, 124–140.
- Billot, S., & Lang, B. (1989). The structure of shared forests in ambiguous parsing. In Proceedings of the 1989 Meeting of the Association for Computational Linguistics.
- Borsley, R. D. (1997). Relative clauses and the theory of phrase structure. Linguistic Inquiry, 28, 629–647.
- Brame, M. K. (1976). In Conjectures and refutations in syntax and semantics (pp. 125–143). Amsterdam : North-Holland.
- Bresnan, J. (Ed.). (1982). The mental representation of grammatical relations. Cambridge , MA : MIT Press.
- Chater, N., Crocker, M. W., & Pickering, M. J. (1998). The rational analysis of inquiry: The case of parsing. In M. Oaksford & N. Chater (Eds.), Rational models of cognition (pp. 441–468). New York : Oxford University Press.
- Chi, Z. (1999). Statistical properties of probabilistic context-free grammars. Computational Linguistics, 25, 131–160.
- Chomsky, N. (1956). Three models for the description of language. IRE Transactions on Information Theory, 2(3), 113–124.
- Chomsky, N. (1977). On Wh-movement. In P. Culicover, T. Wasow & A. Akmajian (Eds.), Formal syntax (pp. 71–132). New York : Academic.
- Chomsky, N. (1995). The minimalist program. Cambridge , MA : MIT Press.
- Frank, R. (2004). Restricting grammatical complexity. Cognitive Science, 28, 669–697.
- Gibson, E. (1991). A computational theory of human linguistic processing: Memory limitations and processing breakdown. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh , PA .
- Gibson, E. (1998). Linguistic complexity: Locality of syntactic dependencies. Cognition, 68, 1–76.
- Gibson, E. (2000). The dependency locality theory: A distance-based theory of linguistic complexity. In Y. Miyashita, A. Marantz & W. O'Neil (Eds.), Image, language, brain (pp. 95–126). Cambridge , MA : MIT Press.
- Goldman-Eisler, F. (1958). Speech production and the predictability of words in context. Quarterly Journal of Experimental Psychology, 10, 96–106.
- Grenander, U. (1967). Syntax-controlled probabilities (Tech. Rep.). Providence , RI : Brown University, Division of Applied Mathematics.
- Hale, J. (2003a). Grammar, uncertainty and sentence processing. Unpublished doctoral dissertation, Johns Hopkins University, Baltimore .
- Hale, J. (2003b). The information conveyed by words in sentences. Journal of Psycholinguistic Research, 32, 101–123.
- Hale, J., & Stabler, E. (2005). Strict deterministic aspects of minimalist grammars. In P. Blache, E. Stabler, J. Busquets & R. Moot (Eds.), Logical aspects of computational linguistics (pp. 162–176). New York : Springer.
- Harkema, H. (2001). Parsing minimalist grammars. Unpublished doctoral dissertation, University of California at Los Angeles, Los Angeles .
- Hawkins, J. A. (1994). A performance theory of order and constituency (Vol. 73). New York : Cambridge University Press.
-
Joshi, A. K. (1985). Tree adjoining grammars: How much context-sensitivity is required to provide reasonable structural descriptions?
In
D. Dowty,
L. Karttunen &
A. M. Zwicky (Eds.), Natural language parsing: Psychological, computational and theoretical perspectives (pp.
206–250).
New York
: Cambridge University Press.
10.1017/CBO9780511597855.007 Google Scholar
- Joshi, A. K., Levy, L. S., & Takahashi, M. (1975). Tree adjunct grammars. Journal of Computer and System Sciences, 10, 136–163.
- Joshi, A. K., Vijay-Shanker, K., & Weir, D. (1991). The convergence of mildly context-sensitive grammatical formalisms. In Foundational issues in natural language processing (pp. 31–81). Cambridge , MA : MIT Press.
- Jurafsky, D. (1996). A probabilistic model of lexical and syntactic access and disambiguation. Cognitive Science, 20, 137–194.
- Kaplan, R. M. (1972). Augmented transition networks as psychological models of sentence comprehension. Artificial Intelligence, 3, 77–100.
- Kay, M. (1986). Algorithm schemata and data structures in syntactic processing. In B. J. Grosz, K. S. Jones & B. L. Webber (Eds.), Readings in natural language processing (pp. 125–170). San Francisco : Morgan Kaufmann.
- Kayne, R. S. (1994). The antisymmetry of syntax. Cambridge , MA : MIT Press.
- Keenan, E. L., & Comrie, B. (1977). Noun phrase accessibility and universal grammar. Linguistic Inquiry, 8, 63–99.
- Keenan, E. L., & Hawkins, S. (1987). The psychological validity of the accessibility hierarchy. In E. L. Keenan (Ed.), Universal grammar: 15 essays (pp. 60–85). London : Croom Helm.
- Kroch, A., & Joshi, A. (1985). The linguistic relevance of tree adjoining grammar (Tech. Rep. No. MS-CIS-85–16). Philadelphia : University of Pennsylvania.
- Kučera, H., & Francis, W. N. (1967). Computational analysis of present-day American English. Providence , RI : Brown University Press.
-
Lang, B. (1974). Deterministic techniques for efficient non-deterministic parsers. In
J. Loeckx (Ed.), Proceedings of the 2nd Colloquium on Automata, Languages and Programming (pp.
255–269).
Saarbrücken
,
Germany
: Springer.
10.1007/978-3-662-21545-6_18 Google Scholar
- Lang, B. (1988). Parsing incomplete sentences. In Proceedings of the 12th International Conference on Computational Linguistics (pp. 365–371).
- Lewis, R., & Vasishth, S. (2005). An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science, 29, 375–419.
- Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., et al. (1994, March). The Penn Treebank: Annotating predicate argument structure. Paper presented at the ARPA Human Language Technology Workshop, Plainsboro, NJ.
- McDaniel, D., McKee, C., & Bernstein, J. B. (1998). How children's relatives solve a problem for minimalism. Language, 74, 308–334.
- Michaelis, J. (1998). Derivational minimalism is mildly context-sensitive. In Proceedings of Logical Aspects of Computational Linguistics ′98. Grenoble .
- Michaelis, J. (2001). On formal properties of minimalist grammars. Unpublished doctoral dissertation, Potsdam University, Potsdam , Germany .
- Miller, G. A., & Chomsky, N. (1963). Finitary models of language users. In R. D. Luce, R. R. Bush & E. Galanter (Eds.), Handbook of mathematical psychology (pp. 419–491). New York : Wiley.
- Morrill, G. (2000). Incremental processing and acceptability. Computational Linguistics, 26(3) 319–338.
- Narayanan, S., & Jurafsky, D. (2001). A Bayesian model predicts human parse preference and reading time in sentence processing. In T. G. Dietterich, S. Becker & Z. Ghahramani (Eds.), Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference. Cambridge , MA : MIT Press.
- Pearlmutter, N. J., & Mendelsohn, A. A. (2000). Serial versus parallel sentence comprehension. Unpublished manuscript, Northeastern University.
- Perlmutter, D., & Postal, P. (1974). Lectures on relational grammar. Amherst , MA : LSA Linguistic Institute.
- Peters, P. S., & Ritchie, R. W. (1973). On the generative power of transformational grammar. Information Sciences, 6, 49–83.
- Pollard, C., & Sag, I. A. (1994). Head-driven phrase structure grammar. Chicago : University of Chicago Press.
- Rohde, D. L. (2002). A connectionist model of sentence comprehension and production. Unpublished doctoral dissertation, Carnegie Mellon University, Pittsburgh , PA .
- Sag, I. A. (1997). English relative clause constructions. Journal of Linguistics, 33, 431–484.
- Schachter, P. (1973). Focus and relativization. Language, 49, 19–46.
- Seki, H., Matsumura, T., Fujii, M., & Kasami, T. (1991). On multiple context-free grammars. Theoretical Computer Science, 88, 191–229.
- Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423, 623–656.
- Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal, 30, 50–64.
- Shieber, S., & Johnson, M. (1993). Variations on incremental interpretation. Journal of Psycholinguistic Research, 22, 287–318.
- Stabler, E. (1994). The finite connectivity of linguistic structure. In C. Clifton, L. Frazier & K. Rayner (Eds.), Perspectives on sentence processing (pp. 303–336). Hillsdale , NJ : Lawrence Erlbaum Associates, Inc.
-
Stabler, E. (1997). Derivational minimalism. In
C. Retoré (Ed.), Logical aspects of computational linguistics (pp.
68–95).
New York
: Springer.
10.1007/BFb0052152 Google Scholar
- Stabler, E. (2004). Varieties of crossing dependencies. Cognitive Science, 28, 699–720.
- Stabler, E., & Keenan, E. (2003). Structural similarity. Theoretical Computer Science, 293, 345–363.
- Stevens, A. L., & Rumelhart, D. E. (1975). Errors in reading: An analysis using an augmented transition network model of grammar. In D. A. Norman & D. E. Rumelhart (Eds.), Explorations in cognition (pp. 136–155). New York : Freeman.
- Stevenson, S., & Smolensky, P. (2006). Optimality in sentence processing. In P. Smolensky & G. Legendre (Eds.), The harmonic mind: From neural computation to optimality-theoretic grammar: Vol. 2. Linguistic and philosophical implications (pp. 307–338). Cambridge , MA : MIT Press.
- Tabor, W., Galantuccia, B., & Richardson, D. (2004). Effects of merely local syntactic coherence on sentence processing. Journal of Memory and Language, 50, 355–370.
- Taylor, W. (1953). Cloze procedure: A new tool for measuring readability. Journalism Quarterly, 30, 415–433.
- Thorndike, E., & Lorge, I. (1968). The teacher's word book of 30,000 words. New York : Teachers' College Press.
-
van Noord, G. (2001). Robust parsing of word graphs. In
J-C. Junqua &
G. Noord (Eds.), Robustness in languages and speech technology (pp.
205–238). Kluwer.
10.1007/978-94-015-9719-7_9 Google Scholar
- Wanner, E., & Maratsos, M. (1978). An ATN approach to comprehension. In M. Halle, J. Bresnan & G. A. Miller (Eds.), Linguistic theory and psychological reality (pp. 119–161). Cambridge , MA : MIT Press.
- Weir, D. J. (1988). Characterizing mildly context-sensitive grammar formalisms. Unpublished doctoral dissertation, University of Pennsylvania, Philadelphia .
- Wilson, K., & Carroll, J. B. (1954). Applications of entropy measures to problems of sequential structure. In C. E. Osgood & T. A. Sebeok (Eds.), Psycholinguistics: A survey of theory and research (pp. 103–110). Bloomington : Indiana University Press.
- Yngve, V. H. (1960). A model and an hypothesis for language structure. Proceedings of the American Philosophical Society, 104, 444–466.
