Type Token Ratios and Success on the Effective Writing Test

I was just reading about the components of lexical richness in John Read’s book Assessing Vocabulary (Read, 2000) when I thought I’d write about one aspect of lexical richness in relation to the performance of NNES students on the Effective Writing Test at the University of Calgary. One of my key findings is that while about 70% of native English speaking (NS) students pass this test on their first attempt, only 23% of NNES students whose first language is of East Asian origin (Chinese, Japanese, Korean, Vietnamese, Laotian) actually pass on their first attempt. Obviously something is affecting the scores of these NNES students. Roessingh’s work (2008) with NNES high school students looking at the role vocabulary plays as an underlying variable in determining success on the written response component of the Grade 12 English 30-1 Diploma examination is one of the factors that has led me to analyse the vocabulary use of novice undergraduate writers at university.

One aspect of my analysis is the type to token ratios of NS versus NNES students on the test. Type to token ratios look at the number of unique words students use in comparison to the total number of words that they use. In his book, Read (2000) explains how students with a high type to token ratio use a variety of different words in their writing. Students with lower type to token ratios are using a limited number of words repetitively in their writing. Through the use of low frequency words, synonyms, hyponyms, and specific vocabulary items, good writers are able to tap into their larger vocabulary knowledge in order to convey a more precise meaning. Another term for type to token ratios is lexical variation.

Looking at my own research into lexical variation (please see the chart accompanying this blog), I found that NS students wrote significantly shorter essays than NNES students. However, the number of unique words employed by each of these groups was statistically the same. This resulted in NS students having a higher type to token ratio than their NNES counterparts. In other words, the NS students were employing a greater range of expression than the NNES students in their essays, and this significant difference may be one of the factors contributing to the low success rate of NNES students on the Effective Writing Test.

It was interesting to see that the NS students were writing shorter essays with more variation than the NNES students. In addition to being repetitive, another reason behind why the NNES students were writing longer essays includes the necessity of using circumlocution when a precise term wasn’t lexically available. More words are needed than necessary to express ideas due to a lack of vocabulary. However, the NS students typically have access to a much larger lexicon, enabling them to employ greater lexical variation in their writing.

On the Effective Writing Test rubric, repetitious diction is one of the aspects of word use that markers are evaluating. While circumlocution isn’t overtly marked according to the rubric, the use of “too many words” is a key component of the word use category. This is what drew me to consider the lower type to token ratios as being a possible underlying factor to the overall quality of undergraduate compositions.

These findings seem to mirror the findings of Cheryl Engber (1995) who was looking at the relationship between lexical proficiency and reader judgments of the overall quality of timed essays. Engber found that reader judgements of the overall quality of these compositions did reflect lexical variation. This leads me to conclude that a similar effect may be taking place in the Effective Writing Test, with the lower lexically varied essays of NNES students having less success than the more lexically varied essays of their NS counterparts.

Some References:

Engber, C. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing 4(2), 139-155.

Read, J. (2000). Assessing Vocabulary. Cambridge: CUP.

Roessingh, H. (2008). Variability in ESL Outcomes: The Influence of Age on Arrival and Length of Residence on Achievement in High School. TESL Canada Journal 26(1), 87-107.