Vocabulary Assessment P. David Pearson University of California - - PowerPoint PPT Presentation

Vocabulary Assessment

• P. David Pearson

University of California Berkeley

There is only one book in Books in Print with the title, Vocabulary Assessment

• What field do you think it is in?

– Reading – Oral Language Development – Intelligence Testing – English as a Foreign/Second Language

Special Characteristics of this presentation

• Data free presentation
• why?
• So little new and/or interesting work in

vocabulary assessment

Pretest in the form of a Jeopardy Game

Reading Jeopardy

• Scientifically based reading research:
• What phrase occurs in the Reading First Act

more often than the word, the.

• Phonemic Awareness:
• What happens when a phoneme experiences

self actualization?

Reading Jeopardy

• Math Second:
• What happens when you put Reading First?
• Reading First:
• What should you have done before you

walked in the wrong gendered rest room?

• Consequential Validity
• What happens to you when you don’t do what

the teacher says?

Reading Jeopardy

• Rate, accuracy, and expression
• Name 3 words that would have different

social consequences for you at a reading conference versus a church social

• Reading Recovery:
• What do you call a 12 step program for first

graders who have overdosed on Accelerated Reader.

• Accelerated Reader:
• What would you call a person who can read

billboards at 120 miles per hour?

A distinction I find important

• The world is filled with lots of good

indicators that make lousy instructional

• goals. It is just fine to get better at them

as long as you don’t do so by teaching them directly.

• Never send a test out to do a curriculum’s

job.

Outline for today

• Define domain of interest
• A short history of vocabulary assessment
• Some important features of the domain of

vocabulary assessment

• A pot pouri of examples of vocabulary

assessments

Domain of interest

• For sure: Knowledge of word meanings and the

conceptual networks in which they exist. BUT

– Which words?

• in general
• a set of specific words from a story, unit, book, etc.
• Perhaps: The ability to use available cues, both

inside words and outside of them, to infer--and maybe to learn--the meanings of words

A very short history of vocabulary assessment

• Vocabulary assessment has been around as

long as we have had

– Assessment of any sort – Reading assessment

• Has always been a part of intelligence testing
• Has always been a part of reading assessment
• Has always been a major part of second

language assessment

Trends over time

• Early on: test isolated words and find

their synonyms or meanings

• Not surprisingly, there has been a

movement toward contextualization

• ver time

– Psycholinguistic and cognitive revolution – Constructivist pedagogies

• Whole language and its kissing cousins
• Communicative competence (ESL)

Circa 1920s thru 1950s

A _______ is used to eat with

1. Plow 2. Fork 3. Hammer 4. Needle

Foolish

1. Clever 2. Mild 3. Silly 4. Frank

1970s

• He discovered a new route through the

mountains.

1. Wanted 2. Found 3. Traveled 4. captured

• Their success came about as a result of

your assistance.

1. according to 2. before 3. because of 4. during

Note: Vocabulary subtest of this sort correlates .85 to .95 with RC Note that context does allow us to assess abstract words

mid 1980s

1950s: deliberately

1. Both 2. Noticeably 3. Intentionally 4. Absolutely

1970s: He was found guilty because he did the act deliberately. deliberately

1. Both 2. Noticeably 3. Intentionally 4. Absolutely

mid 1980s

In a (1) democratic society, individuals are presumed innocent until proven guilty. The (2) establishment of guilt is often a difficult task. One consideration is whether or not there remains a (3) reasonable doubt that the suspected persons committed the act in question. Another consideration is whether the acts were committed (4) deliberately.

(4)

• 1. Both
• 2. Noticeably
• 3. Intentionally
• 4. absolutely

Compared to other formats, this one showed the highest reliability, predictive validity, discrimination

1995: among comprehension questions, insert vocabulary…

…Two reasons are usually advanced to account for this tardy development; namely the mental difficulties…

The word tardy in line 2 is closest in meaning to

1. Historical 2. Basic 3. Unusual 4. Late

Note: Still an open question of whether you report vocabulary separately

Now, in the age of on-line assessment…

• The Southwest has always been a dry country,

where water is scarce, but the Hopi and Zuni were able to bring water from streams to their fields and gardens through irrigation ditches. Because it is so rare, yet so important, water played a major role in their religion.

• Look at the word rare in the passage. Click on

the word in the text that has the same meaning.

NAEP’s likely influence

• That NAEP is in the game will elevate the role of

vocabulary

• NAEP’s standards (achievement levels) and

format will also influence assessment

• Good development to attend to.
• Note: whether it is a separate scale depends on

– Resources (for item development) – Psychometrics (will it scale separately from comprehension)

What will be tested on NAEP?

• Assess words characteristic of written language not oral

language

• Label generally familiar and broadly understood

concepts, even if the words themselves are not familiar (akin to Isabel’s discussion of Tier 2):

• Stunning but not pretty
• Prosperous but not rich
• Demonstrate but not show
• Are required to built a sensible rendition of the text (and

preferably linked to central ideas in the text).

• Are characteristic of grade level material (4, 8, 12)

What won’t be tested on NAEP

• Words that are narrowly defined and not

widely used (appears to be tier 3, technical vocabulary) or just arcane (hamlet or rivulet)

• Words that label the main idea of the text (e.g.,

emancipation in Emancipation Proclamation)

• Words that are part of most students’

speaking vocabulary

• Words with meanings that are readily derived

from context (appositives, parenthetical definitions)

NAEP Distractor features

• Can present a more common meaning for the

word, which must be ignored in favor of the meaning in the text.

• Can present correct information from the text

that is NOT the meaning of the word.

• May be an alternative interpretation of the

context in which the word occurs

• Visually or auditorily similar words.

Note: Useful to have a theory of distractor generation because it gives meaning to errors

NAEP Achievement levels for vocabulary

Advanced

• Advanced readers will have outstanding vocabularies,

– With a sound knowledge of words and terms beyond their grade level.

• In addition, they will have an excellent grasp of the

multiple meanings of an extensive set of words and complex networks of associations to the words they know.

• They will also have a strong base of words that identify

complex and abstract ideas and concepts

• Finally their sophistication with words and word meanings

will enable them to be highly flexible in extending the senses of words they know

– to appropriately fit different contexts.

Proficient

• Proficient readers will have sizeable meaning

vocabularies,

– including knowledge of many word and terms above grade level.

• They will also have greater depth of knowledge of

words, beyond the most common meaning.

• Proficient readers will be flexible with word meanings

and able to extend the senses of words whose meanings they know

– in order to appropriately fit different contexts and understand passage meaning.

Basic

• Readers at the basic level will generally have

limited, concrete vocabularies that consist primarily of words at and below grade level.

• Knowledge of these words will be limited to the

most familiar definition,

– making it difficult to identify the appropriate meaning of a word among the distractors.

Dimensions of Vocabulary Assessment

• After John Read
• Interesting book, entitled Vocabulary

Assessment, Cambridge University Press, 2000/2003

• You can assess vocabulary with an eye

toward these distinctions:

– Discrete--->Embedded (phenomenon) – Selective-->Comprehensive (corpus) – Context independent-->context dependent (format)

Discrete/Embedded

• Discrete: vocabulary as an independent

construct

– (e.g., lots of standardized tests report a separate comprehension score)

• Embedded: vocabulary is assessed but feeds

into a score for a larger construct

– (e.g., added to comprehension aggregate score) – (e.g., on a typical test, you get

• Comprehension
• Vocabulary
• Total reading

Selective-->Comprehensive

• Selective: a targeted set of vocabulary items

– Those in the selection at hand – Those in the unit at hand – Those in the course of study – Those in a certain band of frequency (e.g., 1000 most frequent

• Comprehensive: all the words in some domain or

performance

– All the words in an essay (when we rate the sophistication of an essay) – All the words in a speech – All the words in a text – All the words in a corpus, language,

Context dependence

• Critical aspect is whether you must use the

context

• Context independent

– Don’t have it OR – Don’t need it when you DO have it

• Context dependent: can’t get the right answer

without reference to context

– Rare words – Nonsense words – Multiple meanings

Context absent

• consumed

1.Ate or drank 2.Prepared 3.bought 4.enjoyed

Context present but not critical

• The people consumed their dinner
• 1. Ate or drank
• 2. Prepared
• 3. bought
• 4. enjoyed

Context critical

• The people

consumed their dinner.

1. Ate or drank 2. Used up 3. Spent wastefully 4. Destroyed Note: the distractor set matters!

• The air conditioner

consumed a lot of energy.

1. Ate or drank 2. Used up 3. Spent wastefully 4. Destroyed

Questions to ask in building or evaluating vocabulary assessments

• What does it mean to know a word
• What counts as a word
• How do we choose words
• How do we know whether selected words

are known?

What does it mean to know a word?

• Increasing breadth

– Know what semantic space it fits into in our heads. What it goes with?

• Paradigmatic relations (dog-cat or dog-canine)
• Syntagmatic (dog-bark or dog-chase)

– Know how it is used in discourse (recognition and use)

• Increasing depth:

– awareness--> acquaintanceship--> ownership – Denotation--> Connotation

• Any two words that mean the same at one level of analysis

mean something different at another level

What counts as a word?

• Words or word families

– Extend? Sure – Affixes: Extension, extendable – Inflections: Extends, extended, extending

• Levels of transfer among words in a family (Nagy and

Anderson)

– Level 1: transparent (they clearly belong to the same family: various-vary – Level 3: stretch: (some metaphorical help) collarbone-collar; visualize-visual – Level 5: accidental: (not much help to know the family: need lots of conceptual support to get from a to b) prefix-fix; peppermint-pepper

An aside: New labels or new ideas?

• The RARE words in literary texts tend to emphasize new,

more sophisticated, and more precise labels for partially known ideas (just the right nuance)

– Misanthrope, discomfited

• The RARE words in informational texts tend to be

conceptually central to the selection

– Photosynthesis, chlorophyll

• Some RARE concepts in informational texts tend to be

secondary senses of common words

– Prime, force (a different problem)

How do we choose words

• Depends entirely on purpose
• How well did students learn the words in the X,

where X is a very particular group of words? (story, chapter, unit, text)

• How much general vocabulary growth have

students experienced over time?

• How able are students to use strategies to infer

the meanings of words they don’t already know? (How do you know what words a given child knows or doesn’t know?

– Context – Morphology (see Joanne Carlisle’s work)

Learning words in a specific corpus

• Define the corpus
• Draw a sample
• Pick a format

Some format options and issues

• Checklist test (Anderson-Freebody)
• Read each word and put a check beside

the ones you know

• Put in distractors to correct for kids’ bias

toward saying yes (misspellings [parfome]and synthetic/nonsense words [implaceable] or [brep])

Anderson Freebody example

Target words __democratic __presumed __establishment __reasonable __committed __deliberately Non words __ethocracy __preburbed __retableness __reserb __compt __delb Misspellings __dimecratik __persumed __istableshment __resonible __comitad __deliberately

Note that Anderson Freebody is not unlike some of our informal devices

• I’ve never seen it
• I’ve seen it but am not sure what it means
• I know what it means and can explain it
• I can use it in a sentence or a story

Vocabulary Knowledge Scale

1. I don’t remember having seen this word before. 2. I have seen this word but don’t know what it means. 3. I have seen this word and I think it means ____________ 4. I know this word and it means __________ 5. I can use this word in a sentence (write the sentence).

Paribakht & Wesche, 1997

Format issues

• Pictures make life easier…

– Reduce the reading load

• And harder…

– Ambiguity

• Distractors matter

– They largely determine the cognitive focus of the task – What is varied is what you must pay the most attention to..

What is varied is what is assessed

• Look at the following examples to

determine the focus

Prosperous a.wealthy b.sad

• c. tall

d.happy Prosperous a.doing well b.in trouble

• c. not very happy

d.very lucky

• A person who is prosperous could be said to

be…

– doing well financially – in trouble – not very happy with life – very lucky

• A person who is doing well financially could

be said to be

– anxious – sick – open – prosperous

• With time things got better and many settlers

became prosperous.

– wealthy – sad – tall – happy

• With time things got better and many settlers

became ____________

– anxious – sick – open – prosperous

• With time things got better and many

settlers became _____________.

– prospered – prosper – prosperous – prosperable

A diversion in assessing breadth/depth

• Getting into networks

– Semantic associations

• What class does it belong to?
• What are some examples of it?
• What are some of its key features?

– Episodic associations (syntagmatic)

• Which of these words go with dog?
• Which of these words would you expect to find in the same

sentence as dog?

– Bark, bite, meow, sleep, soup,

Assessing general vocabulary growth

• Usually we resort to some normative assessment:

How much growth did they make compared to other kids in the norming sample?

– PPVT, vocabulary subtest of any reading assessment,

• Logically possible, but not very practical, to draw

samples from a very large corpus

– Words in the band of frequency from 1-1000, 3000-5000, etc. – Words in a large but specifiable domain (a semester long course or in a big textbook)

Assessing strategies for inferring word meanings

Context (already discussed)

• Rare words that naturally occur

– BUT (almost has to be calibrated for individual kids) – What is rare for you may be common for me

• Nonsense words to substitute for real words

– BUT (the meanings may be transparent but the nonce will disturb kids)

• Multiple meanings

– BUT (assumes that kids will know the various meanings and can discriminate them from one another)

Morphological inferences

• Incidental interactions around words

(during instructional episodes)

• I have not figured out a good multiple

choice approach to assessing this important strategy.

Morphological awareness

• John is a __________ worker

– Capacity – Capable – Capacious – Capability

Morphological awareness

• John is good at what he does. He is a

__________ worker

– Capacity – Capable – Capacious – Capability

Morphological awareness

• John is good at what he does. He is a

__________ worker

– Capacity – Capable – Capacious – Incapable

Morphological Awareness

• If laud means admire
• then
• laudable means something that could be

– admired – astonished – admire – admires

So what is a body to do in assessing vocabulary.

D

Vocabulary items (picture)

Our Picture Assessment in

• ur Seeds and Roots work

restrictions

D

Vocabulary (semantic associations)

Our Semantic Associations Assessment in our Seeds and Roots work Stem is a high frequency association:

• Target word
• Word from sane unit and

form class

• Word from a parallel unit
• High frequency word from

same form class and selectional restrictions

D

MAZE

• What do you want to know?
• That will determine the domain and the

sample you select.

Make sure your purpose is clear

Select a format that fits your situation.

• Balances

– what we know about the psychometric properties of different formats and approaches – With the realities of our circumstances

• In an ideal world, we might want a carefully

crafted set of multiple choice questions, but the realities of our situation might dictate

– Fill in the blank – matching

A tip on matching

• 1. Rapid
• 2. Cone
• 3. Incidental

_Used for ice cream _Unintentional _Ugly _Fast _Peppy Have more choices than there are items to be matched, but… what happens from one item to the next?

Some useful examples

• 1. If Karla was astounded to learn she was chosen, she:

(a) expected it (b) was surprised (c) was scared (d) was thrilled

• 2. If a hotel can accommodate large groups, it can:

(a) hire them (b) handle them (c) counsel them (d) train them

From “Teaching Vocabulary to Adolescents to Improve Comprehension” by Mary E. Curtis and Ann Marie Longo.Reading Online, www.readingonline.org Posted November 2001

The context in the stem limits the application of meaning

• 1. astounded

(a) expected (b) surprised (c) scared (d) thrilled

• 2. accommodate

(a) hire (b) handle (c) counsel (d) train Same items without the contextual frame

Unit Grade Report (contributes to provision of ongoing assessment and communication about progress) Recognition of Word Meanings You got XX% correct on your post-test. Analysis of Word Relationships

• You always completed the Word Activity Sheets in a thoughtful

way.

• You usually gave some thought to the completion of the Word

Activity Sheets.

• Too often you did not complete the Word Activity Sheets.

Response to What Has Been Read

• You did a great job responding to the questions that followed the

readings.

• I could tell you did some thinking about the articles we read.
• Too often you didn’t respond to what we read.

Use of Words in Speaking

• You participated fully in our discussions of the words and their

meanings.

• You usually participated in our discussions of the words.
• You need to participate more in class discussions.

Use of Words in Writing

• You did a great job in using your vocabulary words in your writing.
• You used some of your vocabulary words in your writing.
• Your writing assignments were incomplete or not handed in at

all.

Also from Curtis and Longo A nifty rubric for vocabulary use

Unit Grade Report (contributes to provision of ongoing assessment and communication about progress) Recognition of Word Meanings

• You got XX% correct on your post-test.

Analysis of Word Relationships

• You always completed the Word Activity Sheets in a

thoughtful way.

• You usually gave some thought to the completion of

the Word Activity Sheets.

• Too often you did not complete the Word Activity

Sheets.

Response to What Has Been Read

• You did a great job responding to the questions that followed

the readings.

• I could tell you did some thinking about the articles we read.
• Too often you didn’t respond to what we read.

Use of Words in Speaking

• You participated fully in our discussions of the words and their

meanings.

• You usually participated in our discussions of the words.
• You need to participate more in class discussions.

Use of Words in Writing

• You did a great job in using your vocabulary words in your

writing.

• You used some of your vocabulary words in your writing.
• Your writing assignments were incomplete or not handed in at

all. Unit Grade Report

Bottom line

• We have a long history of assessing vocabulary,

but…

• Not much research to guide us in selecting the

perfect approach

• We can get along with many of the tools we have,

but…

• We need some significant work on the construct

validation of vocabulary assessments

• The research community, along with the

publishing community, needs to provide teachers with better tools.

• Don’t forget technology