Week 7
Testing productive skills: Writing
Module 5119
Language Testing and Assessment
2
Summer, 2011
3
Testing writing ability
4
Product mirroring process or ability
Dominance of subjectivity
Reliability issues

Genres of written language
Academic writing
Papers and general subject reports
Essays, compositions
Academically focused journals
Short-answer test responses
Technical reports (e.g., lab reports)
Theses, dissertations
Genres of written language (cont.)
Job-related reading
Messages (e.g., phone messages)
Letters/emails
Memos (e.g., interoffice)
Reports (job evaluations, project reports)
Schedules, labels, signs
Advertisements, announcements
Manuals
Genres of written language (cont.)
Personal reading
Letters, emails, greeting cards, invitations
Messages, notes
Calendar entries, shopping lists, reminders
Financial documents (checks, tax forms, loan applications)
Forms, questionnaires, medical reports, immigration documents
Diaries, personal journals
Fiction (e.g., short stories, poetry)
Types of writing performance
Imitative
learners try to master the mechanics of writing (ability to spell correctly);
primary focus is form, context and meaning are of secondary concern
Intensive (controlled)
skills in producing appropriate vocab. within a context, collocations
correct grammatical features up to the length of a sentence;
meaning and context are of some important, but a focus on form
Responsive
Learners required to perform at a limited discourse level connecting sentences into a prara., creating logically connected sequence of 2 or 3 paras
Tasks respond to pedagogical directives, outlines
Genres of writing include brief narratives, lab reports, interpretation s of charts or graphs
Writers have mastered fundamentals of sentence-level grammar and more focused on discourse conventions
Form-focused attention with a strong emphasis on context and meaning
Extensive
Implies successful management of all processes and strategies of writing for all purposes, up to the length of an essay, a thesis
Writer focus on achieving a purpose, organizing and developing ideas logically, using details to support or illustrate ideas, demonstrating syntactic and lexical variety.
Activity
10
Go through the writing skills in the handouts. In pairs, identify what skills are micro and what macro?
Which ones are commonly used in your context of teaching and testing?
Brainstorm some tasks that assess those skills.
Micro and macroskills of writing
Test development
12
Hughes (2003) suggests three parts:
Setting a representative sample of writing tasks
Eliciting valid samples of writing ability
Ensuring valid and reliable scoring
Sample of writing tasks
13
Purpose of the test
What ability is being assessed?
As many tasks and test formats as practical

Test formats
14
Gap filling
Copying
Form completion
Editing
Open-ended essay
Responding to given information (a prompt or assignment)
Information transfer
Integrated reading and writing: responses to the reading of an article or story
(Weir, 1993)
Designing assessment tasks: imitative writing Copying
Designing assessment tasks: intensive writing Short sentences
Designing assessment tasks: responsive and extensive writing
Open-ended essay
Is seldom completely open-ended: a task defined by the teacher or test administrator, and the writer must fulfill the criterion of the task
In extensive writing, a set of directives stated by the teacher or is implied by the conventions of the genres
e.g, Summaries of Readings
Effectively capture the main and supporting ideas of the original
Maintain objectivity in reporting
Use writer’s own words for the most part
Use quotations effectively when appropriate
Omit irrelevant or marginal details
Conform to an expected length
Content selection
19
Elements of test specification framework:
Operations
Types of texts
Addressees
Topics
Dialect and style
Lengths
(Hughes, 2003)


Eliciting a valid sample of writing
20
Testing only writing
1 Write the conversation you have with a friend about the holiday you plan to have together.

2 You spend a year abroad. While you are there, you are asked to talk to a group of young people about life in your country. Write down what you would say to them.

3 ‘Envy is the sin which most harms the sinner.’ Discuss.

4 Discuss the advantages and disadvantages of being born into a wealthy family.
Restricting candidates


Valid and reliable scoring
21
Giving no choice of tasks
Writing samples are long enough
Create appropriate scales for scoring



Activity: Criteria and standards
22
You are planning to have holiday abroad/read a
novel. Make a list of 4/5 factors that you would
take into consideration in choosing the
country/novel.

Spread out each of the factors in terms of three different levels or conditions. For example,

Security =
Absolutely safe> Somewhat risky> Dangerous





Rating scales
23
An ordered series of level descriptions – a ladder
with a number of rungs.
Cont’d
24
Number of levels
Definition of each level
Level descriptors:
Has fully operational command of the language with only occasional unsystematic inaccuracies and inappropriacies.

Basic competence is limited to familiar situations. Have frequent problems in understanding and expression. Not able to use complex language.















Cont’d
25
Level descriptors: wording and labels
HD> D> C> P> F
9> 8> 7> ….1
Expert> Very good> Good> Competent> Modest>
Limited…Non user
Cut point/hurdle
Top level
Native speaker standard (?)






Assessment criteria
3 major approaches to scoring writing performance:
Holistic
Analytical
Primary trait
Assessment criteria
27
Varies according to the context and the purpose of assessment.
Holistic scoring:
a single score is assigned to one task performance on the basis of overall impression.

Advantages:
Relatively high inter-rater reliability
Rapid scoring
Scores tend to emphasize the writer’s strengths
Scores represent ‘standards’ easily interpreted by lay people






Holistic scoring
28
Disadvantages
No diagnostic information (no washback potential)
Subjective and impressionistic and difficult to explain
Lower reliability


Analytic scoring
29
Considering a set of criteria and assigning scores to each of them (e.g., content, organization, vocabulary, grammar etc).
Advantages:
Consideration of uneven development of sub-skills.
Diagnostic information (enabling S to home in on weaknesses and to capitalize on strengths)
More reliable
Disadvantages:
Halo effect
Performance is more than the sum of its parts (wood and trees).
Time-consuming
Cont’d
30
Disadvantages:
Halo effect
Performance is more than the sum of its parts
(wood and trees).
Time-consuming





Primary trait scoring
31
A kind of holistic scoring which identifies a key characteristic or trait which is crucial to task performance.
(achievement of the primary purpose or trait of an essay is the only factor rated)
Task-specific and data-driven
e.g., the accuracy of the account of the original (summary)
The expression of the writer’s opinion (response to an article)








Cont’d
32
Advantages:
Zeroing on actual performance
Learning-friendly

Disadvantages:

Primary traits are difficult to write
Disregard for individuality
Not integrative enough
Time-consuming and expensive


Multiple trait scoring
33
Scoring on more than one facet or trait exhibited by the text.
Cf. analytic scoring. Task-focused criteria
and disregarding trivial features.
Advantages:
Increased validity and reliability
More diagnostic information
Positive washback effects
Disadvantages:
Difficult to identify traits appropriate for the given context.
Problems of analytic scoring



Steps in scale construction
34
Thinking and deciding:
Purpose of testing
Number of distinctions
Score reporting
Criteria of performance
Feedback
Type of scales – holistic or analytic, or both.
Examining existing scales
Calibrating the scale
Selecting and training raters



Subjectivity in scoring
35
Addressing issues of subjectivity
Rater training and moderation
Adhering to criteria of assessment
Multiple trait scoring more reliable than primary trait or holistic scoring
Multiple rating is more reliable than single rating
Measuring reliability coefficient
Classification analysis












Portfolio assessment
36
A collection of written work by learners over a period of time.
Exemplifies continuous/alternative assessment.

Advantages:
Focus on the process of writing and multiple drafts
Writing ability as developmental and sequential
Classroom environment as the basis of effective writing
Authentic and learner-centred
Activity
Review the advantages and disadvantages of the three kinds of scoring: holistic, primary trait and analytic. Construct a chart that shows how different contexts (types of test, objectives of a curriculum, proficiency levels) may benefit from each kind of scoring.
Leading Discussion
38
Group 5
Group 6