| 1 |
Reverse engineering language test constructs for Messick’s value implications: a sociolinguistic approach
|
|
|
|
Abstract:
Industrial, high volume, high stakes language testing has recently emerged, which annually assesses the language proficiency of tens of millions of people, worldwide. Important decisions are made using test results, so the tests and the scoring procedures should be theoretically sound. This study explores an aspect of Messick’s validity matrix which explicitly identifies the potential for validity threat where there are hidden social and cultural assumptions in the test construct. This facet of validity research and theorisation has been largely ignored by the language testing field. The problem of implicit assumptions in test constructs is approached through a theoretical framework which combines Bernstein’s model of the structuring of pedagogic discourse and the results of Systemic Functional Linguistics (SFL) informed text analysis. The framework allows connections between semantic and lexicogrammatical analysis of test responses and the social and cultural assumptions of the institution which evaluates that discourse. Analysis using SFL was pursued on the basis that the orientation to language which underpins the test construct under investigation, the International English Language Testing System (IELTS) General Training Version Written Module Task One (GTVWMT#1) is ill equipped to address questions of social and cultural assumptions implicit in texts, tasks, and test constructs. The IELTS GTVWMT#1 scoring criteria performance dimension Task Achievement, and its constituent elements ‘tone’ and ‘the requirements of the task’ were the particular focus of the study. The embodiment relationship between a test’s construct and the scoring criteria used to inform evaluation of test texts is an assumption of the study. Responses (n=54) to a single IELTS GTVWMT#1 prompt were collected from a diverse potential candidate population. The prompt demanded a friendly letter which communicated three main messages. The responses were double marked by trained IELTS raters. Quantitative analyses informed targeted qualitative analyses, including text analysis to identify similarities between groups of texts with respect to semantic constituency, realisation of critical semantic text elements, and Tenor. The study found that there are implicit expectations in the test construct related to social and cultural values and assumptions. These come from; 1. The particular task used in the study. There is potentially limited access to the domains of language use specified in this task. 2. The situational context of testing which has an impact on need for explicitness, on order of information, and on inclusion of detail. 3. Community expectations of thank you letters, particularly as they are entwined with notions of gracious behaviour. The major limitation of the study is the small sample size. Major recommendations include further work on the development of a model of text in social context to support language assessment, and consideration of the semantic constituency and potential for the impact of socially originating semantic variation in realisation of texts in high stakes language tests. ; Thesis (Ph.D.) -- University of Adelaide, School of Humanities, 2016.
|
|
Keyword:
assessing writing; Bernstein; Hasan; IELTS; language testing; SFL; systemic functional linguistics; tenor; thankyou letters; tone; validity
|
|
URL: http://hdl.handle.net/2440/113317
https://doi.org/10.25909/5b3d80e347ffe
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Style, structure and ideology in English and Vietnamese business hard news reporting : a comparative study.
|
|
|
|
BASE
|
|
Show details
|
|
|
| 3 |
A longitudinal study of developments in the academic writing of Thai university students in the context of a genre based pedagogy.
|
|
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Managing the subjective: exploring dialogistic positioning in undergraduate essays.
|
|
|
|
BASE
|
|
Show details
|
|
|
| 5 |
New-arrival-ness as a social construct: a qualitative case study.
|
|
|
|
BASE
|
|
Show details
|
|
|
| 6 |
Students’ preparation for IELTS: development of written and oral argumentative texts.
|
|
|
|
BASE
|
|
Show details
|
|
|
|
|
