Core Academic Language Skills (CALS) Assessment Design and Development

Over the last three years, as part of the Catalyzing Comprehension Through Discussion Debate project funded by a grant from IES to the Strategic Educational Research Partnership, our team has produced a research-based, theoretically-grounded and psychometrically robust criterion-referenced Assessment of Academic Language (ALA) skills for students in grades 4-8. This assessment has enabled us to directly measure a larger constellation of academic language skills that go beyond academic vocabulary and to offer direct evidence of strong associations between these skills and reading comprehension (Phillips-Galloway, Stude, Uccelli, in press; Uccelli, Barr, Dobbs, Phillips-Galloway, Meneses, & Sánchez, under review).The  ALA assess students' CoreAL skills, i.e., language skills that co-occur with learning tasks in academic contexts (e.g., discourse markers, genre markers, nominalizations and other forms used to present information compactly and precisely). The ALA is a paper and pencil test, administered in whole-class settings, within one class period (45 minutes).  It includes nine tasks: Connecting Ideas, Tracking Themes, Organizing Texts, Breaking Words, Comprehending Sentences, Identifying Definitions, Epistemic stance (Sure or Unsure?), Metalanguage (Understanding responses). Tasks assess a range of skills through multiple choice, matching, or short written responses. 

Using Rasch modeling, an initial set of items (distributed across the first seven tasks listed above) was selected from a larger pool of items administered to a cross-sectional sample of 4th to 8th graders. Reliability was investigated and found to be robust at .92 as indexed by coefficient alpha and at .82 by split-half reliability of even vs. odd numbered items. A confirmatory factor analysis (CFA) applied to the final set of items suggested a single factor: (CFI=.93, TLI = .92, RMSEA <.05). The zero order within-grade correlations between the ALA total score and the Massachusetts Comprehensive Assessment (MCAS-ELA) (a not vertically-equated state-wide assessment) were statistically significant and ranged from .41 for grade 7 to .77 for grade 6.  Multiple linear regression analyses within-grade revealed that students’ ALA scores were significant predictors of MCAS-ELA scores, even after controlling for reading fluency. Construct validity of the items and the ALA assessment as a whole was further explored through an external review completed by five experts in the field of academic language research. These experts’ reviews generated positive evaluations and their suggestions for improvement of particular items were carefully considered (Uccelli, Barr, Dobbs, Phillips-Galloway, Meneses, & Sánchez, under review). In a second pilot study, data from another cross-sectional sample of 4th to 8th graders revealed that, beyond the contribution of reading fluency and depth of vocabulary knowledge, the performance in the ALA was a significant predictor of reading comprehension as measured by the Gates-McGinitie reading comprehension test. After the final item selection on each form, two single factor CFA models were examined and found to fit the data well (Phillips-Galloway, Stude, Uccelli, in press).

Funding Source

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305F100026 to the Strategic Education Research Partnership as part of the Reading for Understanding Research Initiative. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Presentations & Publications

Uccelli, P., Barr, C. D., Dobbs, C. L., Galloway, E. P., Meneses, A., & Sánchez, E. (2014).Core academic language skills: An expanded operational construct and a novel instrument to chart school-relevant language proficiency in preadolescent and adolescent learners. Applied Psycholinguistics, 1-33 (Read this article).

Uccelli, P., Phillips Galloway, E., Dobbs, C., & Ronfard, S., (2013, April). General Academic Language Proficiency: A Key Predictor of Adolescents’ Reading Comprehension. Presented at the Society of Research in Child Development Biannual Meeting, Seattle, Washington. 

Phillips Galloway, E., Uccelli, P., Barr, C., (2013, April). Modeling the Relationship Between Lexical, Grammatical, and Discourse Structure Knowledge and Academic Writing Proficiency for Middle-Grade Writers. Presented at the American Educational Research Association Annual Conference, San Francisco, California.

Uccelli, P., Barr, C., Dobbs, C., Phillips Galloway, E., Meneses, A., Sanchez, E., (2013, April). Identifying Cross-Disciplinary Academic Language Skills Throughout the Middle School Years. Presented at the American Educational Research Association Annual Conference, San Francisco, California.

Phillips Galloway, E. & Uccelli, P. (2013, March).  Anticipating the challenges of complex texts for middle grade EL learners: A focus on academic language.  Presented at the Regional Educational Laboratory Southeast, Miami, Florida.

Uccelli, P., Phillips Galloway, E.,  & Dobbs, C. (2012, October). Assessing Academic Language: A workshop. Presented at the Center for Research on the Educational Achievements and Teaching of English Language Learners Conference entitled: English language learners in content areas: Teaching for Achievement in the Middle Grades, Orlando, Florida.

Stude, J., Phillips Galloway, E. & Uccelli, P. (2012, August). Negotiating communicative practices in schools: Student’s reelections on the academic register. Presented at Sociolinguistics Symposium 19, Berlin, Germany.

Uccelli, P., Meneses, A. , Phillips Galloway, E., Barr, C. (2012, July). To define nouns: An academic challenge that reveals later-language development in adolescent students. Presented at the Society for Text and Discourse conference, Montreal, Canada.