Guidelines for Rubric Design and Assessment Scoring
Peggy L. Maki is internationally recognized for her work in helping faculty construct assessment practices that support enhanced student learning. Her work guides much of how CSUMB has designed its assessment process. A brief excerpt from her Chapter 5 of her book Assessing for Learning: Building a Sustainable Commitment across the Institution is summarized here.
The purpose of the rubric is to translate an outcome into a set of criteria (elements of the outcome) and standards of performance that can be used to assess how well the student work demonstrates the outcome. As Maki writes, “[r]esults of applying these criteria provide evidence of learning patterns. At both the institution and program levels these patterns identify students’ areas of strength as well as weakness. Interpreting patterns of weakness leads to adjustments or modifications in pedagogy; curricular, co-curricular, and instructional design; and education practices and opportunities” (Maki, p. 121). At the same time, patterns of strength point us to practices that can be adopted and adapted more widely. Maki provides a step-by-step approach in the excerpt (pp. 124-126). TLA has an archive of sample rubrics from a range of disciplines and the AAC&U VALUE rubrics are great starting points for Essential Learning Outcomes.
Once faculty members have agreed on a rubric, it’s important to spend time working together to “calibrate” or “norm.” This generally involves having participants read and score a sample of student work, share their scores and discuss how they chose the scores. The goal is to ensure a shared understanding of the rubric and establish relative agreement on how to apply the standards to specific work. If there is substantial disagreement on the initial sample, the process should be repeated with another sample. Agreement will never be perfect, but establishing a reasonable level of inter-rater reliability is crucial. This is not only to serve the purposes of the assessment, but to ensure that the program faculty are in agreement about what they are expecting of students. Once the norming is completed, at least two readers score each student sample, with the person managing the reading monitoring scores for inter-rater reliability. Where there is a pattern of substantial disagreement between first and second reader (e.g., the first reader is consistently higher than the second), the manager can ask the readers to consult and come to agreement. This should bring readers back into line with the group norm. Again, the excerpt from Maki has a step-by-step approach (pp. 126-127).
CSUMB subscribes to a web-based application, Taskstream, that allows us to enter student samples and rubrics. Faculty can access these electronically and enter their scores and comments. We strongly encourage programs to utilize Taskstream because of the ease of access and, more importantly, because it creates an archive of student work and data about student achievement of learning outcomes that can be used over time. TLA can prepare Taskstream (i.e., enter student work and rubrics), train faculty in its use, help manage assessment, and work with programs to produce reports.