Measurement Error in e-asTTle

Print Friendly

A score of any kind is always an estimate of “true” performance. Usually, if you were to re-test the same student again, their score would be at least slightly different.

Even raw scores, such as ‘8/10’ are often treated as if they are 100% precise, however even these are subject to error.

How can we know how precise a score is?

We can estimate how widely spread out a student’s scores would be, if they were re-tested again and again. This calculation is called the standard error of measurement, or the measurement error.

A common analogy for the SEM is shooting arrows at a target. The SEM is how closely the arrows cluster together. According to the Rasch Model used in e-asTTle, on a given test, it is the very high and very low scores that have the highest SEM, and the scores in the middle with the lower SEM. The length of the test is also a factor – longer tests tend to have smaller SEM.

How precise are e-asTTle scores?

On average, the measurement error of measurement is +/-22 points for e-asTTle Maths and Reading, and +/-40 points in e-asTTle Writing.

One way of interpreting this is:

  • For Reading and Maths, scores that differ by less than 44 points (22 x 2) tend not to be meaningfully different – they could essentially be the same score, given chance factors involved in assessment.
  • For Writing, scores that differ by less than 80 points tend not to be meaningfully different – they could essentially be the same score, given chance factors involved in assessment.

However, of course, this is just an average. The measurement error of measurement can be calculated for each instance of a student sitting a test.

For Reading, Maths and Writing, you can find the SEM for specific students by looking at their Individual Learning Pathway.Image: Measurement Error

The SEM displays in two wayss:

- As a +/- value next to the student’s score

- The circle on the score arrow represents this graphically. There is a good probability that the student’s true score falls within the red circle.

 

Measurement error differences across subjects:

The measurement error is calculated slightly differently depending on subject:

Maths and Reading:

The measurement error is calculated based on the specific student’s score and test.

- Longer tests will generally have smaller measurement errors than shorter tests.

- Students achieving in the middle of the test (i.e. ~ 50% correct) will have smaller measurement errors than students who score high or low.

Writing:

Similarly to Reading and Maths, there is an estimate of the error associated with inconsistencies in how students respond to the Writing test. The measurement error will be higher for students with very low or high rubric scores.

An additional error component is added to this estimate to represent the variability in how different markers apply the marking rubrics.

Maori-Medium subjects (Pangarau, Panui, Tuhituhi):

An average measurement error of measurement (22 points) is used.