New Student Portal and Test Player: Field Trials

Print Friendly

Background 

The focus of the interface enhancement project was to replace the old flash test player.  The challenge was to develop a new player that could replace the old player while modernising the player interface and making it accessible from a range of devices without impacting students test performance.

Student results obtained using the old player needed to be directly comparable to their results obtained using the new player on a broader range of devices.

For this reason, changes were focused on positional changes of the three major elements of the test player. These are the reading passage or question, the response and the question navigation bar.

The reading passage or question previously occupied the top of the screen. It now occupies the left side of the screen. The question previously appeared under the reading passage below a screen splitter. It now appears on the right side of the screen. The navigation bar was on the left side of the screen. It is now at the bottom of the screen. See Test Player Comparison  for more details.

No changes were made to the wording, content of the questions or the reading passages. The response options for each question remain the same, previous student test scores were unchanged and the test generation processes used by e-asTTle are the same. All reports remain the same.

For this reason it was expected that students’ performance would be unaffected by the high level layout changes introduced by the new player.

Field Trialling

To test this expectation, trialling was undertaken using Reading tests with Year 7 students.  Two trials were run with approximately 100 students each. The first trial compared students’ performance on the old and new players. The second trial compared students’ performance using the new player on desktops and tablets.

During these trials, students sat two tests each and the effect of test order was controlled.

Trial 1: Player Comparability trial

Comparison of Flash test player and the new test player. 99 Year 7 students sat a different Level 3-4 reading test on each of the test player versions.  No significant difference was found between students’ scores across the Flash and HTML versions.  80% of students preferred the HTML player.

Trial 2: Device Comparability trial

Comparison of the new player used on laptops (Chromebooks) and tablets (iPads). 98 Year 7 students sat a different Level 3-4 reading test on each of a Chromebook and an iPad. No significant difference was found between students’ scores across the laptop and tablet versions. The results involved different students but produced similar test results to the first trial.

Overall, there was no evidence to suggest that the device used or the layout changes had a significant impact on students’ performance.

Why was trialling done using Reading tests and Year 7 students?

 Year 7 students were used because intermediate students are the heaviest users of e-asTTle. In addition, Year 7 students are halfway between the youngest and oldest students using e-asTTle.

Reading tests were used because more scrolling is typically required in Reading tests. If there was to be a performance difference relating to the reduction in scrolling or the layout differences between tablets and desktops, this difference would be expected to be greater in Reading tests.

Usability Testing

Throughout the e-asTTle interface project four primary schools, four intermediate schools and two high schools participated in usability testing.

Students sat tests on a range of different devices, including tablets (iPad/Android) and laptop/desktop machines (Mac/Windows/Chromebook). Students’ feedback was invaluable and was incorporated into the new interface design.

Teacher input was sought for the new online administration guidelines and for the new practice test scripts.

In addition to the trials focusing on student performance, user acceptance testing was undertaken on the final version of the test player in Term 4, 2015. Teachers from three schools administered a test to a group of students, using the new administration guidelines, practice tests and the new test player. A range of devices were used during the user acceptance testing. In addition, both Reading and Maths tests, as well as Custom and Adaptive tests were used.