So is Quality Rating Tool 2 any better? We will now hone in on our data to see if we have achieved better reliability with Tool 2. The following 4 charts show the inter-rater reliability on three subscales of version 2 of the e-Book Quality Rating Tool. Ten independent raters evaluated 5 books by assigning each book a score between 1 (strongly disagree) and 5 (strongly agree) on ease of use, multimedia and interaction features. Ratings were considered reliable if they fell within +1 point of the rating assigned by the Expert Rater, the project technology expert.

Chart 1 shows strong inter-rater reliability (89%) for Ease of Use ratings using the e-Book Quality Rating Tool.

Chart 2 shows relatively strong inter-rater reliability (80%) for Multimedia ratings using the e-Book Quality Rating Tool 2.

Chart 3 shows that Interaction ratings using the e-Book Quality Rating Tool 2 are not as robust (77%) as the previous 2 scales.

A closer look at the Interaction ratings of the expert and other raters reveals large discrepancies between expert and raters, most predominately in Books 1, 3 and 5.

Here is a summary of the data gathered to date from the use of Tool 2. Our corpus consists of nearly 100 unique e-Book titles that have been rated by one external rater. There is an overlap of 16 e-Book titles that have been rated by 1 or more external rater.

What we see in the data is that e-Book authors and developers have found a relatively decent method of creating simple and easy to use e-Books for both teacher and young children. In essence, we have found a fairly adequate way to digitally simulate the book handling experience. We see attributes such as cover, title and author consistently in e-Books, as well as the ability to turn pages in a simple and effective manner.

The data also reveals that e-Book developers are investing heavily in multimedia components to help “tell the story.” However, among the multimedia components found in our e-Book corpus, relatively few of them are rated as highly supportive of story comprehension for an emerging reader. In fact, we see that most fall within the low end of our rating scale.

Interaction rating data reveals that very few e-Book developers are including opportunities for young children to interact with print, illustration and other book features. We have found that most e-Books do not include interactions that scaffold and support the learning loop for emerging readers, with 95% of the e-Book corpus falling within the low end of our rating scale.

The data shows a garden-variety population of moderate-to-low quality e-Books. The mean overall rating is 2.84 on a 5-point scale. With these findings, it is clear that good e-Books for the purposes of literacy instruction for young children are hard to find. e-Books equipped with learning supports, scaffolds and built-in content resources are, in fact, very rare.


In the week leading up to my AERA 2011 Annual Meeting presentation, I will be sharing parts of the presentation on Raised Digital. The paper I will be presenting focuses on the development of my e-Book Quality Rating Tool and is part of a symposium titled E-Books as Instructional Tools in Preschool Classrooms: Promises and Pitfalls. The symposium will take place Saturday, April 9, 2011 from 2:15pm – 3:45pm, at the Doubletree/Madewood B.