This study examined the validity of data collected from a novel online story retell task. The task was specifically designed for use by junior school teachers with the support of speech-language therapists or literacy specialists. The assessment task was developed to monitor children's oral language progress in their first year at school as part of the Better Start Literacy Approach for early literacy teaching. Teachers administered the task to 303 5-year-olds in New Zealand at school entry and after 20 weeks and 12 months of schooling. The children listened to a story with pictures via iPad presentation and were then prompted to retell the story. The children's spontaneous language used in their story retell was captured and uploaded digitally via iPad audio recording and analyzed using semi-automated speech recognition and computer software. Their responses to factual and inferential story comprehension questions were also analyzed. The data suggested that the task has good criterion validity. Significant correlations between story retell measures and a standardized measure of children's oral language were found. The Better Start Literacy Approach story retell task, which took approximately 6 min for teachers to administer, accurately identified children with low oral language ability 81% of the time. Growth curve analysis revealed that the task was useful for monitoring oral language development, including for English as second language learners. Boys showed a slower story comprehension growth trajectory than girls. The Better Start Literacy Approach story retell task shows promise in providing valid data to support teacher judgement of children's oral language development.