Grounded in the sociocultural nature of literacies and informed of the inherent biases in widely used, English-dominant reading assessments in U.S. schools, this case study traces the planning, development, and pilot administration (n = 52) of a culturally inclusive (i.e., participant informed), online reading assessment. The Critical Reading Assessment (CRA) is designed to gauge elementary students' comprehension and critical reasoning (i.e., identifying potential biases or instances of diversity, equity and inclusion) of digital, multimodal texts. Findings from our analysis of recorded pilot sessions with student participants, who are predominantly Spanish/English multilingual learners, suggest (a) the importance of transparency and feedback from multiple stakeholders in the assessment development process; (b) the potential affordance of multiple textual modalities for clarifying comprehension skills and abilities; (c) the potential negative consequences of using established, dominant-English reading tests for determining comprehension abilities; and (d) the need for greater opportunities to practice critical discussions (i.e., questions about perspectives, representation, and other potential biases) about texts. Implications from this study highlight the need for supporting elementary students and their teachers in dialogic, critical reading practices of multimodal textual information.