Mobile technologies, including apps, have become increasingly popular, and are being used to support daily activities among a variety of individuals. While the use of mobile technologies will not eliminate barriers often faced by individuals with disabilities, these systems have the potential to help minimize some of these barriers. As the popularity of apps is increasing, the purpose of this study was to evaluate the reliability, internal consistency, and social validity among novice raters on two app evaluation rating scales. A total of 17 adults, with and without identified disabilities, evaluated apps using two team-designed app rating scales. Overall, findings indicated that the ratings completed during the pilot phase by the research team were more reliable than those completed by novice raters during the testing phase; that the dimension of individualization was the most reliable among team raters and novice participants without disabilities; and that the highest level of inconsistency in the reliability was among novice participants with disabilities. Practical implications, limitations, and future research directions are discussed.