Considerable debate continues regarding the efficacy of the website RateMyProfessors.com (RMP). To date, however, virtually no direct, experimental research has been reported which directly bears on questions relating to sampling adequacy or item adequacy in producing what favorable correlations have been reported. The authors compare the data provided by RMP relative to experimentally controlled, systematically collected data that offers a more rigorous evaluation of the findings reported by RMP. Six equivalent undergraduate course sections with 252 students were systematically studied with respect to professor evaluations using simple RMP scales versus a more complex, multi-item scale. Major findings were twofold: (a) RMP samples were entirely inadequate to statistically project the evaluations of other students at conventionally accepted research levels; and (b) statistically significant differences between stated RMP scores compared to those using a more ample attribute approach were found, with RMP evaluations biased in a negative direction.