This paper explores problems of construct validity in the evaluation of the Follow Through Planned Variation program. Comments center around political ramifications of the program evaluation design, insufficient knowledge on the part of program planners, the diversity of the experimental options, and the instability of the sample. It is suggested that the political problem of whether the program was to operate in a service or an experimental mode resulted in confusion over what sort of information an evaluation was to provide; this made it impossible to construct and validate instruments for the multidimensional program that Project Follow Through was. It was assumed that expected program impact would be great enough to overcome the marginal biases of the standardized tests used. The fact that planners were forced to start the evaluation with insufficient information about constructs being addressed by different program developers, and insufficient time to develop or select an appropriate testing battery, resulted in a test battery of questionable construct validity. Also, because of the longitudinal nature of the evaluation, evaluators were not free to choose alternate batteries. Another problem resulted from the varying number of replications: among the 22 experimental options funded, replications ranged from one to 14. Finally, sample attrition exceeded 50 percent. It is concluded that more attention to construct validity is needed if meaningful educational evaluation is to be done. (SB)