ERIC Number: EJ1457360
Record Type: Journal
Publication Date: 2025-Feb
Pages: 33
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0013-1644
EISSN: EISSN-1552-3888
Evaluating the Predictive Reliability of Neural Networks in Psychological Research with Random Datasets
Yongtian Cheng; K. V. Petrides
Educational and Psychological Measurement, v85 n1 p5-37 2025
Psychologists are emphasizing the importance of predictive conclusions. Machine learning methods, such as supervised neural networks, have been used in psychological studies as they naturally fit prediction tasks. However, we are concerned about whether neural networks fitted with random datasets (i.e., datasets where there is no relationship between ordinal independent variables and continuous or binary-dependent variables) can provide an acceptable level of predictive performance from a psychologist's perspective. Through a Monte Carlo simulation study, we found that this kind of erroneous conclusion is not likely to be drawn as long as the sample size is larger than 50 with continuous-dependent variables. However, when the dependent variable is binary, the minimum sample size is 500 when the criteria are balanced accuracy [greater than or equal to] 0.6 or balanced accuracy [greater than or equal to] 0.65, and the minimum sample size is 200 when the criterion is balanced accuracy [greater than or equal to] 0.7 for a decision error less than .05. In the case where area under the curve (AUC) is used as a metric, a sample size of 100, 200, and 500 is necessary when the minimum acceptable performance level is set at AUC [greater than or equal to] 0.7, AUC [greater than or equal to] 0.65, and AUC [greater than or equal to] 0.6, respectively. The results found by this study can be used for sample size planning for psychologists who wish to apply neural networks for a qualitatively reliable conclusion. Further directions and limitations of the study are also discussed.
Descriptors: Psychological Studies, Artificial Intelligence, Cognitive Processes, Predictive Validity, Predictor Variables, Test Reliability, Randomized Controlled Trials, Sample Size
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://bibliotheek.ehb.be:2993
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A