Behavioral inhibition (BI) is an early-appearing temperamental trait characterized by intense negative affect and withdrawal behaviors to novel and challenging situations. Inhibited children are more likely to display social withdrawal and experience an increased risk for internalizing problems. Trait inference, the way children interpret behaviors and infer the characteristics of others, may be one potential mechanism. The current study examined the individual differences of BI in trait inference using a cross-sectional design (Study 1, Age 4, N = 80, 38 boys, Chinese) and explored the role of trait inference (Age 4) in relation to early BI (Age 2) and later internalizing behaviors (Age 4) using a longitudinal design (Study 2, N = 163, 93 boys). Trait inference was measured by children's personality judgments after watching four pieces of behavioral information of animated actors in two conditions: a high positive information condition and a high negative information condition. We found that high BI children exhibited a more negative trait inference than low BI children did in two studies, except for the trait inference in the high negative information condition of Study 2. Besides, in Study 2, trait inference in the high positive information condition partially mediated the BI-internalizing behavior link, and such mediation effect was moderated by the level of trait inference. These findings highlight the contribution of social information processing in the social adjustment of children with extreme temperament and have implications for prevention and intervention.