Physical fitness is one of the key factors in healthy aging. Although physical fitness is widely recognized to decline with age, age-related decreases in the individual dimensions of physical fitness in later life are less clear. Purpose: The purpose of the present study was to describe the age-related changes in six fitness components covering a variety of motor dimensions in men and women through a 12-year longitudinal epidemiological study. Method: Participants were randomly selected community-living men (n = 1,139) and women (n = 1,128) aged 40 to 79 years at baseline who were repeatedly tested over a mean (standard deviation) follow-up period of 9.8 (3.4) years with a mean of 4.9 (2.3) examinations. The six physical fitness tests were sit-and-reach, grip strength, sit-ups, one-leg standing, reaction time and leg extension power. The effect of age on each physical fitness task was analyzed using a mixed-effects model controlling for habitual physical activity level. Results: The fixed effect of the interaction of age and time in all fitness tests was significant in men. Higher age at baseline was significantly associated with a greater decrease over time except in one-leg standing. In women, no significant interaction effect of age and time was found in grip strength or reaction time. Age-related changes in grip strength and reaction time were constant with increasing age. Conclusion: Twelve-year longitudinal data showed that the age-related decrease in physical fitness over time, especially musculoskeletal fitness in men, was pronounced, whereas in women, the decrease was less pronounced.