Recent work in reinforcement learning has demonstrated a choice preference for an option that has a lower probability of reward (A) when paired with an alternative option that has a higher probability of reward (C), if A has been experienced more frequently than C (the frequency effect). This finding is critical as it is inconsistent with widespread assumptions that expected value is based on average reward, and instead suggests that value is based on cumulative instances of reward. However, option frequency may also affect instrumental reinforcement of choosing A during training, which may then transfer to choice on AC trials. This study therefore aimed to assess the contribution of action reinforcement and option value to the frequency-effect across 2 experiments. In both experiments we included an additional test phase in which participants were asked to rate the likelihood of reward for each choice option, a response that should be unaffected by action reinforcement. In Experiment 1, participants completed the original choice training phase. In Experiment 2, participants were presented with each option individually, thus removing reinforcement of choice during training. Single cue training reduced the strength of the preference for A compared to choice training, suggesting a contributing role of action reinforcement. However, frequency effects were still evident in both experiments. We found that the pattern of reward likelihood ratings was consistent with the pattern of choice preferences in both experiments, suggesting that action reinforcement may also influence judgements about the likelihood of receiving reward.