Bayesian methods seem a natural choice for combining sources of evidence in meta-analyses. However, in practice, their sensitivity to the choice of prior distribution is much less attractive, particularly for parameters describing heterogeneity. A recent non-Bayesian approach to fixed-effects meta-analysis provides novel ways to think about estimation of an average effect and the variability around this average. In this paper, we describe the Bayesian analogs of those results, showing how Bayesian inference on fixed-effects parameters--properties of the study populations at hand--is more stable and less sensitive than standard random-effects approaches. As well as these practical insights, our development also clarifies different ways in which prior beliefs like homogeneity and correlation can be reflected in prior distributions. We also carefully distinguish how random-effects models can be used to reflect sampling uncertainty versus their use reflecting a priori exchangeability of study-specific effects, and how subsequent inference depends on which motivation is used. With some important theoretical results, illustrated in an applied meta-analysis example, we show the robustness of the fixed effects even for small numbers of studies.