Performance in everyday spatial orientation tasks (e.g., map reading and navigation) has been considered functionally separate from performance on more abstract object-based spatial abilities (e.g., mental rotation and visualization). However, few studies have examined the link between spatial orientation and object-based spatial skills, and even fewer have done so including a wide range of spatial tests. To examine this issue and more generally to test the structure of spatial ability, we used a novel gamified battery to assess six tests of spatial orientation in a virtual environment and examined their association with ten object-based spatial tests, as well as their links to general cognitive ability (g). We further estimated the role of genetic and environmental factors in underlying variation and covariation in these spatial tests. Participants (N = 2660; aged 19-22) were part of the Twins Early Development Study. The six tests of spatial orientation clustered into a single 'Navigation' factor that was 64% heritable. Examining the structure of spatial ability across all 16 tests, three, substantially correlated, factors emerged: "Navigation, Object Manipulation," and "Visualization." These, in turn, loaded strongly onto a general factor of Spatial Ability, which was highly heritable (84%). A large portion (45%) of this high heritability was independent of g. The results point towards the existence of a common genetic network that supports all spatial abilities.