Pathogens evolve different life-history strategies, which depend in part on differences in their host populations. A central feature of hosts is their population structure (e.g. spatial). Additionally, hosts themselves can exhibit different degrees of symptoms when newly infected; this latency is a key life-history property of pathogens. With an evolutionary-epidemiological model, we examine the role of population structure on the evolutionary dynamics of latency. We focus on specific power-law-like formulations for transmission and progression from the first infectious stage as a function of latency, assuming that the across-group to within-group transmission ratio increases if hosts are less symptomatic. We find that simple population heterogeneity can lead to local evolutionarily stable strategies (ESSs) at zero and infinite latency in situations where a unique ESS exists in the corresponding homogeneous case. Furthermore, there can exist more than one interior evolutionarily singular strategy. We find that this diversity of outcomes is due to the (possibly slight) advantage of across-group transmission for pathogens that produce fewer symptoms in a first infectious stage. Thus, our work reveals that allowing individuals without symptoms to travel can have important unintended evolutionary effects and is thus fundamentally problematic in view of the evolutionary dynamics of latency.