##
**The common ancestor process for a Wright-Fisher diffusion.**
*(English)*
Zbl 1127.60079

Summary: Rates of molecular evolution along phylogenetic trees are influenced by mutation, selection and genetic drift. Provided that the branches of the tree correspond to lineages belonging to genetically isolated populations (e.g., multi-species phylogenies), the interplay between these three processes can be described by analyzing the process of substitutions to the common ancestor of each population. We characterize this process for a class of diffusion models from population genetics theory using the structured coalescent process introduced by N. L. Kaplan, T. Darden and R. R. Hudson [The coalescent process in models with selection. Genetics 120, 819–829 (1988)] and formalized by N. H. Barton, A. M. Etheridge and A. K. Sturm [Ann. Appl. Probab. 14, No. 2, 754–785 (2004; Zbl 1060.60100)]. For two-allele models, this approach allows both the stationary distribution of the type of the common ancestor and the generator of the common ancestor process to be determined by solving a one-dimensional boundary value problem. In the case of a Wright-Fisher diffusion with genic selection, this solution can be found in closed form, and we show that our results complement those obtained by P. Fearnhead [J. Appl. Probab. 39, No. 1, 38–54 (2002; Zbl 1001.92037)] using the ancestral selection graph. We also observe that approximations which neglect recurrent mutation can significantly underestimate the exact substitution rates when selection is strong. Furthermore, although we are unable to find closed-form expressions for models with frequency-dependent selection, we can still solve the corresponding boundary value problem numerically and then use this solution to calculate the substitution rates to the common ancestor. We illustrate this approach by studying the effect of dominance on the common ancestor process in a diploid population. Finally, we show that the theory can be formally extended to diffusion models with more than two genetic backgrounds, but that it leads to systems of singular partial differential equations which we have been unable to solve.

### MSC:

60J70 | Applications of Brownian motions and diffusion theory (population genetics, absorption problems, etc.) |

92D10 | Genetics and epigenetics |

92D20 | Protein sequences, DNA sequences |