Symmetry breaking for representations of rank one orthogonal groups.

*(English)*Zbl 1334.22015
Mem. Am. Math. Soc. 1126, v, 110 p. (2015).

The memoir is written by leading experts in representation theory of Lie groups. It contains 16 chapters, including an introduction and an appendix for a total of 112 pages. The main topic is the study of restrictions of representations of real rank one (indefinite) orthogonal groups.

Let \(G\) be a Lie group and \(G'\) a subgroup of \(G\). An important problem is to understand the restriction to \(G'\) of an irreducible representation \(\pi\) of \(G\), also known as symmetry breaking. When \(G\) is compact this question is well understood. For instance, every representation of \(G\) is unitary and \(\pi_{\mid_{G'}}\) is isomorphic to a direct sum of irreducible representations \(\pi'\) of \(G'\) with multiplicity \(m(\pi,\pi')\). Moreover, in the formulation of representation theory of (connected) compact groups in terms of coadjoint orbits, the representation \(\pi\) is associated with the orbit through the highest weight of \(\pi\) and \(m(\pi,\pi')\) can be explicitly described in terms of partition functions. When \(\pi\) is infinite dimensional and \(G'\) is non-compact, the restriction \(\pi_{\mid_{G'}}\) is no longer a direct sum of irreducible representations.

Let \(H\) be a closed subgroup of \(G\) and \((\lambda,V)\) a finite-dimensional complex representation of \(H\). One gets a homogeneous vector bundle \({\mathcal V}:=G\times_HV\) over the homogeneous space \(X:=G/H\). The space \(C^\infty(X,{\mathcal V})\) of smooth sections identifies with \(V\)-valued smooth functions on \(G\) satisfying the equivariance relation: \[ f(gh)=h^{-1}\cdot f(g)\;\;\; \forall g\in G,h\in H. \] The group \(G\) acts by left translations on \(C^\infty(X,{\mathcal V})\). Similarly, if \(H'\) is a subgroup of \(G'\) and \((\nu,W)\) is a finite dimensional complex representation of \(H'\), one gets a \(G'\)-module \(C^\infty(Y,{\mathcal W})\) where \(Y:=G'/H'\) and \({\mathcal W}:=G\times_HW\). A continuous \(G'\)-homomorphism \(C^\infty(X,{\mathcal V})\rightarrow C^\infty(Y,{\mathcal W})\) is called a symmetry breaking operator. The multiplicity \(m(\pi,\pi')\) is the (possibly infinite) dimension of the space \(H(\lambda,\nu)\) of symmetry breaking operators. In the paper under review, the authors provide a complete self-contained study and classification of symmetry breaking operators in the case of indefinite orthogonal groups.

From now on, suppose \(G\) is the indefinite orthogonal group \(O(n+1,1)\) and \(G'\) is the subgroup \(O(n,1)\) realized as follows, \(n\geq 1\). \(G\) is the group preserving the quadratic form \(x_0^2+x_1^2+\dots+x_n^2-x_{n+1}^2\) of signature \((n+1,1)\). The subgroup \(G'\) is the stabilizer of the vector \(e_n=\text{}^{t}(0,\dots,0,1,0)\) in \(\mathbb R^{n+2}\). The subgroup \(K:=O(n+1)\times O(1)\) is a maximal compact subgroup of \(G\) and \(K':=K\cap G'\) is a maximal compact subgroup of \(G'\). Let \({\mathfrak g}={\mathfrak o}(n+1,1)\) (resp. \({\mathfrak g}'={\mathfrak o}(n,1))\) be the Lie algebra of \(G\) (resp. \(G'\)). With the usual notation \(\{E_{i,j}\}_{0\leq i,j\leq n+1}\) for the canonical basis of \(gl(n+2,{\mathbb R})\), fix the hyperbolic element \(H:=E_{0,n+1}+E_{n+1,0}\) in \({\mathfrak g}'\). Then \(\text{ad}(H)\), as an endomorphism of \({\mathfrak g}\), has eigenvalues \(0\) and \(\pm 1\). Set \({\mathfrak a}:={\mathbb R}H\), \({\mathfrak n}_{\pm}:=\text{Ker}(\text{ad}(H)\pm 1)\), \(A:=\exp({\mathfrak a})\), \(N_{\pm}:=\exp({\mathfrak n}_{\pm})\) and \(M:=Z_K({\mathfrak a})\simeq O(n)\times{\mathbb Z}_2\). Then \(P:=MAN_+\) is a Langlands decomposition of a minimal parabolic subgroup of \(G\). Likewise \(P':=M'AN_+'\) is a Langlands decomposition of a minimal parabolic subgroup of \(G'\) with \(M':=Z_{K'}({\mathfrak a})\simeq O(n-1)\times{\mathbb Z}_2\), \(N_{\pm}':=\exp({\mathfrak n}_{\pm}')\) and \({\mathfrak n}_{\pm}':={\mathfrak n}_{\pm}\cap{\mathfrak g}'\). The fact that \(M\) and \(M'\) are both compact, with \(P\) and \(P'\) having the same \(A\)-part, makes the study of the space \(H(\lambda,\nu)\) of symmetry breaking operators more tractable.

Chapter 2 contains the proof of results about \(G'\)-intertwining operators between irreducible composition factors of spherical principal series representations and symmetry breaking operators for spherical principal series representations of \(G\) and \(G'\). Chapter 3 provides a method to study symmetry breaking operators for induced representations by means of their distribution kernels. Chapter 4 contains the realization in the non-compact picture of spherical principal series and a normalization \(\widetilde{{\mathbb T}}_\lambda\) of the Knapp-Stein intertwining operators \({\mathbb T}_\lambda\), while Chapter 5 deals with the double coset decompositions \(G'\backslash G/P\) and \(P'\backslash G/P\). Chapters 6 to 10 prepare the complete classification of symmetry breaking operators from principal series representations of \(G\) to those of \(G'\) contained in Chapter 11, based on some results proved in Chapters 12 and 13. As a byproduct, the authors analyse vector bundles over the anti-de Sitter spaces \(G/G'\) in Chapter 14, while in Chapter 15 they obtain an explicit construction of the complementary series representations of \(G\) along with their branching laws. Finally, Chapter 16 is an appendix which collects useful information about Gegenbauer polynomials, (normalized) \(K\)-Bessel functions and Zuckerman derived functor modules \(A_{\mathfrak q}(\lambda)\).

Let \(G\) be a Lie group and \(G'\) a subgroup of \(G\). An important problem is to understand the restriction to \(G'\) of an irreducible representation \(\pi\) of \(G\), also known as symmetry breaking. When \(G\) is compact this question is well understood. For instance, every representation of \(G\) is unitary and \(\pi_{\mid_{G'}}\) is isomorphic to a direct sum of irreducible representations \(\pi'\) of \(G'\) with multiplicity \(m(\pi,\pi')\). Moreover, in the formulation of representation theory of (connected) compact groups in terms of coadjoint orbits, the representation \(\pi\) is associated with the orbit through the highest weight of \(\pi\) and \(m(\pi,\pi')\) can be explicitly described in terms of partition functions. When \(\pi\) is infinite dimensional and \(G'\) is non-compact, the restriction \(\pi_{\mid_{G'}}\) is no longer a direct sum of irreducible representations.

Let \(H\) be a closed subgroup of \(G\) and \((\lambda,V)\) a finite-dimensional complex representation of \(H\). One gets a homogeneous vector bundle \({\mathcal V}:=G\times_HV\) over the homogeneous space \(X:=G/H\). The space \(C^\infty(X,{\mathcal V})\) of smooth sections identifies with \(V\)-valued smooth functions on \(G\) satisfying the equivariance relation: \[ f(gh)=h^{-1}\cdot f(g)\;\;\; \forall g\in G,h\in H. \] The group \(G\) acts by left translations on \(C^\infty(X,{\mathcal V})\). Similarly, if \(H'\) is a subgroup of \(G'\) and \((\nu,W)\) is a finite dimensional complex representation of \(H'\), one gets a \(G'\)-module \(C^\infty(Y,{\mathcal W})\) where \(Y:=G'/H'\) and \({\mathcal W}:=G\times_HW\). A continuous \(G'\)-homomorphism \(C^\infty(X,{\mathcal V})\rightarrow C^\infty(Y,{\mathcal W})\) is called a symmetry breaking operator. The multiplicity \(m(\pi,\pi')\) is the (possibly infinite) dimension of the space \(H(\lambda,\nu)\) of symmetry breaking operators. In the paper under review, the authors provide a complete self-contained study and classification of symmetry breaking operators in the case of indefinite orthogonal groups.

From now on, suppose \(G\) is the indefinite orthogonal group \(O(n+1,1)\) and \(G'\) is the subgroup \(O(n,1)\) realized as follows, \(n\geq 1\). \(G\) is the group preserving the quadratic form \(x_0^2+x_1^2+\dots+x_n^2-x_{n+1}^2\) of signature \((n+1,1)\). The subgroup \(G'\) is the stabilizer of the vector \(e_n=\text{}^{t}(0,\dots,0,1,0)\) in \(\mathbb R^{n+2}\). The subgroup \(K:=O(n+1)\times O(1)\) is a maximal compact subgroup of \(G\) and \(K':=K\cap G'\) is a maximal compact subgroup of \(G'\). Let \({\mathfrak g}={\mathfrak o}(n+1,1)\) (resp. \({\mathfrak g}'={\mathfrak o}(n,1))\) be the Lie algebra of \(G\) (resp. \(G'\)). With the usual notation \(\{E_{i,j}\}_{0\leq i,j\leq n+1}\) for the canonical basis of \(gl(n+2,{\mathbb R})\), fix the hyperbolic element \(H:=E_{0,n+1}+E_{n+1,0}\) in \({\mathfrak g}'\). Then \(\text{ad}(H)\), as an endomorphism of \({\mathfrak g}\), has eigenvalues \(0\) and \(\pm 1\). Set \({\mathfrak a}:={\mathbb R}H\), \({\mathfrak n}_{\pm}:=\text{Ker}(\text{ad}(H)\pm 1)\), \(A:=\exp({\mathfrak a})\), \(N_{\pm}:=\exp({\mathfrak n}_{\pm})\) and \(M:=Z_K({\mathfrak a})\simeq O(n)\times{\mathbb Z}_2\). Then \(P:=MAN_+\) is a Langlands decomposition of a minimal parabolic subgroup of \(G\). Likewise \(P':=M'AN_+'\) is a Langlands decomposition of a minimal parabolic subgroup of \(G'\) with \(M':=Z_{K'}({\mathfrak a})\simeq O(n-1)\times{\mathbb Z}_2\), \(N_{\pm}':=\exp({\mathfrak n}_{\pm}')\) and \({\mathfrak n}_{\pm}':={\mathfrak n}_{\pm}\cap{\mathfrak g}'\). The fact that \(M\) and \(M'\) are both compact, with \(P\) and \(P'\) having the same \(A\)-part, makes the study of the space \(H(\lambda,\nu)\) of symmetry breaking operators more tractable.

Chapter 2 contains the proof of results about \(G'\)-intertwining operators between irreducible composition factors of spherical principal series representations and symmetry breaking operators for spherical principal series representations of \(G\) and \(G'\). Chapter 3 provides a method to study symmetry breaking operators for induced representations by means of their distribution kernels. Chapter 4 contains the realization in the non-compact picture of spherical principal series and a normalization \(\widetilde{{\mathbb T}}_\lambda\) of the Knapp-Stein intertwining operators \({\mathbb T}_\lambda\), while Chapter 5 deals with the double coset decompositions \(G'\backslash G/P\) and \(P'\backslash G/P\). Chapters 6 to 10 prepare the complete classification of symmetry breaking operators from principal series representations of \(G\) to those of \(G'\) contained in Chapter 11, based on some results proved in Chapters 12 and 13. As a byproduct, the authors analyse vector bundles over the anti-de Sitter spaces \(G/G'\) in Chapter 14, while in Chapter 15 they obtain an explicit construction of the complementary series representations of \(G\) along with their branching laws. Finally, Chapter 16 is an appendix which collects useful information about Gegenbauer polynomials, (normalized) \(K\)-Bessel functions and Zuckerman derived functor modules \(A_{\mathfrak q}(\lambda)\).

Reviewer: Salah Mehdi (Metz)

##### MSC:

22-02 | Research exposition (monographs, survey articles) pertaining to topological groups |

22E46 | Semisimple Lie groups and their representations |

22E45 | Representations of Lie and linear algebraic groups over real fields: analytic methods |

33C80 | Connections of hypergeometric functions with groups and algebras, and related topics |

53C35 | Differential geometry of symmetric spaces |

##### Keywords:

branching law; reductive Lie group; symmetry breaking; Lorentz group; conformal geometry; Verma module; complementary series##### References:

[1] | Jean-Louis Clerc, Toshiyuki Kobayashi, Bent Ørsted, and Michael Pevzner, Generalized Bernstein-Reznikov integrals, Math. Ann. 349 (2011), no. 2, 395-431. · Zbl 1207.42021 |

[2] | Michael G. Eastwood and C. Robin Graham, Invariants of conformal densities, Duke Math. J. 63 (1991), no. 3, 633-671. · Zbl 0745.53007 |

[3] | A. Erdélyi, W. Magnus, F. Oberhettinger, and F. G. Tricomi, Tables of integral transforms. Vol. II, McGraw-Hill Book Company, Inc., New York-Toronto-London, 1954. Based, in part, on notes left by Harry Bateman. · Zbl 0055.36401 |

[4] | J. Faraut, Distributions sphériques sur les espaces hyperboliques, J. Math. Pures Appl. (9) 58 (1979), no. 4, 369-444 (French). · Zbl 0436.43011 |

[5] | I. M. Gel\(^{\prime}\)fand, M. I. Graev, and N. Ya. Vilenkin, Generalized functions. Vol. 5: Integral geometry and representation theory, Translated from the Russian by Eugene Saletan, Academic Press, New York-London, 1966. · Zbl 0144.17202 |

[6] | I. M. Gel’fand and G. E. Shilov, Generalized functions. Vol. I: Properties and operations, Translated by Eugene Saletan, Academic Press, New York-London, 1964. · Zbl 0115.33101 |

[7] | Benedict H. Gross and Dipendra Prasad, On the decomposition of a representation of \({\mathrm SO}_n\) when restricted to \({\mathrm SO}_{n-1}\), Canad. J. Math. 44 (1992), no. 5, 974-1002. · Zbl 0787.22018 |

[8] | I. S. Gradshteyn and I. M. Ryzhik, Table of integrals, series, and products, 7th ed., Elsevier/Academic Press, Amsterdam, 2007. Translated from the Russian; Translation edited and with a preface by Alan Jeffrey and Daniel Zwillinger; With one CD-ROM (Windows, Macintosh and UNIX). · Zbl 1208.65001 |

[9] | Michael Harris and Hans Plesner Jakobsen, Singular holomorphic representations and singular modular forms, Math. Ann. 259 (1982), no. 2, 227-244. · Zbl 0466.32017 |

[10] | Sigurdur Helgason, Geometric analysis on symmetric spaces, 2nd ed., Mathematical Surveys and Monographs, vol. 39, American Mathematical Society, Providence, RI, 2008. · Zbl 1157.43003 |

[11] | Joachim Hilgert, Toshiyuki Kobayashi, and Jan Möllers, Minimal representations via Bessel operators, J. Math. Soc. Japan 66 (2014), no. 2, 349-414. · Zbl 1294.22011 |

[12] | Kenneth D. Johnson and Nolan R. Wallach, Composition series and intertwining operators for the spherical principal series. I, Trans. Amer. Math. Soc. 229 (1977), 137-173. · Zbl 0349.43010 |

[13] | Andreas Juhl, Families of conformally covariant differential operators, \(Q\)-curvature and holography, Progress in Mathematics, vol. 275, Birkhäuser Verlag, Basel, 2009. · Zbl 1177.53001 |

[14] | Masaki Kashiwara, Takahiro Kawai, and Tatsuo Kimura, Foundations of algebraic analysis, Princeton Mathematical Series, vol. 37, Princeton University Press, Princeton, NJ, 1986. Translated from the Japanese by Goro Kato. · Zbl 0605.35001 |

[15] | A. W. Knapp and E. M. Stein, Intertwining operators for semisimple groups, Ann. of Math. (2) 93 (1971), 489-578. · Zbl 0257.22015 |

[16] | Toshiyuki Kobayashi, Discrete decomposability of the restriction of \(A_{\mathfrak q}(\lambda )\) with respect to reductive subgroups. III. Restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), no. 2, 229-256. · Zbl 0907.22016 |

[17] | Toshiyuki Kobayashi, \(F\)-method for constructing equivariant differential operators, Geometric analysis and integral geometry, Contemp. Math., vol. 598, Amer. Math. Soc., Providence, RI, 2013, pp. 139-146. · Zbl 1290.22008 |

[18] | Toshiyuki Kobayashi, F-method for symmetry breaking operators, Differential Geom. Appl. 33 (2014), no. suppl., 272-289. · Zbl 1311.22016 |

[19] | Toshiyuki Kobayashi and Gen Mano, The Schrödinger model for the minimal representation of the indefinite orthogonal group \({\mathrm O}(p,q)\), Mem. Amer. Math. Soc. 213 (2011), no. 1000, vi+132. · Zbl 1225.22001 |

[20] | T. Kobayashi and T. Matsuki, Classification of finite-multiplicity symmetric pairs, Transform. Groups 19 (2014), no. 2, 457-493. · Zbl 1298.22015 |

[21] | Toshiyuki Kobayashi and Bent Ørsted, Analysis on the minimal representation of \(\roman O(p,q)\). I. Realization via conformal geometry, Adv. Math. 180 (2003), no. 2, 486-512. · Zbl 1046.22004 |

[22] | Toshiyuki Kobayashi and Bent Ørsted, Analysis on the minimal representation of \(\roman O(p,q)\). II. Branching laws, Adv. Math. 180 (2003), no. 2, 513-550. · Zbl 1049.22006 |

[23] | Toshiyuki Kobayashi and Bent Ørsted, Analysis on the minimal representation of \(\roman O(p,q)\). III. Ultrahyperbolic equations on \({\mathbb R}^{p-1,q-1}\), Adv. Math. 180 (2003), no. 2, 551-595. · Zbl 1039.22005 |

[24] | T. Kobayashi, B. Ørsted, P. Somberg, V. Souček, Branching laws for Verma modules and applications in parabolic geometry, Part I, preprint, http://arxiv.org/abs/1305.6040 arXiv:1305.6040. · Zbl 1327.53044 |

[25] | Toshiyuki Kobayashi and Toshio Oshima, Finite multiplicity theorems for induction and restriction, Adv. Math. 248 (2013), 921-944. · Zbl 1317.22010 |

[26] | T. Kobayashi, M. Pevzner, Rankin-Cohen operators for symmetric pairs, preprint, 53pp. http://arxiv.org/abs/1301.2111arXiv:1301.2111. · Zbl 1342.22029 |

[27] | Bertram Kostant, On the existence and irreducibility of certain series of representations, Bull. Amer. Math. Soc. 75 (1969), 627-642. · Zbl 0229.22026 |

[28] | Manfred Krämer, Multiplicity free subgroups of compact connected Lie groups, Arch. Math. (Basel) 27 (1976), no. 1, 28-36. · Zbl 0322.22011 |

[29] | Hung Yean Loke, Trilinear forms of \(\mathfrak {gl}_2\), Pacific J. Math. 197 (2001), no. 1, 119-144. · Zbl 1049.22007 |

[30] | Toshio Ōshima, Poisson transformations on affine symmetric spaces, Proc. Japan Acad. Ser. A Math. Sci. 55 (1979), no. 9, 323-327. · Zbl 0485.22011 |

[31] | Y. Sakellaridis, A. Venkatesh Periods and harmonic analysis on spherical varieties, http://arxiv.org/abs/1203.0039 arXiv:1203.0039. · Zbl 06847674 |

[32] | Henrik Schlichtkrull, Eigenspaces of the Laplacian on hyperbolic spaces: composition series and integral transforms, J. Funct. Anal. 70 (1987), no. 1, 194-219. · Zbl 0617.43005 |

[33] | Birgit Speh and T. N. Venkataramana, Discrete components of some complementary series, Forum Math. 23 (2011), no. 6, 1159-1187. · Zbl 1282.11054 |

[34] | Robert S. Strichartz, Harmonic analysis on hyperboloids, J. Functional Analysis 12 (1973), 341-383. · Zbl 0253.43013 |

[35] | Binyong Sun and Chen-Bo Zhu, Multiplicity one theorems: the Archimedean case, Ann. of Math. (2) 175 (2012), no. 1, 23-44. · Zbl 1239.22014 |

[36] | A. M. Vershik and M. I. Graev, The structure of complementary series and special representations of the groups \({\mathrm O}(n,1)\) and \({\mathrm U}(n,1)\), Uspekhi Mat. Nauk 61 (2006), no. 5(371), 3-88 (Russian, with Russian summary); English transl., Russian Math. Surveys 61 (2006), no. 5, 799-884. · Zbl 1148.22017 |

[37] | David A. Vogan Jr., Unitary representations of reductive Lie groups, Annals of Mathematics Studies, vol. 118, Princeton University Press, Princeton, NJ, 1987. · Zbl 0626.22011 |

[38] | David A. Vogan Jr. and Gregg J. Zuckerman, Unitary representations with nonzero cohomology, Compositio Math. 53 (1984), no. 1, 51-90. · Zbl 0692.22008 |

[39] | Nolan R. Wallach, Real reductive groups. II, Pure and Applied Mathematics, vol. 132, Academic Press, Inc., Boston, MA, 1992. · Zbl 0785.22001 |

[40] | Joseph A. Wolf, Spaces of constant curvature, 6th ed., AMS Chelsea Publishing, Providence, RI, 2011. · Zbl 1216.53003 |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.