Analytic extensions of representations, the solvable case. (English) Zbl 0788.22014

Let \(G\) be a real Lie group contained in a complexification \(G_ C\). It was the idea of Gelfand and Gindikin to study those representations of \(G\) that could be holomorphically extended to an appropriate open domain in \(G_ C\) which contained \(G\) in its boundary and thus develop a noncommutative version of Hardy space theory. This program was successfully implemented by G. I. Ol’shanskij in the case of Hermitian simple Lie groups. In general, if there is an \(\text{Ad}(G)\)-invariant pointed convex cone \(W\) in the Lie algebra \(\mathfrak g\) of \(G\) with interior \(W^ o\), then \(\Gamma^ o(W)=(\exp iW^ o)G\) is a holomorphic open subsemigroup. If \(G\) is Hermitian simple and \(W\) is minimal invariant, then the unitary highest weight representations of \(G\) are those that can be analytically extended to a representation of \(\Gamma^ o(W)\) by contractions, and these extensions can then be used to construct a Hardy space of holomorphic functions on \(\Gamma^ o(W)\), which is a Hilbert space with reproducing kernel and carries a natural representation. This is then used to study holomorphic discrete series representations.
In this paper the authors show that an analogous program can be carried out for those solvable Lie groups for which the Lie algebra admits an invariant cone \(W\). The authors show that the analytic continuation procedure to \(\Gamma^ o(W)\) still obtains in this setting. One has to find appropriate notions of highest weight representations and those representations that play the role of the holomorphic discrete series, so the analogy with the simple case is far from being purely formal. In particular, the authors use the Kirillov orbit picture to describe the relevant representations. It is shown that the regular representation of these groups is of Type I and an explicit Plancherel formula is given. Conditions are also given for irreducible unitary representations to be admissible (finite multiplicity of \(K\)-types).


22E27 Representations of nilpotent and solvable Lie groups (special orbital integrals, non-type I representations, etc.)
43A65 Representations of groups, semigroups, etc. (aspects of abstract harmonic analysis)