## Proximal analysis and minimization principles.(English)Zbl 0865.49015

Let $$X$$ be a Hilbert space, and suppose $$f:X\to\mathbb{R}\cup\{+\infty\}$$ is lower semicontinuous. Given a point $$x$$ where $$f(x)<+\infty$$, the set $$\partial_Pf(x)$$ of proximal subgradients of $$f$$ at $$x$$ consists of all $$z$$ in $$X$$ for which there exist a constant $$\sigma>0$$ and a neighbourhood $$\Omega$$ of $$x$$ such that $f(y)-\langle z y-x\rangle+\sigma|y-x|^2\geq f(x)\quad\forall y\in\Omega.$ The authors present a self-contained proof of the following density property: given any point $$x_0$$ where $$f(x_0)<+\infty$$ and any $$\varepsilon>0$$, there exists a point $$x$$ where $$\partial_Pf(x)\neq\emptyset$$ and $$|x-x_0|+|f(x)-f(x_0)|<\varepsilon$$. Combining this result with certain properties of the quadratic infimal-convolution, or Moreau-Yosida approximation, they produce efficient proofs of two basic results: (i) key properties of proximal subgradients for functions of the form $$f(x)=\inf_{s\in S}|x-s|$$, where $$S$$ is a given closed subset of $$X$$; (ii) Hilbert-space versions of the well-known variational principles of Ch. Stegall [Math. Ann. 236, 171-176 (1978; Zbl 0379.49008)] and J. M. Borwein and D. Preiss [Trans. Am. Math. Soc. 303, 517-527 (1987; Zbl 0632.49008)]. The delicacy of the density property cited above is illustrated with a construction of a continuously differentiable function $$f$$ on $$\mathbb{R}$$ for which the set where $$\partial_Pf(x)\neq\emptyset$$ has measure zero and fails to contain a dense $$G_\delta$$, and the same is true of $$\partial_P(-f)(x)$$. Although many of the key facts in this paper are already known, this presentation is unique, since it derives them all from the existence of proximal subgradients on a dense set.
Reviewer: P.Loewen (Bath)

### MSC:

 49J52 Nonsmooth analysis 58E05 Abstract critical point theory (Morse theory, Lyusternik-Shnirel’man theory, etc.) in infinite-dimensional spaces