## Proximal smoothness and the lower-$$C^ 2$$ property.(English)Zbl 0881.49008

The authors call a closed subset $$X$$ of a real Hilbert space $$H$$ proximally smooth if and only if there exists $$r>0$$ for which (i) the distance function $$d_X$$ is continuously differentiable on the set $$U(r) = \{|p |: 0 < d_X(p) < r\}$$. (Recall that $$d_X(p) = \inf\{|p-x |: x\in X\}$$.) Their main result provides a number of equivalent characterizations of property (i), including (ii) the proximal subgradient of $$d_X$$ is nonempty at every point of $$U(r)$$; (iii) at every point of $$U(r)$$, $$d_X$$ is Gâteaux differentiable and there exists a nearest point in $$X$$; and (iv) every proximal normal to $$X$$ can be realized with a ball of radius $$r$$, and every point of $$U(r)$$ has a nearest point in $$X$$. When $$X$$ is weakly closed (not just closed), these conditions are equivalent to (v) every point of $$U(r)$$ has a unique nearest point in $$X$$. (Hence a closed set $$X$$ in $$H$$ is convex if and only if it is proximally smooth of radius $$r$$ for every $$r>0$$.) Further equivalences are obtained when $$\dim(H)<\infty$$. The proofs are elementary, but not easy. In addition to the main result described above, the paper contains a number of independently interesting auxiliary results on the differentiability properties of distance functions. Given an open, convex subset $$G$$ of a Euclidean space $$E$$ and a Lipschitz function $$f\colon G\to\mathbb{R}$$, the authors show that the set $$\text{ epi} f = \{ (x,r): r\geq f(x) \}$$ is proximally smooth if and only if $$f$$ is lower-$$C^2$$ in the sense of Rockafellar.
Reviewer: P.Loewen (Bath)

### MSC:

 49J52 Nonsmooth analysis 90C26 Nonconvex programming, global optimization 26B05 Continuity and differentiation questions
Full Text: