Proximal smoothness and the lower-\(C^ 2\) property. (English) Zbl 0881.49008

The authors call a closed subset \(X\) of a real Hilbert space \(H\) proximally smooth if and only if there exists \(r>0\) for which (i) the distance function \(d_X\) is continuously differentiable on the set \(U(r) = \{|p |: 0 < d_X(p) < r\}\). (Recall that \(d_X(p) = \inf\{|p-x |: x\in X\}\).) Their main result provides a number of equivalent characterizations of property (i), including (ii) the proximal subgradient of \(d_X\) is nonempty at every point of \(U(r)\); (iii) at every point of \(U(r)\), \(d_X\) is Gâteaux differentiable and there exists a nearest point in \(X\); and (iv) every proximal normal to \(X\) can be realized with a ball of radius \(r\), and every point of \(U(r)\) has a nearest point in \(X\). When \(X\) is weakly closed (not just closed), these conditions are equivalent to (v) every point of \(U(r)\) has a unique nearest point in \(X\). (Hence a closed set \(X\) in \(H\) is convex if and only if it is proximally smooth of radius \(r\) for every \(r>0\).) Further equivalences are obtained when \(\dim(H)<\infty\). The proofs are elementary, but not easy. In addition to the main result described above, the paper contains a number of independently interesting auxiliary results on the differentiability properties of distance functions. Given an open, convex subset \(G\) of a Euclidean space \(E\) and a Lipschitz function \(f\colon G\to\mathbb{R}\), the authors show that the set \(\text{ epi} f = \{ (x,r): r\geq f(x) \}\) is proximally smooth if and only if \(f\) is lower-\(C^2\) in the sense of Rockafellar.
Reviewer: P.Loewen (Bath)


49J52 Nonsmooth analysis
90C26 Nonconvex programming, global optimization
26B05 Continuity and differentiation questions
Full Text: EuDML EMIS