The spectrum of random inner-product kernel matrices

We consider n × n matrices whose (i, j)th entry is f(X-{i}-{T}X-{j}), where X1, ..., Xn are i.i.d. standard Gaussian in p, and f is a real-valued function. The weak limit of the eigenvalue distribution of these random matrices is studied at the limit when p, n → ∞ and p/n = γwhich is a constant. We show that, under certain conditions on the kernel function f, the limiting spectral density exists and is dictated by a cubic equation involving its Stieltjes transform. The parameters of this cubic equation are decided by a Hermite-like expansion of the rescaled kernel function. While the case that f is differentiable at the origin has been previously resolved by El Karoui [The spectrum of kernel random matrices, Ann. Statist.38 (2010) 1-50], our result is applicable to non-smooth f, such as the Sign function and the hard thresholding operator of sample covariance matrices. For this larger class of kernel functions, we obtain a new family of limiting densities, which includes the Marčenko-Pastur (M.P.) distribution and Wigner's semi-circle distribution as special cases.