## Abstract

We consider n × n matrices whose (i, j)th entry is f(X-{i}-{T}X-{j}), where X1, ..., Xn are i.i.d. standard Gaussian in p, and f is a real-valued function. The weak limit of the eigenvalue distribution of these random matrices is studied at the limit when p, n → ∞ and p/n = γwhich is a constant. We show that, under certain conditions on the kernel function f, the limiting spectral density exists and is dictated by a cubic equation involving its Stieltjes transform. The parameters of this cubic equation are decided by a Hermite-like expansion of the rescaled kernel function. While the case that f is differentiable at the origin has been previously resolved by El Karoui [The spectrum of kernel random matrices, Ann. Statist.38 (2010) 1-50], our result is applicable to non-smooth f, such as the Sign function and the hard thresholding operator of sample covariance matrices. For this larger class of kernel functions, we obtain a new family of limiting densities, which includes the Marčenko-Pastur (M.P.) distribution and Wigner's semi-circle distribution as special cases.

Original language | English (US) |
---|---|

Article number | 1350010 |

Journal | Random Matrices: Theory and Application |

Volume | 2 |

Issue number | 4 |

DOIs | |

State | Published - Oct 1 2013 |

## All Science Journal Classification (ASJC) codes

- Algebra and Number Theory
- Statistics and Probability
- Statistics, Probability and Uncertainty
- Discrete Mathematics and Combinatorics

## Keywords

- Hermite polynomials
- Kernel matrices
- Stieltjes transform
- limiting spectrum