Two distinct topological phases in the mixed-valence compound YbB6 and its differences from SmB6

We discuss the evolution of topological states and their orbital textures in the mixed-valence compounds SmB6 and YbB6 within the framework of the generalized gradient approximation plus onsite Coulomb interaction (GGA+U) scheme for a wide range of values of U. In SmB6, the topological Kondo insulator (TKI) gap is found to be insensitive to the value of U, but in sharp contrast, Kondo physics in isostructural YbB6 displays a surprising sensitivity to U. In particular, as U is increased in YbB6, the correlated TKI state in the weak-coupling regime transforms into a d-p-type topological insulator phase with a band inversion between Yb-5d and B-2p orbitals in the intermediate coupling range, without closing the insulating energy gap throughout this process. Our theoretical predictions related to the TKI and non-TKI phases in SmB6 and YbB6 are in substantial accord with recent angle-resolved photoemission spectroscopy experiments.