Multiband exact diagonalization and an iteration approach to search for fractional Chern insulators in rhombohedral multilayer graphene

We perform a multi-band exact diagonalization (ED) study of rhombohedral pentalayer graphene twisted on hexagonal boron nitride with a focus on fractional Chern insulators (FCI) in systems with weak moiré gaps, complementing the results of [Yu et al., arXiv:2407.13770]. We consider both the charge-neutrality (CN) and average (AVE) interaction schemes. Saliently and surprisingly, we now find using the particle entanglement spectrum that the FCI at filling factor 1/3 in the CN scheme predicted by single-(Hartree-Fock) band ED is unstable towards a transition to charge density wave once a small fraction of electrons is allowed to occupy the higher bands. Meanwhile, the FCI at filling 2/3 in the AVE scheme remains more robust under similar band mixing until being suppressed when increasing band mixing. To tackle truncation errors that arise from including multiple bands in larger system sizes, we propose an ED iteration method that iteratively optimizes the single-particle basis so that the particles in the ground state should residemainly in the lowest band. Nevertheless, we find that the FCI gap remains absent after convergence when the mixing with higher bands is considered. These findings highlight the delicate sensitivity of FCIs to multi-band effects and the shortcoming of all of the current models to explain the experimental emergence of such phases.