Fractional Chern insulators versus nonmagnetic states in twisted bilayer MoTe2

Fractionally filled Chern bands with strong interactions may give rise to fractional Chern insulator (FCI) states, the zero-field analog of the fractional quantum Hall effect. Recent experiments have demonstrated the existence of FCIs in twisted bilayer MoTe2 without external magnetic fields - most robust at ν=-2/3 - as well as Chern insulators (CIs) at ν=-1. Although the appearance of both of these states is theoretically natural in an interacting topological system, experiments repeatedly observe nonmagnetic (or weakly magnetic) states (lacking FCIs) at ν=-1/3 and -4/3, a puzzling result, which has not been fully theoretically explained. In this paper, we perform Hartree-Fock and exact diagonalization calculations to test whether the standard MoTe2 moiré model with the (greatly varying) parameter values available in the literature can reproduce the nonmagnetic/weakly magnetic states at ν=-1/3 and -4/3 in unison with the FCI at ν=-2/3 and CI state at ν=-1. We focus on the experimentally relevant twist angles and, crucially, include remote bands. We find that the parameters proposed in Wang et al. [arXiv:2306.02501] can nearly capture the experimental phenomena at ν=-1/3,-2/3,-1,-4/3 simultaneously, although the predicted ground states at ν=-1/3 are still mostly FCIs and a larger dielectric constant ϵ>10 than is typical of hexagonal boron nitride (h-BN) substrate ϵ∼6 is required. Our results show the importance of remote bands in identifying the competing magnetic orders and lay the groundwork for further study of the realistic phase diagram.