Moiré fractional Chern insulators. I. First-principles calculations and continuum models of twisted bilayer MoTe2

Recent experiments observed fractional Chern insulators (FCI) in twisted bilayer MoTe2 at zero magnetic field, yet even the single-particle model of this material is controversial, leading to unreliable predictions of the experimental phase diagram as discussed in [Yu, Phys. Rev. B 109, 045147 (2024)10.1103/PhysRevB.109.045147]. In this light, we revisit the single-particle model of twisted bilayer MoTe2. Utilizing large-scale density functional theory, we calculate the band structure of twisted AA-stacked bilayer MoTe2 at various twist angles relevant to experiment. We find that a band inversion occurs near 4.41° between the second and third bands in one valley. Our ab initio band structure is in qualitative agreement with [Wang, Phys. Rev. Lett. 132, 036501 (2024)10.1103/PhysRevLett.132.036501], but shows important differences in the remote bands and in the Γ valley. We incorporate two higher harmonic terms into the continuum model to capture the highest three valence bands per valley. We confirm that the two highest valence bands per valley have opposite Chern numbers with |C|=1 for experimentally relevant angles, and also use our model to predict a variety of Chern states in the remote bands accessible by displacement field. We also perform DFT calculations and build models for the AB-stacking configuration. Our paper serves as a foundation for accurate determination of the correlated phases in twisted bilayer MoTe2.