Observation of an anomalous density-dependent energy gap of the ν=5/2 fractional quantum Hall state in the low-density regime

We have studied the ν=5/2 fractional quantum Hall state in a density-tunable sample at extremely low electron densities. For the densities accessed in our experiment, the Landau level mixing parameter κ spans the 2.52<κ<2.82 range. In the vicinity of 5.8×1010cm-2 or κ=2.6 an anomalously large change in the density dependence of the energy gap is observed. Possible origins of such an anomaly are discussed, including a topological phase transition in the ν=5/2 fractional quantum Hall state.