To make it viable for remote monitoring to scale to large patient populations, the accuracy of detectors used to identify patient states of interests must improve. Patient-specific detectors hold the promise of higher accuracy than generic detectors, but the need to train these detectors individually for each patient using expert labeled data limits their scalability. We explore a solution to this challenge in the context of atrial fibrillation (AF) detection. Using patient recordings from the MIT-BIH AF database, we demonstrate the importance of patient specificity and present a scalable method of constructing a personalized detector based on active learning. Using a generic detector having a sensitivity of 76% and a specificity of 57% as its seed, our active learning approach constructs a detector with a sensitivity of 90% and specificity of 85%. This performance approaches that of a patient-specific detector, which has a sensitivity of 94% and specificity of 85%. By selectively choosing examples for training, the active learning approach reduces the amount of expert labeling needed by almost eight fold (compared to the patient-specific detector) while achieving accuracy within 99%.