Hospital readmission rate is high for heart failure patients. Early detection of deterioration will help doctors prevent readmissions, thus reducing health care cost and providing patients with just-in-time intervention. Wearable devices (e.g., wristbands and smart watches) provide a convenient technology for continuous outpatient monitoring. In the paper, we explore the feasibility of monitoring outpatients using Fitbit Charge HR wristbands and the potential of machine learning models to predicting clinical deterioration (readmissions and death) among outpatients discharged from the hospital. We developed and piloted a data collection system in a clinical study which involved 25 heart failure patients recently discharged from a hospital. The results from the clinical study demonstrated the feasibility of continuously monitoring outpatients using wristbands. We observed high levels of patient compliance in wearing the wristbands regularly and satisfactory yield, latency and reliability of data collection from the wristbands to a cloud-based database. Finally, we explored a set of machine learning models to predict deterioration based on the Fitbit data. Through 5-fold cross validation, K nearest neighbor achieved the highest accuracy of 0.8800 for identifying patients at risk of deterioration using the health data from the beginning of the monitoring. Machine learning models based on multimodal data (step, sleep and heart rate) significantly outperformed the traditional clinical approach based on LACE index. Moreover, our proposed weighted samples one class SVM model can reach high accuracy (0.9635) for predicting the deterioration happening in the future using data collected by a sliding window, which indicates the potential for allowing timely intervention.