Implantable and wearable medical devices (IWMDs) are commonly used for diagnosing, monitoring, and treating various medical conditions. A general trend in IWMDs is towards increased functional complexity, software programmability, and connectivity to body area networks (BANs). However, as medical devices become more "intelligent," they also become less trustworthy - less reliable and more vulnerable to malicious attacks. Various shortcomings - hardware failures, software errors, wireless attacks, malware and software exploits, and side-channel attacks - could undermine the trustworthiness of IWMDs and BANs. The trustworthiness of IWMDs must be addressed aggressively and proactively due to the potential for catastrophic consequences. While some recent efforts address the defense of IWMDs against specific security attacks, a holistic strategy that considers all concerns and types of threats is required. This paper discusses trustworthiness concerns in IWMDs and BANs through a comprehensive identification and analysis of potential threats and, for each threat, provides a discussion of the merits and inadequacies of current solutions.