The fundamental breakthroughs in machine learning, and the rapid advancements of the underlying deep neural network models have enabled the potential use of these systems in specialized, high stakes domains such as medicine. However, the increased capabilities of machine learning systems comes at the cost of greater complexity, with the design of machine learning systems becoming ever more laborious, computationally expensive and opaque. This can result in catastrophic failures and significantly hinders effective collaboration with human experts, which is central to successful deployment. In this talk, I overview steps towards an insight-driven design of machine learning systems, and methods to facilitate collaboration with human experts in medicine. I develop tools that enable the quantitative analysis of the complex hidden layers of deep neural networks, which provide both fundamental insights on central components of the models as well as informing algorithms for efficiently training these systems. I demonstrate how these trained systems can be adapted to work effectively with human experts in the medical setting, resulting in better outcomes than either entity alone. Throughout the talk, I highlight key open questions in these areas of study.