A rapid increase in available computing power has ushered in an era of neural network dominance in tasks such as image classification, yet these models can have severe weaknesses. Due to their generalization mechanism, deep neural networks often assign high confidence to regions far from the decision boundary, leaving them vulnerable to adversarial attacks. The general brittleness of these models leads to other problems, too: catastrophic forgetting, overconfidence, an inability to quantify uncertainty, and an inability to encode prior “common sense” knowledge. Furthermore, in high-risk domains such as defense, intelligence, and healthcare, models do not always have access to large training databases full of labeled samples. If labeled samples are available, they are often sparse, noisy, and the label’s validity may be ambiguous. These domains require models that operate on fewer training samples, recognize uncertain data, and encode prior knowledge when available. In this paper, we present a robust, physics-informed deep learning architecture based on radial basis functions, capable of making interpretable predictions from noisy and sparse raw data. Radial basis functions address the problem of brittleness by penalizing long-distance response, and the architecture uses a similarity metric to learn only highly discriminative, localized features. This allows a model to reject adversarial attacks and out-of-distribution examples. The architecture also provides a mechanism for encoding prior knowledge, enabling a model to generalize from sparse training data. Through several image classification experiments on a modified MNIST dataset, we demonstrate the effectiveness of this architecture in noisy, low-data environments. Finally, we discuss utilizing the proposed method for threat detection and target discrimination from satellite imagery, synthetic aperture radar (SAR) data, and LiDAR data.