Deep models are vulnerable to catastrophic forgetting when finetuned on new data, which will degrade the performance of old tasks. Popular distillation-based learning methods usually neglect the relations between data samples and may eventually forget essential structural knowledge. To solve these shortcomings, we propose a structural graph knowledge distillation-based incremental learning framework to preserve both the positions of samples and their relations. Firstly, a memory knowledge graph (MKG) is generated to fully characterize the structural knowledge of historical tasks. Secondly, we develop a graph interpolation mechanism to enrich the domain of knowledge and alleviate the inter-class sample imbalance issue. Thirdly, we introduce structural graph knowledge distillation to transfer the knowledge of historical tasks, by penalizing the offset of the vertices and edge weights of MKG on the new model during new-task learning. Comprehensive experiments on CIFAR100, ImageNet-100, and ImageNet-1000 confirm the effectiveness of the proposed method.