Abstract: The existing image classification methods based on deep learning have the problems of the large number of classification model parameters, long time consumption and low classification precision. It is difficult to achieve a compromise between the portability of the model and the classification precision. In order to solve the above problems, a coal and gangue image classification model based on improved Feedback-Net is proposed. The Feedback-Net model is built on the basis of the ResNet50 model. The high-order information and the low-order information are fused by building a feedback path in the ResNet50 model. Therefore, the representation capability of the features is improved. The constructed Feedback-Net model consumes more time while improving the classification accuracy. In order to solve this problem, the square convolution block in the Feedback-Net model is improved into an asymmetric convolution block (ACB). The feature extraction capability of the convolution kernel is increased by superposition and fusion. The full connection layer with the largest number of parameters is replaced by global covariance pooling (GCP) to reduce the number of network parameters. By simulating the environment of coal and gangue sorting in coal mines, the performance of the Feedback-Net model and the improved Feedback-Net model (Feedback-Net + ACB and Feedback-Net + ACB + GCP) is verified. The results show the following points. ① The precision of the Feedback-Net model is higher than that of the ResNet50 model, and the loss value is slightly lower than that of the ResNet50 model. ② Compared with the ResNet50 model, the training precision of the Feedback-Net model is improved by 1.20%. The testing precision is improved by 1.21%, but the training time is increased by 0.22%. ③ The precision of the Feedback-Net + ACB + GCP model is high than that of the Feedback-Net and Feedback-Net + ACB model. The Feedback-Net + ACB + GCP model's convergence rate is the fastest among the three models. It has the best performance. ④ Compared with the Feedback-Net model, the testing precision and training precision of the Feedback-Net + ACB model are improved by 1.39%. The time consumption is reduced by 15.53 minutes. Compared with the Feedback-Net model, the training precision and testing precision of the Feedback-Net + ACB + GCP model are improved by 1.62% and 1.59% respectively. The time consumption is reduced by 1.12%. Compared with the Feedback-Net+ACB model, the time consumption of the Feedback-Net+ACB+GCP model is reduced by 50.38 minutes. The performance of the Feed-Net+ACB+GCP model is better.