Abstract: Existing coal and gangue sorting methods mainly use image processing and pattern recognition technology to identify and sort coal and gangue according to texture characteristic values of coal and rock. The grain size of coal and gangue is 25-150 mm, while the coal and gangue above 150 mm still relies on manual sorting. In order to sort coal and gangue with large grain size, a multi-manipulator coal and gangue sorting robot system based on machine vision was proposed.The system uses machine vision to collect coal and gangue information and applies deep learning method to realize coal and gangue identification and grab feature extraction. After obtaining the sequence information of coal and gangue, the sorting work is carried out according to the position of coal and gangue, and the grasping task is assigned to the corresponding manipulator controller by multi-objective task assignment strategy. After the manipulator obtains the task, the target is dynamically monitored according to the acquired task. When the target enters the working space of the manipulator, the visual servo system drives the manipulator to complete the coal and gangue sorting. The test results show that the system can efficiently and quickly sort coal and gangue with grain size of 50-260 mm, and the adopted coal and gangue identification method and sorting strategy have good stability and accuracy under different belt speeds, and the comprehensive accuracy of coal and gangue identification and positioning can reach 93%, which verifies the feasibility of the system.