Coal and gangue instance segmentation and estimation of surface gangue fraction of coal flows based on YOLO-DOSSeg

Coal, gangue, and foreign objects on conveyor belts often exhibit adjacency, local adhesion, blurred boundaries, and pronounced size differences. These characteristics can lead to insufficient separation of instance boundaries, inaccurate contour reconstruction, and inadequate learning from difficult samples, affecting the reliability of subsequent projected-area statistics for coal-flow surfaces and process control. To address these problems, a method for coal and gangue instance segmentation and estimation of surface gangue fraction of coal flows based on YOLO-DOSSeg was proposed. Using YOLO11-seg as the baseline, an improved coal and gangue instance segmentation model, YOLO-DOSSeg, was developed by integrating DySample+, Omni-Dimensional Dynamic Convolution (ODConv), and Slide Loss. A method for estimating and stabilizing coal-flow surface indicators was designed based on the model's instance segmentation results. Prediction confidence was used to screen valid instances through gating, and the projected areas of coal, gangue, and foreign objects were calculated from the retained binary instance masks. The projected-area gangue fraction on coal-flow surfaces and the projected-area fraction of foreign objects were calculated, and a coal-flow surface cleanliness index based on projected area was constructed. The SegClean framework was proposed to stabilize coal-flow surface indicators through temporal smoothing, quality gating, and hysteresis-based decisions, improving the stability of continuous monitoring results and alarm outputs. Experiments on the DsCGF instance segmentation dataset showed that YOLO-DOSSeg achieved a precision of 78.6%, a recall of 90.7%, mAP@0.5 of 88.2%, and mAP@0.5:0.95 of 73.4%, exceeding those of YOLO11-seg by 7.7%, 9.2%, 7.6%, and 6.5%, respectively. With confidence-gated binary-mask statistics, the mean absolute error, root mean square error, and Pearson correlation coefficient for the projected-area gangue fraction on coal-flow surfaces were 1.84%, 3.67%, and 0.95, respectively, with a mean bias of +0.18%. SegClean further reduced its standard deviation, mean interframe change, and maximum interframe change to 3.52%, 1.01%, and 5.67%, respectively, and reduced the numbers of alarm-state transitions and repeated alarms within short periods to 3 and 1, respectively. The proposed method balances instance segmentation accuracy in complex coal and gangue scenes, the reliability of projected-area statistics, and the stability of continuous outputs, providing interpretable and quantifiable data support for monitoring coal-flow surface composition, issuing foreign-object warnings, and adjusting sorting parameters.