Abstract: For complex conditions in open-pit coal mines—characterized by pronounced non-structural features in small-curvature bends, prominent spatiotemporal dynamic evolution, severe terrain undulations, and highly unpredictable obstacles—existing hybrid A* algorithms suffer from path irregularities, frequent turns, low search efficiency, and elevated safety risks. This study proposes an improved hybrid A* path planning method integrating distance penalties, cost incentives, and quadratic smoothing. First, the algorithm's evaluation function is restructured by introducing a distance penalty term. By dividing the area into collision danger zones, potential zones, and safe zones, node expansion priorities are dynamically adjusted to reduce redundant searches and enhance path safety. Second, a cost function incentive mechanism is designed to weight forward driving and heading angle stability, guiding dump trucks to follow the centerline of curves and reducing unnecessary steering frequency. Simultaneously, node expansion ranges are restricted to prevent planning failures caused by abrupt curvature changes. Finally, a nonlinear optimization model incorporating smoothing terms, distance costs, deviation costs, and curvature change rates is constructed. Combining quadratic programming techniques with the IPOPT solver, quadratic smoothing is applied to discrete path points. Position and curvature constraints are introduced to ensure trajectories comply with vehicle kinematic limits. Simulation experiments and field validation at an open-pit coal mine demonstrate that this method reduces path search time by 61.9% to 82.7% compared to traditional hybrid A* algorithms in 150m and 500m low-curvature curve scenarios. Maximum curvature decreases by 60.0%, peak heading angle deviation reduces by 87.5%, and the curvature change rate is controlled within 0.02 m?2, generating paths that are continuous, smooth, and distant from curve boundaries. This effectively reduces the risk of slope collapse and vehicle overturn, significantly enhancing the safety, stability, and efficiency of dump truck navigation in complex, small-curvature bend scenarios within open-pit coal mines, demonstrating excellent engineering application value.