Abstract: For insufficient of safety protection means and characteristics of denial of service attack in MAC layer of WLAN, the paper designed an intrusion detection system based on support vector machine. The system uses characteristics of high classification accuracy of support vector machine to build hyper-plane of optimal classification and classification decision function, and achieves classification and identification of network traffic, and completes detection of abnormal traffic. The intrusion detection of WLAN was simulated on OPNET platform, and the result indicates that the system can detect denial of service attack in MAC layer efficiently.