Abstract: The key issues for roadside sensing system (RSS) include achieving accuracy and real-time sharing of over-horizon perception information. This study proposes a novel and efficient framework dedicated to multi-object detection from the roadside perspective. Firstly, compared to other backbones, the mobile net-based model has superior performance and speed as results of the network parameters obtained from network architecture search (NAS), developed to increase the forward inference speed. Secondly, a method of optimization based on the coordinate attention mechanism is developed to increase the long-range dependence of neural networks on spatial information. Thirdly, the traditional convolution operation in the attention mechanism is optimized by the depthwise over-parameterized convolution (DOPC) to improve the capability of extracting features from high-dimensional feature space. Finally, the lightweight single-stage multi-target detection model from the roadside perspective based on DCM3-YOLOv4 is developed. The test results show that the optimized one-stage lightweight multiple object detection model DCM3-YOLOv4 on the RS-UA dataset produces a mean average precision (mAP) value of 0.930 and a network model with parameter size of 31.12 Million. The inference time is 96.13 ms, which is faster than another basic model on the same platform. The proposed methods can be utilized in a wide range of applications, where the accuracy and speed requirements of RSS must be met.