ESINet——基于空间信息增强的实时语义分割网络

ESINet—Real-Time Semantic Segmentation Network Based on Enhanced Spatial Information

摘要: 为提高语义分割实时性，现有的方法细节分支往往设计简单，对空间上下文信息提取极不充分，另外传统卷积操作伴随大量的冗余特征计算对推理速度形成瓶颈。本文提出实时语义分割模型ESINet，引入改进的轻量化空间优化卷积模块(SpaceOptiConv)，采用少量生成特征图并与本征特征进行拼接，减少冗余特征图的计算，降低计算复杂度和参数量；在混合注意力的基础上引入多尺度特征提取，设计了一种多尺度混合注意力机制(HMA)，高效捕捉不同尺度的特征信息；提出一种复合损失函数，交并比损失(IoULoss)优化分割区域的整体重叠度，在线难例挖掘交叉熵损失(OhemCELoss)聚焦类别不平衡、小目标和复杂边界增强局部分类的准确性。对于2048 × 1024的输入，ESINet在Cityscapes测试集上实现81.6%的mIoU，NVIDIA TITAN RTX上的速度为144.5 FPS。相较于基线模型mIoU和分割速度分别取得5.6%和16.4 fps的提高。

Abstract: To improve the real-time performance of semantic segmentation, existing methods often design detail branches that are relatively simple and fail to extract spatial context information effectively. In addition, traditional convolution operations involve significant redundant feature computations, which become bottlenecks for inference speed. This paper proposes a real-time semantic segmentation model, ESINet, which introduces an improved lightweight spatial optimization convolution module (SpaceOptiConv). The module generates a few feature maps and concatenates them with intrinsic features, reducing redundant feature map computations, thus lowering computational complexity and the number of parameters. Based on a hybrid attention mechanism, a multi-scale feature extraction approach is introduced, and a multi-scale hybrid attention mechanism (HMA) is designed to efficiently capture feature information at different scales. Moreover, a composite loss function is proposed, including Intersection over Union loss (IoULoss) to optimize the overall overlap of segmented regions, and Online Hard Example Mining Cross-Entropy Loss (OhemCELoss) to focus on class imbalance, small targets, and complex boundaries, enhancing the accuracy of local classification. With an input size of 2048 × 1024, ESINet achieves 81.6% mIoU on the Cityscapes test set, with a speed of 144.5 FPS on an NVIDIA TITAN RTX. Compared to the baseline model, ESINet achieves improvements of 5.6% in mIoU and 16.4 FPS in segmentation speed.