[1]赵崟昊,刘炳辰,杨建红,等.RGB-D多模态融合与深度特征增强的固废检测网络[J].华侨大学学报(自然科学版),2025,(2):133-141.[doi:10.11830/ISSN.1000-5013.202410016]
　ZHAO Yinhao,LIU Bingchen,YANG Jianhong,et al.Solid Waste Detection Network With RGB-D Multimodal Fusion and Deep Feature Enhancement[J].Journal of Huaqiao University(Natural Science),2025,(2):133-141.[doi:10.11830/ISSN.1000-5013.202410016]
点击复制

RGB-D多模态融合与深度特征增强的固废检测网络()
分享到：

《华侨大学学报（自然科学版）》[ISSN:1000-5013/CN:35-1079/N]

卷:
期数:
2025年第2期
页码:
133-141
栏目:
出版日期:
2025-03-20

文章信息/Info

Title:
Solid Waste Detection Network With RGB-D Multimodal Fusion and Deep Feature Enhancement
文章编号:
1000-5013(2025)02-0133-09
作者:
赵崟昊;  刘炳辰;  杨建红;  房怀英
华侨大学 机电及自动化学院, 福建 厦门 361021
Author(s):
ZHAO Yinhao;  LIU Bingchen;  YANG Jianhong;  FANG Huaiying
College of Mechanical Engineering and Automation, Huaqiao University, Xiamen 361021, China
关键词:
固废分选;  深度加强;  RGB-D图像;  特征融合;  实例分割
Keywords:
solid waste sorting;  depth enhancement;  RGB-D image;  feature fusion;  instance segmentation
分类号:
TP183;TP249
DOI:
10.11830/ISSN.1000-5013.202410016
文献标志码:
A
摘要:
针对建筑固废在线识别中因相似特征导致的RGB识别准确率不高的问题,搭建双相机采集实验台,同步采集彩色图像和深度图像,提出一种基于彩色图像和深度图像的多模态融合与深度特征增强网络(DFENet).DFENet能够有效融合固废的彩色图像特征和深度图像特征。通过设计深度特征加强融合模块PFPD平衡并加强深度特征,显著提升了网络的识别精度。实验结果表明:与RGB+FPN(特征金字塔网络)方式相比,PFPD方式在IoU=0.50上的识别精度从92.4%提高至94.7%,在IoU=0.75上的识别精度从90.8%提升至92.8%;与实例分割网络(Mask R-CNN)相比,DFEnet识别精度从86.4%提高至89.2%;提出的方法有效地提高了固体废弃物识别的目标检测和实例分割模型识别精度。
Abstract:
Aiming at the problem of low accuracy of RGB recognition due to similar features in online construction identification of solid waste, a dual-camera collection experimental platform is established to collect color images and depth images simultaneously. A multimodal fusion and depth feature enhancement network(DFENet)based on color image and depth image is proposed. DFENet can effectively fuse the color and depth image features of solid waste. By designing a deep feature strengthening fusion module(PFPD), the network balances and enhances depth features, significantly improving recognition accuracy. Experimental results show that compared with RGB+FPN(feature pyramid network)method, the recognition precision of PFPD method increases from 92.4% to 94.7% at IoU=0.50, and from 90.8% to 92.8% at IoU=0.75. Compared with the instance segmentation network(Mask R-CNN), the recognition precision of DFENet improvs from 86.4% to 89.2%. The proposed method can effectively improve the recognition precision of object detection and instance segmentation models for solid waste identification.

参考文献/References:

[1] FRATERNALI P,MORANDINI L,GONZáLEZ S L H.Solid waste detection, monitoring and mapping in remote sensing images: A survey[J].Waste Management,2024,189:88-102.DOI:10.1016/j.wasman.2024.08.003.
[2] BONIFAZI G,SERRANTI S.Recycling technologies[C]//Encyclopedia of Sustainability Science and Technology.New York: Springer,2019:1-57.DOI:10.1007/978-1-4939-2493-6_116-4.
[3] JANK A,MüLLER W,SCHNEIDER I,et al.Waste separation press: A mechanical pretreatment option for organic waste from source separation[J].Waste Management,2015,39:71-77.DOI:10.1016/j.wasman.2015.02.024.
[4] ROSS T Y,DOLLáR G.Focal loss for dense object detection[C]//Proceedings of the IEEE International Conference on Computer Vision.Piscataway: IEEE Press,2017:2980-2988.DOI:10.1109/ICCV.2017.324.
[5] LIN T Y,DOLLáR P,GIRSHICK R,et al.Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE Press,2017:2117-2125.DOI:10.1109/CVPR.2017.106.
[6] WANG C Y,YEH I H,LIAO H Y M.Yolov9: Learning what you want to learn using programmable gradient information[C]//European Conference on Computer Vision.Cham:Springer,2025:1-21.DOI:10.1007/978-3-031-72751-1_1.
[7] WANG C Y,BOCHKOVSKIY A,LIAO H Y M.YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE Press,2023:7464-7475.
[8] KIRILLOV A,MINTUN E,RAVI N,et al.Segment anything[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Piscataway:IEEE Press,2023:4015-4026.DOI:10.48550/arXiv.2304.02643.
[9] ZONG Zhuofan,SONG Guanglu,LIN Yu.Detrs with collaborative hybrid assignments training[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Piscataway: IEEE Press,2023:6748-6758.DOI:10.48550/arXiv.2211.12860.
[10] LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE Press,2015:3431-3440.DOI:10.1109/TPAMI.2016.2572683.
[11] 郑龙海,袁祖强,殷晨波,等.基于机器视觉的建筑垃圾自动分类系统研究[J].机械工程与自动化,2019(6):16-18.DOI:10.3969/j.issn.1672-6413.2019.06.006.
[12] XU Xiong,ZHAO Beibei,TONG Xiaohua,et al.A data augmentation strategy combining a modified pix2pix model and the copy-paste operator for solid waste detection with remote sensing images[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,2022,15:8484-8491.DOI:10.1109/JSTARS.2022.3209967.
[13] DAVIS P,AZIZ F,NEWAZ M T,et al.The classification of construction waste material using a deep convolutional neural network[J].Automation in Construction,2021,122:103481.DOI:10.1016/j.autcon.2020.103481.
[14] LI Pan,XU Jiayin,LIU Shenbo.Solid waste detection using enhanced YOLOv8 lightweight convolutional neural networks[J].Mathematics,2024,12(14):2185.DOI:10.3390/math12142185.
[15] LU Weisheng,CHEN Junjie,XUE Fan.Using computer vision to recognize composition of construction waste mixtures: A semantic segmentation approach[J].Resources, Conservation and Recycling,2022,178:106022.DOI:10.1016/j.resconrec.2021.106022.
[16] DENG Fuqin,FENG Hua,LIANG Mingjian,et al.FEANet: Feature-enhanced attention network for RGB-thermal real-time semantic segmentation[C]//2021 IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway: IEEE Press,2021:4467-4473.DOI:10.1109/IROS51168.2021.9636084.
[17] XIAO Wen,YANG Jianhong,FANG Huaiying,et al.A robust classification algorithm for separation of construction waste using NIR hyperspectral system[J].Waste Management,2019,90:1-9.DOI:10.1016/j.wasman.2019.04.036.
[18] LU Bing,DAO P D,LIU Jianggui,et al.Recent advances of hyperspectral imaging technology and applications in agriculture[J].Remote Sensing,2020,12(16):2659.DOI:10.3390/rs12162659.
[19] LI Jiantao,FANG Huaiying,FAN Lulu,et al.RGB-D fusion models for construction and demolition waste detection[J].Waste Management,2022,139:96-104.DOI:10.1016/j.wasman.2021.12.021.
[20] CAI Zhenxing,FANG Huaiying,JIANG Fengfeng,et al.AMFFNet: Asymmetric multi-scale feature fusion network of RGB-NIR for solid waste detection[J].IEEE Transactions on Instrumentation and Measurement,2023,72:1-10.DOI:10.1109/TIM.2023.3300445.
[21] LI Yangke,ZHANG Xinman.Multi-scale context fusion network for urban solid waste detection in remote sensing images[J].Remote Sensing,2024,16(19):3595.DOI:10.3390/rs16193595.
[22] ZHUANG Jiangteng,FANG Huaiying,XIAO Wen,et al.Recognition of concrete and gray brick based on color and texture features[J].Journal of Testing and Evaluation,2019,47(4):3224-3237.DOI:10.1520/JTE20180523.
[23] HU Xinxin,YANG Kailun,FEI Lei,et al.Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation[C]//IEEE International Conference on Image Processing.Piscataway: IEEE Press,2019:1440-1444.DOI:10.1109/ICIP.2019.8803025.
[24] HE Kaiming,GKIOXARI G,DOLLáR P,et al.Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Xision.Piscataway: IEEE Press,2017:2961-2969.DOI:10.1109/ICCV.2017.322.
[25] HU Jie,SHEN Li,SUN Gang.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE Press,2018:7132-7141.DOI:10.1109/CVPR.2018.00745.
[26] HOU Qibin,ZHOU Daquan,FENG Jiashi.Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE Press,2021:13713-13722.DOI:10.1109/CVPR46437.2021.01350.
[27] MA Wanqi,CHEN Hong,ZHANG Wenkang,et al.DSYOLO-trash: An attention mechanism-integrated and object tracking algorithm for solid waste detection[J].Waste Management,2024,178:46-56.DOI:10.1016/j.wasman.2024.02.014.
[28] XIE S,GIRSHICK R,DOLLáR P,et al.Aggregated residual transformations for deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway: IEEE Press,2017:1492-1500.DOI:10.1109/CVPR.2017.634.

备注/Memo

备注/Memo:
收稿日期: 2024-10-30
通信作者: 房怀英(1978-),女,教授,博士,主要从事固废分选机器人开发等的研究。E-mail:happen@hqu.edu.cn。
基金项目: 福建省高效产学合作项目(2024H6010); 福建省科技计划项目(2023Y3006); 第6批福建省泉州市引进高层次人才团队项目(2023CT003)
更新日期/Last Update: 2025-03-20