正则稀疏优化模型及算法研究综述
A Survey on Regularized Sparse Optimization Models and Algorithms
摘要: 稀疏优化在工业工程等实际问题中具有十分重要的作用。在过去的几十年里,很多实际问题可以归纳成正则稀疏优化模型并求解出欠定系统的稀疏解,因此其改进和算法设计得到广泛的研究。正则稀疏优化模型不仅可以将原问题降维,而且可以将不适定的问题转换为适定问题。关键问题是如何构造正则项,使得模型具有好的稀疏解的同时还有很好的泛化能力。本文,我们重点关注近30年来正则稀疏优化模型及算法的研究进展,总结归纳了最具代表性的几种正则稀疏优化模型及算法。最后,结合最新研究成果,针对损伤识别、故障诊断、超分辨率重建和电阻抗层析成像等实际问题,构造不同的新正则稀疏优化模型,并探讨其可以研究发展的方向以及更广阔的应用前景。
Abstract: Sparse optimization plays a very important role in practical problems such as industrial engineering. In the past decades, many practical problems can be generalized into regularized sparse optimization models to solve the sparse solutions of underdetermined systems. Therefore, the improvement of such models and the design of algorithms have been widely studied. The regularized sparse optimization model can reduce the dimension of the original problem and transform the ill-posed problem into a well-posed problem. The key point is how to choose and construct the regularization terms so that the model can obtain a good sparse solution with good generalization ability. In this paper, we focus on the research progress of regularized sparse optimization models and algorithms in the past 30 years, and summarize the most representative models and algorithms. In addition, according to the latest research results, different new regularized sparse optimization models are constructed to solve practical problems such as damage identification, fault diagnosis, super-resolution reconstruction and electrical impedance tomography, and the development direction and broader application prospect of these models are discussed.
文章引用:程克林. 正则稀疏优化模型及算法研究综述[J]. 人工智能与机器人研究, 2023, 12(3): 155-166. https://doi.org/10.12677/AIRR.2023.123018

参考文献

[1] Bruckstein, A.M., Donoho, D.L. and Elad, M. (2009) From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images. SIAM Review, 51, 34-81. [Google Scholar] [CrossRef
[2] Duarte, M.F., Davenport, M.A., Takhar, D., et al. (2008) Single-Pixel Imaging via Compressive Sampling. IEEE Signal Processing Magazine, 25, 83-91. [Google Scholar] [CrossRef
[3] Wang, Y., Zhou, G.L., Zhang, X., Liu, W. and Caccetta, L. (2016) The Non-Convex Sparse Problem with Nonnegative Constraint for Signal Reconstruction. Journal of Optimization Theory and Applications, 170, 1009-1025. [Google Scholar] [CrossRef
[4] Xiu, X.C., Pan, L.L., Yang, Y. and Liu, W. (2022) Efficient and Fast Joint Sparse Constrained Canonical Correlation Analysis for Fault Detection. IEEE Transactions on Neural Networks and Learning Systems, 1-11. [Google Scholar] [CrossRef
[5] Donoho, D.L. (2006) Compressed Sensing. IEEE Transactions on Information Theory, 52, 1289-1306. [Google Scholar] [CrossRef
[6] Luo, Z.Y., Qin, L.X., Kong, L.C. and Xiu, N.H. (2014) The Nonnegative Zero-Norm Minimization under Generalized Z-Matrix Measurement. Journal of Optimization Theory and Applications, 160, 854-864. [Google Scholar] [CrossRef
[7] Merhej, D., Diab, C., Khalil, M. and Prost, R. (2011) Embedding Prior Knowledge within Compressed Sensing by Neural Networks. IEEE Transactions on Neural Networks, 22, 1638-1649. [Google Scholar] [CrossRef
[8] Hastie, T., Tibshirani, R. and Wainwright, M. (2015) Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman & Hall/CRC, New York. [Google Scholar] [CrossRef
[9] Torfifi, A., Shirvani, R.A., Soleymani, S. and Nasrabadi, N. M. (2018) Attention-Based Guided Structured Sparsity of Deep Neural Networks. ArXiv Preprint ArXiv: 1802.09902.
[10] Zhang, D., Katz-Samuels, J., Figueiredo, M.A.T. and Balzano, L. (2018) Simultaneous Sparsity and Parameter Tying for Deep Learning using Ordered Weighted l1 Regularization. 2018 IEEE Statistical Signal Processing Workshop (SSP), Freiburg im Breisgau, 10-13 June 2018, 65-69. [Google Scholar] [CrossRef
[11] Mallat, S. and Zhang, Z. (1993) Matching Pursuit with Time-Frequency Dictionaries. IEEE Transactions on Signal Processing, 41, 3397-3415. [Google Scholar] [CrossRef
[12] Blumensath, T. and Davies, M.E. (2008) Iterative Thresholding for Sparse Approximations. Journal of Fourier Analysis and Applications, 14, 629-654. [Google Scholar] [CrossRef
[13] Herrity, K.K., Gilbert, A.C. and Tropp, J.A. (2006) Sparse Approximation via Iterative Thresholding. 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, 3, Toulouse, France.
[14] Natraajan, B.K. (1995) Sparse Approximate Solutions to Linear Systems. SIAM Journal on Computing, 24, 227-234. [Google Scholar] [CrossRef
[15] Chen, X.J., Ge, D.D., Wang, Z.Z. and Ye, Y.Y. (2014) Complexity of Unconstrained L2 -Lp Minimization. Mathematical Programming, 143, 371-383. [Google Scholar] [CrossRef
[16] Frank, I.E. and Freidman, J.H. (1993) A Statistical View of Some Chemometrics Regression Tools. Technometrics, 35, 109-135. [Google Scholar] [CrossRef
[17] Tropp, J.A. (2004) Greed Is Good: Algorithmic Results for Sparse Approximation. IEEE Transactions on Information Theory, 50, 2231-2242. [Google Scholar] [CrossRef
[18] Bredies, K. and Lorenz, D.A. (2009) Iterated Hard Shrinkage for Minimization Problems with Sparsity Constraints. SIAM Journal on Scientific Computing, 30, 657-683. [Google Scholar] [CrossRef
[19] Chen, S.S., Donoho, D.L. and Saunders, M.A. (1998) Atomic Decomposition by Basis Pursuit. SIAM Journal on Scientific Computing, 20, 33-61. [Google Scholar] [CrossRef
[20] Candès, E.J. (2008) The Restricted Isometry Property and Its Implications for Compressed Sensing. Comptes Rendus Mathematique, 346, 589-592. [Google Scholar] [CrossRef
[21] Donoho, D.L. (1995) De-Noising by Soft-Thresholding. IEEE Transactions on Information Theory, 41, 613-627. [Google Scholar] [CrossRef
[22] Chambolle, A., DeVore, R.A., Lee, N.-Y. and Lucier, B.J. (1998) Nonlinear Wavelet Image Processing: Variational Problems, Compression, and Noise Removal Through Wavelet Shrinkage. IEEE Transactions on Image Processing, 7, 319-335. [Google Scholar] [CrossRef] [PubMed]
[23] Figueiredo, M.A.T. and Nowak, R.D. (2003) An EM Algorithm for Wavelet-Based Image Restoration. IEEE Transactions on Image Processing, 12, 906-916. [Google Scholar] [CrossRef
[24] Daubechies, I., Defrise, M. and Mol, C.D. (2004) An Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint. Communications on Pure and Applied Mathematics, 57, 1413-1457. [Google Scholar] [CrossRef
[25] Vonesch, C. and Unser, M. (2007) Fast Iterative Thresholding Algorithm for Wavelet-Regularized Deconvolution. In: Van De Ville, D., Goyal, V.K. and Papadakis, M., Eds., Proceedings of the SPIE Optics and Photonics 2007 Conference on Mathematical Methods: Wavelet XII, Vol. 6701, SPIE, Bellingham, 135-139. [Google Scholar] [CrossRef
[26] Kim, S.-J., Koh, K., Lustig, M., Boyd, S. and Gorinevsky, D. (2007) An Interior-Point Method for Large-Scale l1-Regularized Least Squares. IEEE Journal of Selected Topics in Signal Processing, 1, 606-617. [Google Scholar] [CrossRef
[27] Osborne, M.R., Presnell, B. and Turlach, B.A. (2000) A New Approach to Variable Selection in Least Squares Problems. IMA Journal of Numerical Analysis, 20, 389-403. [Google Scholar] [CrossRef
[28] Osborne, M.R., Presnell, B. and Turlach, B.A. (2000) On the Lasso and Its Dual. Journal of Computational and Graphical Statistics, 9, 319-337. [Google Scholar] [CrossRef
[29] Figueiredo, M.A.T., Nowak, R.D. and Wright, S.J. (2007) Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems. IEEE Journal of Selected Topics in Signal Processing, 1, 586-597. [Google Scholar] [CrossRef
[30] Hale, E.T., Yin, W.T. and Zhang, Y. (2007) A Fifixed-Point Continuation Method for l1-Regularized Minimization with Applications to Compressed Sensing. CAAM Technical Report TR07-07.
https://hdl.handle.net/1911/102072
[31] Becker, S., Bobin, J. and Candès, E.J. (2011) NESTA: A Fast and Accurate First-Order Method for Sparse Recovery. SIAM Journal on Imaging Sciences, 4, 1-39. [Google Scholar] [CrossRef
[32] Ben-Tal, A. and Nemirovski, A. (2001) Lectures on Modern Convex Optimization-Analysis, Algorithms, and Engineering Applications. In: MPS-SIAM Series on Optimization, Society for Industrial and Applied Mathematics, Philadelphia. [Google Scholar] [CrossRef
[33] Yang, J. and Zhang, Y. (2011) Alternating Direction Algorithms for l1-Problems in Compressive Sensing. SIAM Journal on Scientific Computing, 33, 250-278. [Google Scholar] [CrossRef
[34] Meinshausen, N. and Yu, B. (2009) Lasso-Type Recovery of Sparse Representations for High-Dimensional Data. Annals of Statistics, 37, 246-270. [Google Scholar] [CrossRef
[35] Zou, H. (2006) The Adaptive Lasso and Its Oracle Properties. Journal of the American Statistical Association, 101, 1418-1429. [Google Scholar] [CrossRef
[36] Fan, J. and Li, R. (2001) Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Properties. Journal of the American Statistical Association, 96, 1348-1360. [Google Scholar] [CrossRef
[37] Candès, E.J., Wakin, M.B. and Boyd, S.P. (2008) Enhancing Sparsity by Reweighted l1 Minimization. Journal of Fourier Analysis and Applications, 14, 877-905. [Google Scholar] [CrossRef
[38] Chen, X.J., Xu, F.M. and Ye, Y.Y. (2010) Lower Bound Theory of Nonzero Entries in Solutions of l2-lp Minimization. SIAM Journal on Scientific Computing, 32: 2832-2852. [Google Scholar] [CrossRef
[39] Xu, Z.B., Zhang, H., Wang, Y., Chang, X.Y. and Liang, Y. (2010) L1/2 Regularization. Science China Information Sciences, 53, 1159-1169. [Google Scholar] [CrossRef
[40] Xu, Z.B., Chang, X.Y., Xu, F.M. and Zhang, H. (2012) L1/2 Regularization: A Thresholding Representation Theory and a Fast Solver. IEEE Transactions on Neural Networks and Learning Systems, 23, 1013-1027. [Google Scholar] [CrossRef
[41] Figueiredo, M.A.T. and Nowak, R.D. (2005) A Bound Optimization Approach to Wavelet-Based Image Deconvolution. IEEE International Conference on Image Processing 2005, Genova, 14-14 September 2005. [Google Scholar] [CrossRef
[42] Figueiredo, M.A.T., Bioucas-Dias, J.M. and Nowak, R.D. (2007) Majorization Minimization Algorithms for Wavelet-Based Image Restoration. IEEE Transactions on Image Processing, 16, 2980-2991. [Google Scholar] [CrossRef
[43] Chen, X.J. and Zhou, W.J. (2010) Convergence of Reweighted l1 Minimization Algorithms and Unique Solution of Truncated lp Minimization. Technical Report, Hong Kong Polytechnic University.
https://www.researchgate.net/publication/228532259_Convergence_of_reweighted_l1_minimization_algorithms_and_unique_solution_of_truncated_lp_minimization
[44] Chen, X.J. and Zhou, W.J. (2014) Convergence of the Reweighted l1 Minimization Algorithm for l2-lp Minimization. Computational Optimization and Applications, 59, 47-61. [Google Scholar] [CrossRef
[45] Lai, M.J. and Wang, J.Y. (2011) An Unconstrained lq Minimization with for Sparse Solution of Underdetermined Linear Systems. SIAM Journal on Optimization, 21, 82-101. [Google Scholar] [CrossRef
[46] Chen, X.J. (2012) Smoothing Methods for Nonsmooth, Nonconvex Minimization. Mathematical Programming, 134, 71-99. [Google Scholar] [CrossRef
[47] Lai, M.-J., Xu, Y.Y. and Yin, W.T. (2013) Improved Iteratively Reweighted Least Squares for Unconstrained Smoothed lq Minimization. SIAM Journal on Numerical Analysis, 51, 927-957. [Google Scholar] [CrossRef
[48] Lou, Y.F., Yin, P.H., He, Q. and Xin, J. (2015) Computing Sparse Representation in a Highly Coherent Dictionary Based on Difffference of L1 and L2. Journal of Scientific Computing, 64, 178-196. [Google Scholar] [CrossRef
[49] Boyd, S., Parikh, N., Chu, E., Peleato, B. and Eckstein, J. (2011) Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends® in Machine Learning, 3, 1-122. [Google Scholar] [CrossRef
[50] Bai, Y.Q. and Shen, K.J. (2016) Alternating Direction Method of Multipliers for l1-l2-Regularized Logistic Regression Model. Journal of the Operations Research Society of China, 4, 243-253. [Google Scholar] [CrossRef
[51] Selesnick, I. (2017) Sparse Regularization via Convex Analysis. IEEE Transactions on Signal Processing, 65, 4481-4494. [Google Scholar] [CrossRef
[52] Li, Q., Bai, Y.Q., Yu, C.J. and Yuan, Y.-X. (2019) A New Piecewise Quadratic Approximation Approach for L0 Norm Minimization Problem. Science China Mathematics, 62, 185-204. [Google Scholar] [CrossRef
[53] Gao, X.R., Bai, Y.Q. and Li, Q. (2021) A Sparse Optimization Problem with Hybrid L2-Lp Regularization for Application of Magnetic Resonance Brain Images. Journal of Combinatorial Optimization, 42, 760-784. [Google Scholar] [CrossRef
[54] Gao, X.R., Bai, Y.Q., Fang, S.-C., et al. (2023) A New Hybrid lp-l2 Model for Sparse Solutions with Applications to Image Processing. Journal of Industrial and Management Optimization, 19, 890-915. [Google Scholar] [CrossRef
[55] 吕昊, 冯仲仁, 王雄江, 等. 基于混合鲸鱼退火算法和稀疏正则化的结构损伤识别[J]. 振动与冲击, 2021, 40(17): 85-91.
[56] 宋泽树, 宋泽树, 石娟娟, 等. 广义平滑对数正则化稀疏分解方法研究及其在齿轮箱复合故障诊断中的应用[J]. 机械工程学报, 2022, 58(23): 123-137.
[57] 杨雪, 李峰, 鹿明, 等. 混合稀疏表示模型的超分辨率重建[J]. 遥感学报, 2022, 26(8): 1685-1697.
[58] 黄淑英, 吴昕, 杨勇, 等. 自适应正则化稀疏表示的遥感图像SR重建[J]. 小型微型计算机系统, 2023, 44(3): 573-581.
[59] 马敏, 于洁, 范文茹. 基于改进低秩稀疏正则化的CFRP电阻抗层析成像算法研究[J]. 振动与冲击, 2022, 41(14): 151-157.
[60] Ma, T.-H., Xu, Z.B., Meng, D.Y. and Zhao, X.-L. (2021) Hyperspectral Image Restoration Combining Intrinsic Image Characterization with Robust Noise Modeling. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14: 1628-1644. [Google Scholar] [CrossRef
[61] Wang, Y.Y., He, Z.M., Zhan, X., Fu, Y.H. and Zhou, L.M. (2022) Three-Dimensional Sparse SAR Imaging with Generalized Lq Regularization. Remote Sensing, 14, Article No. 288. [Google Scholar] [CrossRef
[62] Xu, Y.S. and Zeng, T.S. (2023) Sparse Deep Neural Network for Nonlinear Partial Differential Equations. Numerical Mathematics: Theory, Methods and Applications, 16, 58-78. [Google Scholar] [CrossRef