Many difficult engineering problems cannot be solved by the conventional optimization techniques in practice. Direct searches that need no recourse to explicit derivatives are revived and become popular since the new ...Many difficult engineering problems cannot be solved by the conventional optimization techniques in practice. Direct searches that need no recourse to explicit derivatives are revived and become popular since the new century. In order to get a deep insight into this field, some notes on the direct searches for non-smooth optimization problems are made. The global convergence vs. local convergence and their influences on expected solutions for simulation-based stochastic optimization are pointed out. The sufficient and simple decrease criteria for step acceptance are analyzed, and why simple decrease is enough for globalization in direct searches is identified. The reason to introduce the positive spanning set and its usage in direct searches is explained. Other topics such as the generalization of direct searches to bound, linear and non-linear constraints are also briefly discussed.展开更多
Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stocha...Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stochastic proximal gradient method performs well. However, research on its accelerated version remains unclear. This paper proposes a proximal stochastic accelerated gradient (PSAG) method to address problems involving a combination of smooth and non-smooth components, where the smooth part corresponds to the average of multiple block sums. Simultaneously, most of convergence analyses hold in expectation. To this end, under some mind conditions, we present an almost sure convergence of unbiased gradient estimation in the non-smooth setting. Moreover, we establish that the minimum of the squared gradient mapping norm arbitrarily converges to zero with probability one.展开更多
基金supported by the Key Foundation of Southwest University for Nationalities(09NZD001).
文摘Many difficult engineering problems cannot be solved by the conventional optimization techniques in practice. Direct searches that need no recourse to explicit derivatives are revived and become popular since the new century. In order to get a deep insight into this field, some notes on the direct searches for non-smooth optimization problems are made. The global convergence vs. local convergence and their influences on expected solutions for simulation-based stochastic optimization are pointed out. The sufficient and simple decrease criteria for step acceptance are analyzed, and why simple decrease is enough for globalization in direct searches is identified. The reason to introduce the positive spanning set and its usage in direct searches is explained. Other topics such as the generalization of direct searches to bound, linear and non-linear constraints are also briefly discussed.
文摘Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stochastic proximal gradient method performs well. However, research on its accelerated version remains unclear. This paper proposes a proximal stochastic accelerated gradient (PSAG) method to address problems involving a combination of smooth and non-smooth components, where the smooth part corresponds to the average of multiple block sums. Simultaneously, most of convergence analyses hold in expectation. To this end, under some mind conditions, we present an almost sure convergence of unbiased gradient estimation in the non-smooth setting. Moreover, we establish that the minimum of the squared gradient mapping norm arbitrarily converges to zero with probability one.
