In this paper, we extend a descent algorithm without line search for solving unconstrained optimization problems. Under mild conditions, its global convergence is established. Further, we generalize the search direction to more general form, and also obtain the global convergence of corresponding algorithm. The numerical results illustrate that the new algorithm is effective.
Consider an unconstrained optimization problem (UP)
m i n x ∈ ℜ n f ( x ) , (1)
where f : ℜ n → ℜ is a continuously differentiable function. In general, the iterative algorithms for solving (UP) usually take the form:
x k + 1 = x k + α k d k , (2)
where x k , α k and d k are current iterative point, a positive step length and a search direction, respectively. For simplicity, we denote ∇ f ( x k ) by g k and f ( x k ) by f k .
The main task in the iterative formula (2) is to choose search direction d k and determine step length α k along the direction. There are many classic methods to choose search direction d k , such as the steepest descent methods, Newton-type methods, Variable metric methods (see [
d k = { − g k if k = 1 , − g k + β k d k − 1 if k ≥ 2 , (3)
where β k is a parameter (see [
Based on the above consideration, some authors have started to study the algorithms without line search. Recently, some conjugate gradient algorithms without line search were investigated. In [
Inspired by the above literatures, in this paper we will extend the descent algorithm without line search of [
To proceed, we first assume that [
(H1) The function f has lower bound on £ = { x ∈ ℜ n | f ( x ) ≤ f ( x 1 ) } , where x 1 is available.
(H2) The gradient g is Lipschitz continuous in an open convex set B that contains £ , i.e., there exists L > 0 such that
‖ g ( x ) − g ( y ) ‖ ≤ L ‖ x − y ‖ , ∀ x , y ∈ B . (4)
Now we give the extended algorithm.
Algorithm 2.1. Given a starting point x 1 , a positive constant ϵ , three
parameters μ 1 , μ 2 and ρ such that 0 < μ 1 < 1 2 < μ 2 < 1 , 1 2 ≤ ρ < 1 . Let k : = 1 .
Step 1. If ‖ g k ‖ < ϵ , then stop; otherwise go to Step 2.
Step 2. Compute
s k = { ρ , k = 1 , ρ ‖ g k ‖ 2 ρ ‖ g k ‖ 2 + ( 1 − ρ ) | g k T d k − 1 | , k ≥ 2. (5)
Step 3. Set search direction
d k = { − s k g k , k = 1 , − [ ρ ( 1 − α k − 1 s k 1 + α k − 1 ) g k + ( 1 − ρ ) α k − 1 s k 1 + α k − 1 d k − 1 ] , k ≥ 2. (6)
Step 4. Compute step length by the following rule. When k = 1 , α k is determined by Wolfe line search, i.e., it satisfies that
f ( x k + α k d k ) − f k ≤ μ 1 α k g k T d k , (7)
g ( x k + α k d k ) T d k ≥ μ 2 g k T d k . (8)
When k ≥ 2 ,
α k = − g k T d k L k ‖ d k ‖ 2 , (9)
where L k satisfies that ρ L ≤ L k ≤ m k L and { m k , k = 1 , 2 , ⋯ } is a positive sequence which has a sufficient large upper bound.
Step 5. Set next iterative point
x k + 1 = x k + α k d k . (10)
Step 6. Set k : = k + 1 , and go to Step 1.
Remark 2.1. Note that the formula of s k and d k in Algorithm 2.1 are the generalized forms of those in [
Lemma 3.1. If Algorithm 2.1 generates an infinite sequence { x k , k = 1 , 2 , ⋯ } , then all search directions d k are descent, and ∀ k ≥ 2 , it holds that
− g k T d k ≥ ρ ‖ g k ‖ 2 1 + α k − 1 . (11)
Proof. If k = 1 , it is obvious that − g 1 T d 1 = ρ ‖ g 1 ‖ 2 > 0 . If k ≥ 2 , by (5) and (6), we have
− g k T d k = ρ ( 1 − α k − 1 s k 1 + α k − 1 ) ‖ g k ‖ 2 + ( 1 − ρ ) α k − 1 s k 1 + α k − 1 g k T d k − 1 = ρ ‖ g k ‖ 2 − α k − 1 s k 1 + α k − 1 [ ρ ‖ g k ‖ 2 − ( 1 − ρ ) g k T d k − 1 ] ≥ ρ ‖ g k ‖ 2 − α k − 1 s k 1 + α k − 1 [ ρ ‖ g k ‖ 2 + ( 1 − ρ ) | g k T d k − 1 | ] = ρ ‖ g k ‖ 2 1 + α k − 1 . (12)
This completes the proof. ,
Lemma 3.2 (Mean value theorem, see [
f ( x k + α d k ) − f k = α ∫ 0 1 g ( x k + t α d k ) T d k d t , (13)
where x k , x k + α d k ∈ B , d k ∈ ℜ n . If f ( x ) is twice continuously differentiable on B , then
g ( x k + α d k ) − g k = α ∫ 0 1 ∇ 2 f ( x k + t α d k ) d k d t , (14)
and
f ( x k + α d k ) − f k = α g k T d k + α 2 ∫ 0 1 ( 1 − t ) d k T ∇ 2 f ( x k + t α d k ) d k d t . (15)
Lemma 3.3. ∀ k ≥ 2 ,
‖ d k ‖ 2 ≤ 3 ρ 2 ⋅ ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 . (16)
Proof. Where k ≥ 2 , it holds that ( 1 − ρ ) s k | g k T d k − 1 | = ρ ( 1 − s k ) ‖ g k ‖ 2 by (5). Then ∀ k ≥ 2 , we have
‖ d k ‖ 2 = ‖ ρ ( 1 − α k − 1 s k 1 + α k − 1 ) g k + ( 1 − ρ ) α k − 1 s k 1 + α k − 1 d k − 1 ‖ 2 = ρ 2 ( 1 − α k − 1 s k 1 + α k − 1 ) 2 ‖ g k ‖ 2 + 2 ρ ( 1 − α k − 1 s k 1 + α k − 1 ) ⋅ ( 1 − ρ ) α k − 1 s k 1 + α k − 1 ⋅ g k T d k − 1 + ( 1 − ρ ) 2 ( α k − 1 s k 1 + α k − 1 ) 2 ‖ d k − 1 ‖ 2 ≤ ρ 2 ‖ g k ‖ 2 + 2 ρ ( 1 − ρ ) s k | g k T d k − 1 | + ‖ d k − 1 ‖ 2 = ρ 2 ‖ g k ‖ 2 + 2 ρ 2 ( 1 − s k ) ‖ g k ‖ 2 + ‖ d k − 1 ‖ 2 ≤ 3 ρ 2 ‖ g k ‖ 2 + ‖ d k − 1 ‖ 2 .
Using induction principle and noting that ‖ d 1 ‖ 2 = ρ 2 ‖ g 1 ‖ 2 , it yields that
‖ d k ‖ 2 ≤ 3 ρ 2 ‖ g k ‖ 2 + 3 ρ 2 ‖ g k − 1 ‖ 2 + 3 ρ 2 ‖ g k − 2 ‖ 2 + ⋯ + ρ 2 ‖ g 1 ‖ 2 .
Therefore (16) holds. The proof is completed. ,
Theorem 3.1. If (H1), (H2) hold, and Algorithm 2.1 generates an infinite sequence { x k , k = 1 , 2 , ⋯ } , then
∑ k = 2 + ∞ ‖ g k ‖ 4 ( 1 + α k − 1 ) 2 ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 < + ∞ ; (17)
and
∑ k = 2 + ∞ α k 1 + α k − 1 ‖ g k ‖ 2 < + ∞ . (18)
Proof. When k ≥ 2 , from (13), (4), Lemma 3.1, Lemma 3.3 and ρ L ≤ L k ≤ m k L , it yields that
f k − f k + 1 = − α k ∫ 0 1 g ( x k + t α k d k ) T d k d t = − α k g k T d k − α k ∫ 0 1 [ g ( x k + t α k d k ) − g k ] T d k d t ≥ − α k g k T d k − α k ∫ 0 1 ‖ g ( x k + t α k d k ) − g k ‖ ⋅ ‖ d k ‖ d t ≥ − α k g k T d k − α k 2 L ∫ 0 1 t ‖ d k ‖ 2 d t = − α k g k T d k − 1 2 α k 2 L ‖ d k ‖ 2
= ( 1 L k − 1 2 L k 2 ) ( g k T d k ) 2 ‖ d k ‖ 2 ≥ ( 2 ρ − 1 ) ( g k T d k ) 2 2 L m k 2 ‖ d k ‖ 2 ≥ ( 2 ρ − 1 ) ⋅ ρ 2 ‖ g k ‖ 4 2 L m k 2 ( 1 + α k − 1 ) 2 ⋅ 3 ρ 2 ⋅ ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 = ( 2 ρ − 1 ) ‖ g k ‖ 4 6 L m k 2 ( 1 + α k − 1 ) 2 ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 , (19)
which implies that { f k , k = 1 , 2 , ⋯ } is a decreasing sequence. And it is clear that the sequence { x k , k = 1 , 2 , ⋯ } generated by Algorithm 2.1 is contained in B by (H1), and there exists a constant f * such that lim k → ∞ f k = f * . Therefore
∑ k = 2 + ∞ ( f k − f k + 1 ) = lim N → + ∞ ∑ k = 2 N ( f k − f k + 1 ) = lim N → + ∞ ( f 2 − f N + 1 ) = f 2 − f * .
Thus
∑ k = 2 + ∞ ( f k − f k + 1 ) < + ∞ ,
which combining with (19) yields
∑ k = 2 + ∞ ‖ g k ‖ 4 m k 2 ( 1 + α k − 1 ) 2 ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 < + ∞ . (20)
Since { m k , k = 1 , 2 , ⋯ } has an upper bound, (17) holds.
On the other hand, by (9) and Lemma 3.1, we have
f k − f k + 1 ≥ − α k g k T d k − 1 2 α k 2 L ‖ d k ‖ 2 = − α k g k T d k + L α k g k T d k 2 L k = − ( 2 L k − L ) ( α k g k T d k ) 2 L k ≥ − ( 2 ρ − 1 ) ( α k g k T d k ) 2 ρ ≥ ( 2 ρ − 1 ) α k ‖ g k ‖ 2 2 ( 1 + α k − 1 ) . (21)
By the same analysis as the above proof, (18) holds. The proof is completed. ,
Lemma 3.4 (see [
Theorem 3.2. If the conditions in Theorem 3.1 hold, then
lim inf k → + ∞ ‖ g k ‖ = 0. (22)
Proof. Suppose lim inf k → + ∞ ‖ g k ‖ ≠ 0 , then there exists a positive γ such that
‖ g k ‖ ≥ γ , ∀ k ≥ 1. (23)
In the following, we carry out our proofs in two cases.
Case 1. We complete the proof by utilizing reduction to absurdity. Suppose that sup k ≥ 1 { α k } < + ∞ . By (17), we have
∑ k = 2 + ∞ ‖ g k ‖ 4 ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 < + ∞ (24)
From Lemma 3.4, we know that there exists M > 0 such that ‖ g k ‖ ≤ M , ∀ k ≥ 1 . Combining (23), we have
‖ g k ‖ 4 ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 ≥ γ 4 k ⋅ M 2 .
It is known that
∑ k = 2 + ∞ γ 4 k ⋅ M 2 = γ 4 M 2 ∑ k = 2 + ∞ 1 k = + ∞ ,
So
∑ k = 2 + ∞ ‖ g k ‖ 4 ∑ 1 ≤ i ≤ k ‖ g i ‖ 2 = + ∞ , (25)
which contradicts with (24). Therefore (22) holds.
Case 2. When sup k ≥ 1 { α k } = + ∞ , the proof is the same as that in [
It follows from the proofs of Case 1 and Case 2 that (22) holds. This completes the proof. ,
Remark 3.1. Search direction of Algorithm 2.1 can be extended to more general form as follows:
d k = { − s k g k , k = 1 , − ρ ( 1 − φ ( α k − 1 ) s k ) g k ± ( 1 − ρ ) ϕ φ ( α k − 1 ) s k d k − 1 , k ≥ 2 , (26)
where the function φ ( α ) satisfies the following conditions(see [
a) It is continuous and strictly monotone increasing when α ∈ [ 0, + ∞ ) ;
b) lim α → 0 + φ ( α ) = φ ( 0 ) = 0 and lim α → + ∞ φ ( α ) = 1 ;
c) α ( 1 − φ ( α ) ) is continuous, strictly monotone increasing when α ∈ [ 0, + ∞ ) , and
lim α → + ∞ α ( 1 − φ ( α ) ) = 1.
Evidently, there are many functions satisfying the conditions (a)-(c). For
example, α 1 + α , α 2 1 + α + α 2 , α 3 1 + α 2 + α 3 , etc (see [
2.1 in which d k is determined by (26) as Algorithm 3.1. By using proof technique of above Theorem 3.2, it is easy to get its convergence theorem.
In this section, we report some preliminary numerical experiments. The test problems and their initial values are drawn from [
In numerical experiment, we take the parameter L k = 100 ,and stop the iteration if the inequality ‖ g k ‖ ≤ 10 − 5 is satisfied. The detailed numerical results
Problem | Dim | NI | NF | NG | Problem | Dim | NI | NF | NG |
---|---|---|---|---|---|---|---|---|---|
ROSE | 2 | 8 | 10 | 11 | TRID | 3 | 66 | 67 | 68 |
FROTH | 2 | 6 | 7 | 8 | 50 | 84 | 85 | 86 | |
GAUSS | 3 | 126 | 128 | 129 | 100 | 86 | 87 | 88 | |
BOX | 3 | 1 | 51 | 52 | 200 | 85 | 87 | 89 | |
SING | 4 | 20 | 21 | 22 | BAND | 3 | 22 | 23 | 24 |
WOOD | 4 | 6 | 7 | 8 | 50 | 27 | 28 | 29 | |
BD | 4 | 5 | 7 | 9 | 100 | 23 | 25 | 27 | |
ROSEX | 8 | 9 | 11 | 12 | 200 | 24 | 26 | 28 | |
50 | 8 | 9 | 10 | LIN | 2 | 1 | 3 | 5 | |
100 | 8 | 9 | 10 | 50 | 1 | 3 | 5 | ||
SINGX | 4 | 20 | 21 | 22 | 500 | 1 | 3 | 5 | |
PEN2 | 50 | 3 | 4 | 5 | 1000 | 1 | 3 | 5 | |
VARDIM | 2 | 99 | 100 | 101 | LIN1 | 2 | 18 | 19 | 20 |
50 | 4 | 5 | 6 | 10 | 1 | 3 | 5 |
are reported in
In this paper, we extended the descent algorithm without line search of [
We gratefully acknowledge the scholarship fund of education department of Guangxi Zhuang autonomous region, Guangxi basic ability improvement project fund for the middle-aged and young teachers of colleges and universities (2017KY0068, KY2016YB069), Guangxi higher education undergraduate course teaching reform project fund (2017JGB147), NNSF of China (11761014), Guangxi natural science foundation (2017GXNSFAA198243).
Chen, C.L., Luo, L.L., Han, C.H. and Chen, Y. (2018) Global Convergence of an Extended Descent Algorithm without Line Search for Unconstrained Optimization. Journal of Applied Mathematics and Physics, 6, 130-137. https://doi.org/10.4236/jamp.2018.61013