STABILIZATION BY PERTURBATION
OF A 4N 2 TOEPLITZ SOLVER
PER CHRISTIAN HANSENy AND PLAMEN Y. YALAMOVz
Abstract. We consider a 4n2 -algorithm for computing the Gohberg-Semencul representation of
the inverse of a nonsymmetricToeplitz matrix and for solution of a Toeplitz system. The algorithm is
unstable when one or more of the leading principal submatrices are ill-conditioned. In our stabilized
version we slightly perturb the matrix when we encounter a division by a small number. As a
consequence, the solution is also perturbed, and its accuracy can be improved by taking a small
number of iterative renement steps. We present a roundo error analysis of the new algorithm as
well as numerical results that support our analysis.
Key words. General Toeplitz matrices, O(n2 ) algorithms, stabilization.
AMS(MOS) subject classications. 65F30, 65G05
1. Introduction. Toeplitz matrices and Toeplitz systems appear in many applications, such as signal processing, statistics, integral equations, Pade approximations,
etc., see [5]. A number of algorithms have been proposed for solving Toeplitz systems
and for inversion of Toeplitz matrices. A survey of many algorithms in this area is
given in [4].
All these algorithms are faster than the usual algorithms for general dense matrices because they exploit the structure of the Toeplitz matrices. Many of them require
O(n2 ) operations, where n is the order of the matrix, instead of O(n3) operations for
a general dense solver. But the O(n2)-algorithms can break down or produce completely wrong results. For example, this happens in Levinson's algorithm [12, x4.7.3],
[17] if one or more of the leading principal submatrices are singular or ill-conditioned.
Pivoting techniques are not useful because they spoil the Toeplitz structure.
In order to handle the ill-conditioning of leading principal submatrices in the
Levinson algorithm, Chan and Hansen incorporated look-ahead in the algorithm [6].
Their look-ahead algorithm is based on the idea that we can take a block step instead of the usual step in the Levinson algorithm, so that any ill-conditioned leading
principal submatrix (which is implicitly to be inverted) is avoided. This algorithm,
as well as the related algorithm by Freund and Zha [10], needs some overhead in time
and storage which grows with the size of the block step, and a maximum block size
is required in a practical implementation. Hence, the look-ahead algorithms are only
useful for matrices that require look-ahead steps with small block size.
In this paper we consider a dierent approach for improving the stability of fast
Toeplitz solvers. This stabilization does not require look-ahead, and a xed size of
additional storage, 3n locations, is needed. Our approach is based on the idea that
if we add relatively small perturbations to ill-conditioned submatrices then we can
improve their conditioning. If the whole matrix is well-conditioned it will not be
inuenced strongly by these perturbations, and by taking a few steps of iterative
renement we can compute the solution of a system of equations, or a representation
P. Y. Yalamov was partially supported by Grant MM-434/94 from the Bulgarian Ministry of
Education and Science, and by Grant No. 21-0804/26-1821 from the Danish Research Councils.
y Department of Mathematical Modelling, Building 305, Technical University of Denmark, DK2800 Lyngby, Denmark ([email protected]).
z Center of Applied Mathematics and Informatics, University of Rousse, 7017 Rousse, Bulgaria
([email protected]).
1
of the inverse of the Toeplitz matrix, to satisfactory accuracy.
The same type of stabilization was proposed by Concus and Saylor [7] for indenite
symmetric Toeplitz matrices in connection with Trench's inversion algorithm [12,
x4.7.4].
We apply the stabilization to an algorithm for computing the Gohberg-Semencul
representation of the inverse of a non-symmetric Toeplitz matrix. This algorithm is
discussed by, e.g., Heinig and Rost [13] and Voevodin and Tyrtyshnikov [21], and it
is also the basis algorithm in Argonne's Toeplitz package [2]. It is a modication of
Durbin's algorithm [12, x4.7.2] for solving the Yule-Walker problem. The algorithm
requires 4n2 + O(n) operations, and we shall demonstrate that the computational
overhead in our stabilization is O(1).
When the 4n2 -algorithm is used to solve a system of equations, we need to multiply the right-hand side with the inverse of the Toeplitz matrix, and the complete
algorithm is sometimes referred to as Jain's algorithm [16]. The multiplications can
be performed in O(n log2 n) operations, and we refer to [1] and [20, x4.2.4] for details
about fast methods for multiplication with Toeplitz matrices. When the stabilized
algorithm is used, a few iterative renement steps are necessary, and these steps can
also be performed in O(n log2 n) operations. Hence, the stabilized algorithm needs
4n2 + O(n log2 n) operations to solve a system of equations. Even for moderate n, this
is faster than the classical Levinson algorithm which requires 6n2 + O(n) operations,
as long as the number of renement steps is moderate.
Consider the Toeplitz system
(1)
Tn x = b;
where Tn is a nonsymmetric (n + 1) (n + 1) Toeplitz matrix,
0 t0
B t1
Tn = B
@ ..
.
t?1 t?n 1
t0 t1?n C
C
..
.
...
tn tn?1 ..
.
t0
A;
and let Tk ; k = 0; : : :; n; denote the (k + 1) (k + 1) leading principal submatrix
of Tn . The algorithm computes representations of Tk?1 successively. The inverses are
represented via their rst and last columns x(k) = Tk?1e1 and y(k) = Tk?1 ek+1. In
each step of the algorithm we update recursively a scaling factor pk and the scaled
vectors x~(k) = p?k 1 x(k) and y~(k) = p?k 1y(k) , to arrive at pn , x~(n) = p?n 1 Tn?1 e1 and
y~(n) = p?n 1Tn?1 en+1 .
As we show in the next section, ill-conditioning of Tk leads to division by a small
number and growth of the scaling factor pk . The only dangerous place (i.e., division
by a small number) where the error can blow up is in the computation of the scaling
factors pk . Therefore, we need to apply stabilization only to the computation of pk .
In a practical implementation of the look-ahead Levinson algorithm it is necessary
that the number of successive ill-conditioned submatrices is relatively small, while our
algorithm does not need this assumption. Moreover, our algorithm needs less storage
than the look-ahead algorithm: one array of length 2n ? 1 to store the stabilized
matrix, and one n-array for storage of the residuals in the iterative renement part.
Although the basic idea in our algorithm is the same as that used in the algorithm
by Concus and Saylor [7], there are important dierences. Our algorithm works
for general Toeplitz matrices, and it computes the Gohberg-Semencul representation
2
x~(0) = y~(0) = 1; p0 = 1=t0;
for k = 1; : : :; n
fk = tk x~(0k?1) + + t1 x~(kk??11);
gk = t?1y~0(k?1) + + t?k y~k(k??11) ;
k = pk?1fk ; k = pk?1gk ;
pk = pk?1=(1 ? k k ); (k?1)
0
x
~
(
k
)
;
x~ =
?
0 x~(k?1) k y~(k?01) y~(k) = ?k
+ y~(k?1) ;
0
end
x(n) = pnx~(n); y(n) = pny~(n) ;
. The original 4n2 -algorithm.
Fig. 1
of Tn?1 . Moreover, we provide a roundo error analysis, as well as theory that supports
our choice of the necessary perturbation parameter.
The stabilization idea should carry over to other algorithms that also depend on
well-conditioned leading submatrices, such as the Levinson and Schur algorithms. The
systolic version of the Bareiss algorithm, derived in [4], is another potential application
of the stabilization-by-perturbation idea, since systolic arrays require algorithms with
very simple structure.
The outline of our paper is as follows. In x2 we present a brief summary of the
4n2-algorithm, and we illustrate how unstable the original algorithm can be. In x3
the new stabilized version of the algorithm is presented. An error analysis of both
algorithms is given in x4. Finally, numerical experiments are presented in x5.
2. A Brief Summary of the Original Algorithm. The 4n2-algorithm for
computation of the rst and last columns x(n) and y(n) of Tn?1 is summarized in
Fig. 1. Inside the loop this algorithm computes the scaled rst and last columns
0 (k) 1
x0
B
?
1
(
k
)
(
k
)
x~ = pk x = @ ... C
A;
0 (k) 1
y0
B
?
1
(
k
)
(
k
)
y~ = pk y = @ ... C
A
x(kk)
yk(k)
of the matrix Tk?1 , with scaling factor pk . This is a simplied version of the algorithm
proposed in [21], where there are two separate scaling factors for x~(k) and y~(k) , and
the initial values x~(0) and y~(0) are arbitrarily chosen. We simplify this choice and,
thus, simplify the computation of the scaling factors. As a consequence of our choice,
we have
(2)
x~(0k) = y~k(k) = 1;
k = 0; : : :; n;
which follows from the persymmetry of Tk?1 .
Once pn , x~(n) and y~(n) are computed, the explicit inverse can be obtained either
via Trench's algorithm [12, x4.7.4] (which we shall not pursue here) or from the rst
Gohberg-Semencul formula [11]. Using pn and the scaled vectors x~(n) and y~(n) , we
3
t?001 = pn;
for i = 1; : : :; dn=2e
t?ii 1 = t?i?11;i?1 + pn(~xi(n) y~n(n?)i + x~n(n?)i+1y~i(?n)1 );
end
. Computation of the rst dn=2e diagonal entries t?ii 1 of Tn?1 .
Fig. 2
obtain:
20 (n)
1 0 y~n(n) y~n(n?)1
x~0
0
66B x~(n) x~(n)
CC BB
y~n(n)
B
1
0
?
1
B
6
Tn = pn 6B .
C
4@ (..n) . . . .(.n.) (n) A B@
x~n
x~1
x~0
0
y~0(n) 1
. . . ... C
CC ?
. . . y~(n) C
n?1 A
y~n(n)
0 0
1 0 0 x~n(n) x~1(n) 13
0
y~0(n) 0
B
CC BB 0 . . . ... CC77 :
B
.
.
.
@ .. . . . . A B@
. . . x~(n) C
A75
n
(n)
(n)
y~n?1 y~0
0
0
0
The computation of x~(n) and y~(n) requires 4n2 + O(n) operations. If we want to
solve the system (1) then we need an additional O(n log2 n) operations to compute
Tn?1 b, because the product of a Toeplitz matrix times a vector can be computed via
a few FFTs [20], [21]. Hence, the solution of the system Tn x = b requires 4n2 +
O(n log2 n) operations.
In some applications the diagonal entries of Tn?1 are needed, but not the rest
of Tn?1 . From the representation of Tn?1 and from the relation x~0(n) = y~n(n) = 1
(cf. (2)), it is not dicult to derive an algorithm for computation of these entries.
Due to the persymmetry of Tn?1 , we only need to compute the rst dn=2e diagonal
entries, and the algorithm requires only d5n=2e ops, see Fig. 2.
The only place where the 4n2-algorithm can break down is in the computation of
the scaling factors pk , namely, when k k = 1. More seriously, as we show below, it
is dangerous when the quantity 1 ? k k becomes very small compared to pk?1. For
better understanding of this situation we derive another expression for 1 ? k k .
Theorem 2.1. In the absence of breakdown in the 4n2-algorithm, we have pk 6=
0; k = 0; : : :; n; and
(3)
1 ? k k = x(0k?1)k = pk?1k ;
k = 0; : : :; n;
where k is the Schur complement of Tk?1 in the matrix Tk .
Proof. The rst relation in (3) is evident from the way pk is computed. To prove
the second relation in (3), we write the block LU factorization of the matrix Tk in the
following way:
Tk =
T
k?1
kT
Ik
0
?k =
t0
kT Tk??11 1
4
T
k?1
0
?k ;
k
where
(4)
0t 1
?k
?k = B
@ ... CA ;
0 tk 1
k = @ ... A
t1
t?1
and k = t0 ? kT Tk??11?k is the Schur complement of Tk?1. Next, we need the last
component yk(k) of the vector y(k) . After some simple manipulations, we obtain that
the last component of the k +1 vector Tk y(k) equals k yk(k) . But this quantity is equal
to 1 because y(k) is the last column of Tk?1 . Using (2) and taking into account the
persymmetry of Tk?1 , i.e., x(0k) = yk(k) , we then obtain
k yk(k) = k x(0k) = k pk :
But we have that
pk = pk?1=(1 ? k k )
and, hence,
k pk?1=(1 ? k k ) = 1;
from which the theorem follows.
From the above theorem it follows that
(5)
pk = pk?1=(1 ? kk ) = 1=k :
Now from (5) we see that the algorithm can break down when k = 0. This is
equivalent to Tk being singular. Moreover, the quantity jk?1 j is closely connected to
kTk?1k2 and therefore to the condition number of Tk ; see [10, x6] for details. Therefore,
in practice, trouble sets in when jk j becomes small, which happens when j1 ? k k j
is small relative to jpk?1j. In the next section we propose a stabilized modication of
the algorithm that avoids small jk j.
3. The Stabilized Algorithm. One strategy for avoiding breakdown or nearbreakdown in the 4n2-algorithm was proposed by Huckle [15]. His strategy is to switch
back and forth between two dierent iteration schemes, of which the second is more
expensive and only used when the rst is inaccurate.
3.1. Derivation of Our Algorithm. Our strategy is dierent and simpler to
implement: we stay with one iteration scheme and instead perturb k by Sgn(k ) ,
when necessary, such that jk j increases. Here, is a small positive number whose
magnitude is discussed in x4.5, and Sgn is dened as
1; for 0;
Sgn(k ) = ?1; for k < 0:
k
This approach is inspired by similar perturbations used to \stabilize" Strassen's recursive matrix inversion algorithm in [3]. In order to analyze our approach, we express
the perturbation of k as a perturbation in k and k . Suppose that the perturbed
quantities are
^k = k + ;
^k = k + ;
k
5
k
and that !k = 1 ? k k . Then from the identity
(6)
1 ? ^k ^k = (k + Sgn(k ) )pk?1
we obtain
(7)
k + k + = ?Sgn(k )pk?1:
This equation is nonlinear with respect to and . It has an innite number of
solutions, and we prefer a simple solution. Therefore, we choose either
(8)
= 0
= ?Sgn(k )=gk ;
or
(9)
= ?Sgn(k )=fk :
= 0;
In this way we transfer the perturbation of k to a perturbation of k or k . Whether
to use (8) or (9) is explained below.
We shall now show that the perturbation of in (8) cannot change k dramatically (a similar analysis holds for k ). Notice that if k is zero or very small then a
perturbation is not necessary. Therefore we can assume that
(10)
jk j c > 0;
where c is a constant. From (6) we obtain
(11)
jp^k j = jpk?1=(1 ? ^ k ^k )j = j1=(k + Sgn(k ) j 1=:
k
k
k
k
k
k
k
k
k
k
k
Then it follows that
^k = k ? Sgn(k )=gk = k ? Sgn(k )pk?1=k
and therefore
jk j + jpk?1j=c jkj + 1=c:
We see that ^ k can only increase mildly, as long as jk j is not too small. This analysis
shows that we should always choose to perturb the smaller of jkj and jk j.
For the analysis in the next section we need to express the perturbation as a
perturbation of Tn . Assume again that k is perturbed (a \symmetric" analysis exists
for k ). The perturbation of k is achieved by perturbing the matrix element tk . To
derive the size of the perturbation tk , we equate ^ k with pk?1f^k , where
f^k = (tk + tk )~x(0k?1) +
to obtain
X
k?1
j =1
tk?1x~(jk?1)
k
= ? Sgn((k?k )1) = ? Sgn(k ) :
(
k
?
1)
k
pk?1x~0
pk?1x~0 gk
A perturbation of k corresponds to a perturbation of t?k by t?k = ?Sgn(k )=k .
tk =
6
Obviously, the perturbation of Tn at stage k of the algorithm is a Toeplitz matrix
with a single nonzero o-diagonal at position k below (or above) the diagonal with
elements tk (or t?k ), and the 1-norm of this matrix equals jtk j (or jt?kj). The
total perturbation matrix Tn of Tn , after completion of the algorithm and possibly
a number of intermediate perturbations, is the sum of the individual perturbation
matrices. From (10) we have
kTnk1 K=c:
Here, K n + 1 is the number of perturbation matrices; usually we have K = O(1),
and K = 1 in all our numerical examples.
If, instead of perturbing tk explicitly, we merely perturb k and keep using the
original Tn as input (which would make the implementation easier), then it can be
shown that we compute the inverse of a Toeplitz matrix T~n that diers from Tn in
all elements on and below the kth diagonal. This T~n can dier signicantly from Tn ,
and therefore it is not a useful approach.
We need extra storage for the perturbed matrix, because once an entry has been
perturbed it is this perturbed entry that is used in the remaining steps of the algorithm. In principle, we could save only the perturbed entries along with information
about which entries were perturbed, but that requires some bookkeeping which is
not easy to pipeline. Instead, we suggest a much simpler solution, namely, to use
an additional array of length 2n ? 1 to store the perturbed matrix, of which most of
the elements are identical to those of the unperturbed matrix. If iterative renement
is not needed, then the perturbed matrix can overwrite the original matrix, and no
additional storage is needed.
3.2. Summary of the Stabilized Algorithm. The stabilized algorithm {
without the renement step { is summarized in Fig. 3. Notice that the parameter acts both as a threshold for deciding when to add a small perturbation, and as
a parameter controlling the size of this perturbation. By adding the perturbation, we
avoid division by a small number, such that pk stays reasonably small. The choice of
the parameter is discussed in the next section.
In practice we do not want to perturb the data unnecessarily. It could happen
that after adding a perturbation at some step of the algorithm, then in some of the
next steps the quantity !k =pk?1 is slightly smaller than the threshold , and therefore
we should again add the same perturbation. To avoid such unnecessary perturbations,
we reduce the threshold by some factor once we have added a perturbation. In our
implementation, we use a factor 0.1 (which was also used by Chan and Hansen [6]
and found to work well in their algorithm). This approach also works well in our
algorithm, and in all our examples the perturbation is added only once.
3.3. Iterative Renement. Our perturbation(s) lead to an inaccurate solution
that can be made more accurate by taking a few steps of iterative renement [12,
x3.5.3], which is relatively cheap for Toeplitz systems. The output of our algorithm is
a representation of the matrix T^n?1 which approximates the inverse Tn?1, and during
the iterative renement we need to compute the products of Tn and T^n?1 times a
vector. This can be done relatively fast because such products can be computed in
O(n log2 n) operations using FFTs, see [1], [20, x4.2.4], [21].
The stabilization overhead is O(n) operations at most, if stabilization is used in
every step. In practice, even in the case of successive singular submatrices, only the
rst singular submatrix is usually perturbed, while the subsequent ones do not need
7
if (jt0j < )
t0 = t0 + Sgn(t0 ) ;
= 0:1 ;
end
x~(0) = y~(0) = 1; p0 = 1=t0;
for k = 1; : : :; n
fk = tk x~(0k?1) + + t1x~(kk??11);
gk = t?1 y~0(k?1) + + t?k y~k(k??11);
k = pk?1fk ; k = pk?1gk ;
!k = 1 ? k k ;
if (j!k =pk?1j < )
if (jk j < jk j )
k = k ? Sgn(k )=gk ;
tk = tk ? Sgn(k )=k ;
else
k = k ? Sgn(k )=fk ;
t?k = t?k ? Sgn(k )=k ;
end
!k = 1 ? k k ;
= 0:1 ;
end
pk = pk?1=!k ; 0 (k?1)
x
~
(
k
)
;
? k
x~ =
0 x~(k?1) y~(k?01) y~(k) = ?k
+ y~(k?1) ;
0
end
. The stabilized algorithm.
Fig. 3
8
to be perturbed. Thus, in most cases we need only O(1) operations for the stabilization. The operations count for the whole algorithm including iterative renement is
therefore 4n2 + O(n log2 n), where the constant in front of n log2 n depends on the
number of renement steps. Our experience is that 2 or 3 iterations are enough in
most of the cases.
Note that for symmetric Toeplitz matrices only the rst column is needed for
representation of the inverse, and the number of operations is halved. The stabilization part and the error analysis are analogous, and therefore we do not discuss the
symmetric case here.
4. Error Analysis. Error analysis of Toeplitz algorithms is quite rare in the
literature. The classical paper is Cybenko's analysis in [8] of Levinson's algorithm;
see also Bunch's paper [5]. Our error analysis is based on a method developed by
Voevodin and Yalamov [22], which uses the concept of equivalent perturbations for
each piece of data in the algorithm (input, intermediate, and output). We perform
a detailed analysis of one step of the algorithm, for a xed k, from which the overall
error in the computed solution can be determined.
4.1. Equivalent Perturbations in Step k. In the kth step of the algorithm,
we write the relations between the rounding errors and the equivalent perturbations
in the data in the following linear form, where all second-order term are neglected:
(12)
B k k = k :
Here Bk is a sparse matrix consisting of the Frechet derivatives of all the local operations in step k, k is a vector of all the equivalent perturbations in step k, and k is a
vector of all the corresponding rounding errors in step k. The details of this system
are given below.
Solving the underdetermined system (12), we obtain bounds for the equivalent
perturbations in step k. If the equivalent perturbations of all outputs are zero then
we have backward analysis; if the equivalent perturbations of all inputs are zero we
have forward analysis; and if some of the inputs and some of the outputs have nonzero
perturbations the analysis is mixed. In this paper we use mixed analysis to study the
roundo error propagation.
Our analysis is non-structured, i.e., the entries tk and t?k are considered as different entries at each step of the algorithm. Throughout our analysis we assume that
pk 6= 0 and 1 ? k k 6= 0 for k = 0; : : :; n. These assumptions are valid for our
stabilized algorithm because we always perturb the denominators 1 ? k k if they are
zero or very small.
The vector k consists of all the local rounding errors involved in computing the
scalars fk , gk , k , k , pk , and the vectors x~(k) and y~(k) :
kT = (fk ; gk ; k ; k ; pk ; xT~(k) ; yT~(k) ):
(13)
Both x~ and y~ are vectors of length k +1, and thus k is a vector of length 2k +7.
The vector k consists of the equivalent perturbations of the following data: k?1,
?(k?1), x~(k?1), y~(k?1), tk , t?k , fk , gk , pk?1, k , k , pk , x~(k), and y~(k) , where the
vectors k?1 and ?(k?1) are dened in (4). Thus, k has length 6k + 8 and has the
form
(k)
(14)
(k)
Tk = (k?1 ; ?(k?1) ; : : :; yT~(k) ):
9
Finally, the matrix Bk is very sparse, and we partition it as follows
0
k Ek Jk
(2k+7)(6k+8)
Bk = C0k D
Pk Qk 0 ?I2k+2 2 <
where I2k+2 is the identity matrix of order 2k + 2, and the other submatrices are
0 x~(k?1)
BB 1 0
Ck = B
B@ 0
0
0
x~(kk??11) 0
0 y~0(k?1)
0
0
0
0
0
0
0 tk
B0
Dk = B
B
@ 00
0 t1
0
0 t?1
0 0
0 0
0 0
01
BB 0
Ek = B
B@ 00
0 ?1 0
0
1 0 ?1
0
0 pk?1 0
fk
0 0 pk?1 gk
0 0 0
0 1?1 k k
0
B
Jk = B
B@
0 1
B
B
B
B
B
B 0
Pk = B
B
?k
B
B
B
B
B
@
0
00
0
B
B
0
B
Qk = B
B
@ 00
0
0
0
0
0
0 0
...
0
0
0
0
0
0
?k
1
0
?k
0
1
0
0
0
0
0
0
0
1
t?k C
0 C
CA 2 <52k
0
0
0
0
0
?1
0
pk?1 k
(1?kk )2
0
0
0
?1
pk?1 k
(1?kk )2
1
CC
CC 2 <57
A
01
0C
0C
CA 2 <51
0
?1
0
...
0 1
y~k(k??21) C
C 2 <52(k?1)
0 C
C
0 A
0
...
...
1
CC
CC
C
?k C
C 2 <(2k+2)2k
0 C
CC
CC
CA
0
1
0
0k?1)
0 1
(
0 ?y~0:k?2
0 C
0 ?1
0 C
C 2 <(k+2)7
0
0
?(k1?1) C
C
0
0
?x~1:k?1 A
0
0
0
10
The rst three block columns of Bk correspond to the input (k?1, ?(k?1), x~(k?1),
y~(k?1) , tk , t?k , pk?1) and the intermediate quantities (fk , gk , k , k ) of step k, and
the last two block columns correspond to the output (pk , x~(k), y~(k) ) of the step. We
have used the fact that x~(0k) = y~k(k) = 1, and the corresponding entries of Bk are
replaced by 1.
4.2. The Local Rounding Errors. Now we consider the local absolute rounding errors in the computation of each output. Let 0 denote the machine precision.
Then simple roundo analysis [14, p.69] of the computation of fk , gk , k , k , pk , x~(ik),
and y~i(k) gives (ignoring terms of higher order in 0 )
(15)
fk =
gk =
X
k?1
i=0
tk?i x~(ik?1)i ;
k
X
i=1
jij k0 ; i = 0; : : :; k ? 1
t?i y~i(?k?1 1)i ;
jij k0 ; i = 0; : : :; k ? 1
j1j 0 ; j2j 0
k = k 1 ; k = k 2;
pk =
(16)
pk?1
pk?1k k
1 ? k k 1 + (1 ? k k )2 2;
j1j 20 ; j2j 0
x~i(k) = x~(ik?1)1(i) + k y~i(?k?1 1) 3(i)
y~(k) = k x~(ik?1)4(i) + y~i(?k?1 1) 2(i)
i
j1(i)j 0 ; j2(i) j 0 ; j3(i) j 20; j4(i) j 20
9
>
=
>
;
i = 0; : : :; k
where the quantities
(k)
(k)
(0)
y~?(k1?1) = x~(kk?1) = (0)
1 = 2 = 3 = 4 = 0
are introduced to make the above notation more compact.
4.3. Simplication of the System. As already mentioned, the system in (12)
is underdetermined, and we can simplify the system by xing some of the equivalent
perturbations. In each step k we choose
? = ? ? = 0; = k 2 =2; = k 2=2:
With this simplication, the entries
C ? , ? ? , , and are removed from the
vector k , the block column 0k is removed from Bk , and the last two columns
of Ek are removed. Moreover, a few upper bounds are altered,
j1j 1:50; j2 j 1:50 ; j3(i)j 2:50; j4(i) j 2:50;
and the expression for p becomes
k 1
k
k
(k 1)
1k
(k 1)
k
k
pk?1
1 ? k k 1; j1j 20 :
It is this modied system that we use to derive our bounds.
pk =
11
k
4.4. Derivation of the Bounds. From (12) we see that if the errors x~(k?1) and
y~(k?1) are given, then we can compute x~(k) and y~(k) from the second block equation
as follows:
x~(k)
y~(k)
x~(k?1)
y~(k?1)
+ x~ :
y~
This means that we can do forward analysis for the pairs x~(k), y~(k) because their
equivalent perturbations are dened independently of the rest of the equivalent perturbations in (12). Applying (17) recursively we obtain
(17)
(18)
= Pk
(k)
(k)
0 i+1 1 X
k
Y A x~
x~
@
=
Pj
;
(k)
(i)
y~
i=1
(k)
y~
j =k
(i)
k = 1; : : :; n:
Here we assume that x~ = y~ = 0. Now we bound the local errors x~ and y~ .
From (16) we have
(0)
(i)
(0)
(i)
jx~(k?1)j j j x~
jy~ j jPkj jy~(k?1) j 2:50 :
(19)
(k)
(k)
From the updating equations for x~(k) and y~(k) (the last two statements in the algorithms), we nd that
(20)
x~(k?1) x~(k?2) = P P ;
=
P
k
?
1
k?1
1
(
k
?
1)
y~
y~(k?2)
where
(0)
0
= 11 :
P1 = x~(0)
y~0
(21)
Now (18), (19) and (20) imply that
(22)
+1 1 j j k iY
X
jP j Y P ;
x~
P
2
:
5
j
0
i j=i?1 j jy~ j
i=1 j =k (k)
(k)
and, hence,
(23)
kx~ k1 2:5Mkk0 ; ky~ k1 2:5Mk k0 ;
(k)
(k)
where we have dened
(24)
i+1 1 Y Y P Mk max Pj jPij 1ik j =k j=i?1 j 1
;
in which j j denotes elementwise absolute values.
Hence, we have bounded the equivalent perturbations of x~(k) and y~(k) by means
of forward analysis. As we shall see, Mk appears in the nal bounds. Unfortunately,
there is no simple way to analyze or bound Mk , and we shall therefore resort to
numerical experiments, cf. Eq. (33).
12
Having obtained the equivalent perturbations x~ ? , y~ ? from the previous
step, we obtain the equivalent perturbations p ? , t , and t? by means of backward
error analysis. After inverting Ek (which is possible because of our assumption that
pk?1 6= 0 and 1 ? k k 6= 0), we get
p ? = (1 ? k k )p + pk?11 = pk?1(p + 1);
where we have used the fact that pk = pk?1=(1 ? k k ); and p = p =pk is the
relative equivalent perturbation of pk . Hence,
j1j 20 ;
p ? = p + 1;
(k 1)
(k 1)
k 1
k
k
k
k
k 1
k
k
k
k 1
which when applied recursively to the previous steps gives
(25)
jp ? j 2(n ? k + 1)0:
Analogously, we obtain
(26)
jf j j1j + jp ? j (2n ? 2k + 3:5)0;
k 1
k 1
k
jg j j2j + jp ? j (2n ? 2k + 3:5)0:
k 1
k
Finally, from (15), (23) and (26) we have
jt j jf j + jf j +
k
k
k
k
X
k?1
X
k?1
i=1
jtk?i(xk?1)j
i
jtk?ix~(ik?1)j0 + (2n ? 2k + 3:5)
k?1
X
jtk?ix~(ik)j0 +
i=0
(2n + 3:5)kx~(k?1)k1 kTk k1 0 + 2:5Mk kTk k1 0 :
i=0
X
k?1
i=1
jtk?ix
?
(k 1)
i
j
Here, we need to bound kx~(k?1)k1 . From (20) we obtain
x~(k?1) y~(k?1) = kPk?1 P1k1
1
and it follows from the denition of Mk in (24) that Mk is an upper bound for the
above norm and therefore also for kx~(k?1)k1 . Hence, we get
(27)
jt j 2(n + 3)Mk kTk k1 0 ;
Analogously, we obtain
(28)
jt? j 2(n + 3)Mk kTk k1 0 :
It remains to analyze the rst step of the algorithm, i.e., p0 = 1=t0. For this
operation we have t =t20 ? p = p in the absolute case, and ?t ? p = , j j 0
in the relative case. Then from (25) we get
(29)
jt j (2n + 1)0 ;
showing that the perturbation of t0 is very small.
k
k
0
0
0
0
0
13
0
If Mk < 1 for all k, it is clear that all the bounds are small. Now we shall suppose
that Mk 1; which is the interesting case. From (27), (28) and (29) we obtain the
following bound:
Mk ;
kT k1 (n2 + 5n)M kTn k10 ; M max
k
n
which is a norm-wise bound.
We have thus bounded the forward error of x~(n) and y~(n) (see (23)). Note that
the solution x of the system Tn x = b is computed from x~(n) , y~(n) , pn and b via FFTs.
It is well-known that the FFT is very stable, and therefore we do not analyze these
transformations here but instead use the following bound:
kxk1 cn kTn?1k1 kbk1 0 ;
where cn is polynomial in n.
4.5. Main Result and Discussion. We summarize our analysis as follows.
Theorem 4.1. For the 4n2-algorithm we have
(Tn + T )(^x + x ) = b;
where x^ is the computed solution of Tnx = b, and
kT k1 2(n2 + 3n) M kTnk1 0 ;
n
n
kx k1 cn kTn?1 k1 kbk1 0 ;
M = max Mk :
k
Here, Mk is dened in (24), and 0 is the machine precision.
Theorem 4.1 presents a mixed error analysis for both the original and the stabilized algorithms. Now let us consider the stabilized algorithm in more details. By
perturbing some of the entries tk or t?k , we change the inputs as noted in Section 3.
Instead of solving a system with the matrix Tn , we solve a system with the matrix
T^n = Tn + Tn, where
(30)
For this system we have
(31)
kTn k1 (K=c) :
(T^n + T^ )(^x + x ) = b;
n
where T and x are bounded in Theorem 4.1, with Tn replaced by T^n .
Let x = x^ ? x; where x is the exact solution of the unperturbed problem
Tn x = b. Then after some standard manipulation, and ignoring higher order terms,
we get from (31):
x = ?Tn?1(Tn + T^ )x ? x :
Provided that kTn k1 1, and kTn + Tnk1 2 kTn k1 (which is true for our choice
of ), Theorem 4.1 and (30) then lead to the bound
kxk1 (K=c) kTn?1k1 kxk1 + 4(n2 + 3n)M kTn?1k1 kTnk1 kxk10
+ cn kTn?1k1 kbk1 0
c0n kTn?1k1 kTn k1 kxk1( + M0 );
n
n
14
where c0n is a new polynomial in n. The relative error is bounded as
kxk1 c0 ( + M ); = kT ?1 k kT k :
(32)
kxk1
0
n
n
1 n 1
The last bound depends on M which, in turn, depends on the growth of the k and k ,
and thus on the growth of the scaling factors pk . Since all pk are bounded by (11) we
use a model of the form
(33)
M ?s ;
where s is a parameter. It is very dicult to nd a bound on s. From practical
experience, cf. the next section, we nd that s usually lies in the range 1 s 2.
Now from (32) and (33) we can derive the recommended value of . The right-hand
side of (32) is minimal when
= (s0 )1=(s+1);
and if 1 s 2 then
03 02 :
1
1
For example, if 0 10?16 (double precision) then 10?5 10?8. In all the
experiments we used the value = 10?6 which turns out to be suitable.
This choice of assures us that the computed solution of the perturbed problem is
not far away from the exact solution of the unperturbed problem for well-conditioned
matrices Tn (because the bound (32) depends on the condition number ). Hence,
the computed solution x^ is a good starting vector for the iterative procedure which
converges very fast (usually in two or three iterations).
5. Numerical Experiments. The numerical experiments are done in Matlab,
where the roundo unit is 0 2:22 10?16, and all matrix-vector multiplications
are performed with Matlab's FFT routine. The exact solution in all the examples
is x = (1; : : :; 1)T , and we measure the forward error kx ? x^k1 , where x^ is the
solution computed with rounding errors. We use the standard iterative renement
(IR) procedure from [12, x3.5.3] with stopping criteria kTnx^ ? bk1 =kbk1 10000
and Nit = 100, where Nit is the number of IR steps.
Table 1 presents the results for \random" test matrices TnCH . For each row of this
table we generated 1000 random matrices of order n = 500, in such a way that at least
one leading principal submatrix is ill conditioned. The order k of this ill-conditioned
submatrix is chosen randomly in the range 1 k 250. This is the strategy from [6].
The conditioning of the submatrix is controlled by a real parameter , which is given
in the rst row of Table 1. In order to generate these test matrices, we rst generate
a random nonsymmetric Toeplitz matrix Tn with positive entries. Let denote the
numerically smallest real eigenvalue of the leading principal matrix Tk of Tn (due to
Perron's theorem [18], a positive matrix always has at least one real eigenvalue, which
is identical to the spectral radius of the matrix). Then we set our test matrix to
TnCH = Tn ? ( ? )In ;
and we know that the leading principal submatrix TkCH has an eigenvalue equal to .
Since the numerically smallest eigenvalue is bounded below by the smallest singular
15
value [12, x7.1.6], we are guaranteed that the smallest singular value of TkCH is bounded
by , and in this way we can control the conditioning of TkCH by means of .
Table 1 presents average and worst-case errors for the two versions of the algorithm
with and without iterative renement. It is clear that the stabilized algorithm can
produce much more correct solutions than the original algorithm. The average number
of IR steps is small, and almost independent of . For = 0 the original algorithm
breaks down in 12 out of 1000 cases. Therefore, the results for the original algorithm
in the last column of Table 1 take into account only the remaining cases where the
algorithm did not break down.
Next, we applied the algorithms to three symmetric test matrices. The rst is
the shifted Kac-Murdock-Szego matrix TnKMS from [19] with entries given by
t0 = 10?14; ti = t?i = (1=2)i?1; i = 1; : : :; n;
for which every third leading principal submatrix TkKMS , k = 0; 3; 6; : : : is very illconditioned. The results are given in Table 2. After IR, the stabilized algorithm
produces results which are as accurate as can be expected, and the number of IR
steps is constant. In contrast to this, the solutions computed by the original algorithm
with IR are less accurate, and the number of IR steps increases with n. For large n,
exemplied by the case n = 1200 in Table 2, the condition number and the initial error
are so large that IR diverges. We conclude that the perturbation in our algorithm is
essential for the stability and the robustness of the algorithm.
F is taken from [9]. The entries ti = t?i ,
The second symmetric test matrix T311
F
i = 0; : : :; 311 of T311 are given as the coecients of the Taylor expansion
f (z ) =
1
X
i=0
of the function
ti z i
1
3
6
24
48
96
1 ? z ? 1 ? z 3 + 1 ? z 6 ? 1 ? z 24 + 1 ? z 48 ? 1 ? z 96 :
All leading principal submatrices are well-conditioned with the exception of the submatrices TkF , k = 50; : : :; 56 which are exactly singular. The forward error kx ? x^k1
is 0.82 for the original algorithm before IR (it does not break down in the presence
of roundo errors), and the IR process diverges. The forward error for the stabilized
algorithm is 3:14 10?7 before IR and 5:24 10?14 after 2 IR steps.
The third symmetric test matrix is the tridiagonal matrix Tnt = tridiag(1; 10?13; 1)
which is well-conditioned for odd n and almost singular for even n, i.e., each second
principal submatrix is almost singular. The results are given in Table 3. The original algorithm is not capable of solving this system, while the stabilized algorithm
computes solutions to almost full precision after 3 IR steps.
These numerical tests demonstrate that our stabilized algorithm is always capable
of computing a solution to a satisfactory accuracy. In most case, 2 or 3 steps of iterative renement are enough, and only one perturbation is needed in all the examples
independently of the number of singular or ill-conditioned submatrices.
Finally, in Table 4 we give some numerical evidence that the range of s is probably
1 s 2. The rst row shows dierent choices of , and the other rows show the
number of IR steps for three test matrices. We see that the minimum number of
iterative steps is achieved for in the range 10?7 10?5. This means that f (z ) = 18 +
16
Table 1
The forward error kx ? x^k1 and the number of iterations for the \random" matrices TnCH.
Original Average
before IR Worst
Original Average
after IR Worst
Nit
Average
Worst
Stabilized Average
before IR Worst
Stabilized Average
after IR Worst
Nit
Average
Worst
10?7
1:26 10?5
2:05 10?3
2:00 10?12
5:66 10?10
2.12
3
1:80 10?5
2:05 10?3
1:81 10?12
5:66 10?10
2.21
3
10?10
1:34 10?2
9:79 10?1
1:79 10?11
1:62 10?8
3.61
10
1:43 10?4
1:56 10?2
1:92 10?12
5:05 10?10
2.60
5
10?13
6:11
8:38 102
1:06 104
7:21 106
14:0
100
1:77 10?4
3:02 10?2
1:92 10?12
5:08 10?10
2.59
5
0
8:00 101
1:70 104
4:44 104
2:09 107
13:9
100
1:71 10?4
5:18 10?2
1:85 10?12
6:18 10?10
2.60
5
Table 2
The forward error kx ? x^k1 and the number of iterations for the KMS matrices.
n
15
30
60
120
1200
Original
before IR
2:39 10?3
2:82 10?3
2:85 10?3
2:92 10?3
7:94 10?3
Original Nit Stabilized
after IR
before IR
1:55 10?15 5 1:50 10?6
6:77 10?14 6 2:00 10?6
2:35 10?13 7 2:00 10?6
3:18 10?13 9 2:00 10?6
divergence { 2:00 10?6
Stabilized Nit Condition
after IR
number
2:22 10?15 3 2:55 101
2:66 10?15 3 5:13 101
3:77 10?15 3 1:01 102
1:63 10?14 3 2:01 102
5:95 10?14 3 1:98 103
probably should be chosen in the this range, which is equivalent to s being in the
range 1 s 2. We did not nd examples in which is outside the above mentioned
range. We thus recommend to use the value = 10?6.
Acknowledgements. We wish to thank Thomas Huckle for sharing his insights
about Toeplitz algorithms, as well as his Matlab routines, with us.
REFERENCES
[1] G. S. Ammar & W. B. Gragg, Superfast solution of real positive denite Toeplitz systems,
SIAM J. Matrix Anal. Appl. 9(1988), pp. 61{76.
Table 3
The forward error kx ? x^k1 and the number of iterations for the matrix tridiag(1; 10?3 ; 1).
Original Original Nit
before IR after IR
100
2.27
divergence {
200
2.27
divergence {
400
2.27
divergence {
n
17
Stabilized Stabilized Nit
before IR after IR
8:10 10?5 6:43 10?15 3
7:90 10?5 9:99 10?15 3
7:87 10?5 3:14 10?14 3
Table 4
Number of iterative renement steps for dierent choices of .
10?3 10?4 10?5 10?6 10?7
KMS, n = 120
10
5
3
3
2
tridiag(1,0,1), n = 400 10
5
3
3
7
T311
4
3
3
2
3
[2] O. B. Arushanian, M. K. Samarin. V. V. Voevodin, E. E. Tyrtyshnikov, B. S. Garbow,
J. M. Boyle, W. R. Cowell & K. W. Dritz, The Toeplitz Package users' guide, Report
ANL-83-16, Argonne National Laboratory. October 1983.
[3] S. M. Balle, P. C. Hansen & N. J. Higham, A Strassen-type matrix inversion algorithm for
the Connection Machine, APPARC PaA2 Deliverable, ESPRIT BRA III Contract # 6634;
Report UNIC-93-11, October 1993 (29 pages).
[4] R. Brent, Parallel algorithms for Toeplitz systems, in: Numerical Linear Algebra, Digital Signal
Processing and Parallel Algorithms, Eds. G. H. Golub and P. Van Dooren, Springer, Berlin,
1991, pp. 75{92.
[5] J. R. Bunch, Stability of methods for solving Toeplitz systems of equations, SIAM J. Sci. Stat.
Comput., 6(1985), pp. 349{364.
[6] T. F. Chan, P. C. Hansen, A look-ahead Levinson algorithm for indenite Toeplitz systems,
SIAM J. Matrix Anal. Appl., 13(1992), pp. 490{506.
[7] P. Concus & P. Saylor, A modied direct preconditioner for indenite symmetric Toeplitz
systems, Num. Lin. Alg. Appl., 2(1995), pp. 415{429.
[8] G. Cybenko, The numerical stability of the Levinson-Durbin algorithm for Toeplitz systems of
equations, SIAM J. Sci. Stat. Comput., 1(1980), pp. 303{319.
[9] R. W. Freund, A look-ahead Bareiss algorithm for general Toeplitz matrices, Numer. Math.
68(1994), pp. 35{69.
[10] R. W. Freund, H. Zha, Formally biorthogonal polynomials and a look-aheadLevinson algorithm
for general Toeplitz matrices, Lin. Alg. Appl. 188(1993), pp. 255{303.
[11] I. Gohberg, A. Semencul, On the inversion of nite Toeplitz matrices and their continuous
analogs, Mat. Issled., 2(1972), pp. 201{233.
[12] G. H. Golub, C. F. Van Loan, Matrix Computations, 3nd Ed., The Johns Hopkins University
Press, Baltimore, 1996.
[13] G. Heinig, F. Rost, Algebraic Methods for Toeplitz and Toeplitz-Like Operators, Birkhauser,
1984.
[14] N. J. Higham, Accuracy and Wtability of Numerical Algorithms, SIAM, 1996.
[15] T. Huckle, Computations with Gohberg-Semencul formulas for Toeplitz Matrices, Report, Institut fur Informatik, TU Munchen, Germany, December 1996. Submitted to Lin. Alg.
Appl.
[16] J. R. Jain, An ecient algorithm for a large Toeplitz set of linear equations, IEEE Trans.
Acoust., Sppech, Signal Proc., ASSP-27(1979), pp. 612{615.
[17] N. Levinson, The Wiener rms (root-mean-square) error criterion in lter design and prediction,
J. Math. Phys., 25(1947), pp. 261{278.
[18] J. M. Ortega, Matrix Theory, Plenum, New York, 1987.
[19] W. F. Trench, Numerical solution of the eigenvalue problem for Hermitian Toeplitz matrices,
SIAM J. Matrix Anal. Appl., 10(1989), pp. 135{146.
[20] C. F. Van Loan, Computational Frameworks for the Fast Fourier Transform, SIAM, Philadelphia, 1992.
[21] V. V. Voevodin, E. E. Tyrtyshnikov, Computational Processes with Toeplitz Matrices, Nauka,
Moscow, 1987. (in Russian)
[22] V. V. Voevodin, P. Y. Yalamov, A new method of roundo error estimation, Parallel and
Distributed Processing, (K. Boyanov, Ed.), Elsevier, Amsterdam, 1990, pp. 315{333.
[23] J. H. Wilkinson, The Algebraic Eigenvalue Problem, Clarendon Press, Oxford, 1965.
18