to denote set difference : A \ B = { x E A I x e B 1. some function in O(f (n)) . there is one case that must be treated cautiously. but this is of no consequence (see Problem 2. If the symbol "-" were used to denote the differ- ence of two sets. to belong to O(f (n)) x O(f (n)).1. where " exp " denotes the binary exponentiation operator and " Id 2 " is the constant function 1d2(n)=2 for all n . including pointwise subtraction of functions. Similarly. O(g(n))) . Although they both denote the same set as 0([f this requires a proof (Problem 2. we stretch the notation by writing X op g to denote X op { g } .19(i)). Prove the following identities : i. if N is the set of vertices of a directed graph. On the other hand. it suffices for g(n) to be the pointwise product of two possibly different functions. +O(f (n)). for example ? To avoid this ambiguity. The hidden constants that multiply f (n) and g(n) may well be different. a function g(n) must be the pointwise square of (n)]2). This notation occasionally conflicts with the usual mathematical conventions. we use X op a to denote X op Ida. Intuitively this represents the order of the time taken by an algorithm composed of a first stage taking a time in the order of f (n) followed by a second stage taking a time in the order of g(n). Let f and g be arbitrary functions from IN into IR*.19(ii)). Still. We also use the symmetrical notation g op X and a op X. Notice the subtle difference between [O(f (n))]2 and O(f (n)) x O(f (n)). 2 obtained by adding pointwise any function in 0 (f (n)) to any function in 0 (g(n)). If g is a function from IN into IR*. then "X op Y " denotes { t :IN IR* I (3f EX )(3gEY)(3n0EIN)(b'n>no)[t(n) = f(n) op g(n)] }. if X and Y are sets of functions from IN into IR* and if op is any binary operator.19. however. For instance.O (n 2) mean. .44 Analysing the Efficiency of Algorithms Chap. Furthermore. n x O(f (n)) denotes { t :IN -4 JR I (3g(n)EO(f(n)))(3noEIN)(Vn ?no) [t(n) = n xg(n)] }.

Example 2. The notation 0 (f (n)) defined previously is thus equivalent to 0 (f (n) I P(n)) where P(n) is the predicate whose value is always true.X)[te0(f(n))] }. f(n) E Hi=o0(1) Another kind of operation on the asymptotic notation allows it to be nested.Sec.1. the natural way to express the execution time required by Dixon's integer factorization algorithm (Section 8. O([f(n)]2) _ [0(f(n))]2=O(.1. 2. as is the notation with several parameters. [I +0(1)]n = 2°(n). iii.12. Conditional asymptotic notation handles this situation.5 Conditional Asymptotic Notation Many algorithms are easier to analyse if initially we only consider instances whose size satisfies a certain condition. A function f : IN -* R* is . 2. 12(X) and O(X) are defined similarly. such as being a power of 2.1. Let f : IN -4 R* be any function and let P : IN -* 113 be a predicate. which we read as the order of f (n) when P(n). is the set of all functions t(n) bounded above by a real positive multiple of f (n) whenever n is sufficiently large and provided the condition P(n) holds. In other words. We define O(f(n) I P(n)) t :lN -4 R* I (3cER+)(3noeIN)(Vn ? no) [P(n) = t(n) <_ cf (n)] }. Let X be a set of functions from IN into R*.3) is 0(eo(mi )) where n is the value of the integer to be factorized. but [0(1)]n # 2°(") . The principal reason for using this conditional notation is that it can generally be eliminated once it has been used to make the analysis of an algorithm easier. The notation S2(f (n) I P(n)) and O(f (n) I P(n)) is defined similarly. and iv. 0 (f (n) I P(n)).5. You probably used this idea for solving Problem 2.1.f(n))xO(f(n)).R* I (3f (=. Now 0 (X) denotes LJ0(f(n)) = { t :IN . Although this expression can be simplified. possibly defined by some asymptotic notation.1 Asymptotic Notation 45 H.

Such a function is b-smooth if. where a and b are arbitrary real positive constants. Give two specific examples to illustrate that the conditions "t(n) is eventually nondecreasing" and "f (bn ) E O (f (n))" are both necessary to obtain these results.3. and let t : IN --> IR* be an eventually nondecreasing function such that t(n) EX (f (n) I n is a power of b).6. in particular Problem 2. However. <_ t (L(n + l )/2]) and t ([n / 21) <_ t (((n + 1)12 ). By the induction particular. Let b ? 2 be any integer. as well as being eventually nondecreasing. * Problem 2. The proof that (Vn >_ 1) [t(n) <_t(n + 1)] is by mathematical induction.20. One might be tempted to claim that t(n) is eventually nondecreasing because such is obviously the case with n log n . we shall therefore in future simply refer to such functions as being smooth.2. This argumentation is irrelevant and fallacious because the relation between t(n) and n log n has only been demonstrated thus far when n is a power of 2. if we only consider the cases when n is a power of 2. First. which implies by mathematical induction that (3 n0 E l) (`d n >_ n0) (V m >_ n) [f (n) <_ f (m)]. S2. Example 2. or e. note that t (1) = a <_ 2(a +b) = t (2). the equation becomes t (n) Ja ifn=l 2t (n / 2) + bn if n > I is a power of 2.1. In order to apply the result of the previous problem to conclude that t(n) E O(n log n). prove that t (n) is also smooth.4. where X stands for one of 0.Analysing the Efficiency of Algorithms 46 Chap. Prove that t(n) E X (f (n)). The presence of floors and ceilings makes this equation hard to analyse exactly. if 1(n) E O(f (n)). Let t (n) be defined by the following equation : t(n) _ Ja ifn = 1 t(Ln/2J)+t([n/21)+bn otherwise. The proof that t(n) is nondecreasing must use its recursive definition. . 2 eventually nondecreasing if (3 n 0 E IN) ('d n > n 0) [f (n) <_ f (n + 1)]. It turns out that any function that is b-smooth for some integer b >_ 2 is also c-smooth for every integer c ? 2 (prove it!). allow us to infer immediately that t (n) E O(n log n { n is a power of 2).3. The following problem assembles these ideas. we need only show that t(n) is an eventually nondecreasing function and that n log n is smooth. t (Ln / 2j) t(n)=t(Ln/2j)+t([n/2])+bn St(L(n+1)12j)+t([(n+l)/21)+b(n+l)=t(n+l). Let b ? 2 be any integer. assume that (V m < n) [t(m) <_ t (m + 1)]. A word of caution is important here. it satisfies the condition f (bn ) E O (f (n)). Furthermore. Let n be greater than hypothesis. let f : IN -4 IR* be a smooth func- tion. 0 We illustrate this principle using an example suggested by the algorithm for merge sorting given in Section 4. The techniques discussed in Section 2. Therefore In 1.1.

to generalize our results automatically to the case where n is an arbitrary integer. n log n is smooth because it is clearly eventually nondecreasing and 2n log(2n) = 2n (log2 + log n) = (21og2)n + 2n log n E O (n + n log n) = O (max(n. we do not always find ourselves faced with equations as precise as those in Example 2. since they do not allow us to conclude that t(n) is eventually nondecreasing. Our asymptotic notation allows these constraints to be expressed succinctly as t(n) E t(Ln/2])+t([n121)+0(n). 2. it is convenient to convert them first to equalities. We immediately conclude that t(n) E O(f (n)) = O(n log n).1. let u = max(c. This could not have been done directly with the original inequalities.Sec.1. It is easy to prove by mathematical induction that v <_ t(n)/f(n) <_ u for every integer n. and for appropriate initial functions t 1.2 in the preceding section. which in turn prevents us from applying Problem 2. however. using the conditional asymptotic notation and the technique explained in Problem 2. max I t I (n) I n Sn0}) and v =min(d. Obviously it saves having to prove independently both t (n) c O (n log n) and t (n) E Q (n log n). and simultaneously t(n) for some constants c. More often we have to deal with inequalities such as t(n) < tI(n) ifn :5 no t (Ln / 2]) + t (Fn / 21) + cvi otherwise t2(n) ifn <_n0 t (Ln / 2]) + t ([n / 21) + do otherwise. n log n)) = O (n log n). Coming back now to the function t(n) satisfying the preceding inequalities. To this end. We saw in the previous section that f (n) E O(n log n). To solve such inequalities.20.20. d E 1R+. t2: IN -f IR+. More importantly.6 Asymptotic Recurrences When analysing algorithms.1.1. This change from the original inequalities to a parametrized equation is useful from two points of view. .min {t7(n)/f(n) I n Sn0}). It is then possible. 2. it allows us to confine our analysis in the initial stages to the easier case where n is a power of 2.1 Asymptotic Notation 47 Finally. n 0 E IN. define f : IN -* IR by f(n)= 1 ifn = 1 f (Ln / 2]) + f ([n / 21) + n otherwise.

Supposing that HI (n . and c by constructing three linear equations using the values of HI (n) forn =0. the value of c being as yet unconstrained. Let the function f : IN -> IN be defined by the following recurrence : 0 . but that you are looking for some such formula. b. We begin with a simple example. However. we conclude that c = 0 and that f (n) = n 2/ 2 + n / 2 is true for every integer n . Obviously f (n) = YJ%o i <_ J70n = n 2. more complete hypothesis.3. . In the case of the preceding example it would have been simpler to determine the values of a. 1. The technique of constructive induction consists of trying to prove this incomplete hypothesis by mathematical induction.2.4 and 2.Analysing the Efficiency of Algorithms 48 Chap.b + c = c. Along the way we hope to pick up sufficient information about the constants to determine their values. we obtain two nontrivial equations for our three unknowns : 1 + b . 2 2. We have just shown that if HI (n -1) is true for some n ? 1.f (n) = ifn =0 n + f (n . We now have therefore a new. which we continue to call HI (n) : f (n) = n 2/ 2 + n / 2 + c. Knowing that f (O) = 0.1) is true for some n ? 1. Pretend for a moment that you do not know that f (n) = n (n + 1)/ 2. we can simultaneously prove the truth of a partially specified assertion and discover the missing specifications thanks to which the assertion is correct. Example 2. By applying this technique. It remains to establish the truth of HI (0) in order to conclude by mathematical induction that HI (n) is true for every integer n. their origin remains mysterious. If we are to conclude HI (n). By equating the coefficients of each power of n.1. This hypothesis is partial in the sense that a. then so is HI (n). and c are not yet known. and 2. Now HI (0) says precisely that f (O) = a 02 + b 0 + c = c. this technique of constructive induction is especially useful for solving certain recurrences that occur in the context of the analysis of algorithms. This suggests that we formulate a hypothesis that f (n) might be a quadratic polynomial. Too often it is employed to prove assertions that seem to have been produced from nowhere like a rabbit out of a hat. While the truth of these assertions is thus established.1.2a = b and a . but also its exact statement.5. From these it follows that a = b = '-f. mathematical induction is a tool sufficiently powerful to allow us to discover not merely the proof of a theorem. As we shall see in Examples 2. We therefore try the partially specified induction hypothesis HI (n) according to which f (n) = an 2 + bn + c.1) otherwise. and so f (n) E O (n 2 ). it must be the case that f (n) = an 2 + bn + c.7 Constructive Induction Mathematical induction is used primarily as a proof technique.1. b. It is clear that f (n) = Yi"_o i. we know that f(n)=n +f(n-1)=n +a(n-1)2+b(n-1)+c =an2+(1+b-2a)n +(a-b+c).

1.u (n -1)! for some n > 1. b = Z. this is the same as saying that u <. and thus that t (n) E U(M). However. and c = 0. it suffices to show that this is true for n = 1. It seems then that constructive induction has nothing to offer in this context.W.1) otherwise. As usual.v(n . To establish this.1. In this setting constructive induction can be exploited to the hilt. Thus we see that t(n) ? un! is always true. Let the function t : IN+ -> IR+ be given by the recurrence t(n) _ (a ifn=1 jl bn z + nt(n . Suppose by the partially specified induction hypothesis that t(n -1) <. 2.1 Asymptotic Notation 49 c=0 a+b+c=1 4a +2b +c=3 Solving this system gives us immediately a = 2.bn2 + nu (n -1)! = bn2 + un! >. or perhaps even that the hypothesis to the effect that t(n) EO (n!) is false. For simplicity. since we are only interested in establishing an upper bound on the quantity of interest. By definition of t(n). we know that t(n) = bn2 + nt(n -1) >. Encouraged by this success. However our aim is to show that t(n) :. Unfortunately no positive value of v allows us to conclude that t(n) <. an exact solution of the recurrence equations is generally unnecessary. However. that is. regardless of the value of u. Even the techniques we shall see in Section 2.3 will prove insufficient on occasion. The technique of constructive induction is useful in both cases. we begin by proving that t(n)ES2(n!).Sec.bn2 + vn!. Example 2.1)! for some n > 1.3. we have established that t(n) ? an! for every positive integer n . it is sufficiently similar to the recurrence that characterizes the factorial (n! = n x(n -1)!) that it is natural to conjecture that t (n) E O(n!).vn!. where a and b are arbitrary real positive constants. this allows us to affirm that t(n) = bn2 + nt(n -1) <. .1) ? u (n . Since t (l) = a. since nothing allows us to assert a priori that f (n) is in fact given by a quadratic polynomial. in the context of asymptotic notation. we shall prove independently that t (n) E O (n!) and that t(n) E S2(n!). t (l) >. provided that t(n . Although this equation is not easy to solve exactly.bn2 + vn!.1)!. Thus once the constants are determined we must in any case follow this with a proof by mathematical induction.a. Taking u = a. In order to conclude that t(n) >. using this approach does not prove that f (n) = n 2/ 2 + n / 2. we now try to show that t (n) E O (n!) by proving the existence of a real positive constant v such that t(n) 5 vn! for every positive integer n.un! for every positive integer n. Suppose by the partially specified induction hypothesis that t(n -1) >. Some recurrences are more difficult to solve than the one given in Example 2. that! given that t(n) :.un! for every positive integer n.4. there exists a real positive constant u such that t (n) >.

3. by straightforward mathematical induction.22. we know that t(l) = a.a + 5b. we conclude that t (n) = bn 2 + nt (n -1) <. Since n /(n -2):5 3 for every n >. All that remains is to adjust the constant v to take care of the cases n <. since t(n) <.wn is a stronger statement than t(n) <.1. Let IR+ be the function defined by the recurrence . When n = 2.3b. When n = 1. Using the definition of t(n).vn!. on the grounds that if the statement to be proved is stronger. Problem! .3bn for every positive integer n. which we were unable to prove.1. In particular. you may wish to prove this assertion. it is necessary and sufficient that (b -w )n + w <.6b . To conclude that t(n) <.Analysing the Efficiency of Algorithms 50! . If you got lost in the preceding argument.c <. we may choose v = a + 5b. solve exactly the recurrence * Problem 2. defining t(n). it is necessary and sufficient that v >. If we are to conclude that t(n) <.a + 5b.3 and w > bn /(n -2). then so too is the induction hypothesis it allows us to use.wn is a consequence of the hypothesis t(n -1) <.vn! .vn!. If we are to conclude that t(n) <.w (n -1) for some n > 1.2.v .2v . provided that n >. however.4. Rather than trying to prove directly that t(n) <. independently of the value of v. we may in particular choose w = 3b to ensure that t(n) <.vn! . Determine in terms of a and b the real positive constant c such that t(n) lim n-*= n! = c.v (n . We may hope for 2 + n (v (n -1)! . This idea may seem odd. thanks to which we were able to prove that t(n) E O(n!) without ever finding an exact expression for t(n). b E IR+ be arbitrary constants.21. it is necessary and sufficient that v >. it illustrates well the advantage obtained by using constructive induction.a + 3b.-w. Verify that a <. Suppose then by the partially specified induction hypothesis that t(n -1) < v (n -1)! . To complete Example 2.1)! .w (n -1)) = vn! + ((b -w )n + w )n. 2 In fact. This inequality holds if and only if n >. we use constructive induction to determine real positive constants v and w such that t(n) <.w (n -1).wn for any positive integer n.3. which is now completely specified. The conclusion from all this is that t(n) E 9(n!) since an! S t(n) S (a + 5b)n! . we can apply the recurrence definition of t(n) to find t(2) = 4b + 2t(1) = 4b + 2a . which is stronger than the previous condition. 11 The following problem is not so easy . g : IN+ Let k E IN and a. it is possible to obtain the result we hoped for.1.

Use of such "one-way equalities" (for one would not write O (n3) = n2) is hard to defend except on historical grounds. Unfortunately. Furthermore. We often find 0(. You may encounter three major differences in other books. 2. with this definition. On the other hand. This corresponds more closely to our intuitive idea of what a lower bound on the performance of an algorithm should look like.1 Asymptotic Notation g(n) = 51 a ifn=1 bnk + ng (n .IR 1 (3cclR+)(2noEIN)(Vn ? n0)[It(n)I <_ cf(n)] 1 where It(n)I denotes (here only) the absolute value of t(n). Problem 2. 11 .Sec. The second difference is less striking but more important. The most striking is the widespread use of statements such as n 2 = O (n 3) where we would write n 2 E O (n 3).1. The third difference concerns the definition of 0. since it can lead an incautious reader astray. Some authors define Q(f(n)) = { t:IN -> IR* I (3ce1R+)(VnoeIN)(3n ?no) [t(n)?cf(n)] I. Prove that g(n) e O(n!). 1 2. Notice the quantifier reversal.13 is neatly avoided. the asymmetry between 0 and Q noted after Problem 2.f(n)) _ { t :IN .1. g(n))) is no longer true. one would write n 3 .23. the meaning of "such-and-such an algorithm takes a time in 0 (n 2)" does not change since algorithms cannot take negative time. the notation becomes difficult to handle. With this definition we say that the execution time of some algorithm is of the order of f (n) (or is 0 (f (n))) rather than saying it is in the order of f (n). With this definition it suffices that there exist an infinite number of instances x that force some algorithm to take at least cf ( I x I) steps in order to conclude that this algorithm takes a time in Q (f (n)).1) otherwise. and because it makes O asymmetric. Why ? When we want the equivalent of this definition of 0 (f (n)) we write ±O (f (n)).8 For Further Reading The notation used in this chapter is not universally accepted. Of course. Using this definition. a statement such as O(f (n)) + O(g(n)) = O(max(f (n). in particular because Q thus defined is not transitive.n 2 E n 3 + O (n 2).1.

One trip round the outer loop is therefore bounded above by c +b + a (n-i ). the complete algorithm takes a time not greater than d + 7-11 [c + b + a (n -i )]. However. Example 2.4. there are cases where it is necessary to take account of the implicit control of the loops.1.2 ANALYSIS OF ALGORITHMS There is no magic formula for analysing the efficiency of an algorithm.3.0 for i . for a fourth constant d. 2 2. Here are some examples. 0 In this first example we gave all the details of our argument. where b is a second constant introduced to take account of the time spent initializing the loop.T[i] <. as is the case with selection sort. When an algorithm includes several nested loops. from which we conclude that the algorithm takes a time in 0(n2). Example 2.2. A similar analysis for the lower bound shows that in fact it takes a time in O(n 2). It is largely a question of judgement. one possible barometer is the test in the inner loop. however. It is often sufficient to choose some instruction in the algorithm as a barometer and to count how many times this instruction is executed. We can simplify this expression to 2 n 2 + (b +c -a /2)n + (d -c -b ).1 to T [i ] do k +-k +T[j] . In the selection sort example. and finally. How much time does the algorithm take? .i for every i <. where c is a third constant.1): k E. Selection sort. Let s be the sum of the elements of T.Analysing the Efficiency of Algorithms 52 Chap. that such simplifications should not be made incautiously. Details like the initialization of the loops are rarely considered explicitly.2. including the implicit control statements for this loop. The following example shows. The next step is to simplify this function using the asymptotic notation as well as the techniques explained in Section 2. Consider the following algorithm (which is reminiscent of the countsort algorithm discussed in Section 10. Often a first analysis gives rise to a complicated-looking function.1)/ 2 times when n items are sorted. The time taken by each trip round the inner loop can be bounded above by a constant a. where T is an array of n integers such that 0 <. which is executed exactly n (n . The complete execution of the inner loop for a given value of i therefore takes at most a time b + a (n -i ). This figure gives us the exact order of the execution time of the complete algorithm. and experience. Most of the execution time is spent carrying out the instructions in the inner loop. involving summations or recurrences.n.2. Consider the selection sorting algorithm given in Section 1. provided that the time taken to execute the chosen instruction can itself be bounded above by a constant. any instruction of the inner loop can usually be used as a barometer. intuition.1 to n do for j f. Choosing a barometer.

nor the comparison j>0". With a little experience the same algorithm can be analysed more succinctly. We now give an example of analysis of the average behaviour of an algorithm. This time is not zero when T[i]=O. If indeed this instruction could serve as a barometer. s )). In this case s = [I J.k + T [ j ] " is executed T [i ] times. given in Section 1. The time that this algorithm takes to sort n elements depends on their original order. As Section 1. The problem arises because we can only neglect the time spent initializing and controlling the loops provided we include something each time the loop is executed. The instruction in the inner loop is executed exactly s times.1. Thus the algorithm makes i -1 comparisons. To this we must add n to take account of the control of the outer loop and of the fact that the inner loop is initialized n times. The time therefore depends on two independent parameters n and s and cannot be expressed as a function of just one of these variables. Let a be the time taken by one trip round the inner loop. Problem 2. Finally. including the time spent on loop control. To execute the inner loop completely for a given value of i therefore takes a time b + aT [i]. This situa- tion is typical of algorithms for handling graphs. This can happen for . as we shall see in Chapter 10. the complete algorithm takes a time d + J:i" 1(c + b + aT [i]). Let x = T[i]. we would conclude that the algorithm takes a time in the exact order of s. T [i -2] . where the constant b represents the initialization time for the loop. Coming back to our algorithm. The worst case arises when x is less than T [ j ] for every j between 1 and i . and they presuppose that we know a priori the probability distribution of the instances to be solved. where c is a new constant.2. The detailed analysis of this algorithm is as follows. for yet another constant d. as in the algorithm. . (Here we do not count the implicit comparisons involved in control of the for loop. the algorithm clearly takes a time in 11(n) since each element of T is considered at least once. this expression yields (c +b )n + as + d . The number of comparisons carried out between elements is a good measure of the complexity of most sorting algorithms.) Suppose for a moment that i is fixed..Sec.4. since in this case we have to compare x to T [i . However. the time taken to execute one trip round the outer loop is c + b + aT [i ].3. The total time taken by the algorithm is therefore in O(n+s). When simplified. 2.1.4 points out. A simple example is sufficient to convince us that this is not the case. Next.1 ] . Suppose that T [i] = 1 whenever i is a perfect square and that T [i] = 0 otherwise. Consider the insertion sorting algorithm Example 2.7 says that we can express its execution time in asymptotic notation in two ways : O(n +s) or O(max(n . .. analyses of average behaviour are usually harder than analyses of the worst case. The total number of times it is executed is therefore E° T [i] = s times. Insertion sort. T [ 1 ] before we leave the while loop because j = 0. where the execution time often depends on both the number of vertices and the number of edges. We use the comparison "x < T [ j ] " as our barometer.2 Analysis of Algorithms 53 For each value of i the instruction " k .

. When we analyse an algorithm. T[i] can therefore be situated with equal probability in any position relative to the items T [I]. The same probability applies to any given number of comparisons up to i-2 included. the probability is 2/i that i .i.1)/ 2 E O(n 2). . the probability that T[i] is the kth largest element among T [1]. suppose that the n items to be sorted are all distinct and that each permutation of these items has the same probability of occurrence. . T[i] < T [i -1 ] is false at the outset. but this number is still in O(n2).1) = n (n . In the worst case insertion sorting thus takes a time in O(n2). The average number of comparisons made for a given value of i is therefore i-2 ci = . the n th term of the harmonic series.1 comparisons will be carried out. is negligible compared to the dominant term n 2/4 because H" E O(log n).Analysing the Efficiency of Algorithms 54 Chap.2(i-1)+ Y.1 . The insertion sorting algorithm makes on the average about half the number of comparisons that it makes in the worst case. With probability 1/i . For a given value of i. To determine the time needed by the insertion sorting algorithm on the average. T [i] is 1/i because this happens for [n] (i -1)! (n -i )! = n!/i of the n! possible permutations of n elements. The total number of comparisons is therefore It 2(i .. T [i -1]. The average number of comparisons made by the algorithm when sorting n items is therefore " " Eci=z i=2 i+1 2 i=2 - -1 i n2+3n -H" 4 E 0(n2).. as shown in Problem 2. . we often have to evaluate the sum of arithmetic. 2 every value of i from 2 to n when the array is initially sorted in descending order. Here H" = E i "_ I i. On the other hand. k k=I _ (i -1)(i+2) _ i + I _ I 2i 2 i These events are independent for different values of i.17. . since this happens both when x < T [ 1 ] and when T [ 1 ] S x < T [2].k <.1. If i and k are such that 1 <. Although the algorithm takes a time in c2(n2) both on the average and in the worst case. a time in 0 (n) is sufficient for an infinite number of instances. T [2]. T [2].. and other series. . and the first comparison x < T [j ] gets us out of the while loop. Notice that selection sorting systematically makes the same number of comparisons between elements that insertion sorting makes in the worst case. geometric.

Making a heap. But it is impossible for k (and thus j ) to exceed n.9.2. ? 2jr_1 for 1 < t <.k " on the t th trip round the loop. therefore k >. (*) To simplify this expression. Consider the "make-heap " algorithm given at the end of Section 1. so that this can be done in a time in 0 (n).4: this algorithm constructs a heap starting from an array T of n items. Since any algorithm for constructing a heap must look at each element of the array at least once.1). the value of j after execution of the assignment "j . As a barometer we use the instructions in the repeat loop of the algorithm used to sift down a node.m. Consequently n >_Jm >2Jm-I?4Jm-2? . 55 Prove that for any positive integers n and d d k=o Rather than simply proving this formula by mathematical induction.m.I > lg (n / 8).2. Let m be the largest number of trips round the loop that can be caused by calling sift-down (T. But d = Llg (n / 2)] implies that d + 1 <. Obviously j.E2 k lg(n/2k) < 2dllg(n/2d i=1 1) k=O (by Problem 2..2 Analysis of Algorithms Problem 2. try to see how you might have discovered it for yourself. This shows that j. which implies that m <. n. we obtain our . Hence Ln/2J lg(n/i)<-3n From (*) we thus conclude that Ln / 2j + 3n trips round the repeat loop are enough to construct a heap. = i. notice first that for any k ? 0 2--. 2. Moreover.2j.2. i).4. % =GA The interesting part of the sum (*) can therefore be decomposed into sections corresponding to powers of 2.1 + lg (n / i ). Let d = Llg (n / 2)].1. Ln/2i d E lg(n/i) <. X lg(n/i)<-2klg(n/2k). then at the end of the (t -1)st trip round the loop we had j # k . Example 2. Denote by j..Sec. >2m-li Thus 2m n / i. The total number of trips round the repeat loop when constructing a heap can now be bounded above by Ln/2] (I +lg(n/i)). if 1 < t <.lg n and d .

In our analysis.3).4.. Recursive calculation of determinants. For a given number of elements. what are the best and the worst ways to arrange the elements initially insofar as the execution time of the algorithm is concerned? Example 2. In order to construct the heap.2.3. The algorithm then sifts the root down a path whose length is at most k. But a heap containing n elements is of height Llg nj. both in the worst case and on the average. additions and multiplications are considered to be elementary operations. in particular Example 2. Let t(n) be the time taken by some implementation of this algorithm working on an n x n matrix.1 (the right hand subtree could be of height k .2).2. Find the exact order of the execution time for Williams's heapsort. The techniques of Section 2. most of the work done by the algorithm consists of calling itself recursively n times to work on (n -1) x (n -1) matrices.9. which takes a time in the order of k in the worst case.2. We now analyse the algorithm derived from the recursive definition of a determinant (Section 1. Analysis of heapsort. When n is greater than 1.2. hence it can be built in at most t (Llg n J) steps. We thus obtain the asymptotic recurrence t (k) E 2t (k -1) +O (k). 2 final result that the construction of a heap of size n can be carried out in a time in O(n). n ]) IT is an array to be sorted) make-heap (T) for i F n step -1 to 2 do exchange T [1] and T [i ] sift-down (T [1. We therefore ignore for the time being the problems posed by the fact that the size of the operands can become very large during the execution of the algorithm.4 we saw another algorithm for making a heap (slow-make-heap ).3. procedure heapsort (T [ 1 . In Section 1. Let t (k) stand for the time needed to build a heap of height at most k in the worst case. Assume k >.2. 1) { T is sorted) What is the order of the execution time required by this algorithm in the worst case ? * Problem 2. which is in O(n) since 2UgnJ <.56 Analysing the Efficiency of Algorithms Chap. .2.5. can be used to conclude that t (k) E O (2k).5.n. Williams invented the heap to serve as the underlying data structure for the following sorting algorithm. the algorithm first transforms each of the two subtrees attached to the root into heaps of height at most k . Problem 2.3.4.. Analyse the worst case for this algorithm and compare it to the algorithm analysed in Example 2. i -1].7. Problem 2.2. A different approach yields the same result.

and so Ln / m] = 1. is in O(n3). however.22. Recall that Euclid's algorithm calculates the greatest common divisor of two integers (Section 1. which means that n modm =n -m < n -n/2=n12. * Problem 2.4).5.k. If m > n / 2. This gives us the following asymptotic recurrence : t (n) E nt(n . let ni and mi be the values of n and m at the end of the i th trip round the loop.Sec.2. where mo and no are the initial values of m and n : ni =mi_i mi =ni_i modmi_1. Example 2. For each integer i <. By Problem 2. By Problem 2. then 1 <_ n / m < 2.2. Analysis of Euclid's algorithm.1. the matrices for the recursive calls have to be set up and some other housekeeping done. function Euclid (m. Let k be the number of trips round the loop made by the algorithm working on the instance < m.1.5). taking account this time of the fact that the operands may become very large during execution of the algorithm. Analyse the algorithm again.i <.5 supposes that the time needed to compute a determinant. The values of mi and ni are defined by the following equations for 1 <.n mod m n <. n >.2. Show that this time can be reduced to O(n). n) while m > 0 do t .7. Problem 2. this does not affect the fact that the complete algorithm takes a time in O(n!). In particular.22 the algorithm therefore takes a time in O(n!) to calculate the determinant of an n x n matrix.6.6.then (n mod m)<m <-n/2.2.m m -t return n We first show that for any two integers m and n such that n >_ in. 2. it is always true that n modm <n12.1) + O(n 3 ). Assume that you know how to add two integers of size n in a time in O(n) and that you can multiply an integer of size m by an integer of size n in a time in O(mn).2 Analysis of Algorithms 57 Besides this. Ifm<_n/2.2. which takes a time in O(n2) for each of the n recursive calls if we do this without thinking too much about it (but see Problem 2. Example 2. excluding the time taken by the recursive calls. Mk = 0 causes the algorithm to ter- minate and mi > 1 for every i < k.k. .

and give values for these constants. and c such that a f. provided the instructions in the loop can be considered as elementary.t(n) <. Then mk_1 < mk_3/2 < Mk-5/4 < . for appropriate constants a. as in Example 2. for appropriate constants a and b. Finally. Let t(n) be the time taken by some implementation of this algorithm working on the integer n. Prove that the algorithm fibl takes a time in even if we Problem 2..2. the number of trips round the loop made by Euclid's algorithm working on the integers in and n.. However. Prove that the worst case for Euclid's algorithm arises when we calculate the greatest common divisor of two consecutive numbers from the Fibonaccisequence. b. < mo/2d. 0(4") to calculate the Using constructive induction.7. b.9._1 mod m.2.b. Analysis of the algorithm fib2. k = 2d + 1 <. and c. The algorithm fib 1 therefore takes a time in n th term of the Fibonacci sequence. prove that af <. Once again.7. where 0 = (1+' )/ 2.1 + 21g m o .7. * Problem 2. we have mi =n.d. We give without explanation the corresponding asymptotic recurrence: t(n) E t(n-1)+t(n-2)+O(1). rithm fib l of Section 1. . Using the preceding observation. Define d by k = 2d + 1. But recall that Mk-1 ? 1. In conclusion.8. take into account that we need a time in O(n) to add two integers of size n.2.2.2. it is easy to use this technique to find three real positive constants a. This time is therefore in 0(n). its size is in 0(n lg o) = E) (n). (Since the value of f is in O(4"). and so m o >. remembering that m 1 = no mod m o < m 0.8. Suppose for the moment that k is odd. We now analyse the algoExample 2. constructive induction cannot be used directly to find a constant d such that t(n) <.) Example 2.5.c Problem 2. 2 1.Analysing the Efficiency of Algorithms 58 Clearly n. are in 0 (log m).c for any positive integer n._1 <n1_1/2=m1_2/2 for every i >. the recurrence looks so like the one used to define the Fibonacci sequence that it is tempting to suppose that t(n) must be in 0(f ). and therefore the time required by the algorithm. . <. . On the other hand.t (n) <. > m. . still not taking account of the large size of the operands involved. for each i Chap. The case when k is even is handled similarly.2d. .4. Analysis of the algorithm . It is clear that the algorithm fib2 takes a time equal to a + bn on any instance n .

however. The preceding equation shows that nn 5 n/2m < 1. and so nn = 0.t <_ m. 2. Then the time taken by ffb2 on an integer n > 2 is bounded above by n d + 2(c + 2a) + Y. If you find the result disappointing.. which implies that the algorithm fib3 takes a time in 0 (log n). but not. Compare your result to that obtained in Example 2. Prove that the execution time of the algorithm fib3 on an integer n is in @(log n) if no account is taken of the size of the operands.7. Assume that addition of two integers of size n takes a time in O(n) and that multiplication of an integer of size n by an integer of size m takes a time in O(mn).11.3.2. be the value of n at the end of the t th trip .7 we shall see a multiplication algorithm that can be used to improve the performance of the algorithm fib3 (Problem 4.2 Analysis of Algorithms 59 What happens. Problem 2.Sec. . in particular n 1 = Ln / 2j. Let m = I + LlgnJ. and let b be a constant such that the size of fn is bounded above by bn for every integer n ? 2.1 (where we take f _1 = 1). ** Problem 2. Analysis of the algorithm fib3.2. ab (2k -1) = abn 2 + (d + 2c +4a . k=3 which is in 0 (n 2). It is obvious that nr = Lnr_1/ 2]<. The analysis of fib3 is relatively easy if we do not take account of the size of the operands. let n. that of fib2 (why not?).7.5 and remember that the hidden constants can have practical importance! In Section 4.1) for k >.8.9. if we take account of the size of the operands involved? Let a be a constant such that the time to add two numbers of size n is bounded above by an. the time is bounded above by c +2a. plus some constant time c to carry out the assignments and the loop control. which takes a time bounded above by ab (2k . look back at the table at the end of Section 1.5).n. It is easy to see by symmetry that the algorithm takes a time in 0(n 2). But nn is a nonnegative integer. Determine the exact order of the execution time of the algorithm fib3 used on an integer n. We conclude that the loop is executed at most m times.-1/2 for every 2 <. take the instructions in the while loop as our barometer.10.n12' .2. For each of the first two trips round the loop. To evaluate the number of trips round the loop. which is the condition for ending the loop.2. of . Let d be an appropriate constant to account for necessary initializations.. Notice first that the values of i and j at the beginning of the k th trip round the for loop are respectively fk_2 and fk. To see this. < n 1/ 2t-1 <. The k th trip round the loop therefore consists of calculating fk_2 + fk _ 1 and fk . Consequently nt 5 nr-1/2 5 nr-2/4 <_ . Example 2.4ab) .

Example 2.2.i while j# 0 do j E.. this for i -0ton do j F. and loop control can all be carried out at unit cost.j div 2 Supposing that integer division by 2.i rank [i] 0 cost [i ] . It can happen that the analysis of an algorithm is facilitated by the addition of extra instructions and counters that have nothing to do with the execution of the algorithm proper. assignments.0 fori F. For instance. Prove your answer.5.9. which indicates that i is the root of its tree. define the group of an element of rank r as G (r).2.1 to N do set [i] F. it is clear that this algorithm takes a time in U(n) n 0 (n log n). N ] plays the role of height [ 1 . Their purpose will be explained later.2.10.9. Finally. We also introduce a strictly increasing function F : IN -* IN (specified later) and its "inverse" G : IN -* N defined by G (n) = min { m E IN I F (m) >_ n ). Chap. The analysis of these algorithms is the most complicated case we shall see in this book.j div 2 Show a relationship between this algorithm and the act of counting from 0 to n + 1 in binary. except when set [i ] = i. * Problem 2.0 . The array set [1 . The algorithms become procedure init (initializes the trees) global <. We begin by introducing a counter called global and a new array cost [ 1 .. Analysis of disjoint set structures. N ] keeps the meaning given to it in algorithms find3 and merge3: set [i ] gives the parent of node i in its tree.Analysing the Efficiency of Algorithms 60 Problem 2. The array rank [ 1 .13.. Find the exact order of its execution time.. 2 Consider the following algorithm : for i F-0ton do j F. this is so when we look at the algorithms find3 and merge3 used to handle the disjoint set structures introduced in Section 1. N ] in algorithm merge3 : rank [i] denotes the rank of node i (see Section 1.12. N J.i while j is odd do j F. time for the algorithm Answer the same question as in the preceding problem.5).

cost [i ] + 1 j .set [r] { r is the root of the tree) i E. In order to obtain an upper bound on these values.2 61 function find (x) { finds the label of the set containing object x I r -x while set [r r do r F. the rank of a node that is not a root is always strictly less than the rank of its parent .rank [a] + 1 set[b] f. the following remarks are relevant : 1. Therefore the total time required to execute an arbitrary sequence of n calls on find and merge. 2. .set [i ] set [i] F. once an element ceases to be the root of a tree. 2.a else if rank [a] > rank [b] then set [b] F a else set (a] b With these modifications the time taken by a call on the procedure find can be reckoned to be in the order of I plus the increase of global + EN cost [i] occasioned by the call. it never becomes a root thereafter and its rank no longer changes .global + 1 else cost [i] F. is in N O (N + n + global + cost [i ]). including initialization.Analysis of Algorithms Sec. b) { merges the sets labelled a and b we suppose that a b [ if rank [a] = rank [b] then rank [a] t. The time required for a call on the procedure merge can be bounded above by a constant.x while i * r do if G (rank [i]) < G (rank [set [i]]) or r = set [i] then global E. i=1 where global and cost [i] refer to the final values of these variables after execution of the sequence.r i-j return r procedure merge (a.

Analysing the Efficiency of Algorithms


Chap. 2

3. the rank of an element never exceeds the logarithm (to the base 2) of the number
of elements in the corresponding tree ;

4. at every moment and for every value of k, there are not more than N i 2k elements of rank k ; and

5. at no time does the rank of an element exceed L1gNJ, nor does its group ever
exceed G ( Llg NJ).

Remarks (1) and (2) are obvious if one simply looks at the algorithms. Remark

(3) has a simple proof by mathematical induction, which we leave to the reader.
Remark (5) derives directly from remark (4). To prove the latter, define subk (i) for
each element i and rank k : if node i never attains rank k, subk (i) is the empty set ; oth-

erwise sub, (i) is the set of nodes that are in the tree whose root is i at that precise
moment when the rank of i becomes k. (Note that i is necessarily a root at that
moment, by remark (1).) By remark (3), subk (i) # 0
# subk (i) ? 2k . By remark
(2), i # j
subk (i) n subk (j) = 0. Hence, if there were more than N/ 2k elements i
such that subk (i) 0, there would have to be more than N elements in all, which
proves remark (4).

The fact that G is nondecreasing allows us to conclude, using remarks (2) and
(5), that the increase in the value of global caused by a call on the procedure find
cannot exceed 1 + G ( LlgNJ). Consequently, after the execution of a sequence of n
operations, the final value of this variable is in 0 (1 + nG (L1g NJ)). It only remains to
find an upper bound on the final value of cost [i] for each element i in terms of its final

Note first that cost [i ] remains at zero while i is a root. What is more, the value
of cost [i ] only increases when a path compression causes the parent of node i to be
changed. In this case the rank of the new parent is necessarily greater than the rank of
the old parent by remark (2). But the increase in cost [i ] stops as soon as i becomes
the child of a node whose group is greater than its own. Let r be the rank of i at the
instant when i stops being a root, should this occur. By remark (1) this rank does not
change subsequently. Using all the preceding observations, we see that cost [i ] cannot
increase more than F (G (r)) - F (G (r) -1) -1 times. We conclude from this that the
final value of cost [i] is less than F (G (r)) for every node i (=-final (r), where final (r)
denotes the set of elements that cease to be a root when they have rank r >- 1 (while,
on the other hand, cost [i] remains at zero for those elements that never cease to be a
root or that do so when they have rank zero). Let K = G (Llg NJ) - 1. The rest is
merely manipulation.



cost [i ] = Y

E cost [i ]

g=0 r=F(g)+1 iefina((r)




I F (G (r))

g=0 r=F(g)+I iefinal(r)

Analysis of Algorithms

Sec. 2.2




E (N12`)F(g+1)

<_ 1

g=O r=F(g)+l


It suffices therefore to put F (g + 1) = 2F(g) to balance global and 1"' cost [i] and so
to obtain E 7l cost [i] NG (Llg NJ). The time taken by the sequence of n calls on
find and merge with a universe of N elements, including the initialization time, is
therefore in

O(N +n +global +

cost[i]) c O(N +n +nG(L1gNJ)+NG(L1gNJ))

= O(max(N,n)(l +G(L1gN]))).
Now that we have decided that F (g + 1) = 2F(g), with the initial condition
F (0) = 0, what can we say about the function G ? This function, which is often
denoted by lg*, can be defined by

G(N)=lg*N =min{k I lglg ...lgN <0)
k times

The function lg* increases very slowly : lg*N <- 5 for every N 5 65,536 and
lg*N <- 6 for every N < 265,536 Notice also that lg*N - lg*(L1gN j) 2, so that
lg*(Llg NJ) E O(lg*N ). The algorithms that we have just analysed can therefore execute a sequence of n calls on find and merge with a universe of N elements in a time
in 0 (n lg*N ), provided n >_ N, which is to most intents and purposes linear.
This bound can be improved by refining the argument in a way too complex to
give here. We content ourselves with mentioning that the exact analysis involves the
use of Ackermann's function (Problem 5.8.7) and that the time taken by the algorithm
is not linear in the worst case.







Figure 2.2.1


The towers of Hanoi.



Analysing the Efficiency of Algorithms


Example 2.2.11. The towers of Hanoi.

Chap. 2

It is said that after creating the

world, God set on Earth three rods made of diamond and 64 rings of gold. These rings
are all different in size. At the creation they were threaded on one of the rods in order

of size, the largest at the bottom and the smallest at the top. God also created a
monastery close by the rods. The monks' task in life is to transfer all the rings onto
another rod. The only operation permitted consists of moving a single ring from one
rod to another, in such a way that no ring is ever placed on top of another smaller one.
When the monks have finished their task, according to the legend, the world will come
to an end. This is probably the most reassuring prophecy ever made concerning the
end of the world, for if the monks manage to move one ring per second, working night
and day without ever resting nor ever making a mistake, their work will still not be
finished 500,000 million years after they began!

The problem can obviously be generalized to an arbitrary number of rings. For
example, with n = 3, we obtain the solution given in Figure 2.2.1. To solve the general problem, we need only realize that to transfer the m smallest rings from rod i to
rod j (where 1 <- i <- 3, 1 <- j <- 3, i # j, and m ? 1), we can first transfer the smallest
m -1 rings from rod i to rod 6 - i -j, next transfer the m th ring from rod i to rod j,

and finally retransfer the m -1 smallest rings from rod 6 - i -j to rod j. Here is a
formal description of this algorithm ; to solve the original instance, all you have to do
(!) is to call it with the arguments (64, 1, 2).

procedure Hanoi(m, i, j)
(moves the m smallest rings from rod i to rod j }

ifm > 0 then Hanoi(m -1, i, 6 - i -j)
write i "-*" j
Hanoi(m - 1, 6 - i j, j)
To analyse the execution time of this algorithm, let us see how often the instruction write, which we use as a barometer, is executed. The answer is a function of m,
which we denote e (m). We obtain the following recurrence :


2e(m-1)+1 ifm > 1,

from which we find that e (m) = 2" - 1 (see Example 2.3.4). The algorithm therefore
takes a time in the exact order of 2" to solve the problem with n rings.
Problem 2.2.14. Prove that the algorithm of Example 2.2.11 is optimal in the
sense that it is impossible with the given constraints to move n rings from one rod to
another in less than 2" - 1 operations.
* Problem 2.2.15.
Give a nonrecursive algorithm to solve this problem. (It is
cheating simply to rewrite the above algorithm using an explicit stack to simulate the
recursive calls.)

Sec. 2.3

Solving Recurrences Using the Characteristic Equation



We have seen that the indispensable last step when analysing an algorithm is often to
solve a system of recurrences. With a little experience and intuition such recurrences
can often be solved by intelligent guesswork. This approach, which we do not illustrate here, generally proceeds in four stages : calculate the first few values of the
recurrence, look for regularity, guess a suitable general form, and finally, prove by
mathematical induction that this form is correct. Fortunately there exists a technique
that can be used to solve certain classes of recurrence almost automatically.

2.3.1 Homogeneous Recurrences
Our starting point is the resolution of homogeneous linear recurrences with constant
coefficients, that is, recurrences of the form

aptn +altn-1 + ... +aktn-k =0



i. the t, are the values we are looking for. The recurrence is linear because it does
not contain terms of the form t, ti+j , t, 2, and so on ;
ii. the coefficients a, are constants ; and
iii. the recurrence is homogeneous because the linear combination of the t, is equal
to zero.

After a while intuition may suggest we look for a solution of the form


where x is a constant as yet unknown. If we try this solution in (*), we obtain


+a1xn-I + ... +akx"-k = O.

This equation is satisfied if x = 0, a trivial solution of no interest, or else if


+alxk-I + ... +ak =0.

This equation of degree k in x is called the characteristic equation of the recurrence

rk of this characteristic
Suppose f o r the time being that the k roots r I , r2r 2 ,-.
equation are all distinct (they could be complex numbers). It is then easy to verify that
any linear combination

t" _



ci rin

Analysing the Efficiency of Algorithms


Chap. 2

of terms rrn is a solution of the recurrence (*), where the k constants c 1 , c 22 ,-.- Ck
are determined by the initial conditions. (We need exactly k initial conditions to determine the values of these k constants.) The remarkable fact, which we do not prove
here, is that (*) has only solutions of this form.
Example 2.3.1.

Consider the recurrence

n >2

to -3tn-1 -4tn_2=0
subject to to = 0, t i = 1.
The characteristic equation of the recurrence is

x2-3x -4=0
whose roots are -1 and 4. The general solution therefore has the form
=C1(-1)n +C24n.


The initial conditions give

c1+ c2=0
that is, c 1 =- 5. C2 =

n =0



We finally obtain

[4n - (-1)n j.

to =

Example 2.3.2. Fibonacci.

Consider the recurrence

to = to-1 + to-2

n >2

subject to to = 0, t 1 = 1.

(This is the definition of the Fibonacci sequence ; see Section 1.7.5.)

The recurrence can be rewritten in the form to - to-1 - to-2 = 0, so the characteristic equation is

x2-x - 1 =0
whose roots are

2 and


The general solution is therefore of the form

to =Cir +c2r2
The initial conditions give

Sec. 2.3

Solving Recurrences Using the Characteristic Equation

c1+ c2=0

n =0


n =1


from which it is easy to obtain




Thus t" = 5 (r i - r2" ). To show that this is the same as the result obtained by De
Moivre mentioned in Section 1.7.5, we need to note only that r I = 0 and r2 = - 0-I.

* Problem 2.3.1.

Consider the recurrence

t" =2t"_I -2tn-2

n >2

subject to to = 0, t I = 1.

Prove that t, = 2'12sin(n it/4), not by mathematical induction but by using the
characteristic equation.

Now suppose that the roots of the characteristic equation are not all distinct. Let



I + ...

+ ak

be the polynomial in the characteristic equation, and let r be a multiple root. For every
n >- k, consider the n th degree polynomial defined by
h (x) = x [x" -k p (x)]' = a o n x" + a 1(n - 1)x"- I +

+ ak (n -k


Let q (x) be the polynomial such that p (x) = (x -r )2q (x). We have that
h (x) = x [(x-r )2 x


q (x)]' = x [2(x-r

)x"-k q (x) + (x-r

)2[x"-k q (x)]7

In particular, h (r) = 0. This shows that

aonr" +ai(n-1)r"-I+

+ak(n-k)rn-k =0,

that is, to = nr" is also a solution of (*). More generally, if m is the multiplicity of the
root r, then t" = r" , t" = nr" , t" = n 2r"
t" = n m - I r n are all possible solutions
of (*). The general solution is a linear combination of these terms and of the terms
contributed by the other roots of the characteristic equation. Once again there are k
constants to be determined by the initial conditions.
Example 2.3.3.

Consider the recurrence

tn =5tn_I-Stn-2+4tn-3
subject to to = 0, t I = 1, t2 = 2.
The recurrence can be written

n >_3

Analysing the Efficiency of Algorithms


Chap. 2

to - Stn -I + 8tn-2 - 4tn-3 = 0
and so the characteristic equation is

x3-5x2+8x -4=0
or (x-1)(x-2)2 = 0.
The roots are 1 (of multiplicity 1) and 2 (of multiplicity 2). The general solution
is therefore
to =C11n +C22n +C3n2n.

The initial conditions give


n =0

c1+ C2

n =2

from which we find c 1 = -2, c 2 = 2, c 3 = - z . Therefore
to = 2n +1

- n2n- - 2.

2.3.2 Inhomogeneous Recurrences
We now consider recurrences of a slightly more general form.

aotn +a1tn-1+ ... +aktn-k = b n p(n)

( ** )

The left-hand side is the same as (*), but on the right-hand side we have bn p (n),

i. b is a constant ; and
H. p (n) is a polynomial in n of degree d.
For example, the recurrence might be

tn -2tn_1=3n.

In this case b = 3 and p (n) = 1, a polynomial of degree 0. A little manipulation
allows us to reduce this example to the form (*). To see this, we first multiply the
recurrence by 3, obtaining
3tn - 6tn - 1 =

3n + 1

If we replace n by n + 1 in the original recurrence, we get

Finally, subtracting these two equations, we have

we can see that the factor (x-2) comes from the left-hand side of the original recurrence. Example 2. replace n in the recurrence by n + 1 and then multiply by .Sec. The characteristic equation is x2-5x +6=0 that is.8th+1 + 21tn . th . proceed as in the homogeneous case. we can show that to solve (**) it is sufficient to take the following characteristic equation : (aoxk+alxk-I+ . Here is a second example.2t.3. whereas the factor (x-3)2 is the result of our manipulation..2tn+l = -6th+I + 12th (n+5)3n+2 (n+7)3' 2 = -6(n+6)3n+1 .4.3. The number of movements of a ring required in the Towers of Hanoi problem (see Example 2. +ak)(x-b)d+1=0. (x-2)(x-3)2 = 0. obtaining respectively 9th . Adding these three equations. Intuitively we can see that the factor (x-2) corresponds to the left-hand side of the original recurrence. and c.6..2.18tn_I = to+2 . which can be solved by the method of Section 2.1. Once again.3 Solving Recurrences Using the Characteristic Equation 69 to+1 -5th +6th_I =0.11) is given by . Once this equation is obtained. multiply the recurrence by 9 b. 2. (x-2)(x-3) = 0. we obtain th+2 .18tH-I = 0. The characteristic equation of this new recurrence is X3-8X2+21x-18=0 that is. replace n in the recurrence by n +2.-I = (n+5)3n The necessary manipulation is a little more complicated : we must a. whereas the factor (x-3) has appeared as a result of our manipulation to get rid of the right-hand side. Generalizing this approach.

2tn -I =c1+C22n -2(c1+C22n-') Whatever the initial condition. . We know that to = 0. Substituting the general solution back into the original recurrence. we find 1 = to . the number of movements of a ring required. 2 n>?1 subject to to = 0. -I = 1. which is of the form (**) with b = 1 and p (n) = 1. since clearly to >.. there is no need to calculate the constants in the general solution. so the general solution of the recurrence is =Ciln +C22n. a polynomial of degree 0. and the conclusion follows. In fact we can obtain a little more. to find a second initial condition we use the recurrence itself to calculate t1=2to+I=1. it is therefore always the case that c 1 must be equal to -1. Therefore C2 > 0.n. If all we want is the order of to . We finally have c1+ c2=0 n =0 c1+2c2=1 n=1 from which we obtain the solution to=2n-1. The characteristic equation is therefore (x-2)(x-1) = 0 where the factor (x-2) comes from the left-hand side and the factor (x-1) comes from the right-hand side. For this it is sufficient to notice that to . In the previous example.2t. once we know that to = C I In + C22n we can already conclude that to E 8(2n). to We need two initial conditions. The roots of this equation are 1 and 2. is certainly neither negative nor a constant.Analysing the Efficiency of Algorithms 70 tn=2tn_1+I Chap. The recurrence can be written to .

2. Conclude that all the interesting solutions of the recurrence must have c i > 0.3.. . we are always looking for a solution where to >.3 Solving Recurrences Using the Characteristic Equation 71 Problem 2. Example 2. Solve tn =2tn_I+n +2" n >.0 for every n. which is of the form (**) with b = 1 and p(n) = n. (***) where the b. The recurrence can be written t -2tn_I =n +2n. . =0 which contains one factor corresponding to the left-hand side and one factor corresponding to each term on the right-hand side. and to solve the problem as before. and hence that they are all in O(2"). Consider the recurrence to = 2tn -I + n. on the contrary! Why? Example 2. are distinct constants and the p. The characteristic equation is therefore (x-2)(x-1)2 = 0 with roots 2 (multiplicity 1) and 1 (multiplicity 2).. +ak)(X-b1)d'+I(x-b2)°. The general solution is to =c12" +C2In +C3nIn. Problem 2.3. A further generalization of the same type of argument allows us finally to solve recurrences of the form aotn +aitn .. prove that in the preceding example C2 = -2 and C3 = -1 whatever the initial condition.1 subject to to = 0. In the problems that interest us. By substituting the general solution back into the recurrence.2tn -I = n. If this is so. we can conclude immediately that to must be in 0 (2"). +aktn-k = b?p1(n)+bnP2(n)+ .+I .I + .2. There is nothing surprising in the fact that we can determine one of the constants in the general solution without looking at the initial condition .6. a polynomial of degree 1.Sec. (n) are polynomials in n respectively of degree d. .I+ .. This can be written to . It suffices to write the characteristic equation (aoxk +aIxk .3.3. .5. .3..

=-2-n +2n+1+n2". Problem 2. Replace n by 2k (so that k = lg n) to obtain T (2k) = 4T(2'. How many constraints on these constants can be obtained without using the initial conditions ? (See Problems 2. We can now determine C1 . This can be written . C2. and tk for the term of a new recurrence obtained by a change of variable. we can calculate t 1 = 3.Analysing the Efficiency of Algorithms 72 Chap. pi(n) = n .5. b2 = 2. both of multiplicity 2.3 and 2.3. The degree of p 1(n) is 1. Problem 2. Here is how we can find the order of T (n) if n is a power of 2 and if T(n)=4T(n12)+n n > 1. C 2 .1) + 2k . and p2(n) is of degree 0. We could obviously have concluded that t o e O (n 2") without calculating the constants. t2 = 12. In the following examples we write T (n) for the term of a general recurrence. 2 which is of the form (***) with b 1 = 1. Using the recurrence.7.4. regardless of the initial condition. cm . Example 2. p2(n) = 1. The characteristic equation is (x-2)(x-1)2(x-2) = 0. If the characteristic equation of the recurrence (***) is of degree m =k +(d1+1)+(d2+1)+ .3.3.) 2.3. C3 and C4 from =0 n =0 c1+ c2+2c3+ 2c4= 3 n=1 c1 +2c2+4c3+ 8c4= 12 c1+3c2+8c3+24c4=35 n =2 n =3 + c3 Cl arriving finally at t. Prove that all the solutions of this recurrence are in fact in O(n 2").3. then the general solution contains m constants C 1 .4. The general solution of the recurrence is therefore of the form to =C1I" +C2nI" +C32" +Cgn2". t3 = 35. which has roots 1 and 2.3.3 Change of Variable It is sometimes possible to solve more complicated recurrences by making a change of variable.

Hence. we obtain successively T(2k)=4T(2k-1)+4k tk =4tk-I +4k.3. and so tk =c14k +c2k4k T(n) =c1n2+C2n2lgn. We know how to solve this new recurrence : the characteristic equation is (x-4)(x-2) = 0 and hence tk = c 14k + c 22k . Here is how to find the order of T (n) if n is a power of 2 and if T(n)=4T(n/2)+n2 n > 1. As before. 2.9. Here is how to find the order of T (n) if n is a power of 2 and if T (n) = 2T (n / 2) + n lg n n > 1. T (n) E 0 (n log2n I n is a power of 2). we obtain T(2k)=2T(2k-1)+k2k tk =2tk-I+k2k The characteristic equation is (x-2)3 = 0. The characteristic equation is (x-4)2 = 0. Example 2. Putting n back instead of k. we find T(n)=C1n2+C2n. and so tk =c12k +C2k2k +C3k22k T(n)=cin +c2nlgn +c3nlg2n. Example 2. .8.3 Solving Recurrences Using the Characteristic Equation 73 tk =4tk-1+2k if tk = T (2k) = T (n).3. T (n) is therefore in 0 (n 2 1 n is a power of 2). Thus T (n) e O (n 2log n I n is a power of 2).Sec. Proceeding in the same way.

1. Chap. b >. The characteristic equation is (x-3)(x-2) = 0.3. T (n) = 3T(n12) + cn We obtain successively T(2k)=3T(2k-l)+c2k tk =3tk_i+c2k. 0 Finally. The constants no ? 1. however. Let T : IN -) 1R+ be an eventually nondecreasing function such that T(n)=aT(n/b)+cnk n > no when n In 0 is a power of b. Show that the exact order of T (n) is given by T (n) E O(nk) ifa < hk O(n k log n) ifa = b k 'ogn a ifa > b k O(n ) Rather than proving this result by constructive induction. n = 2k > 1). it is sufficient to add the condition that T (n) is eventually nondecreasing to be able to conclude that the asymptotic results obtained apply unconditionally for all values of n.Analysing the Efficiency of Algorithms 74 Example 2. T (n) E O (n 'g 3 I n is a power of 2). In each of these four cases. It is therefore inevitable that the solution obtained should be in conditional asymptotic notation. This follows from problem 2.13.3. In Examples 2. n 2 log n .10 the recurrence given for T (n) only applies when n is a power of 2.3. Remark. .10. 2 We want to find the order of T (n) if n is a power of 2 and if (c is constant. This result is generalized in Problem 2.3. and so tk = c 13k + C22k T(n)=c13'g" +c2n and hence since algb = b1g° T(n)=cInIg3+c2n. whereas a and c are positive real numbers. * Problem 2. n log2n and n I g 3 are smooth.20 since the functions n 2.6.2 and k >.3. obtain it using the techniques of the characteristic equation and change of variable.0 are integers.7 to 2.

Finally.4 Range Transformations When we make a change of variable. At first glance. 2. we transform the domain of the recurrence. one of the coefficients is not constant. Problem 2. From VO= I + lg 3. 75 Solve the following recurrence exactly for n a power of 2: T(n)=2T(n/2)+lgn n >>-2 subject to T(1) = 1. We give just one example of this approach. The characteristic equation is (x-2)(x-1)2 = 0. and furthermore. We want to solve n>1 T (n) = n T 2(n / 2) subject to T (l) = 6 for the case when n is a power of 2. none of the techniques we have seen applies to this recurrence since it is not linear. which gives k>0 tk =2ktk2 I subject to to = 6. To transform the range. using tk = 2t'" and T(n) = tlgn . and V2 = 8 + 41g 3 we obtain c i = 3 + lg 3.8.3. and c 3 =-J. and so Vk =c12k +c2lk +c3klk. It is sometimes useful to transform the range instead in order to obtain something of the form (***). we create a new recurrence by putting Vk = lg tk . C2=-2.3 Solving Recurrences Using the Characteristic Equation Problem 2. 2. Express your solution as simply as possible using the O notation.7. Express your solution as simply as possible using the 0 notation. V.2. = 3 + 21g 3. Solve the following recurrence exactly for n of the form )+lgn 22A n >_4 subject to T(2) = 1. The first step is a change of variable : put tk = T (2k ).3. which yields k>0 -I subject to V() = lg 6.k . and hence Vk = (3 + lg 3)2k .Sec. we obtain 23n-23n T(n)= n .3.

Let f : X . Problem 2. (Multiplication of large integers: see Sections Solve the following recurrence exactly : to = to -1 + 2tn-2 . Consider any constants c E R+ and no E N.3. Recurrences arising from the analysis of divide-and-conquer algorithms (Chapter 4) can usually be handled by Problem 2. Define the function T : X -* R* by the recurrence T(n)= d if n =no aT (nlb)+ f(n) if nEX. Solve the following recurrence exactly : tn =tn_I +tn_3-tn_4 n >>>4 subject to to = n for 0 5 n < 3.6. whereas a and d are real positive constants.20 to conclude for T (n). and * Problem 2. Problem 2. use Example 2.7. Express your answer as simply as possible using the 0 notation. 2 2.15n + 106 for 0!5 n :5 2. a more general result is required (and the technique of the characteristic equation does not always apply).1.Analysing the Efficiency of Algorithms 76 Chap. Prove that T (n) E O (n ig 3 ). Hint : observe that T (n) < 3 T (1 + [n / 21) + cn for n > n 0.3. Express your answer as simply as possible using the 0 notation. 4. Let T AN 4 R* be an eventually nondecreasing function such that T(n) ST(Ln/2])+T([n/2])+T(1+[n/21)+cn n > no.9.2 subject to T(1) = 1.2. In some cases. and use problem 2.5 Supplementary Problems Problem 2.11. Problem 2.3.10 to solve for T '(n) when n is a power of 2. The constants no >-1 and b >.n>no . make the change of variable T '(n) = T (n+2).7).2 are integers. however.3.R* be an arbitrary function. Express your answer as simply as possible using the O notation.2tn-3 n >3 subject to to = 9n 2 .10. Solve the following recurrence exactly for n a power of 2: T (n) = 5T (n / 2) + (n lg n)2 n >. Define X={n ENIlogb(n/no)EIN}={nENNI(3iEIN)[n=nob`I).

then T (n) E < O(no) if f(n) E O(nP/(logn)1+q ) O(f (n) log n log log n) if f (n) E O(n P / log n) O(f(n)logn) if f(n) E O(nP(log n)q Off (n)) if f (n) E O(n p+q) 1) . find the simplest but most general additional constraint on f (n) that q 1 <_ q 2 such that f (n) E O (n P (log n)q' 1) and f (n) E Q(n P (log n)' suffices to imply T (n) E O(f (n) log n). n >2 .3. As a special case of the first alternative. 2. It turns out that the simplest way to express T (n) in asymptotic notation depends on how f (n) compares to n P .3 Solving Recurrences Using the Characteristic Equation 77 Let p = loge a . If we set f (n o) = d (which is of no consequence for the definition of T ). Problem 2. iv. Note that the third alternative includes f (n) E O(nP) by choosing q=1.15. Let q be any strictly positive real constant . the value of T (n) is given by a simple summation when n EX logn(n In. Solve the following recurrence exactly as a function of the Problem 2. If you disprove it. The last alternative can be generalized to include cases such as f (n) E O(n P+q log n) or f (n) E O(n P+q / log n) . all asymptotic notation is implicitly conditional on n E=X.14. In what follows. we also get T (n) E O(f (n)) if there exist a function g : X -* IR* and a real constant a> a such that f(n)EO(g(n)) and g(bn)>-ag(n) for all v. W. Prove that i.3. Solve the following recurrence exactly : t = 1/(4-tn_1) n>1 subject tot 1 = 1/4. initial conditions a and b : T(n+2)=(1+T(n+l))/T(n) subject to T (O) = a. T (I) = b.) T (n) = E a' f (n l b') i=o ii. Prove or disprove that the third alternative can be generalized as follows : T(n) E O(f (n) log n) whenever there exist two strictly positive real constants ). T (n) e O(n P) whenever f (n) E O (n' ) for some real constant r < p .Sec.

for instance. . 1983).4 REFERENCES AND FURTHER READING The asymptotic notation has existed for some while in mathematics: see Bachmann (1894) and de Bruijn (1961). Example 2.10 is adapted from Hopcroft and Ullman (1973). The more precise analysis making use of Ackermann's function can be found in Tarjan (1975.1 corresponds to the algorithm of Dixon (1981). who also suggests that "one-way inequalities" should be abandoned in favour of a notation based on sets.15. Knuth (1976) gives an account of its history and proposes a standard form for it. Chap. n?3 El 2.1.3 comes from Williams (1964). Haken.3. including the characteristic equation and change of variable. are explained in Lueker (1980). For a more rigorous mathematical treatment see Knuth (1968) or Purdom and Brown (1985).16.2. The book by Purdom and Brown (1985) presents a number of techniques for ana- lysing algorithms. and Saxe (1980) is particularly relevant for recurrences occurring from the analysis of divide-and-conquer algorithms (see Chapter 4).2. Rudin (1953). The main mathematical aspects of the analysis of algorithms can also be found in Greene and Knuth (1981). Several techniques for solving recurrences. The analysis of disjoint set structures given in Example 2.Analysing the Efficiency of Algorithms 78 Problem 2. Problem 2. Conditional asymptotic notation and its use in Problem 2. Buneman and Levy (1980) and Dewdney (1984) give a solution to Problem 2. 2 Solve the following recurrence exactly : T(n)='T(n/2)-zT(n/4)-1/n subject to T (1) = 1 and T (2) = 3/2. consult any book on mathematical analysis. The paper by Bentley. For information on calculating limits and on de l'Hopital's rule.20 are introduced by Brassard (1985).1.2.

1 INTRODUCTION Greedy algorithms are usually quite simple. as the case may be) the value of the 79 . a selection function that indicates at any time which is the most promising of the candidates not yet used . and so on). whether or not it is possible to complete the set in such a way as to obtain at least one solution (not necessarily optimal) to our problem (we usually expect that the problem has at least one solution making use of candidates from the set initially available). the set of candidates that have already been used . a function that checks whether a set of candidates is feasible. the nodes of the graph. In the most common situation we have a set (or a list) of candidates : the jobs to be executed. and so on. that is. ignoring questions of optimality for the time being . To solve our optimization problem. They are typically used to solve optimization problems : find the best order to execute a certain set of jobs on a computer. this is the function we are trying to optimize. we look for a set of candidates constituting a solution that optimizes (minimizes or maximizes. a function that checks whether a particular set of candidates provides a solution to our problem.3 Greedy Algorithms 3. the length of the path we have found. find the shortest route in a graph. and an objective function that gives the value of a solution (the time needed to execute all the jobs in the given order. or whatever .

it is never reconsidered. A greedy algorithm proceeds step by step. representing for instance 1. and 25 units. it is there for good . Each time we enlarge the set of chosen candidates. and containing at least one coin of each type . function greedy (C : set) : set { C is the set of all the candidates) S f.1. element of C maximizing select(x) C-C\{x} if feasible (S u {x }) then S . Initially. The selection function is usually based on the objective function . We want to give change to a customer using the smallest posExample 3.0 (S is a set in which we construct the solution) while not solution (S) and C # 0 do x . If the enlarged set of chosen candidates is no longer feasible. our choice being guided by the selection function. they may even be identical. However.S u {x } if solution (S) then return S else return "there are no solutions" It is easy to see why such algorithms are called "greedy": at every step. we try to add to this set the best remaining candidate. It never changes its mind : once a candidate is included in the solution. sible number of coins. the candidate we tried and removed is never considered again. However. the first solution found in this way is always optimal. When a greedy algorithm works correctly.1. the procedure chooses the best morsel it can swallow. we shall see in the following examples that at times there may be several plausible selection functions. without worrying about the future. and the objective function : the number of coins used in the solution.Greedy Algorithms 80 Chap. a feasible set : the total value of the chosen set does not exceed the amount to be paid the selection function : choose the highest-valued coin remaining in the set of candidates. a solution : the total value of the chosen set of coins is exactly the amount we have to pay . if the enlarged set is still feasible. we remove the candidate we just added . the set of chosen candidates is empty. 3 objective function. 0 . so that we have to choose the right one if we want our algorithm to work properly. then the candidate we just added stays in the set of chosen candidates from now on. The elements of the problem are the candidates: a finite set of coins. 5. we check whether the set now constitutes a solution to our problem. once a candidate is excluded from the solution. Then at each step.

The problem is to find a subset T of the edges of G such that all the nodes remain connected when only the edges in T are used. The following lemma is crucial for proving the correctness of the forthcoming algorithms. T > is called a minimal spanning tree for the graph G. A > be a connected undirected graph where N is the set of nodes and A is the set of edges. Let B C N be a strict subset of the nodes of G. and the cost of an edge (a. Prove that with the values suggested for the coins in the preceding example the greedy algorithm will always find an optimal solution provided one exists. then a minimal spanning tree of G shows us how to construct at the lowest possible cost a road system linking all the towns in question.1. Let . on the other hand. and it is feasible if it does not include a cycle. Let G = < N. 3. this change of terminology does not affect the way we solve the problem. In particular.2 Greedy Algorithms and Graphs 81 * Problem 3.1. Prove.1. b } is the cost of building a road from a to b. Each edge has a given non-negative length.1 Minimal Spanning Trees Let G = < N. The graph < N. We give two greedy algorithms to solve this problem.2 GREEDY ALGORITHMS AND GRAPHS 3. Show that it can even happen that the greedy algorithm fails to find a solution at all despite the fact that one exists. Lemma 3. that the greedy algorithm no longer gives an optimal solution in every case if there also exist 12-unit coins. Using integer division is also more efficient than proceeding by successive subtractions. In the terminology we have used for greedy algorithms. (Instead of talking about length. 3. a feasible set of edges is promising if it can be completed so as to form an optimal solution. the empty set is always promising since G is connected. Prove that the partial graph < N. T > formed by the nodes of G and the edges in T is a tree.) Problem 3. In this case the problem is to find a subset T whose total cost is as small as possible. we can associate a cost to each edge. It is obviously more efficient to reject all the remaining 25-unit coins (say) at once when the remaining amount to be represented falls below this value.2. or if one type of coin is missing from the initial set. This problem has many applications. Moreover.Sec.2. Obviously. For instance. Finally. an edge touches a given set of nodes if exactly one end of the edge is in the set.1. and the sum of the lengths of the edges in T is as small as possible. a set of edges is a solution if it constitutes a spanning tree. if the nodes of G represent towns.2. A > be a connected undirected graph where the length of each edge is given. by giving specific counterexamples.

The main difference between the various greedy algorithms to solve this problem lies in the order in which the edges are selected.1. If e e U. At the end of the algorithm only one connected component remains. there is nothing to prove. each node of G forms a distinct trivial connected component. Let e be the shortest edge that touches B (or any one of the shortest if ties exist). Then T v f e } is promising. which touches B. since e touches B. Proof. But since the length of e is by definition no greater than the length of e'. Kruskal's algorithm.Greedy Algorithms 82 Chap. (Initially. the total length of the edges in U' does not exceed the total length in U. Let U be a minimal spanning tree of G such that T E_ U (such a U must exist since T is promising by assumption).2. so that T is then a minimal spanning tree for all the nodes of G. . that also touches B (otherwise the cycle could not close . we create exactly one cycle (this is one of the properties of a tree). and it includes e.) The elements of T that are included in a given connected component form a minimal spanning tree for the nodes in this component. N\B B Figure 3. At every instant the partial graph formed by the nodes of G and the edges in T consists of several connected components. say. As the algorithm progresses. when T is empty. cannot be in T. If we now remove e'. The initial set of candidates is the set of all the edges. 3 T c A be a promising set of edges such that no edge in T touches B. Otherwise when we add the edge e to U. the cycle disappears and we obtain a new tree U' that spans G. e'. there necessarily exists at least one other edge. A greedy algorithm selects the edges one by one in some given order.2. Therefore U' is also a minimal spanning tree. In this cycle.see Figure 3. we note that T c U' because the edge e'.1). A cycle is created if we add edge e to U. Each edge is either included in the set that will eventually form the solution or eliminated from further consideration. edges are added to T. To complete the proof. The set T of edges is initially empty.

5).2. A graph and its minimal spanning tree. (} (1. and consequently. (2.5 } .5) rejected 7 14.5. is by induction on the number of edges selected until now. In increasing order of length the edges are : (1. 3 } .71 5 (1. 31.2.71. its total length is 17.2).3.2.4).2.2.Sec.6}. (2. the two connected components now form only one component. . El Figure 3.7) (1. Prove that Kruskal's algorithm works correctly.2. 13. (1.7) 6 t2.4).1. 5) . This minimal spanning tree is shown by the heavy lines in Figure 3.7). (2.51 (61 (7) 4 16.71.21 (3} (4} 151 (61 (71 3 (2.2.7).7) 1 2 T contains the chosen edges (1.2.7) (1.2.3. (4. 2) .3} (4.2.3} (4. (6.5) (1. (5. 12. Otherwise the edge is rejected : it joins two nodes in the same connected component and cannot therefore be added to T without forming a cycle since the edges in T form a minimal spanning tree for each component.2.2. (6. Step Edge considered Initialization - Connected components (11 (2) (3) (4) 151 (6) f7) 11.7).3} (41 (5} (6) 171 (1. we add it to T. (3.4. The algorithm proceeds as follows. (4.4}. If an edge joins two nodes in different connected components. consider the graph in figure 3. The algorithm stops when only one connected component remains.5} (6.2 Greedy Algorithms and Graphs 83 To build bigger and bigger connected components.2. The proof. Problem 3.21 11. 3. which uses lemma 3.5} (6. we examine the edges of G in order of increasing length.4.5). To illustrate how this algorithm works. and (4.3) (4. 14.6). (5.

B) to merge two disjoint sets. function Kruskal (G = < N. by the analysis given in example 2.4. in the worst case 0 ((2a+n -1)lg*n) for all the find and merge operations.3. where is this possibility reflected in the algo- 0 rithm ? To implement the algorithm. On a graph with n nodes and a edges the number of operations is in 0 (a log a) to sort the edges.n . which tells us in which component the node x is to be found. length : A -i IR*) : set of edges 1 initialization) Sort A by increasing length n -#N T .9. each containing one distinct element of N { greedy loop } repeat { u . We conclude that the total time for the algorithm is in 0 (a log n) because 0 (lg*n) c 0 (log n).find (u ) vcomp f. Here is the algorithm. that is not connected? What happens if. We therefore use disjoint set structures (Section 1. by mistake.0 1 will contain the edges of the minimal spanning tree) initialize n sets. We have to carry out rapidly the two operations find (x).10. Is this the case in our example.v}} until #T = n -1 return T Problem 3. vcomp) T -T u {{u.2. 3 Problem 3. we have to handle a certain number of sets : the nodes in each connected component.shortest edge not yet considered ucomp F-.find (v) if ucomp # vcomp then merge (ucomp.1. and merge (A . Although this does not .5).2. and if so.2. For this algorithm it is preferable to represent the graph as a vector of edges with their associated lengths rather than as a matrix of distances. A > : graph . O (a) for the remaining operations. A graph may have several different minimal spanning trees. we run the algorithm on a graph We can estimate the execution time of the algorithm as follows. and at worst. For a connected graph we know that a >. since there are at most 2a find operations and n -1 merge operations on a universe containing n elements . v) f.Greedy Algorithms 84 Chap. which is equivalent to 0 (a log n) since n-1 <-a Sn(n-1)/2. 0 (n) to initialize the n disjoint sets .

6. which again uses Lemma 3. the set B of nodes contains a single arbitrary node. length : A -> IR*) : set of edges ( initialization) T F. It then adds u to B and (u. Prove that Prim's algorithm works correctly.2. the minimal spanning tree grows "naturally". function Prim (G = < N. To illustrate how the algorithm works. In this way the edges in T form at any instant a minimal spanning tree for the nodes in B. v) to T . The proof.4 . is by induction on the number of nodes in B. consider once again the graph in Figure 3.9. 3.Sec. . leaving to the algorithm the job of working out which edges exist ? Prim's the heap property should be inverted so that the value of each internal node is less than or equal to the values of its children). it is preferable to keep the edges in a heap (Section 1. the original algorithm wastes time sorting all these useless edges. the user supplies a matrix of distances. We continue thus as long as B # N. instead of providing a list of edges. There results a forest of trees that grows somewhat haphazardly. At each stage we add a new branch to the tree already constructed. and the set T of edges is empty. In such cases. on the other hand. Here is an informal statement of the algorithm. This is particularly advantageous in cases when the minimal spanning tree is found at a moment when a considerable number of edges remain to be tried. v } of minimum length such that u E N \ B and v E B T 6-T u {(u.2.(an arbitrary member of N } while B # N do find { u .2.2. although each search for a minimum in the repeat loop will now take a time in 0 (log a) = 0 (log n). This allows the initialization to be carried out in a time in 0 (a). except that we are careful never to form a cycle. A > : graph .2.v}} B F-B U (u) return T Problem 3.2 Greedy Algorithms and Graphs 85 change the worst-case analysis. In Kruskal's algorithm we choose promising edges without worrying too much about their connection to previously chosen edges. v J such that u E N \ B and v E B .1. At each step Prim's algorithm looks for the shortest possible edge { u . We arbitrarily choose node I as the starting node. In Prim's algorithm.5. Initially.0 (will contain the edges of the minimal spanning tree } B (-. and the algorithm stops when all the nodes have been reached. starting from an arbitrary root. What can you say about the time required by Kruskal's algoProblem 3. rithm if.



The modifications to the algorithm are simple : Figure A directed graph. j) exists and L [i. which is why the main loop is only repeated n . D would not change if we did one more iteration to remove the last element of C..30.20.2 times.3. 10] Initialization Clearly. n. n ]) : array[2. n ] (initialization) C F. simply follow the pointers P backwards from a destination to the source. j ] ? 0 if the edge (i. D [v] +L[v.Greedy Algorithms 88 Chap. If we want not only to know the length of the shortest paths but also where they pass.2.30. . j ] = 00 otherwise.30. .some element of C minimizing D [v ] C (-.. 3 gives the length of each directed edge : L [i.4) [50. 10] (35.10] 2 4 12.10] 1 5 (2.3.. I . function Dijkstra (L [1 .3. n ) { S = N \ C exists only by implication ) for i -2tondoD[i]*-L[l...S u { v } } for each w e C do D[w] f. Step V - C D (2.i] { greedy loop } repeat n -2 times v ..5) [50. Here is the algorithm.20.100.{ 2.4. 3.2. .C \ { v) { and implicitly S .w]) return D The algorithm proceeds as follows on the graph in Figure 3.min(D[w].31 3 3 {2) [40. n ]. where P [v ] contains the number of the node that precedes v in the shortest path. To find the complete path. it suffices to add a second array P [2.

3. there are two possibilities for the shortest special path from the source to w : either it does not change.4).2. it will now belong to S. Problem 3. Consider now a node w 0 S different from v When v is added to S. 3. This follows immediately from the induction hypothesis for each node i that was already in S before the addition of v.3. I. We prove by mathematical induction that i. replace the contents of the inner for loop by ifD[w]>D[v]+L[v. Show how the modified algorithm works on the graph of Figure 3. or else it now passes through v. As for node v.Greedy Algorithms and Graphs Sec. Proof of correctness. is a special path.D[v] +L[v. consider the inductive step.w] P[w]-v. then D [i] gives the length of the shortest path from the source to i H. Next. then D [i] gives the length of the shortest special path from the source to i. the first node encountered that does not belong to S is some node x distinct from v (see Figure 3.2 89 initialize P [i] to 1 for i = 2. n . if a node i is not in S. . The initial section of the path. . We therefore have to verify that the shortest path from the source to v does not pass through a node that does not belong to S. as far as x. the base for our induction is thus obtained. and suppose by the induction hypothesis that these two conditions hold just before we add a new node v to S.2.. if a node i is in S.2. In the latter case it seems at first .10.. . We must therefore check that D [v ] gives the length of the shortest path from the source to v. Look at the initialization of D and S to convince yourself that these two conditions hold at the outset . Consequently. Suppose the contrary : when we follow the shortest path from the source to v. ii. By the induction hypothesis D [v ] certainly gives the length of the shortest special path.w] then D[w] <-. the total distance to v via x is distance to x (since edge lengths are non-negative) D [x] (by part (ii) of the induction) (because the algorithm chose v before x) D [v] and the path via x cannot be shorter than the special path leading to v. We have thus verified that when v is added to S. part (i) of the induction remains true.

Is it still sensible to talk about shortest paths if negative distances are allowed? S Figure 3. w] that we examined at a previous step when x was added to S. However. can be negative. At this point it is clear that the shortest path from the source to the remaining node is a special path.2. We have to compare explicitly the length of the old special path leading to w and the length of the special path that visits v just before arriving at w .11.2.4. 3 The shortest path The shortest special path Figure 3. we can ignore the possibility (see Figure 3. The shortest path from the source to v cannot go through node x.2. the algorithm does this.5) that v is visited. . Thus the algorithm ensures that part (ii) of the induction also remains true when a new node v is added to S. To complete the proof that the algorithm works. glance that there are again two possibilities : either v is the last node in S visited before arriving at w or it is not. Show by giving an explicit example that if the edge lengths Problem 3. we need only note that when its execution stops all the nodes but one are in S (even though the set S is not constructed explicitly). then Dijkstra's algorithm does not always work correctly.5 The shortest path from the source to w cannot visit x between v and w. but not just before arriving at w : a path of this type cannot be shorter than the path of length D [x] + L [x.Greedy Algorithms 90 Chap. because D [x] <_ D [v].2.

we compare the value of a node to the value of its parent rather than making comparisons with both children).. . If a << n 2. * Problem 3. . 1 iterations. n .. the element v of C that minimizes D [v] will always be found at the root. Let k = max(2.2. We might therefore consider modifying the definition of a heap slightly to allow percolation to run faster still. This allows us to save time in the inner for loop. If so. The straightforward implementation is therefore preferable if the graph is dense. for each element w of C adjacent to v. The instruction " C . a > n -1 and this time is in 0 (a log n). since we only have to consider those nodes w adjacent to v.2 times and to percolate at most a nodes. n . choosing v in the repeat loop requires all the elements of C to be examined. . The time required by this version of the algorithm is therefore in O (n 2). we have to remove the root of the heap exactly n . but how are we to avoid taking a time in c(n2) to determine in succession the n -2 values taken by v ? The answer is to use a heap containing one node for each element v of C. To sum up. to see whether D [v] + L [v. This does not happen more than once for each edge of the graph. we must modify D [w] and percolate w up the heap. we saw that up to a nodes can be percolated. This is interesting when we remember that eliminating the root has for effect to sift down the node that takes its place. If the graph is connected.9.1 .. ordered by the value of D [v].1. which takes a time in O (log n).in the matrix L. giving for each node its direct distance to adjacent nodes (like the type lisgraph of Section 1. With this in mind. Initialization of the heap takes a time in 0 (n). .2). it could be preferable to represent the graph by an array of n lists. Using the representation suggested up to now. and that percolating up is somewhat quicker than sifting down (at each level.2 Greedy Algorithms and Graphs 91 Analysis of the algorithm. it now consists of looking. La In j ).. n J. the instance is given in the form of a matrix L [1 . Suppose Dijkstra's algorithm is applied to a graph having n nodes and a edges. Show how your modification allows you to calculate the shortest paths from a source to all the other nodes of a graph in a time in . w ] < D [w]. so that we look at n .2.12. 2 values of D on successive iterations. Initialization takes a time in O (n). giving a total time in 0 ((a +n) log n). If we remember to invert the heap. at the cost of slowing down sifting. which again takes a time in 0 (log n ).C \ { v } " consists of eliminating the root from the heap. As for the inner for loop. In the preceding analysis. What modification to the definition of a heap do you suggest ? ii.. 1 . The inner for loop does n .Sec. it seems we might be able to avoid looking at the many entries containing . If a E O(n 2/ log n).. for a total also in 0 (n 2). giving a total time in 0 (n 2). whereas it is preferable to use a heap if the graph is sparse. i. 3. In a straightforward implementation. the choice of algorithm may depend on the specific implementation. whereas less than n roots are eliminated. n -2.

Since the number of customers is fixed. . we add customer j. 1 <_ i <_ n. 3. show that the modification suggested in the previous problem applies just as well to Prim's algorithm.Greedy Algorithms 92 Chap... . The service time required by each customer is known in advance : customer i will take time t. Note that this gives 0 (n 2) if a = n 2 and 0 (a log n) if a = n . a petrol pump. . Show that Prim's algorithm to find minimal spanning trees can also be implemented through the use of heaps. customer 1 is served immediately. i.) Problem 3. Show that it then takes a time in 0(a log n ). We want to minimize n T= (time in system for customer i ). 3 O (a logk n). it therefore gives the best of both worlds. minimizing the total time in the system is equivalent to minimizing the average time.. t3=3. a cashier in a bank. Imagine an algorithm that builds the optimal schedule step by step..3. and so on) has n customers to serve. customer 2 waits while customer 1 is served and then gets his turn. then six orders of service are possible.1.2.optimal 321: 3 + (3+10) + (3+10+5)=34 231 : In the first case. i 2. just as Kruskal's algorithm would. Finally. The increase in T at this stage is . Problem 2. and customer 3 waits while both 1 and 2 are served and then is served himself: the total time passed in the system by the three customers is 38.3 GREEDY ALGORITHMS FOR SCHEDULING 3.17(i) does not apply here since k is not a constant. Order T 123 : 5 + (5+10) + (5+10+3)=38 5 + (5+3) + (5+3+10)=31 10 + (10+5) + (10+5+3)=43 132: 213: 312: 10 + (10+3) + (10+3+5) = 41 3 + (3+5) + (3 + 5 + 10) = 29 E.1 Minimizing Time in the System A single server (a processor. Suppose that after scheduling customers i I . t2=10. For example. (Still faster algorithms exist.13. if we have three customers with t1=5.

... In the preceding example this algorithm gives the correct answer 3. 2.1 . If we exchange the positions of these two customers. k=1 kta. We now prove that this algorithm is always optimal. . k=1 Suppose now that I is such that we can find two integers a and b with a < b and t. in ) be any permutation of the integers { 1 . we obtain a new order of service I' obtained from I by interchanging the items is and ib This new order is preferable : .1).+(n-b+1)ti + E (n-k+l)t.. in other words..1 l tab 1o Exchanging the positions of two customers.+ =nti +(n-1)ti.. the a th customer is served before the b th customer even though the former needs more service time than the latter (see Figure 3.+(n-2)ti + n _ k(n-k+1)t.+ti)+(ti. because n T(I')=(n-a+1)ti. t" . . .+ To minimize this increase.. add to the end of the schedule the customer requiring the least service among those who remain. tin After exchange of is and !b I Service duration Served customer (from I') Figure 3. b a n i.. t.. we need only minimize tj ..3. t' . fin . 3.b Service order 1 Served customer . n 1.+ti.. 11 (from I) Service duration 2 t.+(ti.. t' . .. tea .3.. the total time passed in the system by all the customers is T(I)=ti.Sec. If customers are served in the order 1. 2... > t. +ti. Let I = (i I i 2 . 1.3 Greedy Algorithms for Scheduling 93 +ti +tj .. This suggests a simple greedy algorithm : at each step.

The only schedules that remain are those obtained by putting the customers in nondecreasing order of service time. suppose the customers are numbered so that s . and the speed of the tape drive is also constant. i. Prove by giving an explicit example that it is not necessarily optimal to hold the programs in order of decreasing values of pi iii. A magnetic tape contains n programs of length l 1 . loss of generality.i <. Prove that T is minimized if the programs are held in order of decreasing pi /li . Problem 3. . Prove that this algorithm always yields an optimal schedule. i 2.3.tin ) >0..3. (This of course implies that Ji 1 pi = 1. must serve customers number t 1 <.3. . the average time required to load a program is n I T =cE Pr Eli - j=1 k=1 where the constant c depends on the recording density and the speed of the drive. n ] as data and produces an optimal schedule ? The problem can be generalized to a system with s servers.Greedy Algorithms 94 Chap. If the programs are held in the order i 1.. 1 2 . Prove by giving an explicit example that it is not necessarily optimal to hold the programs in order of increasing values of 1i . i + 2s .2. In this context.. How much time (use the 0 notation) is required by a greedy algorithm that accepts n and t [I . We know how often each program is used : a fraction pi of requests to load a program concern program i. 1 <. Each time a program has been loaded. We want to minimize T. and therefore they are all optimal.n.t2 <S to . We can therefore improve any schedule in which a customer is served before someone else who requires less service. Without Problem 3..) Information is recorded along the tape at constant density. 3 and thus (b -a )(ti. .3. 1 i9 i +s .. as can the algorithm. in that order.. l2 Problem 3.-. server i. i. All such schedules are clearly equivalent. .1. . the tape is rewound to the beginIn ning. ii.

adding at each step the job with the highest value of g. Our solution . The sequence 3. which in fact can only be done in the order 4. for instance. 2. To maximize our profit in this example.41. 3. is not considered because job 2 would be executed at time t = 2. 41. among those not yet considered. Finally we try (1.. Job i.3 2. It remains to prove that this algorithm always finds an optimal schedule and to find an efficient way of implementing it. d. 3 the schedules to consider and the corresponding profits are Sequence: 1 2 Profit: 50 10 3 15 4 30 1. we should execute the schedule 4. . An obvious greedy algorithm consists of constructing the schedule step by step.optimal in this case . . 1 80 . Happily this is not the case.3 45. A set of jobs is feasible if there exists at least one sequence (also called feasible) that allows all the jobs in the set to be executed in time for their respective deadlines. 41. provided that the chosen set of jobs remains feasible. 1 <_ i <_ n .2 Scheduling with Deadlines We have a set of n jobs to execute. In the preceding example we first choose job 1.. 41 is feasible because it can be executed in the order 4. 2. which turns out not to be feasible . after its deadline d2 = 1. we choose job 4: the set 11. therefore to execute the set of jobs 11. At any instant t = 1. so job 2 is also rejected. which is also infeasible .3 Greedy Algorithms for Scheduling 95 3. Let J be a set of k jobs.1 65 60 2. each of which takes unit time. we can execute exactly one job.3. we try the set 11. . 3. earns us a profit g. . 2.Sec. job 3 is therefore rejected. Next. 1. if and only if it is executed no later than time d. At first glance it seems we might have to try all the k ! possible permutations of these jobs to see whether J is feasible. 1. 1.optimum 4. with n = 4 and the following values : i 50 2 10 15 4 30 2 1 2 1 1 g.3 25 3. For example.1 65 4.

) S... one could substitute a for b in J and improve it. This is also impossible since it did not do so. The only remaining possibility is that some task a is scheduled in S. from right to left. . .2 for an example. (by the construction of a and the minimality of a) (by the definition of b). we can obtain two feasible sequences S. Finally. Proof. it is clear that b > a . each having at least one more position in agreement with a. > i . The result is a new feasible sequence. The job ra could therefore be executed later. For the "only if " : rk) such If J is feasible.. . . we obtain a series of feasible sequences. Since rb can certainly be executed earlier than planned. Continuing thus. and S are distinct sequences since I # J. Proof of optimality. which is therefore feasible. 1 <. whereas there is a gap in S. Suppose a # p. and S. the set I u { b } is feasible. Also dr. at the time when rb is at present scheduled. such that every job common to both I and J is scheduled for execution at the same time in the two sequences. and let a = (s 1s 2 . in order of increasing deadlines. to know whether a set of jobs is or is not feasible. we obtain a itself. = dr.i S k.. and S. .. Let J be a set of k jobs. and S. . See Figure 3.. this implies that a does not appear in J and that b does not appear in I.Greedy Algorithms 96 Chap. If some task b is scheduled in S.. which is the same as a at least in positions 1. a. This goes against the optimality of J. 2. We now prove that the greedy algorithm outlined earlier always finds an optimal schedule. . Let a be the smallest index such that Sa # rQ and let b be defined by rb = sQ . and S. whereas a different task b is scheduled in S. for the two sets of jobs in question. If some task a is scheduled in S. 11 This shows that it suffices to check a single sequence. Again. ? d. By making appropriate interchanges of jobs in S. there exists at least one sequence of jobs p = (r 1r2 that dr. sk) be a permutation of these jobs such that ds <. If gQ > gb . . The "if " is obvious.1. Then the set J is feasible if and only if the sequence a is feasible. Consider two feasible sequences S.3. (We may have to leave gaps in the schedule. whereas there is a gap in S.3. The necessary interchanges are easily found if we scan the two sequences S. (and therefore task a does not belong to J ).. This is not possible since J is optimal by assumption. .ds2 ds. 3 Lemma 3. after a maximum of k -1 steps of this type. hence the greedy algorithm should have included b in I. the set J u { a } is feasible and would be more profitable than J. Let us consider an arbitrary time when the task scheduled in Sj is different from that scheduled in S. Suppose that the greedy algorithm chooses to execute a set of jobs I whereas in fact the set J # I is optimal. we can interchange the items ra and rb in p.

. the same task. .2 to n do { in decreasing order of g I r-k while d [ j Ir ]] > max(d [i]. n ] d [0]. q V q w that one will be b common tasks Figure 3. j [0] f.Sec. To allow us to use sentinels.The only remaining possibility is therefore that go = gn In conclusion. 3. V q w Si after reorganization. the greedy algorithm should have chosen h before even considering a since (I \ { a }) u { h } would be feasible. suppose further that n > 0 and that di > 0.0 (sentinels) k.3.g . Rearranging schedules to bring identical tasks together. array[ 1 . and S. S" .3 Greedy Algorithms for Scheduling 97 .g 2 ? >.. or two distinct tasks yielding the same profit. This is not possible either since it did not include h in I.. if this task is a x Y P r u S P r S. < gl. 1 <_ i <.2.k] P Y q x r r s r P u S. sequences S. function sequence (d [0. This implies that the total worth of I is identical with that of the optimal set J. either schedule no tasks. For our first implementation of the algorithm suppose without loss of generality that the jobs are numbered so that g I >. k ] array j [0. n ]) : k. ..i k4-k+I return k. j [ l ] ..If g.n . and thus I is optimal as well. r) do r F. for each time slot. j[I .r - ifd[j[r]]<-d[i]andd[i]>r then for/ F-kstep-] tor+lldoj[I+1]E-j[l] j[r+I] .1 { task 1 is always chosen } {greedy loop I for i <-.

but no later than its deadline. these sets will merge to form larger sets . We obtain an algorithm whose essential steps are the following : I Positions of the same set i n.5. Verify that the algorithm works. For any position t. For a given set K of positions. max({ di 1 <_ i <_ n 1)).Greedy Algorithms 98 Chap. are unnecessary provided the jobs are correctly numbered in order of decreasing profit. disjoint set structures are intended for just this purpose. This obviously does not affect the feasibility of a set of jobs. where t is the largest integer such that 0 < t <_ min(n. = nj j Free position Occupied position Figure 3.3. then the set J is infeasible. define n. 11 The lemma suggests that we should consider an algorithm that tries to fill one by one the positions in a sequence of length l = min(n.2. define a fictitious position 0.3. = max{k <_ t I position k is free }. di) and the job to be executed at time t has not yet been decided. If a job cannot be executed in time for its deadline. the schedule can be compressed once all the jobs are included.3. In other words. . Problem 3. A set of jobs J is feasible if and only if we can construct a feasible sequence including all the jobs in J as follows : for each job i e J. Prove Lemma 3. It may happen that there are gaps in the schedule thus constructed when no job is to be executed. Also define certain sets of positions : two positions i and j are in the same set if ni = nj (see Figure 3.2. Sets of positions. Problem 3. Clearly. A second. execute i at time t. consider each job i e J :n turn.3. and add it to the schedule being built as late as possible. 3 The exact values of the g. and show that it requires quadratic time in the worst case. If we so wish.4. as we assign new jobs to vacant positions.3). more efficient algorithm is obtained if we use a different technique to verify whether a given set of jobs is feasible. Finally. Lemma 3.3. let F (K) be the smallest member of K. which is always free.3.3.

. Here is a more precise statement of the fast algorithm.i initialize set { i } {greedy loop } for i F.3. the value of F for this new set is the old value of F (L). reject the job . let this be set L (it is necessarily different from K) .5 illustrate the workings of the slow and fast algorithms. initialization : each position 0..0<-i <<-l. .find the set that contains F (K) ..0 F[i] . Consider a problem involving six jobs : i 1 2 3 4 5 6 gi 20 15 10 7 5 3 d. 1 ) { it remains to compress the solution } k -0 for i F.. F [0.1. . 1 is in a different set and F({i})=i. d ).1 to n do { in decreasing order of g } k . Example 3. . let this be set K .merge K and L .3. . To simplify the descrip- tion.Sec.find (min(n.3.. respectively.k] j[k] E. ii.4 and 3.1. 99 2.3 Greedy Algorithms for Scheduling i.0 to l do j [i] E. array[1 . function sequence 2(d [ i n ]) : k. 1 ] 1 -min(n. addition of a job with deadline d : find the set that contains min(n. j[l . 3 1 1 3 1 3.i 1 <--find (m -1) F[k] F-F[1] merge (k.max{d[i]I1<i <n1) { initialization) for i F. we have assumed that the label of the set produced by a merge operation is necessarily the label of one of the sets that were merged. 1. k ] array j.F [k] if m * 0 then j[m]F.. if F (K) = 0. d [i])) m .j[i] . ifF(K) 0.I to 1 do if j[i]>0then k-k+1 return k. Figures 3.assign the job to position F (K) . 3.

We content ourselves with giving two examples of this technique. which is essentially linear. most of the time will be spent manipulating disjoint sets. If the instance is given to us with the jobs already ordered by decreasing profit. 3. the jobs are given to us in arbitrary order.3. Since there are at most n+1 find operations and I merge operations to execute. Illustration of the slow algorithm. These examples also serve to illustrate that the greedy approach does not always yield an optimal solution. so that an optimal sequence can be obtained merely by calling the preceding algorithm.Greedy Algorithms 100 Chap. 1. and since n > 1. . value = 42 Figure 3. so that we have to begin by sorting them. 3 dU[il] 3 Initialization: Ad k 3 Try 2: 2 Try 3: unchanged 3 Try 4: 2 3 4 Try 5: unchanged Try 6: unchanged Optimal sequence: 2. the required time is in 0 (n lg*1).4 GREEDY HEURISTICS Because they are so simple. we need a time in 0 (n log n) to obtain the initial sequence. If. 4.4. on the other hand. greedy algorithms are often used as heuristics in situations where we can (or must) accept an approximate solution instead of an exact optimal solution.

4. 1. and blue for nodes 2 and 5. then they must be of different colours.5. no free position available Try 6: d6 = 3.4. A > be an undirected graph whose nodes are to be coloured. Our aim is to use as few different colours as possible. 3 and 4. painting it with this colour if possible. An obvious greedy algorithm consists of choosing a colour and an arbitrary starting node. 3. max(d)) = 3 F= 0 1 2 3 Try 1: d. If two nodes are joined by an edge.3. 3. no free position available since the Fvalue is 0 Try 4: d4 = 3. no free position available Optimal sequence: 2. When no further nodes can be painted. assign task 1 to position 3 F= 0 1 2 Fry 2: d2 = 1. Illustration of the fast algorithm.1 can be coloured using only two colours: red for nodes 1. For instance.4 Greedy Heuristics 101 Initialization: I = min(6. = 3. value = 42 Figure 3. assign task 2 to position 1 F= 0 2 Try 3: d3 = 1. 4. assign task 4 to position 2 F= 0 Try 5: d5 = 1. the graph in Figure 3. and then considering each other node in turn.Sec. we choose a new colour and a new .1 Colouring a Graph Let G = < N.

we paint as many nodes as we can with this second colour. we can colour nodes 2 and 5 and finish the job using only two colours . 3 A graph to be coloured. if we systematically consider the nodes in the order 1. 3. For a large-scale instance these algorithms cannot be used in practice. having travelled the shortest total distance possible. If we start again at node 2 using blue paint.1. this is an optimal solution. starting node that has not yet been painted. Let C (G) be the optimal (smallest) number of colours. (VG)(3(. R. 3.1. and lastly node 5 may not be painted. we are not allowed to paint node 2 with the same colour. The algorithm is therefore no more than a heuristic that may possibly. to visit each other town exactly once.Greedy Algorithms 102 Figure 3. This is an example of the NP-complete problems that we shall study in Chapter 10. terms of the graph colouring problem. Why should we be interested by such algorithms? For the colouring problem and many others the answer is that all the exact algorithms known require exponential computation time.4.)[c6(G)=c(G)]. The travelling salesperson wants to leave one of these towns. 2. in this case the result is not optimal. but it may also give an arbitrarily bad answer. then node 2 is painted blue. but not certainly. Prove the following assertions : i.2. be the number of colours used by the greedy algorithm. and to arrive back at the starting point. We . In other words.2 The Travelling Salesperson Problem We know the distances between a certain number of towns. However. Chap. In our example if node 1 is painted red. find a "good" solution. let c(G) Problem 3. the greedy heuristic may find the optimal solution. 5. and so on.4. nodes 3 and 4 can be red. Find two or three practical problems that can be expressed in Problem 3:4. (t/(XEIR+)(3G)(3o)[ca(G)1c(G) > a]. and we are forced to make do with an approximate method (of which this greedy heuristic is among the least effective).4. but now nodes 3 and 4 require us to use a third colour. For a graph G and an ordering a of the nodes of G. 4. we get a different answer: nodes I and 5 are painted red.

3. (Hint : start by constructing a minimal spanning tree and then use the answer to the previous problem. Edge (1. 5. ii. What happens to this greedy algorithm if the graph is not complete.5). In this instance the greedy algorithm does not find an optimal tour since the tour (1. Problem 3. which completes the salesperson's tour) . and also because it would have been the third edge incident on node 5.2). (4. (1. if chosen. 3. 6. a distance matrix is said to be Euclidean if the triangle inequality holds : for any towns i. all the known exact algorithms for this problem require exponential time (it is also NP-complete). and k. Hence they are impractical for large instances. 3.4.) One obvious greedy algorithm consists of choosing at each step the shortest remaining edge provided that it does not form a cycle with the edges already chosen (except for the very last edge chosen. if it is not possible to travel directly between certain pairs of towns ? Problem 3. 4. it is true that distance (i . if our problem concerns six towns with the following distance matrix : From 1 2 3 To : 2 3 4 5 6 3 10 11 7 25 6 12 8 26 9 4 20 5 15 4 5 18 edges are chosen in the order (1. In some instances it is possible to find a shorter optimal tour if the salesperson is allowed to pass through the same town several times. 2. 1). 1) has a total length of only 56. 5 ). As for the previous problem. (The graph can also be directed if the distance matrix is not symmetric: see Section 5. On the other hand.4.3). 6. Problem 3.4. was not kept when we looked at it because it would have completed a circuit (1.6. The problem can be represented using a complete undirected graph with n nodes.5. 5.6) to make the circuit (1. 5. Your algorithm must find a solution whose length is not more than double the length of an optimal tour. that is. 2.Sec. (3. 1) whose total length is 58. Give a heuristic greedy algorithm to solve the travelling salesperson problem in the case when the distance matrix is Euclidean. 2. (2.5). j) <_ distance (i .6). j.4 Greedy Heuristics 103 assume that the distance between two towns is never negative. Give an explicit example illustrating this.) . 3. Show that in this case it is never advantageous to pass through the same town several times. for example. 3. k) + distance (k . For example. (4. 4. i. it will not be the third chosen edge incident on some node.4. j ).

Other ideas concerning shortest paths can be found in Tarjan (1983).Greedy Algorithms 104 Chap.4. Problem 3. Try it ! Problem 3.5 REFERENCES AND FURTHER READING A discussion of topics connected with Problem 3.8. An important greedy algorithm that we have not discussed is used to derive optimal Huffman codes. in particular. Other more sophisticated algorithms are described in Yao (1975).13) is from Johnson (1975). The algorithm to which Prim's name is attached was invented by Jarnik (1930) and rediscovered by Prim (1957) and Dijkstra (1959). Prove that if a directed graph is complete (that is. The implementation of Dijkstra's algorithm that takes a time in 0 (n 2) is from Dijkstra (1959). The problem of minimal spanning trees has a long history.4.7. and give an algorithm for finding such a path in this case. see Schwartz (1964).5 of this book. There exists a greedy algorithm (which perhaps would better be called "abstinent") for solving the problem of the knight's tour on a chessboard : at each step move the knight to the square that threatens the least possible number of squares not yet visited.2. but without coming back to the starting node. Kruskal's algorithm comes from Kruskal (1956). the same reference gives an efficient heuristic for finding a solution to the travelling salesperson problem with a Euclidean distance matrix that is not more than 50% longer than the optimal tour.4.5 is given in Christofides (1976) . and Tarjan (1983). use of the Fibonacci heap allows them to implement Dijkstra's algorithm in a time in 0 (a + n log n ). The first algorithm proposed (which we have not described) is due to Boruvka (1926). if each pair of nodes is joined in at least one direction) then it has a Hamiltonian path.12 can be found in Johnson (1977). Faster algorithms for both these problems are given in Fredman and Tarjan (1984). .1 can be found in Wright (1975) and Chang and Korsh (1976) . 3. In a directed graph a path is said to be Hamiltonian if it passes exactly once through each node of the graph. The details of the improvement suggested in Problem 3.2. 3 Invent a heuristic greedy algorithm for the case when the disProblem 3.1. Cheriton and Tarjan (1976). The solution to Problem 3. tance matrix is not symmetric.8.4.6. Other greedy algorithms for a variety of problems are described in Horowitz and Sahni (1978). see also Problem 5. which is discussed in Graham and Hell (1985). Similar improvement for the minimal spanning tree problem (Problem 3.

Let d be a constant such that the time needed to carry out the decomposition and the recombination is t(n) <_ dn. Suppose you already have some algorithm A that requires quadratic time. By using both your old algorithm and your new idea. solving these subinstances. Although this improvement is not to be sneezed at. you obtain a new algorithm B whose implementation takes time tB(n)=3tA([n/21)+t(n)<_3c((n+l)/2)2+dn =4cn2+(4c+d)n+4c The term 4cn 2 dominates the others when n is sufficiently large. The efficiency of the divide-and-conquer technique lies in the answer to this question.1 INTRODUCTION Divide-and-conquer is a technique for designing algorithms that consists of decomposing the instance to be solved into a number of smaller subinstances of the same problem. Let c be a constant such that your particular implementation requires a time tA (n) <_ cn2 to solve an instance of size n.4 Divide-and-Conquer 4. you have not managed to change the order of the time required : algorithm B still takes quadratic time. solving successively and independently each of these subinstances. nevertheless. 105 . You discover that it would be possible to solve such an instance by decomposing it into three subinstances each of size Fn/21. and combining the results. which means that algorithm B is essentially 25% faster than algorithm A. The first question that springs to mind is naturally "How should we solve the subinstances ?". and then combining the subsolutions thus obtained in such a way as to obtain the solution of the original instance.

When implemented in a conventional language such as Pascal on a conventional . The number of subinstances. the decision when to use the basic subalgorithm rather than to make recursive calls must be taken judiciously. x 2.l tokdoy1 E-DQ(x.) recombine the y. when the subinstances are sufficiently large. 's to obtain a solution y for x return y . Although this choice does not affect the order of the execution time of our algorithm. xk fori . is usually both small and also independent of the particular instance to be solved.10. the more this improvement is worth having.59 The improvement compared to the order of n 2 is therefore quite substantial. the basic subalgorithm. After looking at the question of how to choose the optimal threshold. and 4. and in this case it goes by the name of simplification (see sections 4. We shall see in the following section how to choose no in practice.10). might it not be better to use our new algorithm recursively? The idea is analogous to profiting from a bank account that compounds interest payments ! We thus obtain a third algorithm C whose implementation runs in time tc (n) _ tA(n) ifn <no 3tc( [ n/2 1 )+t(n) otherwise where no is the threshold above which the algorithm is called recursively. k. Here then is the general outline of the divide-and-conquer method: function DQ (x) { returns a solution to instance x ) if x is sufficiently small or simple then return ADHOC (x) decompose x into smaller subinstances x 1. We shall see that it is sometimes possible to replace the recursivity inherent in divide-and-conquer by an iterative loop. it is hard to justify calling the technique divide-and-conquer.6). . this chapter shows how divide-and-conquer is used to solve a variety of important problems and how the resulting algorithms can be analysed. 4. where ADHOC. We should also mention that some divide-and-conquer algorithms do not follow the preceding outline exactly. it is possible that algorithm A may still be the best way to proceed. which is similar to the one in Example 2.3.8. is used to solve small instances of the problem in question. For this approach to be worthwhile a number of conditions are usually required : it must be possible to decompose an instance into subinstances and to recombine the subsolutions fairly efficiently.106 Divide-and-Conquer Chap. 4 To do better than this.3. This equation. and the bigger n is. and the subinstances should be as far as possible of about the same size. but instead. we are also concerned to make the hidden constant that multiplies nlg3 as small as possible. we come back to the question posed in the opening paragraph : how should the subinstances be solved? If they are small. When k = 1. However. they require that the first subinstance be solved even before the second subinstance is formulated (Section 4. gives us a time in the order of ntg3 which is approximately n 1.

Must we conclude that divide-and-conquer allows us to go from a quadratic algorithm to an algorithm whose execution time is in 0(n 1g 3 ). Choosing the threshold . To determine the value of the threshold no that minimizes tc (n). 4. In this case. it is better to apply the basic subalgorithm.2 Determining the Threshold 107 machine. if no = 1.1. it may be possible to save a substantial amount of memory space in this way : for an instance of size n. that is. Find all the values of the threshold that allow an instance of 0 size 1024 to be solved in less than 8 minutes. it takes more than half an hour to solve this instance. 4. it is not sufficient to know that to (n) E O(n 2) and that t (n) E O(n).Sec. consider once again algorithm C from the previous section. If the algorithm proceeds recursively until it obtains subinstances of size 1. On the other hand. the answer is no : in our example. Problem 4.2. This is ridiculous. To illustrate this. consider an implementation for which the values of to (n) and t(n) are given respectively by n2 and 16n milliseconds. since the instance can be solved in little more than a quarter of an hour by using the basic subalgorithm directly. and t (n) is the time taken to do the decomposition and recombination.2. For instance. Problem 4. This example shows that the choice of threshold can have a considerable influence on the efficiency of a divide-and-conquer algorithm.2 DETERMINING THE THRESHOLD An algorithm derived by divide-and-conquer must avoid proceeding recursively when the size of the subinstances no longer justifies this. Prove that if we set no= 2k for some given integer k >_ 0. where to (n) is the time required by the basic subalgorithm. by setting no = oo. provided we choose the threshold no intelligently. although only by a constant multiplicative factor. Suppose we have an instance of size 1024 to solve.2. whose execution time is given by tc (n) _ ifn <no f tA(n) 3tc ([n / 21) + t (n) otherwise. the instance of size 1024 can be solved in less than 8 minutes. then for all 1 >_ k the implementation considered previously takes 2k 31-k (32+2k) -21 +5 milliseconds to solve an instance of size 21. but only at the cost of an increase in the hidden constant so enormous that the new algorithm is never economic on instances that can be solved in a reasonable time? Fortunately. the recursive algorithm uses a stack whose depth is often in 1 (log n) and in bad cases even in S2(n). an iterative algorithm is likely to be somewhat faster than the recursive version. that is.

It is often possible to estimate an optimal threshold simply by tabulating the results of these tests or by drawing a few diagrams. On the other hand. Finally.1. Moreover. changes in the value of the threshold may have no effect on the efficiency of the algorithm when only instances of some specific size are considered.6. This remark may appear trivial.000 (1982) Canadian dollars worth of machine time ! On the other hand.8 makes clear.7. Coming back to our example. as Problem 4. that it is not usually enough simply to vary the threshold for an instance whose size remains fixed. but also on the particular implementation. We vary the value of the threshold and the size of the instances used for our tests and time the implementation on a number of cases. if we systematically replace In / 21 by (n + 1)/2. We shall in future abuse the term "optimal threshold" to mean nearly optimal. the optimal threshold can be determined empirically. .6 describes an algorithm for which the ultimate threshold is less obvious. The optimal threshold can then be estimated by finding the value of n at which it makes no difference. 4 is complicated by the fact that the best value does not generally depend only on the algorithm concerned. Obviously. we find n = 70. but Section 4. over a certain range. consists of determining theoretically the form of the recurrence equations. Several groups of students tried to estimate the optimal threshold empirically. the preceding problem shows that.2. each group using in the attempt more than 5. whether we apply the basic subalgorithm directly or whether we go on for one more level of recursion. which we recommend.108 Divide-and-Conquer Chap. So how shall we choose no? One easy condition is that we must have no > 1 to avoid the infinite recursion that results if the solution of an instance of size 1 requires us first to solve a few other instances of the same size.2 makes it clear. Problem 4. since we saw in Problem 4. There is nothing surprising in this. This approach may require considerable amounts of computer time. corresponding to the fact that the average value of Fn / 21 is (2n + 1)/4.2. and then finding empirically the values of the constants used in these equations for the implementation at hand. in order to compare it with the classic algorithm from Section 1. given that it varies from one implementation to another. If we neglect this difficulty. there is in general no uniformly best value of the threshold : in our example. We once asked the students in an algorithmics course to implement the algorithm for multiplying large integers given in Section 4. we must avoid thresholds below the ultimate threshold. The hybrid approach. whereas it is best to use a threshold between 33 and 65 for instances of size 66. A reasonable compromise. for an instance of size n. Given a particular implementation. The presence of a ceiling in this equation complicates things. we obtain n = 64. a purely theoretical calculation of the optimal threshold is rarely possible. the optimal threshold can be found by solving to (n) = 3tA ([n / 21) + t (n). a threshold larger than 66 is optimal for instances of size 67. is to choose n o = 67 for our threshold. because tc ([n / 21) = to ([n / 21) if [n / 21 <_ no.2 that in fact no uniformly optimal threshold exists. however.

Although bn + c becomes negligible compared to an 2 when n is large.2. If the item we are looking for is not in the array.) In practice. Show that this choice of no = 67 has the merit of being suboptimal for only two values of n in the neighbourhood of the threshold. 1 < j 5 n = T [i] <. then instead we want to find the position where it might be inserted. (Notice that this would not cause an infinite recursion because the threshold is strictly larger than zero. it is the algorithm used to look up a word in a dictionary or a name in a telephone directory. that is.2. we wish to find the index i such that 0 <_ i <_ n and T [i] S x < T [i + I]. one more complication arises.3. 4.) The obvious . with the logical convention that T [0] _ . Furthermore. Formally. In essence. Supposing. for instance. probably using a regression technique. (For purists : even if the domain of f. (By logical convention. prove that there are no instances that take more than I % longer with threshold 67 than they would with any other threshold.T [ j ]. n] be an array sorted into increasing order. 4. if instances of fractional size were allowed.) Problem 4. we mean that these values are not in fact present as sentinels in the array.3 BINARY SEARCHING Binary searching predates computers. measure to (n) a number of times for several different values of n . that to (n) is quadratic. Let T [I . and let x be some item. consider the function f5 : IR* -. the problem can be solved without recourse to transfinite induction. and then estimate all the necessary constants. The problem consists of finding x in the array T if indeed it is there. and c depending on the implementation. it may happen that tA (n) = an 2 + bn + c for some constants a. then fu (x) 5 f. that is.1R* defined by the recurrence x2 f(x)= if X <_ s 3fs (x / 2) + bx otherwise.3 Binary Searching 109 * Problem 4.oo and T [n + 1] _ + oo.4. and f is not countable. Instead. the basic subalgorithm is used in fact precisely on instances of moderate size. Notice that this u is chosen so that au 2 = 3a (u / 2)2 + bu . Prove by mathematical induction that if u = 4b /a and if v is an arbitrary positive real number. precisely because infinite recursion is not a worry.. (x) for every real number x. For each positive real number s. The following problem shows that a threshold of 64 would be optimal were it always possible to decompose an instance of size n into three subinstances exactly of size n / 2.Sec. Let a and b be real positive constants. b. It is therefore usually insufficient merely to estimate the constant a. It is probably the simplest application of divide-and-conquer.

where r is the index returned : this is S2(n) in the worst case and O (1) in the best case..1. We obtain the following algorithm. j ]. x) function binrec (T [i .u < j . Conclude from these two results that a call on binsearch always terminates.I to n do if T[i] >x then return i -1 return n This algorithm clearly takes a time in O(1 + r). .. x) [ binary search for x in subarray T [i . otherwise it is sufficient to search T [1+Ln / 2j . then the average number of trips round the loop is (n 2+ 3n -2)/2n.. therefore.i. On the average..x) Prove that the function binrec is never called on T [i . divide-and-conquer suggests that we should look for x either in the first half of the array or in the second half.2. and that it is to be found with equal probability at each possible position.. function sequential (T [ 1 . x) f binary search for x in array T [ if n = 0 or x < T [1] then return 0 return binrec (T. that x is indeed somewhere in the array.. If we assume that all the elements of T are distinct. Show finally that the values T [0] Problem 4. n ]. it is easy to produce an iterative version... with j < i . To speed up the search..3. x) [ sequential search for x in array T [ for i F. v ]. x) makes a recursive call binrec (T [u . we compare x to an element in the middle of the array : if x < T [ 1+ Ln / 2j]. it is always true that v . Because the recursive call is situated dynamically at the very end of the algorithm. k -1 ]. and T [n + 1] are never used (except in the comments ! ). Show that the algorithm takes a time in 0( log n) to find x in T [1 . To find out which of these searches is appropriate. n J. [n12]]. as well as in the worst case. Problem 4. 4 approach to this problem is to look sequentially at each element of T until we either come to the end of the array or find an item bigger than x.. The algorithm in fact executes only one of the two recursive calls.. sequential search takes a time in O(n).. j ].110 Divide-and-Conquer Chap.3. x). n ] whatever the position of x in T. j ] Prove too that when binrec (T [i . then the search for x can be confined to T [I . x) else return binrec (T [k . j this procedure is only called if T[i] S x < T [j + 1] and i j[ if i = j then return i k -(i+j+1)div2 if x < T [k] then return binrec (T [i . so that technically it is an example of simplification rather than of divide-and-conquer. function binsearch (T [ 1 . n ]. j ].

Show by examples that the preceding algorithm would be incorrect if we replaced i. n ].3. On the second trip round the loop i = 9.n while i < j do ( T [ i ] Sx <T [j+1] } k <-. "i k" by " i E--k+l".k+1 otherwise : i .(i+j+1)div2 if x < T [k] then j <-. binary searching. 11 A first inspection of this algorithm shows what is apparently an inefficiency. and k = 13. function iterbin 2(T [1 . ii. 4. This comparison could allow us to end the search immediately. The comparison between x and T [9] causes the assignment i . The following algorithm leaves the loop immediately after we find the element we are looking for.k -1" by " j <-. and k = 9.k return i It is easy to go wrong when programming the concept of Problem 4. "k F(i+j+1)div2" by " k <-. regardless of the position of x in T. j = 17. and so the assignment i <.k ". simple though this is. or iii.Sec. Suppose T contains 17 distinct elements and that x = T [13].n while i < j do (T[i] <-x <T [j+1]} k <-(i+j)div2 case x< T [k] : k-1 x?T[k+1]:i -. On the first trip round the loop. i = 1.9 to be executed.l.j'. while it is possible that the variant will only make one or two trips round the loop if x is favourably .k return i Which of these algorithms is better ? The first systematically makes a number of trips round the loop in O(log n). "j F.x) ( variant on iterative binary search } if n = 0 or x < T [1] then return 0 iE.x) ( iterative binary search for x in array T } if n = 0 or x < T [1] then return 0 iE--1.. j <-.k-1 else i <. j = 17.3. n].3 Binary Searching 111 function iterbin (T [1 . but no test is made for equality.13 is carried out.(i+j)div2".j<-.. Two more trips round the loop are necessary before we leave with i = j = 13. A comparison is made between x and T [13].

a (n) <. after which the algorithm starts over on an instance reduced to k -1 elements. respectively. n ?2 a(1)=0. Suppose to make life simpler that T contains n distinct elements and that x is indeed somewhere in T. Analysis of the Second Version. Define a (n) and b (n) as n A (n) and n B (n). which causes the assignment j . With probability I .B(Ln/2j). x > T [k]. Let A (n) and B (n) be the average number of trips round the loop made by the first and the second iterative versions. To compare them. already shown to hold when n is a power of 2.(k . respectively. we obtain the recurrence B(n)= 1+ Fn/ 21 . In a similar way. is that n Llg n j <. A likely hypothesis. Analysis of the First Version. b(2)=2.3. Llg n] = lg n *. a trip round the loop in the variant will take a little longer to execute on the average than a trip round the loop in the first algorithm.n Fig n 1.k. which yields a (n) = n lg n using the techniques of Section 2. which causes the assignment i E. The equations then become a(n)=n +a(Ln/2])+a([n/2l). (*) The first equation is easy in the case when n is a power of 2. B(2)=I. guessing the likely form of the answer and determining the missing parameters in the course of a tentative proof by mathematical induction. taking k = In / 21. With probability (k -1)/n .k . so the average number of trips round the loop is given by the recurrence A(n)=1+ Ln/2]A(Ln/2])+ rn/21A(Ln/21). One trip round the loop is carried out before the algorithm starts over. since it then reduces to a(n)=2a(n/2)+n.1)/n. Exact analysis for arbitrary n is harder.Divide-and-Conquer 112 Chap. after which the algorithm starts over on an instance reduced to n -k + 1 elements. occupying each possible position with equal probability. In particular.1. Let k = 1 + Ln / 2j. n >-3 B(1)=0. We proceed by constructive induction.IB(Fn/21-1)+ Ln/2. n >-2 b(n)=n +b([n/21-1)+b(Ln/2j). On the other hand. It seems reasonable to hope that . n >-3 a(1)=b(1)=0. we shall analyse exactly the average number of trips round the loop that each version makes. 4 situated. n?2 A(1)=0. What might we add to n Llg n j to arrive at a (n) ? Let n * denote the largest power of 2 that is less than or equal to n. x < T [k].

Denote this hypothesis by HI (n). (Ln / 2] )* =(n+1)14 and Fn/21 = (rn / 21)* = n * = (n + 1)/2.Sec. then (Ln / 2j)* _ (rn / 21)* = n */ 2. 2 4 2 2 2 2 4 2 that is 4c + 2d = 4c + 3d + 2 and 2d = 3d+2. When n > 1 is of the form 2'-1. c and d being still unconstrained. that is A (n) = Llg n] + 2(1.3 113 Binary Searching a (n) = n lg n * + cn + do * + e lg n * +f for appropriate constants c.2. e. our initial guess holds : Llg nJ <.2n * true for the base n = 1. is given by A (n) = a (n)l n . 4. then we shall have proved by mathematical induction that it is true for every positive integer n. To prove HI(n) in this case. The average number of trips round the loop executed by the first iterative algorithm for binary searching. when looking for an element that is in fact present with uniform probability distribution among the n different elements of an array sorted into increasing order. To prove HI(n) in this case. If our hypothesis is correct.n */ n) In particular. there still being no constraints on c. it is thus necessary and sufficient that n lgn*+cn +dn*+e lgn*+f = n lgn*+cn +dn*+2e lgn*+(2f -2e) that is. we therefore know a (n) = n lg n * + cn + do * . we need e = 0 and f = 0. HI (Ln / 2]) and HI (rn / 2 1 ).A (n) <. d.rig n I . using the recurrence (*). At this point we know that if only we can make the hypothesis a (n) = n lg n * + cn . which gives c = 2 and implies that the general solution of the recurrence (*) for a (n) is a(n)=nlgn*+2(n-n*). Our final constraint is therefore 0=a(1)=c -2. These two equations are not linearly independent. . and f . They allow us to conclude that d = . then Ln / 2] _ (n -1)/2. When n > 1 is not of the form 2'-1. it is necessary and sufficient that n1gn+1 +(c+d)n+d = nlgn+l +(c+3d+!)n+(3d+ i).

n(n) <. solve equation (*) exactly for b (n) when n is of the form 2' -1. Nonetheless. The general solution of the recurrence for b (n) is more difficult than the one we have just obtained. * Problem 4. which can easily be solved to give the same c. hypothesis was wrong. Given that the first algorithm takes less time on the average than the variant to execute one trip round the loop. our efforts were not entirely in vain. and such that the exact solution of the recurrence is b (n) = n lg n * + n .5. A simple modification of the argument allows us to conclude that nlgn*+n/2-2n*+lgn*+3/2 <. 1N+ such that Show that there exists a function it : IN+ (n + 1)/3 <.6.3. Explain why this is insufficient. The problem arises because the two basic cases are incompatible in the sense that it is impossible to obtain them both from the recurrence starting from some artificial definition of b (0). e . Equivalently lt(n -1) = [n * + On / n *j -2)(2n -3n*)] / 2 We are finally in a position to answer the initial question : which of the two algorithms for binary searching is preferable? By combining the preceding analysis of the function a (n) with the solution to Problem 4.2n* + lg n * + (1 +c). Problem 4.(n + 1)/2 for every positive integer n. Stated more elegantly : * Problem 4. and 4 for n in the hypothesis HI (n). Using the techniques presented in Section 2. The hypothesis therefore becomes b (n) = n lg n * + cn . thus obtaining four linear equations in four unknowns. and f .3.b(n) <. b (1) = 0 = c = 2 . d.6.7. Unfortunately.3. 3. d. 2. and f .Divide-and-Conquer 114 Chap. Show that the function n(n) of the previous exercise is given by lt(n -1) = jl n */ 2 if 2n < 3n * n-n * otherwise for all n > 2. A seemingly simpler approach to determine the constants would be to substitute the values 1.4. we obtain A(n)-B(n)=1+ lt(n)-LngnJ-2 3 Thus we see that the first algorithm makes on the average less than one and a half trips round the loop more than the second.2n * + lg n * + 2 .n(n). is whereas b (2) = 2 which inconsistent and shows that the original c=3. The case when n ? 4 is a power of 2 obliges us to choose d = . we conclude that the . Constructive induction yields e = 1 and f = 1 + c to take account of the case when n >. e. 4 Problem 4. It seems reasonable to formulate the same incompletely specified induction hypothesis : b (n) = n lg n * + cn + do * + e lg n * + f for some new con- stants c.3.3 is not a power of 2.3.nlgn*+2n/3-2n*+lgn*+5/3.2.3.

2.. and your algorithm must work in linear time. On the other hand.. V [ 1 ..4 and Problem 2. n ] be an array of n elements for which there exists a total ordering. Problem 4. n] mergesort (U) . a relatively simple algorithm is used..3). 4. but without using an auxiliary array : the sections T [I . where insert (T) is the algorithm for sorting by insertion from Section 1. n div 2].1.. and merge (T.. or by heapsort (Example 2. .V). Repeat the previous problem. n div 2] V -T[1+(n div 2) . The situation is similar if the element we are looking for is not in fact in the array. that is.4). n ] of an array are sorted independently.4 SORTING BY MERGING Let T [I . Give an algorithm capable of merging two sorted arrays U and V in linear time. the threshold beyond which the first algorithm is preferable to the variant can be very high for some implementations. This sorting algorithm is a good illustration of all the facets of divide-andconquer. whereas both the former methods take quadratic time.4 Sorting by Merging 115 first algorithm is more efficient than the second on the average whenever n is sufficiently large. (n + 1) div 2] U F T[ 1 . when this is justified by the number of elements. 4. You may only use a fixed number of working variables to solve the problem. in a time in the exact order of the sum of the lengths of U and V.2. being careful to preserve the order.4. We obtain the following algorithm : procedure mergesort (T [ 1 . and then merging the solutions for each part. We are interested in the problem of sorting these elements into ascending order. n ]) { sorts array T into increasing order } if n is small then insert (T) else arrays U [ 1 . V) merges into a single sorted array T two arrays U and V that are already sorted.4... We have already seen that the problem can be solved by selection sorting and insertion sorting (Section 1. sorting these parts by recursive calls. n I.2. k ] and T [k + 1 ..U. However. Recall that an analysis both in the worst case and on the average shows that the latter method takes a time in O(n log n).Sec. ** Problem 4. U.4. and you wish to sort the whole array T [I . When the number of elements to be sorted is small. mergesort (V ) merge (T. The obvious divide-and-conquer approach to this problem consists of separating the array T into two parts whose sizes are as nearly equal as possible.

1) + t'(1) + O(n). as we shall see in several subsequent examples. L(n + 1)/3j.6. Separating T into U and V takes linear time. Simply forgetting to balance the sizes of the subinstances can therefore be disastrous for the efficiency of an algorithm obtained using divide-and-conquer. to sort each of these recursively. and why? 4. t(n) E t(Ln / 2j) + t (Fn / 21) + 0(n). Rather than separate T into two half-size arrays. we might choose to separate T into about LI arrays. we might choose to separate it into three arrays of size Ln/3]. Unlike sorting by merging.4. We obtain t'(n) E t'(n .5 QUICKSORT The sorting algorithm invented by Hoare. As a first step.5.4.1. Show that t'(n) E O(n 2). respectively.4. allows us to conclude that the time required by the algorithm for sorting by merging is in O(n log n).3. This equation. Which one. To see why. Let t'(n) be the time required by this variant to sort n items. which we analysed in Section 2. This poor sorting algorithm is very like one we have already seen in this book. Let t(n) be the time taken by this algorithm to sort an array of n elements. each containing approximately L' elements. Consequently. have two points in common. 4 mergesort separates the instance into two subinstances half the size. usually known as "quicksort".4.1. The merge sorting algorithm we gave. the nonrecursive part of the work to be done is spent constructing the subinstances rather than combining their solutions. and those suggested by the two previous problems. the fact that the original instance is divided into subinstances whose sizes are as nearly as possible equal is crucial if we are to arrive at an efficient algorithm. this algorithm chooses one of the items in the array to be sorted as the pivot. Problem 4. the final merge also takes linear time. solves each of these recursively. By the result of Problem 4. look at what happens if instead we decide to separate T into an array U with n -1 elements and an array V containing only 1 element. Following up the previous problem. and analyse its performance.6. and analyse its execution time. and then to merge the three sorted arrays.4. Problem 4. ** Problem 4. Develop this idea. The array is then partitioned on either side of the pivot : elements .116 Divide-and-Conquer Chap. and then combines the two sorted half-arrays to obtain the solution to the original instance. Problem 4. and L(n + 2)/3]. Give a more formal description of this algorithm. The fact that the sum of the sizes of the subinstances is equal to the size of the original instance is not typical of algorithms derived using divide-and-conquer. On the other hand.4. is also based on the idea of divide-and-conquer.

procedure pivot (T [i .. One good way of pivoting consists of scanning the array T [i . no subsequent merge step being necessary.1 until T [1] <. Pointers k and 1 are initialized to i and j + 1. Here is the algorithm. j ] just once. might arise. the final result is a completely sorted array. T [i] and T [1] are interchanged to put the pivot in its correct position. . j ] . however. crucial in practice that the hidden constant be small.i is small then insert (T [i j ]) { Section 1.j.Sec. Let p = T [i ] be the pivot..k + 1 until T [k] > p repeat 1 f.j repeat I E. finding the median takes more time than it is worth.. I . j ]. and simulate the pivoting algorithm on these examples. j ]) = T[k]>T[1]} Designing a linear time pivoting algorithm is no challenge.5 117 Quicksort are moved in such a way that those greater than the pivot are placed on its right. This process continues as long as k < 1.k < 1 1 <k 5j and quicksort(T [i .) Unfortunately. j ] into increasing order } if j . var 1) { permutes the elements in array T [i .p.k + 1 until T [k] > p or k >. the elements of T [i . T[11 =p.1] and T [1 + 1 . respectively. j ] in such a way that..1. Now T [k] and T [1] are interchanged. j ] are greater than p.p while k < I do interchange T [k] and T [1] repeat k F. Pointer k is then incremented until T [k] >p. and pointer I is decremented until T [1] <. and the elements of T J1+1 . If now the two sections of the array on either side of the pivot are sorted independently by recursive calls of the algorithm. whereas all the others are moved to its left. 1-1] are not greater than p. j ] to be sorted are severely unbalanced..1) T [k] 5 T [l] { after pivoting.1<-j+1 repeat k.. It is..6. i <. (For a definition of the median. Quicksort is inefficient if it happens systematically on most recursive calls that the subinstances T [i .. i <. see Section 4.. we would like to use the median element as the pivot.l <.4 } else pivot (T [i ...1 until T [1] p interchange T [i] and T [1] Invent several examples representing different situations that Problem 4. j ]) { sorts array T [i . procedure quicksort (T [i . For this reason we simply use the first element of the array as the pivot. 4. at the end.. Finally.1. but starting at both ends. where p is the initial value of T [i ] } p -T[i] kF-i. I -1]) quicksort (T [1 +1 .1.5. To balance the sizes of the two subinstances to be sorted.

taking f (x) = x lg x. 4 Problem 4.) Let a and b be real numbers. if the array to be sorted is initially in random order. j = no and k = n . nevertheless. Show that in the worst case quicksort requires quadratic time. The pivot chosen by the algorithm is situated with equal probability in any position with respect to the other elements of T. The value of I returned by the pivoting algorithm after the initial call pivot (T [I . a < b . let d and no be two constants such that n-1 t (n) <.IR be a nondecreasing function. and let f : [a. To use this approach we need an upper bound on i =no+ 1 i lg + n t (k) for n > n 0 k=0 An equation of this type is more difficult to analyse than the linear recurrences we saw in Section 2. Consequently. Let j and k be two integers such that a <. It remains to sort recursively two subarrays of size 1 -1 and n -1. each value having probability 11n. we obtain n-1 E i lg i i=no+1 n <- f x lg x dx x=no+1 . we assume that all the elements of T are distinct and that each of the n! possible initial permutations of the elements has the same probability of occurring. it is. a + m ]) for 0 <. Give an explicit example of an array to be sorted that causes such behaviour.1) can therefore be any integer between 1 and n. respectively. To determine the average time required by quicksort to sort an array of n items. n ]. reasonable to hope that t(n) will be in 0 (n log n) and to apply constructive induction to look for a constant c such that t (n) S cn lg n . This pivoting operation takes a time in O(n).n -m.. (t(1-1) + t(n -1)) . On the other hand.3. b ] . By analogy with sorting by merging. This is obtained with the help of a simple lemma. Then k-1 k f(i) < J f(x) dx .j < k <. t(n) E O(n) + n 1=1 A little manipulation yields t(n)eO(n)+ n n-1 t(k) k =O To make this more explicit.b. The average time required to execute these recursive calls is t(I -1) + t(n -1). it is likely that most of the time the subinstances to be sorted will be sufficiently well balanced.m <..2.n and 0:5 a <. Let t (m) be the average time taken by a call on quicksort (T [a + 1 . In particular. (We suggest you find a graphical interpretation of the lemma.Divide-and-Conquer 118 Chap.

we can obtain a version of quicksort that takes a time in 0 (n log n) even in the worst case. Show by a simple argument that.Igex2 4 2 n . . k=o Quicksort can therefore sort an array of n distinct elements in an average time in O (n log n). where c= 2d lge + proof by 4 (no+l)2lge mathematical induction that t (k) .5. Intuitively. T[(i+j)div21 andT[j]. quicksort as described here always takes a time in S2(n2) in the worst case. Problem 4. this is the sorting algorithm to be preferred among all those presented in this book. The probability of suffering an execution time in U(n 2) can be greatly diminished.5. 4. n ] I T [ i ] < m f >_n/2.6 SELECTION AND THE MEDIAN Let T [I . . the median of T is that element m in T such that there are as many items in T smaller than m as there are items larger than m.4. at the price of a small increase in the hidden constant. The hidden constant is in practice smaller than those involved in heapsort or in merge sort. What could be easier than to find the smallest element or to calculate the mean of all the elements ? However. Outline a modification to the algorithm to avoid this. or that the elements of T may not all be distinct. We mention this possibility only to point out that it should be shunned : the hidden constant associated with the "improved" version of quicksort is so large that it results in an algorithm worse than heapsort in every case. by choosing as pivot the median of T [i]. By combining the modification hinted at in the previous problem with the linear algorithm from the following section.Ige n2 provided no > 1. it is not obvious that the median can be found so easily.x=no+1 2 < 2 lgn .Sec. 4.3. The formal definition takes care of the possibility that n may be even. . t (n) <. Complete the Problem 4. Thus we define m to be the median of T if and only if m is in T and # I i E [ I . n ] I T [ i ] < m } <n/2 and # { i E [ I .6 Selection and the Median 119 [-lx lg n for all n > n o >_ 1. whatever the choice of pivot. n ] be an array of integers. If an occasional long execution time can be tolerated.

n ] I T [i] < p } v #{i a [1.the elements of T larger than p { the k th smallest element of T is also the (k-v )th smallest of V } return selection(V. function selection(T [ 1 . n] element m such that I T [i ] <.1. Problem 4.. Generalize the notion of pivoting from Section 4. and no auxiliary arrays should be used. solves the selection problem in a way suggested by quicksort. Do not use any auxiliary arrays.. u ] U F the elements of T smaller than p { the k th smallest element of T is also the k th smallest element of U } return selection(U. this algorithm assumes that 1 k <_ n } if n is small then sort T return T RI p .m } >_ k .2. .. it is the k th item in T if the array is sorted into ascending order.6. k) { finds the k th smallest element of T . n ]... n ]. Using ideas from the iterative version of binary searching seen in Section 4..6.. Can we do better? To answer this question. 4 The naive algorithm for determining the median of T consists of sorting the array into ascending order and then extracting the In / 21th entry. n ] { to be specified later) u . In other words.# { i e [l . k -v ) Problem 4. the median of T is its In / 21th smallest element. The kth # { i e [ l . j]. Your algorithm should scan T once only. which is not yet completely specified. The values i and j should be returned by the pivoting procedure. k) if k <..3 and the pivoting procedure of the previous problem.n]IT[i]Sp } if k S u then array U[1 .Divide-and-Conquer 120 Chap. T [i . equal to p. Your algorithm is allowed to alter the initial order of the elements of T.v then { got it ! } return p otherwise { k > v } array V[1 . not calculated beforehand. n -v ] V . and T [j + 1 . respectively. n i smallest element of T is that l T[i] <m } <k..some element of T [I .5 to partition the array T into three sections. containing the elements of T that are smaller than p. this algorithm takes a time in 0 (n log n) to determine the median of n ele- ments. For instance.. we consider a more general problem : selection. and let k be an integer between 1 and n. Let T be an array of n elements. give a nonrecursive version of the selection algorithm. and greater than p. If we use heapsort or merge sort. i -1 ]. T [I .. The following algorithm. whereas # { i e [ I .

(The hidden constant must not depend on k. we can once again borrow an idea from quicksort and choose simply p F-T[1].. n . When we do this. whatever the value of k. Let tm (n) be the time required by this method in the worst case to find the k th smallest element of an array of at most n elements.4. Consequently. which causes a recursive call on n -1 elements.6.3.selection (T.e average. (n + 1) div 2) "? Suppose first that the median can be obtained by magic at unit cost. still supposing that the median can be obtained magically. and k > 1. notice first that. therefore. taking the size to be zero if there is no recursive call. . by definition of the median.v <. not by divide-and-conquer. Assume the n elements of the array T are distinct and that each of the n! permutations of the elements has the same probability of occurring. We have tm(n)E0(n)+max{ tm(i) I i 5 Ln/2] }. Let E (n . u < [n / 21 and v ? [n / 21. If there is a recursive call.6. n ]. * Problem 4. even if occasionally we meet an instance where u = 0. we hope that on the average the sizes of the arrays U and V will not be too unbalanced. Show that tm (n) is in O (n). For the time being. consider the selection algorithm obtained by choosing "p f.Sec.Ln / 2]. Problem 4. so that the sizes of the arrays U and V will be as similar as possible (even if at most one of these two arrays will actually be used in a recursive call).5. But what shall we do if there is no magic way of getting the median? If we are willing to sacrifice speed in the worst case in order to obtain an algorithm reasonably fast on tf. prove that this selection algorithm takes linear time on the average. independently of the value of k. the arrays U and V therefore contain a maximum of Ln / 2] elements. The remaining operations.6.k) < n + k (n-k) < 3n 2 n 4 Assuming that the pivoting algorithm preserves random ordering in the subarrays it produces for the recursive calls.6 Selection and the Median 121 Which element of T should we use as the pivot p ? The natural choice is surely the median of T.T [I] ".) Note that the technique hinted at in this exercise only aprlies because the average time turns out to be linear : the average time taken on several instances is not otherwise equal to the time taken on an average instance. v = 1. the algorithm works by simplification. What happens to the selection algorithm if the choice of p is Problem 4. Prove that E(n. 4. Thus we have an algorithm capable of finding the k th smallest element of an array in a time linear in the size of the array. k). take a time in O (n). To analyse the efficiency of the selection algorithm. k) stand for the expected size of the subarray involved in the first recursive call produced by a call on selection (T [I . made using "p <. In this problem.

n ] Problem 4. By the transitivity of ":5 ". s ] for i E. as is each of the Ln / 5] columns.Divide-and-Conquer 122 Problem 4. to the value of m returned by the algorithm. consider the following algorithm : function pseudomed (T [1 . Similarly. i3 between Si . 4 Show.. Notice that the box contains approximately three-fifths of one-half of the elements of T. 5i ]) return selection(S. Assuming n ? 5. Now suppose that the middle row is sorted by magic. Since m is the exact median of the array S..s]IS[i]<-m }> Is/21.1). To visualize how these factors arise. This can be done with a little cunning.m .n div 5 array S [I . that is. Therefore # { i E [ 1 . I T[i]m }>31s/21=31Ln/5]/21>(3n-12)/10.6. imagine that all the elements of T are arranged in five rows. i2.adhocmed5(T [5i-4 . (s + 1) div 2) . that is... with the possible exception of one to four elements left aside (Figure 4. although nothing in the execution of the algorithm pseudomed really corresponds to this illustration.7. . about 3n / 10 elements. n ]) { finds an approximation to the median of array T) s . The conclusion is that although m is perhaps not the exact median of T. Show similarly that #{iE[I. for every i such that S [i ] <. respectively..n]IT[i]<m }<(7n-3)/10. Note that the time taken by adhocmed5 is bounded above by a constant. But each element of S is the median of five elements of T.6.4 and 5i such that T [i 1] <_ T [i 2] <_ T [i 3] = S [i ] m. the element in the circle corresponds to the median of this array. we have #{ie[l.1 to s do S [i] t.pseudomed(T) . however. Chap.. that in the worst case this algorithm requires quadratic time. Consequently. the smallest elements going to the left and to the top. each of the elements in the box is less than or equal to m. We now look at the efficiency of the selection algorithm given at the beginning of this section when we use p E. The middle row corresponds to the array S in the algorithm. This quadratic worst case can be avoided without sacrificing linear behaviour on the average : the idea is to find quickly a good approximation to the median. yet its rank is approximately between 3n / 10 and 7n / 10. there are three i 1. where adhocmed 5 is an algorithm specially designed to find the median of exactly five elements. Let m be this approximation. We look first at the value of the approximation to the median found by the algo- rithm pseudomed.6.6.

f (LpnJ) +. The version of the algorithm suggested by Problem 4. l7 Problem 4.2 is preferable in prac- . 123 Visualization of the pseudomedian.9 (with p =1/5 and q = 3/4) such that t(n) <_ f(n) for every integer n. Let f : IN -* IR* be any function such that f(n) =. We have to ensure that n > 4 in the preceding equation. Argue that t(n)ES2(n). because the array S can be constructed in linear time.12)/10. In particular.6. still independently of the value of k. and let t (n) be the time required in the worst case by this algorithm to find the k th smallest element of T.(7n + 12)/10 } . * Problem 4. there exists a constant c such that t(n)<-t(Ln/5j)+max{t(i)Ii <-(7n+12)/10}+cn for every sufficiently large n.7 and the preceding discussion show that u <. At the first step.Sec.10. it is possible to find the median of an array of n elements in linear time. and let b be some positive real constant.6.6 Selection and the Median Figure 4. yet simpler problem of the same type. Problem 4. so n -v <. The recursive call that may follow therefore takes a time bounded above by max { t (i) I i <. calculating pseudomed (T) takes a time in O (n) + t( Ln / 5]).1.(7n + 12)/10.6. Problem 4.9.(7n . and thus t(n)EO(n). let no be a positive integer. What is it in the algorithm that makes this restriction necessary? This equation looks quite complicated. The initial preparation of the arrays U and V takes linear time.8. Calculating u and v also takes linear time.3)/ 10 and v >.(3n . Let t (n) be the time required in the worst case to find the k th smallest element in an array of n elements using the selection algorithm discussed earlier. Use constructive induction to prove that f (n) E O(n).6. Hence.6. Give explicitly a nondecreasing function f(n) defined as in Problem 4. Conclude that t(n)eO(n). Let p and q be two positive real constants such that p + q < 1. Let n be the number of elements in T.6.6.f (Lqn]) + bn for every n > n 0. 4. First let us solve a more general.

2 on a computer using the representation you invented for Problem 4.7 ARITHMETIC WITH LARGE INTEGERS In most of the preceding analyses we have taken it for granted that addition and multiplication are elementary operations. Representing these numbers in floating-point is not useful unless we are concerned solely with the order of magnitude and a few of the most significant figures of our results.7. a computer. This was necessary. when the Japanese calculated the first 134 mil- lion digits of It in early 1987. even though this does not constitute an iterative algorithm : it avoids calculating u and v beforehand and using two auxiliary arrays U and V.2. we are obliged to implement the arithmetic operations in software. For some applications we have to consider very large integers. To use still less auxiliary space. Problem 4. this feat constitutes an excellent aerobic exercice for the computer !) The algorithm developed in this section is not. On the other hand. 4 tice. 4. Can we do better? Let u and v be two integers of n decimal digits to be multiplied. Implement your solution to Problem 4.1).7. Your solution to Problem 4. Also implement the classic algorithm for multiplying large integers (see Section 1. as well as additions and subtractions. If results have to be calculated exactly and all the figures count.1.7. Give an algorithm able to add an integer with m digits and an integer with n digits in a time in O(m + n). From a more practical point of view. Problem 4. the time required to execute these operations is bounded above by a constant that depends only on the speed of the circuits in the computer being used. the classic algorithm and multiplication a la russe both take quadratic time to multiply these same operands. Divide-and-conquer suggests that we should separate . (At the very least. Design a good data structure for representing large integers on Problem 4. This is only reasonable if the size of the operands is such that they can be handled directly by the hardware.8).7.1. sufficiently efficient to be used with such operands (see Chapter 9 for more on this). this is no longer true when the operands involved are very large.7. for instance. we can also construct the array S (needed to calculate the pseudomedian) by exchanging elements inside the array T itself. It must also allow negative numbers to be represented. Your representation should use a number of bits in O (n) for an integer that can be expressed in n decimal digits. alas. large integers are of crucial importance in cryptology (Section 4. that is. Although an elementary multiplication takes scarcely more time than an addition on most computers.2 shows how to add two integers in linear time. and it must be possible to carry out in linear time multiplications and integer divisions by positive powers of 10 (or another base if you prefer).3.7.124 Divide-and-Conquer Chap.

4. the integer divisions and multiplications by 102s and 10'. since these are equivalent to an integer division. (For convenience. wz +xy. function mult (u .) n u w x V y z fn/21 Ln/2 Figure 4. v : large-integers) : large-integer n . See Figure 4.7. z) + mult (x .3. Splitting the operands for large integer multiplication.7. Thus tb (n) E 3tb ([n / 21) + tb (Ln / 2j) + O(n). By Example 2. y) x 102s + (mult (w .2. a multiplication.u mod IOs y E--v div 101z 4-v mod 10. as well as the additions. The product that interests us is uv = 102swy + 10S(wz+xy)+xz We obtain the following algorithm. y )) x 101 +mult(x.smallest integer so that _u and v are of size n if n is small then multiply u by v using the classic algorithm return the product thus computed s f-n div 2 w <-.7 Arithmetic with Large Integers 125 each of these operands into two parts of as near the same size as possible : u = 10S w + x and v =10'y +z. and s = Ln / 2j. are executed in linear time. and xz by executing less than four half-length multiplications.7 the time taken by the preceding algorithm is therefore quadratic. even if this means that . 0!5z< 10S .7. The same is true of the modulo operations.z) Let tb (n) be the time required by this algorithm in the worst case to multiply two n digit integers. each of which serves to multiply two integers whose size is about n12. so we have not made any improvement compared to the classic algorithm.7. even if it is not greater than or equal to 10i -1. The last statement of the algorithm consists of four recursive calls.1. x . we have only managed to increase the hidden constant ! The trick that allows us to speed things up consists of calculating wy. The integers w and y therefore both have Fn/21 digits. we say that an integer has j digits if it is smaller than 10j. If we use the representation suggested in Problem 4.Sec. and a subtraction.u div lOs . where 0<_ x< 10'.1. This equation becomes tb (n) E 4tb (n / 2) + O(n) when n is a power of 2. In fact. return mult (w .1 and the algorithms of Problem 4.

345 and v = 6.q -mult(x.005 = 15.7. our example is so small that it would be quicker to use the classic multiplication algorithm in this case. and z = 89. It is thus possible to multiply two n digit integers in a time in 0 (n 1g3).z) return 102sp + l0s (r -p -q) + q Let t(n) be the time required by the modified algorithm to multiply two integers of size at most n . 4 we have to do more additions. Taking account of the fact that w + x and y + z may have up to 1 + Fn / 21 digits. y +z) p <. Recall that the performance of this algorithm and of the classic algorithm are compared empirically at the end of Section 1.59).789.541x104 + (10.mult(w.000 + 506.205 . w = 23. respectively. However. x = 45.4.200 + 405 = 15. The initial Example 4.005)x102 + 4. y = 67. the algorithm we have just seen can multiply u and v.608 -1. After only one multiplication this includes the three terms we need in order to calculate uv. This resembles the equation tc (n) E 3tc J n / 21) + 0 (n). Two other multiplications are needed to isolate these terms. We obtain successively and r = (23 + 45)(67 + 89) = 68 x 156 = 10. Let u and v be integers of exactly m and n digits. which is unacceptable if m is very much smaller . Suppose without loss of generality that m <. A good implementation will probably not use base 10. since it will simply treat u as though it too were of size n. decomposition of the operands gives n = 4. This suggests we should replace the last statement of the algorithm by r E.7. Finally. s = 2.608.11 (notice that t(n) is nondecreasing by definition).2. which is in 0 (n 1. The classic algorithm obtains the product of u by v in a time in 0 (mn).t(Ln12.Divide-and-Conquer 126 Chap. Problem 4. Of course.y).mult (w +x.n. we conclude that t (n) E 0 (n 1g3). q = 45 x 89 = 4. We require to multiply u = 2. which we met at the beginning of the chapter. we find that there exist constants c E IIt+ and no E IN such that t(n) <.7. the largest base for which the hardware allows two "digits" to be multiplied directly.J)+t([n/21)+t(l+Fn121)+cn for every n ? no. Consider the product r =(w+x)(y+z)=wy +(wz+xy)+xz .1. This gives an execution time in 0 (n'3). p = 23 x 67 = 1541. This is sensible because addition is much faster than multiplication when the operands are large. the hidden constants are such that this algorithm only becomes interesting in practice when n is quite large.920. but rather. Even when m # n.410. the required product uv is obtained by calculating 1.541-4.3.005. Using Problem 2.

Show that it is possible in this case to multiply u and v in a time in O (nm lg(3i2)) ** Problem 4.7. Problem 4. it is nevertheless possible to multiply two n digit integers in a time in 0 (n log n log log n) by separating each operand to be multiplied into about parts of about the same size and using Fast Fourier Transforms (Section 9. so as to obtain the required product using five multiplications of integers about one-third as long (not nine as would seem necessary at first sight). Rework Problem 2. * Problem 4. Compare empirically this implementation and the one you made for Problem for every real number a > 1.7. Although the idea tried in Problem 4. where a = 1 + (lg lg n) / lg n.5. Problem 4.11 (analysis of the algorithm fib3) in the context of our new multiplication algorithm. Show by a simple argument that the preceding problem is impossible if we insist that the algorithm A a must take a time in the exact order of n a.5).8. v : large-integers) : large-integer n . that can multiply two n digit integers in a time in the order of n a. Following up Problem 4. .2. Following up Problem 4. (Remark: Integer division of an n digit integer by an arbitrary constant k can be carried out in a time in O(n). Analyse the efficiency of the algorithm suggested by this idea.7. Show that it is possible to separate each of the operands to be multiplied into three parts rather than two. consider the following algorithm for multiplying large integers.3. * Problem 4. although the value of the hidden constant may depend on k. Find two fundamental errors in this analysis of supermul. implement on your machine the algorithm we have just discussed.Sec.10 does not work.7.7.7 Arithmetic with Large Integers 127 than n. an algorithm Aa. and compare your answer once again to Example 2. in a time in 0 (n log n).smallest integer so that u and v are of size n if n is small then multiply u by v using the classic algorithm elsea .7.7. by showing that there exists.1+(lglgn)/lgn multiply u and v using algorithm A a return the product thus computed At first glance this algorithm seems to multiply two n digit numbers in a time in the order of n a.8.7.) ** Problem 4.7.7. Your algorithm must allow for the possibility that the operands may differ in size. 4. function supermul (u. Generalize the algorithm suggested by Problem 4.10. that is.

(If you wish to think about the problem. The cyclic multiplicative group ZP is defined as { x E N I 1 <. Nevertheless.x < p } under multiplication modulo p. the secret value exchanged can now be used as the key in a conventional cryptographic system. 4. * Problem 4. delay reading the rest of this section!) A first solution to this problem was given in 1976 by Diffie and Hellman. 4 Multiplication is not the only interesting operation involving large integers.8. The security of the secret they intend to establish is not compromised should Eve learn these two numbers.g A mod p and transmits this result to Bob . may well be inherently harder to compute .Divide-and-Conquer 128 Chap.8 EXPONENTIATION : AN INTRODUCTION TO CRYPTOLOGY Alice and Bob do not initially share any common secret information. such as calculating the greatest common divisor. As a first step. They cannot therefore use this protocol to exchange directly a message chosen beforehand by one or the other of them.1. she can neither add nor modify messages on the communications line. Alice computes x = b A mod p and Bob calculates y = a B mod p. neither of them can control directly what this value will be. and on some other integer g between 2 and p . They do not want Eve to be privy to their newly exchanged secret. Bob sends Alice the value b = g B mod p. similarly. Integer division. Now x = y since both are equal to g' mod p. An integer g in this group is called a generator if each member of the group can be obtained as some integral power of g. ** Problem 4. Let p be an odd prime number.3). modulo operations. To simplify the problem.2. and the calculation of the integer part of a square root can all be carried out in a time whose order is the same as that required for multiplication (Section 10. although Eve can overhear conversations. Next Alice computes a =. Such a generator always exists. Alice and Bob agree openly on some integer p with a few hundred decimal digits. which they suspect is being tapped by Eve. For some reason they wish to establish such a secret. we assume that. Their problem is complicated by the fact that the only way they can communicate is by using a telephone. This value is therefore a piece of information shared by Alice and Bob.8. they are not treated here.2. Find a protocol by which Alice and Bob can attain their ends. Finally.1. Several other protocols have been proposed since. a malevolent third party. Clearly. Some other important operations. At the second step Alice and Bob choose randomly and independently of each other two positive integers A and B less than p. the condition that g be a generator is necessary if Alice and Bob require that the secret . Clearly.

A' < p . since it makes p 12 trips round the loop on the average when the conditions of Problem 4.2 hold.Sec. One way for her to deduce x would be to find an integer A' such that a = g A'mod p . none of them is able to solve a randomly chosen instance in a reasonable amount of time when p is a prime with several hundred decimal digits. For the time being. g and a is called the problem of the discrete logarithm. still b A mod p =b A' mod p provided that g A mod p = g A' mod p and that there exists a B such that b = g B mod p. At the end of the exchange Eve has been able to obtain directly the values of p. If Eve needs to be able to calculate discrete logarithms efficiently to discover the secret shared by Alice and Bob. although no one has yet been able to prove this.) function dlog (g. and then to proceed like Alice to calculate x' = b A' mod p . and b that does not involve calculating a discrete logarithm. there is no integer A such that 3=2Amod7. g. The obvious algorithm for this is no more subtle or efficient than the one for discrete logarithms. p) A <.8 Exponentiation : An Introduction to Cryptology 129 exchanged by the protocol could take on any value between 1 and p .x 4. then A' is necessarily equal to A. An attentive reader may wonder whether we are pulling his (or her) leg. and sox =x and the secret is correctly computed by Eve in this case. Show that even if A #A'. For instance. The value x' calculated by Eve in this way is therefore always equal to the value x shared by Alice and Bob.O. g a generator of ZP . There exists an obvious algorithm to solve it. Although there exist other more efficient algorithms for calculating discrete logarithms. it seems therefore that this method of providing Alice and Bob with a shared secret is sound. a. a. . and b only. it is equally true that Alice and Bob must be able to calculate efficiently exponentiations of the form a = g A mod p. the algorithm returns the value p. Furthermore. Calculating A' from p. however. 4. Problem 4. g. Prove.3. a. even if A and B are chosen randomly with uniform probability between 1 andp -1. (If the logarithm does not exist. If each trip round the loop takes 1 microsecond.8. that regardless of the choice of p and g some secrets are more likely to be chosen than others.1.xg until (a = x mod p) or (A = p) return A This algorithm takes an unacceptable amount of time.8. and 1 <. there is no known way of recovering x from p. this average time is more than the age of Earth even if p only has two dozen decimal digits. If p is an odd prime.l repeat A<-A+1 x 4.

p) return (a2 mod p) Let h (A) be the number of multiplications modulo p carried out when we calculate dexpo (g . dexpo2 as a function of the value of A and of the size of p. suppose that g is approximately equal to p/2. and p allows us to avoid accumulation of extremely large integers in the loop. A . 4 function dexpo l (g . p ) a 4. which is necessary if we hope to execute each trip round the loop in 1 mod p return a Analyse and compare the execution times of dexpol and * Problem 4.A-l.) function dexpo2(g. Repeat the problem using the divide-and-conquer algorithm from Section 4. A . including the squarings. x 24 = (x 12)2. (The same improvement can be made in dlog. x25 = (((x2x)2)2)2x Thus x25 can be obtained with just two multiplications and four squarings. p ) a<-.dexpo(g. z. For simplicity. function dexpo (g . y. By inspection of the algorithm we find . Use the classic algorithm for multiplying large integers.1 to A do a E.1 for i F. A . where M (p) is an upper bound on the time required to multiply two positive integers less than p and to reduce the result modulo p. This idea can be generalized to obtain a divide-and-conquer return a mod p The fact that x y z mod p = ((x y mod p) x z ) mod p for every x.A/2. p ) if A = 0 then return 1 if A is odd then a . A . We leave the reader to work out the connection between 25 and the sequence of bits 11001 obtained from the expression (((x2x)2x 1)2x1)2x by replacing every x by a 1 and every 1bya0. The preceding formula for x 25 arises because x 25 = x 24x .8.dexpo(g.1 to A do a F.4. assume that calculating a modulo takes a time in the exact order of that required for multiplication. 0 Happily for Alice and Bob.Divide-and-Conquer 130 Chap. In both cases. there exists a more efficient algorithm for computing the exponentiation. An example will make the basic idea clear. and so on.7 for the multiplications. p ).p) return (ag mod p) else a f.1 for i . These operations dominate the execution time of the algorithm. which consequently takes a time in O (h (A) x M (p )).

y F. that is.a .8. function dexpoiter (g . On the other hand.A. In both cases the number of multiplications can easily be reduced to six by avoiding pointless multiplications by the constant 1 and the last squaring carried out by dexpoiter. dexpo calculates x 15 as (((lx)2x)2x)2x.5. This means that Alice and Bob can use numbers p. the computation of dexpo (g . there exists a similar iterative algorithm. A .5. provided A >_ 1. A and B of 200 decimal digits each and still finish the protocol after less than 3.7. with seven multiplications.) Without answering Problem 4. 4. As a function of the size of the base and of the value of the exponent.8.g. As was the case for binary searching. dexpoiter calculates x 15 as 1 x x 2x 4x 8.ay mod p y <__ y2modp n(-ndiv2 return a Problem 4. By suppressing all the reductions modulo p in the preceding * Problem 4. A .1 while n > 0 do if n is odd then a . Find an explicit formula for h (A) and prove your answer by Problem 4.000 multiplications of 200-digit numbers and 3. which corresponds intuitively to calculating x25 as x 16x8x 1. For example. how much time do the algorithms corresponding to dexpo2 and dexpo . which involves eight multiplications (the last being a useless computation of x 16). P) takes a time in 0 (M (p ) x log A ). p ) n F.8.8 Exponentiation : An Introduction to Cryptology 0 h(A)= 131 ifA =0 l + h (A -1) if A is odd 1 + h (A /2) otherwise . Nonetheless. (Do not try to use characteristic equations. This recursive call is not at the dynamic end of the algorithm.000 computations of a 400-digit number modulo a 200-digit number. let us just say that h (A) is situated between once and twice the length of the binary representation of A. which makes it harder to find an iterative version. algorithms. mathematical induction. we obtain algorithms for handling large integers capable of calculating efficiently all the digits of g'. the algorithm dexpo only requires one recursive call on a smaller instance. The efficiency of these algorithms depends in turn on the efficiency of the algorithm used to multiply large integers.8. Show that in fact x 15 can be calculated with only five multiplications. The algorithms dexpo and dexpoiter do not minimize the number of multiplications (including squarings) required. for an arbitrary base g and an arbitrary exponent A. It is therefore an example of simplification rather than of divide-and-conquer.Sec.6. which is entirely reasonable. More generally.

7. 4 take when the classic multiplication algorithm is used? Rework the problem using the divide-and-conquer multiplication algorithm from Section 4.Divide-and-Conquer 132 Chap. Since there are n2 entries to compute in order to obtain C. Consider the following operations : m1 =(a21 +a22-a11)(b22-b12+b11) m2=a11b11 M3 = a 12b21 m4=(a11-a21)(b22-b12) m5=(a21+a22)(b12-b11) m6=(a12-a21 +all -a22)b22 m7=a22(b11+b22-b12-b21). Let A= all a12 a21 a22 and B= b11 b12 b21 b22 be two matrices to be multiplied. We leave the reader to verify that the required product AB is given by the following matrix : m2+m3 m1+m2+m5+m6 m 1+m2+m4-m7 m 1+m2+m4+m5 J .9 MATRIX MULTIPLICATION Let A and B be two n x n matrices to be multiplied. and let C be their product. assuming that scalar addition and multiplication are elementary operations. First we show that two 2 x 2 matrices can be multiplied using less than the eight scalar multiplications apparently required by the definition.7 for multiplying large integers. The basic idea is similar to that used in the divide-and-conquer algo- rithm of Section 4. the product of A and B can be calculated in a time in 8(n 3). The classic algorithm comes directly from the definition : Cij = E Aik Bkj k=1 Each entry in C is calculated in a time in O(n). The preceding problem shows that it is sometimes not sufficient to be only halfclever ! 4. Strassen caused a considerable stir by improving this algorithm. Towards the end of the 1960s.

376) . Strassen's original algorithm takes 18 additions and subtractions as well as 7 multiplications.Sec.640 scalar multiplications. propose a threshold that will minimize the number of scalar operations.9. The asymptotically fastest matrix multiplication algorithm known at the time of this writing can multiply two n x n matrices in a time in 0 (n 2.1. Bearing in mind what you learnt from Section 4. At first glance. however. The algorithm discussed in this section is a variant discovered subsequently by Shmuel Winograd.81.000. 4. asymptotically more and more efficient.81). have been discovered subsequently. show that it is possible to multiply two n x n matrices in a time in 0 (n 2.) Numerous algorithms.9.000 and to which exceeds 150. Following publication of Strassen's algorithm. it was discovered by Coppersmith and Winograd in September 1986. . Your answer will depend on the threshold used to stop making recursive calls. find the exact number of scalar additions and multiplications executed by your algorithm for Problem 4.3. a number of researchers tried to improve the constant w such that it is possible to multiply two n x n matrices in a time in 0 (n °). The number of additions and subtractions needed to calculate the product of two 2 x 2 matrices using this method seems to be 24. none of the algorithms found after Strassen's is of much practical use. Problem 4. Given that matrix additions can be executed much faster than matrix multiplications. provided n is sufficiently large. What do you do about matrices whose size is not a power of 2 ? Problem 4.9 Matrix Multiplication 133 It is therefore possible to multiply two 2 x 2 matrices using only seven scalar multiplications. Building on the preceding discussion. as well as a number of additions and subtractions of n x n matrices. * Problem 4. Show that this can be reduced to 15 by using auxiliary variables to avoid recalculating terms such as m1+m2+m4.9. taking account of the idea from Problem 4. Because of the hidden constants.2.9.1. This is possible because the basic algorithm does not rely on the commutativity of scalar multiplication. the few additional additions compared to the classic algorithm are more than compensated by saving one multiplication.2. If we now replace each entry of A and B by an n x n matrix. we obtain an algorithm that can multiply two 2n x 2n matrices by carrying out seven multiplications of n x n matrices. 703 = 343. (Compare this to 702. once again based on divide-and-conquer: he found a way to multiply two 70 x 70 matrices that involves only 143. Assuming that n is a power of 2.2. Almost a decade passed before Pan discovered a more efficient algorithm.9. this algorithm does not look very interesting : it uses a large number of additions and subtractions compared to the four additions that are sufficient for the classic algorithm.

Otom .n -m + 1 (see Fig. i) exchange (k -i . k) i . j +m -1] in a time in 0(m).k. if T is the array d a e f and k = 3. We have .2 shows how a block of three elements and a block of eight elements are transposed. j. Then i T(i.10. T [i . j) be the number of elementary exchanges that have to be made to transpose a block of i elements and a block of j elements. without making use of an auxiliary array. provided that m < i +m <. we ignore the recursive formulation and give only an iterative version.10.j <.. i+m -1] procedure exchange (i. Here the arrows indicate the part of the array where there are still some changes to make.10 EXCHANGING TWO SECTIONS OF AN ARRAY For this additional example of an algorithm based on simplification.. Let T (i . The general algorithm is as follows. j -n -k.j)= j + T(ij. m) to interchange the elements and T [ j .l do interchange T [i +p ] and T [ j +p ] procedure transpose (T [I .1).2. j . k . m ) forp *. 4. For instance. n ].Divide-and-Conquer 134 Chap.k 4-k+1 while i # j do if i > j then exchange (k -i . After each exchange this part is smaller than before : thus we can affirm that each exchange simplifies the solution of the instance. we can solve our problem as illustrated in Figure 4. k+j. 4 4.10.j-i) ifi=j if i > j if i < j For instance.. We wish to interchange the first k elements and the last n -k. the required result in T is d e f g i g 0 h k i a j k b c It is easy to invent an algorithm exchange (i .j) i +T(i. i ) The analysis of this algorithm is interesting. j ) i 4i -j else j4-j-i exchange (k -i . k . Figure 4. With its help. Let T be an array of n elements.

a c d e I g h i j k h a b c k a b c exchange (1. 4. = 8+T(l. 6. 2) d e I h I a c a c exchange (3. The progression of the parameters of the function T recalls an application of Euclid's algorithm and leads to the following result : Problem 4. in elements m elements T J exchange Figure 4. 8) = 3 + T(3. 1) = 10.1. 3) I h J exchange (1. 1) d e g Figure 4. 3).5) = 6 + T(3. 5.1. h i Progress of transpose (T.4). Effect of the exchange algorithm.2. 1) h e exchange (3. where gcd(i. Prove that T(i. 3) k J d e I g exchange (1.j). b c .2)=9+T(1. 4.10 Exchanging Two Sections of an Array 135 T(3.Sec.10. j) denotes the greatest common divisor of i and j (Section 1.10.j)=i + j -gcd(i. 9.

j) and the matrix F? What happens if i and j are two consecutive numbers from the Fibonacci sequence? Use this idea to invent a divide-and-conquer algorithm to calculate this sequence. as well as another algorithm capable of multiplying two polynomials of degree i in a time in 0 (i log i ) Problem 4.11.11 SUPPLEMENTARY PROBLEMS Problem 4.10. Represent the polynomial p (n) = a o + a 1 n + a 2 n + + ad n d of degree d by an array P [0 . n 2. T[ind]*-T[1] min *. max FT[1]. Consider the matrix F=(011..2. The Fibonacci sequence. Let n 1. Problem 4. we exclude implicit comparisons in the control of the for loop.11.T [2] for i F-3tondo if min > T [i] then min T[i] Find an algorithm that can find both the largest and the smallest elements of an array of n elements by making less than 2n -3 comparisons between elements. .see Chapter 9. space ? 4.1.Divide-and-Conquer 136 Chap. n ] be an array of n elements.3. 4 Can we do better if we are allowed unlimited auxiliary Problem 4. ind F i We only count comparisons between elements . Let T [1 . Does this help you to understand the algorithm fib3 of Section 1. Smallest and largest elements. Give an efficient algorithm based on divide-and-conquer to find the unique monic polynomial p (n) of degree d such that p (n 1) = p (n 2) = = p (nd) = 0.2 to n do if max < T [i ] then max 4..) Analyse the efficiency of your algorithm. We could subsequently find the smallest element of T by making n . (A polynomial is monic if its coefficient of highest degree ad = 1.ind t-1 for i +. . It is easy to find the largest element of T by making exactly n -1 comparisons between elements.2 more comparisons. d ] containing its coefficients.7. 2 . You may . What is the product of the vector (i . Suppose you already have an algorithm capable of multiplying a polynomial of degree i by a polynomial of degree 1 in a time in 0 (i ).5 ? Polynomial interpolation.T [i ].2. Ill 1 Let i and j be any two integers.11.. nd be any integers..

if i = 3. the output is 0101. Your algorithm should take a time in 0 (log n) in the worst case.1. It counts (in binary) the number of bits equal to 1 among the inputs. j = 5. n ] be a sorted array of integers.. iii. If you could not manage the previous problem.11.7. Give an algorithm that can decide whether an array T [I . provided such an index exists. n ] includes a majority element (it cannot have more than one). some of which may be negative but all of which are different. j)]-bit output. but allow your algorithm to take a time in 0 (n log n).Sec. give the simplest expression you can for the number of 3-tallies needed in the construction of your n-tally. Give an algorithm that is able to find an index i such that I <_ i <_ n and T [i] = i. You may therefore not assume that an order relation exists between the elements. For this reason the 3-tally is often called a full adder. * Problem 4.see Figure 4. It adds its two inputs in binary. if n = 9 and the inputs are 011001011. and if so find it. or input A with output B and input B with output A. try again. An element x is said to be a majority element in T if # [ i T [i ] = x } > n 12. You may not suppose that n has any special form. n ] be an array of n elements. show how to build an efficient n-tally. and two outputs.4. For example. Your algorithm must run in linear time. ii. Use these switches to construct a network with n inputs and n outputs able to implement any of the n! possible permutations of the inputs. Let T [I .9. j )-adders you might have used. Problem 4.11.5. It is always possible to construct an (i . depending on the position of the control . . i.11. Tally circuit. for the number of 3-tallies needed to build your n-tally. * Problem 4.8. An (i . Telephone switching. j )-adders as primitive elements.11 Supplementary Problems 137 assume that n is a power of 2. Exactly how many comparisons does your algorithm require? How would you handle the situation when n is not a power of 2 ? Problem 4..11.6. Problem 4. It connects input A with output A and input B with output B. Rework Problem 4. A switch is a circuit with two inputs. and the inputs are 101 and 10111 respectively.11.11. Do not forget to count the 3-tallies that are part of any (i .11. Let T [1 .5 with the supplementary constraint that the only comparisons allowed between elements are tests of equality. Justify your answer. j)-adder is a circuit that has one i bit input. Using full adders and (i . a control. and one [I+ max(i. 4. The number of switches used must be in 0 (n log n ).11.. Problem 4. Majority element. An n-tally is a circuit that takes n bits as inputs and produces 1 + LIg n] bits as output. j)-adder using exactly max(i. Give the recurrence. one j bit input. including the initial conditions. j) 3-tallies. the output is 011100. Using the O notation. For example.

the F4 merge circuit shown in Figure 4. show how to construct an efficient sorting circuit S .1. * Problem 4. Following up the previous problem. The smaller input appears on the upper output. A comparator is a circuit with two inputs and two outputs. 4 Divide-and-Conquer 138 4 N A A A A B B B B Figure 4. show how to construct a merge circuit F whose size and depth are exactly 1 + n lg n and 1 + lg n . Merge circuit.3 gives an example of S4.11. For instance. 7 sorted outputs . The depth is interesting because it determines the reaction time of the circuit.2 shows an F4 circuit. For example. By convention. and the depth is the largest number of comparators an input may have to pass before it reaches the corresponding output. a merge circuit F has two groups of n inputs and a single group of 2n outputs.11. illustrating how the inputs are transmitted to the outputs. There are two ways to measure the complexity of such a circuit : the size of the circuit is the number of comparators it includes.10.2 has size 9 and depth 3. For a given integer n. and the larger input appears on the lower output. it sorts the inputs presented to it. inputs are on the left and outputs are on the right. For n a power of 2.11. then each input appears on one of the outputs. For n a power of 2. has n inputs and n outputs .11. Figure 4.Chap. which is of size 5 and depth 3.11.11. A merge circuit. Each rectangle represents a comparator.2. Problem 4. a sorting circuit S.11. Telephone switches.11. By "efficient" we mean that the depth of your circuit must be significantly less than n 3 first sorted group 1 1 3 3 3 3 6 4 4 4 2 of inputs L37: 6 4 It 2 5 5 2 2 4 5 second sorted group of inputs 2 5 5 5 6 6 7 7 7 7 7 7 6 2 4 Figure 4. Batcher's sorting circuit. Figure 4. and the outputs are also sorted. respectively. Provided each of the two groups of inputs is already sorted.

. Solve these 0 equations exactly. Tournaments. for the size T and the depth P of your circuit S. Continuing the two previous problems. 4. Player n=5 1 2 3 4 1 2 1 - 5 2 3 5 1 - 2 3 4 3 2 1 - 4 5 - 4 3 1 - 4 5 2 3 2 3 4 5 6 Player 1 n=6 5 Day 4 Day 1 2 1 6 5 4 3 2 3 5 1 6 2 4 3 4 3 2 1 6 5 4 5 6 4 3 1 2 5 6 4 5 2 3 1 Figure 4. including the initial conditions.11. show that it is pos- ** Problem 4.11. .11. whenever n is sufficiently large.4. O Problem 4. Each competitor must play exactly once against each of his opponents.12.Sec. sible to construct a sorting circuit for n elements whose size and depth are in 0(n log n) and 0(logn). Give recurrences.11 Supplementary Problems 2 3 139 2 1 3 1 2 3 2 2 1 2 2 4 2 4 1 /-3 4 1 3 3 3 4 4 4 1 Figure 4.13. You are to organize a tournament involving n competitors. A sorting circuit.11. Timetables for five and six players. respectively. but their depth and size must then be taken into account.3. and express T and P in O notation as simply as possible. You may use merge circuits to your heart's content.

The survey article by Brassard. and the algorithm that is asymptotically the most efficient known at present is by Coppersmith and Winograd (1987). The algorithm that multiplies two n x n matrices in a time in O (n 2. points in the plane. 1.59) is attributed to Karatsuba and Ofman (1962). and Zuffellato (1986) covers computation with very large integers. Problem 4. that the cryptosystem based on the knapsack problem. Closest pair of points. Rivest. Pratt.7. For example.8.4 gives possible timetables for tournaments involving five and six players.12 REFERENCES AND FURTHER READING Quicksort is from Hoare (1962). has since been broken. The original solution to Problem 4. For more information about cryptology. with the possible exception of a single day when he does not play at all. ** Problem 4. If n is a power of 2. the first positive success was obtained by Pan (1978). however.7 and 4.6 is examined in Knuth (1969). or in n days if n is odd.11. Subsequent efforts to do better than Strassen's algorithm began with the proof by Hopcroft and Kerr (1971) that seven multiplications are necessary to multiply two 2x2 matrices in a non-commutative structure .81) comes from Strassen (1969).1 is due to Diffie and Hellman (1976). consult the introductory papers by Gardner (1977) and Hellman (1980) and the books by Kahn (1967). The algorithm for multiplying large integers in a time in 0 (n 1.7 to be worthwhile. The algorithm linear in the worst case for selection and for finding the median is due to Blum. Mergesort and quicksort are discussed in detail in Knuth (1973).11.7. Bear in mind. For any integer n > 1 give an algorithm to construct a timetable allowing the tournament to be finished in n -1 days if n is even. Floyd. Kranakis (1986). as described in Hellman (1980). and Brassard (1988).14. Monet. 2. On the other hand.2. see the solution to Exercise 18 of Section 5. The natural generalization of Problem 4. The importance for cryptology of the arithmetic of large integers and of the theory of numbers was pointed out by Rivest.10 comes from Gries (1981). and Adleman (1978). The algorithm of Section 4.2 was solved by Kronrod. Give an of points in a time in O (n log n ). Shamir. .8 is discussed in Knuth (1969). 4 Moreover. efficient exponentiation as described in Section 4. Denning (1983). Notice too that the integers involved in these applications are not sufficiently large for the algorithms of Section 4.4 of Knuth (1973).8.4. give an algorithm to construct a timetable allowing the tournament to be finished in n -1 days.8 is crucial. each competitor must play exactly one match every day. You are given the coordinates of n algorithm capable of finding the closest pair 4. and Tarjan (1972).140 Divide-and-Conquer Chap. Figure 4. The answer to Problems 4.

at least in principle.10 and is discussed in Pohl (1972) and Stinson (1985). Komlos.7 for more on this problem. Problem 4. and Szemeredi (1983).11.11. Problems 4. Problem 4. but consult Section 8.14 is solved in Bentley and Shamos (1976). in Ajtai.11.Sec. Problem 4.1 can be found in Gries and Levin (1980) and Urbanek (1980).11.12 References and Further Reading 141 The solution to Problem 4. .11 are solved in Batcher (1968). 4.12 is solved.

5 Dynamic Programming 5. then a more efficient algorithm will result. saving the solution for later use. usually by keeping a table of known results. When a problem is solved by divide-and-conquer. If. If we pay no attention to this duplication. If we solve each of these independently. 142 . and hence the simplest. until finally we arrive at the solution of the original instance. which we fill up as subinstances are solved. which we then divide into smaller and smaller subinstances as the algorithm progresses. is a top-down method. on the other hand. It sometimes happens that the natural way of dividing an instance suggested by the structure of the problem leads us to consider several overlapping subinstances. we obtain the answers to subinstances of increasing size. and then to combine the solutions of the subinstances so as to solve the original instance. on the other hand. By combining their solutions. Divide-and-conquer. Dynamic programming is a bottom-up technique. we take advantage of the duplication and solve each subinstance only once. The underlying idea of dynamic programming is thus quite simple : avoid calculating the same thing twice.1 INTRODUCTION In the last chapter we saw that it is often possible to divide an instance into subinstances. to solve the subinstances (perhaps by further dividing them). we immediately attack the complete instance. they will in turn create a large number of identical subinstances. subinstances. it is likely that we will end up with an inefficient algorithm. We usually start with the smallest.

k) Figure 5. 143 Consider calculating the binomial coefficient + n Inkl j.7. each subsequence must also be optimal.7.1.Introduction Sec.1. If we calculate [ k] directly by function C (n .5 and Example 2. Prove that the total number of recursive calls made during the computation of C (n.1). k .1.k-1)C(n-1. k ) if k = 0 or k = n then return 1 else return C (n . 5.5 uses dynamic programming? Dynamic programming is often used to solve optimization problems that satisfy the principle of optimality: in an optimal sequence of decisions or choices.. we use a table of intermediate results (this is of course Pascal's triangle . it is not even necessary to store a matrix : it is sufficient to keep a vector of length k. Although this principle may appear obvious.k) C(n. Thus the algorithm takes a time in 0 (nk) and space in 0 (k). 0<k <n k otherwise . i < n .1. Since the final result is obtained by adding up a certain number of Is. We have already met a similar phenomenon in algorithm fib] for calculating the Fibonacci sequence (see Section 1.I k 0 1 C(n-1. we obtain a more efficient algorithm. If. The table should be filled line by line. if we assume that addition is an elementary operation.1. j < k are calculated over and over. .1. on the other hand. which we update from left to right.2. j).1 Example 5. k .1. k) is exactly 2 [ n -2- Problem 5.1. see Figure 5.2. the execution time of this algorithm is certainly in S2( [ k 1). it does not always apply. Pascal's triangle. In fact.7)..1) + C (n . k) many of the values C (i.1. 0 1 1 1 1 2 1 2 2 3 . representing the current line. Which of the algorithms presented in Section 1.l . Problem 5. Calculating the Fibonacci sequence affords another example of this kind of technique.

1. 5. dynamic programming efficiently solves every possible subinstance in order to figure out which are in fact relevant. and the principle of optimality does not apply.2. Without this restriction the longest path might be an infinite loop. For this reason the problem considered in this section is not one of optimization. Let P (i. j) be the probability that team A will win the series given that they still need i more victories to achieve this. Problem 5. that the results of each match are independent. and only then are these combined into an optimal solution to the original instance. whereas team B still needs j more victories if they are to win. We assume that there are no tied games. If the shortest route from Montreal to Toronto goes via Kingston.1. Show that the principle of optimality does not apply to the problem of finding the longest simple path between two cities.3. Imagine a competition in which two teams A and B play not more than 2n -1 games.1. The difficulty in turning this principle into an algorithm is that it is not usually obvious which subinstances are relevant to the instance under consideration. it does not follow that we should drive from Montreal to Kingston as quickly as possible : if we use too much petrol on the first half of the trip. However. (A path is simple if it never passes through the same place twice. maybe we have to stop to fill up somewhere on the second half. This difficulty prevents us from using a divide-and-conquer approach that would start from the original instance and recursively find optimal solutions precisely to those relevant subinstances. but rather concentrate on the control structure and the order of resolution of the subinstances.2 THE WORLD SERIES As our first example of dynamic programming. The subtrips Montreal-Kingston and Kingston-Toronto are not independent. the winner being the first team to achieve n victories.) The principle of optimality can be restated as follows for those problems for which it applies : the optimal solution to any nontrivial instance is a combination of optimal solutions to some of its subinstances.2. losing more time than we gained by driving hard. let us not worry about the principle of optimality. Argue that this is due to the fact that one cannot in general splice two simple paths together and expect to obtain a simple path. if the fastest way to drive from Montreal to Toronto takes us first to Kingston. before the first game of the series the probability that . 5 Example 5. then that part of the journey from Montreal to Kingston must also follow the shortest route between these two cities : the principle of optimality applies. and that for any given match there is a constant probability p that team A will be the winner and hence a constant probability q = 1 -p that team B will win. it is not immediately obvious that the subinstance consisting of finding the shortest route from Montreal to Ottawa is irrelevant to the shortest route from Montreal to Toronto. For example.144 Dynamic Programming Chap. Instead. Coming back to Example 5.

j)=pP(i-1. Finally.2) k . 0) = 0.1. n) : both teams still need n victories.2. P(i.1) P(i.2 145 team A will be the overall winner is P (n . where k = i +j.1. Similarly P (i. 1 <. Combining these results. then it is of course certain that they will win the series : P (0.1. Problem 5.1) k .The World Series Sec.1) + d . The total number of recursive calls is therefore exactly 2(' ' I .2.i < n. n) that team A will win given that the series has not yet started. since team A wins any given match with probability p and loses it with probability q. j). j) + qP (i. ll the required time is thus in S2( [ 2n n JJ P(i. this problem does have other applications ! ) . Recursive calls made by a call on function P (i . j .2 matches left etc. j) = C ((i -1) + j . j) + C (i + (j -1). i) = 1. If team A has already won all the matches it needs. if we look at the way the recursive calls are generated. 5. 1 < i <. To calculate the probability P (n . j) if i = 0 then return 1 else if j = 0 then return 0 else return pP (i -1. j . k>1 where c and d are constants.1. Figure 5. we see that the time required to calculate P (n . 0) is undefined. P (0.1). j .1. T (k) is therefore in 0 (2k) = 0 (4") if i = j = n. In fact. j -1). j -1) Let T (k) be the time needed in the worst case to calculate P (i. which is identical to that followed by the naive calculation of the binomial coefficient C (i + j .1.2 (Problem 5. j) P(i . we find the pattern shown in Figure 5. j) P(i.2.n.j ? 1 Thus we can compute P (i. we see that T(1)=c T(k):5 2T(k .2. (Although sporting competitions with n > 4 are the exception. >. With this method. This method is therefore not practical for large values of n. j) k matches left calls P(i .4" /(2n + 1).1 matches left that call P(i . j) using function P (i.j-1) i ? l. n) is in 0(4 n ) and Q(4 n In). Prove that (nn.j)+gP(i. j).

n-k] .. p ) array P[0.2.0 to n -s do P[s+k.3 CHAINED MATRIX MULTIPLICATION We wish to compute the matrix product M=M1M2.pP[s+k-1. function series (n . its execution time is in 0(n 2).146 Dynamic Programming Chap. and since a constant time is required to calculate each entry.P[s.0]-0 fork .see Section 8. Show that a memory space in O(n) is sufficient to implement this algorithm. 5 * Problem 5. Using this algorithm.1. however.3. Here is the algorithm to calculate P (n . n) using the preceding algorithm is in 0(4"I[ ).5.. Show how to compute P (n. we proceed more or less as with Pascal's triangle : we declare an array of the appropriate size and then fill in the entries. This time.6.2.45 and if four victories are needed to win.0.2. Problem 5.2.s-k] <.1] for s 1 to n do fork .s -k .. so we can compute the product in a number of ways : .I to n do P[0.n] qE-1-p for s F.4. instead of filling the array line by line. Problem 5.2.n-k]+qP[s+k.n. n) . we work diagonal by diagonal.pP[k-l. calculate the probability that team A will win the series if p = 0.) 5.1 to s-1 do P[k.s-k] + gP[k. (Hint : use a completely different approach . n ] Problem 5. Prove that in fact the time needed to calculate P (n . To speed up the algorithm.. n) in a time in O(n). Since in essence the algorithm has to fill up an n x n array.n-k-1] return P [n.s].M" Matrix multiplication is associative.

... Let T (n) be the number of essentially different ways to parenthesize a product of n matrices. There are five essentially different ways of calculating this product..) In each case.Mn) = (M1(M2(M3 . and D is 3 x 34. (In the second case that follows. To measure the efficiency of the different methods.1. Problem 5. Mn) .582 54.1..1. we could simply parenthesize the expression in every possible fashion and count each time how many scalar multiplications will be required.. Suppose we decide to make the first cut between the i th and the (i + 1)st matrices of the product : M = (M 1 M 2 ..Sec. we obtain successively (AB) (AB )C ((AB )C )D 5.3.418 The most efficient method is almost 19 times faster than the slowest. we count the number of scalar multiplications that are involved. M. we do not differentiate between the method that first calculates AB and the one that starts with CD.055 26. Example 5. B is 5 x 89. 0 To find directly the best way to calculate the product..((M1M2)M3). (Mn-1Mn) . i=1 ... 5. if we calculate M = ((AB )C )D.856 4.326 multiplications for a total of 10. C is 89 x 3. here is the corresponding number of scalar multiplications.582 multiplications.i) ways to parenthesize the right-hand term. For example.. Show that calculating the product AB of a p x q matrix A and a q x r matrix B by the direct method requires pqr scalar multiplications.3.3 Chained Matrix Multiplication 147 M=(. Since i can take any value from 1 to n . We wish to calculate the product ABCD of four matrices : A is 13 x 5. ))) The choice of a method of computation can have a considerable influence on the time required. ((AB )C )D (AB) (CD) (A (BC ))D A ((BC )D) A (B (CD )) 10. we obtain the following recurrence for T (n) : nI T(n)= E T(i)T(n-i)..785 multiplications 3.201 2. There are now T (i) ways to parenthesize the left-hand term and T (n .471 multiplications 1. ) (Mi + 1 Mi +2 .

i <.785. Mk ) (Mk + 1 . Among other values. It is only for clarity that the second case is written out explicitly.k < i + s ..n.3.1 and 5.i = s. This method is therefore impracticable for large values of n : there are too many ways in which parentheses can be inserted for us to look at them all. i<k<i+s The third case represents the fact that to calculate Mi Mi+l . as it falls under the general case with s = 1.... For instance.n-s.i+s= min (mik+mk+l. We thus obtain in succession : s = 0: mii = 0. the principle of optimality applies to this problem.1) 1 <S <n:mi. then both the subproducts M1M2 Mi and Mi + A12 . the required MM of the required pronumber of scalar multiplications . where mid gives the optimal solution . n s = 1 : mi. 5 Adding the initial condition T (l) = 1.. if the best way of multiplying all the matrices requires us to make the first cut between the ith and the (i + l)st matrices of the product. Suppose the dimensions of the matrices Mi are given by a vector di .335. The values of T (n) are called Catalan numbers. We construct a table mid .for the part Mi Mi +I duct.that is. we find n 1 2 3 4 5 10 15 T (n) 1 1 2 5 14 4. m23 = 1. Continuation of Example 5.440. *Problem 5..i+s+di-ldkdi+s). for s = 2 we obtain ....89. i =1. .2. Mn must also be calculated in an optimal way.3. . . 2. 2. We have d =(I 3. Fortunately.2). i = 1. 0 <.674. We build the table mid diagonal by diagonal : diagonal s contains the elements mid such that j . Mi +s) and choose the best for i <. i + 1 = di -1di di + 1.3.3.. m34 = 9..2.3. The solution to the original problem is thus given by m 1. i = 1.=(13.2. Mi+s we try all the possibilities (Mi Mi + 1 .1.862 2.3.n. . Prove that T(n)= 1 (2n-2] n-1 n For each way that parentheses can be inserted it takes a time in U(n) to count the number of scalar multiplications required (at least if we do not try to be subtle). This suggests that we should consider using dynamic programming. For s = 1. finding the best way to calculate M using the direct approach requires a time in Q(4"/n). such that the matrix Mi is of dimension di -I by di .. we find m 12 = 5. .078. . Since T(n) is in S2(4"/n2) (from Problems 5. Example 5. Next.34).2.i <.j <.Dynamic Programming 148 Chap. I <. we can thus calculate all the values of T. n -1 (see Problem 5.5.

845 Finally. j=1 2 4 3 0 1 \ 5 785 \ 1 530 i=1 . calculate the value of m In .856 .1)/2 .1.Sec.3. there are n .y s2 = n 2 (n . Example of the chained matrix multiplication algorithm. How must the algorithm be modified if we want not only to Problem 5.3. Problem 5.n (n . m23+m44+5x3x34) = min(24208.3. 9256) = 1. 5.4. for s = 3 m14=min({k=1} mll+m24+13x5x34. 2 856 s=3 2 s=2 3 S=1 4 s=0 Figure 5. {k=2} m12 m34+ 13x89x34 .530 m24=min(m22+m34+5x89x34. m12+m33+ 13x89x3) = min(1530. we must choose between s possibilities (the different possible values of k). 1845) = 1.3.1. The execution time of the algorithm is therefore in the exact order of n-1 s=l n-1 n-I s=1 s=l (n-s)s =n2:s . The array m is thus given in figure 5. 2856) = 2. Write the algorithm to calculate m In . but also to know how to calculate the product M in the most efficient way ? For s > 0.3 Chained Matrix Multiplication 149 m13 =min(mll +m23+ 13x5x3. for each.1)/6 = (n 3-n )/6 The execution time is thus in 0(n 3).1)(2n .s elements to be computed in the diagonal s .5 01. .3. {k=3} m +m44+ 13x3x34) = min(4055.

. and that a matrix L gives the length of each edge. a call on minmat (1.. .. 2. where the array d [0.5. . At itera- tion k. i ] = 0.2. Each edge has an associated nonnegative length. We construct a matrix D that gives the length of the shortest path between each pair of nodes.4 SHORTEST PATHS Let G = <N. . n) is no greater than a 3" .5 is faster than naively trying all possible ways to parenthesize the desired product. j)) return ans . the source. n) of Problem 5. After iteration k .. j] = oc if the edge (i. (Hint: for the "0 " part. and L [i. 2. The principle of optimality applies : if k is a node on the shortest path from i to j..3. . must also be optimal.Dynamic Programming 150 Chap. both of which recursively solve BCDEF from scratch. use ccnstructive induction to find constants a and b such that the time taken by a call on minmat (1. j) if i = j then return 0 ans .min(ans. 2. it is still much slower than the dynamic programming algorithm described previously. k -1 }. We want to calculate the length of the shortest path between each pair of nodes. where we were looking for the length of the shortest paths from one particular node. the algorithm has to check for each pair of nodes (i. N is the set of nodes and A is the set of edges. to all the others.3. N = { 1. After n iterations we therefore obtain the result we want. Prove that with the following recursive algorithm function minmat (i. minmat recursively solves 12 subinstances. 5 * Problem 5. It then does n iterations. (Compare this to Section 3. j) whether or not there exists a path passing through node k that is better than the present optimal path passing only through nodes in { 1.) As before.) Although a call on the recursive minmat (1. d [i -1 ]d [k]d [ j ] + minmat (i. . n ] is global. then that part of the path from i to k. j] ? 0 if i # j. The algorithm initializes D to L. k) + minmat (k + 1. It is this duplication of effort that causes the inefficiency of minmat.2. n }.b.. Let Dk be the matrix D after the k th iteration. suppose that the nodes of G are numbered from 1 to n. A > be a directed graph . n) takes a time in O(3" ). L [i. j) does not exist.. This behaviour illustrates a point made in the first paragraph of this chapter. . D gives the length of the shortest paths that only use nodes in 11. k) as intermediate nodes. including the overlapping ABCDEF and BCDEFG. In order to decide on the best way to parenthesize the product ABCDEFG. The necessary check can be written as . with L [i . 5.00 fork <--i toj-1 do ans F. and that from k to j.

The algorithm. D [i. each time choosing a different node as the source. in O(n 3 ).1 to n do D [i.j]. n.4.4 Shortest Paths 151 Dk[i. n. The order is the same as for Floyd's algorithm. procedure Floyd(L [1 .n.2. and hence with lists of the distances to adjacent nodes. where we make use of the principle of optimality to compute the length of the shortest path passing through k. 5. At the k th iteration the values in the k th row and the k th column of D do not change.2) to solve the same problem. k ] is always zero. If the graph is not very dense (a << n 2). On the other hand. that is. known as Floyd's algorithm.j]). j] P[i.. the total time is in n x O ((a +n) log n). It is obvious that this algorithm takes a time in O(n3). since D [k .l. j] then D[i.k]+D[k.1 to n do for j E. it may be preferable to use Dijkstra's algorithm n times . I. whereas at first sight a matrix n x n x 2 (or even n x n x n) seems necessary.n] D -L fork <-Iton do for i .k]+D[k. not just its length. in O ((an +n 2) log n). n ] array D[I. j] <D[i. We can also use Dijkstra's algorithm (Section 3. We usually want to know where the shortest path goes... If we use the version of Dijkstra's algorithm that works with a matrix of distances. In this case we have to apply the algorithm n times. where a is the number of edges in the graph. that is. k ] +D [k. j] .. j]. n ]) : array[ l .j]-k .1 gives an example of the way the algorithm works.Sec.. j] <--D[i. I. We have also implicitly made use of the fact that an optimal path through k does not visit k twice. It is therefore not necessary to protect these values when updating D. follows. if we use the version of Dijkstra's algorithm that works with a heap. In this case we use a second matrix P initialized to 0. but the simplicity of Floyd's algorithm means that it will probably be faster in practice. . if the graph is dense (a = n 2). j]) return D Figure 5.j] =min(Dk_i[i. The innermost loop of the algorithm becomes ifD[i.k]+Dk-1[k. the total computation time is in n x O(n 2). it is better to use Floyd's algorithm.min(D [i.. This allows us to get away with using only a twodimensional matrix D.Dk-i[i.

Dynamic Programming 152 Chap. Look recursively at P [i. j]. j]. j) . j ] contains the number of the last iteration that caused a change in D [i. j] = 0. j ] = k. k ] and P [k.4. 5 50 0 15 5 30 00 0 15 15 00 5 0 0 0 5 50 0 D2 = 5 30 35 0 15 15 20 5 0 5 20 0 D4 = 20 10 15 0 5 15 oo o 15 10 10 5 30 35 0 15 15 20 5 0 Floyd's algorithm at work. the shortest path is directly along the edge (i.1. . When the algorithm stops. If P [i. 5 15 0 Do=L = 0 D D 1 = 5 00 00 50 0 15 5 30 35 0 15 15 20 5 0 0 20 10 5 45 0 3 = 15 30 35 0 15 20 5 Figure 5. To recover the shortest path from i to j. the shortest path from i to j passes through k. look at P [i. if P [i. otherwise. P [i. j] to find any other intermediate nodes along the shortest path.

the length of the edges is of no interest . the notion of "shortest path" loses much of its meaning : the more often we go round the negative cycle. .1. only their existence is important. and L [i . Initially. No efficient algorithm is known for finding shortest simple paths in graphs that may have edges of negative length.3). P becomes 0 0 4 2 4 0 4 0 P = 0 1 0 0 0 1 0 0 Since P [1. j] = false otherwise.1. 4] and P [4. 3.4. If G includes cycles whose total length is negative. j] = true if there exists at least one path from i to j. j) exists. the shortest path from 1 to 3 passes through 4. We want to find a matrix D such that D [i. Even if a graph has edges with negative length.1.4. 2. case when the matrix L is symmetric (L [i. On a graph that includes a negative cycle ? ii. the shorter our path will be ! Does Floyd's algorithm work i. (We shall see an asymptotically more efficient algorithm for this problem in Section 10. These two problems are NP-complete (see Section 10.4. 3]. L [i.) Adapt Floyd's algorithm for this slightly different case. In this case. For the graph of Figure 5.4 153 Shortest Paths Example 5. i ]). The shortest path from I to 3 is thus 1.2.3.) Find a significantly better algorithm for Problem 5. This is the situation we encountered in Problem 5. we discover that between 1 and 4 we have to go via 2. j] = true if the edge (i. Suppose we allow edges to have negative lengths. Warshall's algorithm.Sec. but that from 4 to 3 we proceed directly. and D [i.4.2. Looking now at P [1. (We are looking for the reflexive transitive closure of the graph G. 5. Problem 5. 3] = 4. j] = L [ j. the notion of a shortest simple path still makes sense.2. On a graph that has some edges whose lengths are negative. but that does not include a negative cycle ? Prove your answer. Finally we see that the trips from 1 to 2 and from 2 to 4 are also direct.1. j] = false otherwise. Problem 5.2 in the * Problem 5.

2 contains the same keys as those in Figure 5." Figure 5.5 OPTIMAL SEARCH TREES We begin by recalling the definition of a binary search tree. we first examine the key held in the root. if X < R.1. (It provides an example of simplification : see chapter 4. we only need look at the right-hand subtree. the information attached to the key we are looking for. tinct keys ? How many different search trees can be made with eight dis- . .) Problem 5. If X=R. For a given set of keys.5. but rather. we have found the key we want.1. B.1.5. . search trees will be understood to be binary. Problem 5. 5 5. H. Suppose this key is R. and less than or equal to the key of its righthand child. and the search stops . The nodes may also contain further information related to the keys : in this case a search procedure does not simply return true or false. A binary tree each of whose nodes contains a key is a search tree if the value contained in every internal node is greater than or equal to (numerically or lexicographically) the values contained in its left-hand descendants.5. we only need look at the left-hand subtree . Problem 5.5. A recursive implementation of this technique is obvious. and less than or equal to the values contained in its righthand descendants. the tree in Figure 5.1 shows an example of a binary search tree containing the keys A.2. Show by an example that the following definition will not do : "A binary tree is a search tree if the key contained in each internal node is greater than or equal to the key of its left-hand child.5.5. several search trees may be possible : for instance.154 Dynamic Programming Chap.) To determine whether a key X is present in the tree. Write a procedure that looks for a given key in a search tree and returns true if the key is present and false otherwise.3. A binary search tree.5.. Figure 5. and if X > R. (For the rest of this section. C .

i=1 This is the function we seek to minimize.5. then di + 1 comparisons are necessary to find it. ci + i .1 two comparisons are needed to find the key E . . For a given tree the average number of comparisons needed is set c 1 < c 2 < * C= pi (di + 1)..5.3..2. that is. n . cj . . Suppose we have an ordered < cn of n distinct keys.5. give a tree that minimizes the average number of comparisons needed. Let the probability that a request refers to key c. the depth of its children is 1.. .) In Figure 5. and (4+3+2+3+1+3+2+3)/8=21/8 comparisons on the average in Figure on the other hand. If T (n) is the number of different search trees we can make with n distinct keys. For the case when the keys are equiprobable. be pi . find either an explicit formula for T (n) or else an algorithm to calculate this value. For the time being. . we shall solve a more general problem still.5. Repeat the problem for the general case of n equiprobable keys. suppose that Ein=1 pi = 1. If all the keys are sought with the same probability.5. it takes (2+3+1+3+2+4+3+4)/8 = 22/8 comparisons on the average to find a key in Figure 5. Problem 5.4. Consider the sequence of successive keys ci . Recall that the depth of the root of a tree is 0. If some key ci is held in a node at depth di . and so on. is held in a node of depth dk in the sub- tree. In fact. in Figure 5.5 155 Optimal Search Trees *Problem 5.i.Sec. j >.2.2 Another binary search tree. 5.. . all the requests refer to keys that are indeed present in the search tree. 2. Suppose that in an optimal tree containing all the n keys this sequence of j-i + 1 keys occupies the nodes of a subtree. (Hint: reread Section 5.5. If the key Ck . i = 1. a single comparison suffices. i < k < j. the average number of comparisons carried out in this subtree when we look Figure 5.

k - I+ Ck + I . must occupy the root of the subtree.. and a change in the subtree does not affect the contribution made to C by other subtrees of the main tree disjoint from the one under consideration.Dynamic Programming 156 Chap. In this case one comparison is made with Ck . We thus arrive at the principle of optimality : in an optimal tree all the subtrees must also be optimal with respect to the keys they contain. . Ck . . with five keys c1 to c5 are . respectively. A subtree of the optimal search tree is optimal. In Figure 5. To obtain a dynamic programming scheme.3. ci + I .. . k. k=i We observe that this expression has the same form as that for C.. the probability that it is in the sequence ci . Figure 5. . (It is convenient to define CiJ = 0 if j = i .3. ci + I .1. c j is mil . The average number of comparisons carried out is therefore CiJ = mil + Ci.I and R is an optimal subtree containing Ck +I . say. .1.. and let CiJ be the average number of comparisons carried out in an optimal subtree containing the keys ci . it remains to remark that the root k is chosen so as to minimize Cij : CiJ =mil + In particular..J) (*) . cJ when a key is sought in the main tree.. 5 for a key in the main tree (the key in question is not necessarily one of those held in the subtree) is CY. J where the three terms are the contributions of the root. Cii = pi mm n ISk<J (Ci. . To find the optimal search tree if the probabilities associated Example 5.5. L and R. and others may then be made in L or R.. Let mij = Yk=i Pk . Pk(dk+l). When we look for a key in the main tree.5. .5. ci +I .) One of these keys.k-I +Ck+I.. L is an optimal subtree con- taining the keys ci . . cJ .

38. 0.6. 1. 5.85. C11+C33.91.35 + min(0. C24+C65) = 0.12 Now.58 + min(0.49 C25 = m25 + min(C21 +C35.12 we first calculate the matrix m.69.05 0. we use (*) to calculate the other values of Cii. 0.70 0. C12+C43) = 0.35 0.76) = 1.61) = 1.73.15. We know how to find the minimum number of comparisons Problem 5.88 + min(0.88 1.58 0. 0.69.61. 0. C13+C54) = 0.Sec. 0.40) = 0.45 0. 0. C22+C44.18.61.13 0. C14+C65) = 1.43 + min(0. and next.00 + min(1. 0.20. C34=0. 1. 0. C11+C35.30.5. 0.85 C14=m14+min(C1o+C24.05. 0. 0.74.50. 0. m = 0. The optimal search tree for these keys requires 1. 1. we note that Cii = pi .30 0.65 + min(0.00 0.5 157 Optimal Search Trees i 1 2 3 Pi 0. C33+C55.30 0. necessary in the optimal tree.09. C12 = m12 + min(CI0+C22.57 0.18) = 0.49) =1.30) = 0.5.73. C23+C55.76 C35 = m35 + min(C32+C45.76. C11 +C34.05 0.61 C24 = m24 + min(C21+C34. CII+C32) = 0.4). C12+C45. C23+C54) = 0.08 4 0.00. 1 <_ i <_ 5.43 0. 0.73 comparisons on the average to find a key (see Figure 5. C12+C44.53 0.18. C22+C45. but how do we find the form of this tree ? 0 .45 5 0. C34+C65) = 0. C45=0.61) = 0.85.08 0.40 Similarly C23=0. C13+C55. Then C13 = m13 + min(C10+C23.65 0.00 C15=m15+min(C1o+C25. 0.70 + min(0.

i = 0. .k <. ** Problem 5. ] } .5.9. The required computation time is therefore in . and so on. . k . Generalize the preceding argument to take account of the possibility that a request may involve a key that is not in fact in the tree. n.i <. Give an algorithm that can determine the optimal search tree in this context.n.. or to ascertain that it is missing.. (Provided the keys are sorted. 2. Problem 5. 1. 2.. n .5. m=1 Problem 5. We now have n n .10. be the probability that a request concerns a key ci that is in the tree. Specifically. each involving a choice among m + 1 possibilities.. . i = 1. if it is present in the tree..j <.I+ Ck + 1. * Problem 5. let rid = max { k I i <. n . .(In-t (n-m)(m+l)) =8(n3) ..5.pi +Iqi=1. then for j -i = 2. The optimal tree must minimize the average number of comparisons required to either find a key. to calculate. An optimal binary search tree. When j -i = m .5. be the probability that it concerns a missing key situated between ci and ci+1 (with the obvious interpretation for q0 and qn ). .) In this algorithm we calculate the values of Cij first for j -i = 1. 2. Prove this last equality. and that produces a description of the optimal search tree for these probabilities. 5 Write an algorithm that accepts the values n and pi . For 1 <.j and C. =Mid + Ci .7. i = 1 .8. there are n -m values of C.Dynamic Programming 158 Chap. and let qj .. let pi .5. . we do not need their exact values.4. Figure 5.

9. we see that g(1. i. Define g (i. .N\{1})=2<j<n min(Ljj +g(j. explicit example that this greedy algorithm does not always find the optimal search tree. S # 0... and i 0 S. j). j }. at the root. . 2. . g (1. ck _ 1 and Ck + i ...j for every 1 <_i j Sn. then so is the path from j to 1 : the principle of optimality holds.. followed by a path from j to 1 that passes exactly once through each node in N \ 11.10 and 5. c. Suppose without loss of generality that the circuit begins and ends at node 1.Sec.10 to show how to calculate an optimal search tree in a time in 0(n2).5. There is an obvious greedy approach to the problem of constructing an optimal search tree : place the most probable key. with i =1 allowed only if S = N \ { 1 } .and right-hand subtrees for c i . By the principle of optimality. j] = oo if the edge (i.. j # 1. say. (Problems 5. we take N = { 1.6 The Travelling Salesperson Problem 159 be the root of an optimal subtree containing c. S # N \ { 1 )... Give an optimal search tree for your example. If the circuit is optimal (as short as possible). and calculate the average number of comparisons needed to find a key for both the optimal tree and the tree found by the greedy algorithm.5. +I . N \ { 1 }) is the length of an optimal circuit. How much time does this algorithm take in the worst case. . jES (*) . Prove that r. .S\ { j})). j) does not exist. we are required to find the shortest possible circuit that begins and ends at the same node.2. recursively in the same way. if i # 1. .j +g(j. -I = i .11 generalize to the case discussed in Problem 5. As usual.11. More generally.12.5. 5. It therefore consists of an edge (1. .) Problem 5. Problem 5. Write also r. Ck. 5. Show with the help of a simple. and the lengths of the edges are denoted by L. g(i.. assuming the keys are already sorted ? ii.+1. S) as the length of the shortest path from node i to node 1 that passes exactly once through each node in S. c 2.5. L [i . Using this definition. i ] = 0.j})).S)=rni (L.N\{l.5. Let G = < N. after having gone exactly once through each of the other nodes.6 THE TRAVELLING SALESPERSON PROBLEM We have already met this problem in Section 3. A > be a directed graph. j ] >_ 0 if i and L [i.j _i <_rj <r. Ck+2 . Given a graph with nonnegative lengths attached to the edges.5. and construct the left.4. cj . . with L [i .j . Consider a set of nodes S c N \ 111 and a node i EN \ S. n j. c. Use the result of Problem 5.

6. then we can apply (**) again to calculate g for all the sets S that contain two nodes (other than 1). and so on. . Example 5.. 0) = 5. 15 6 Figure 5. 5 Furthermore. g(i.Dynamic Programming 160 Chap. N \ { 1.6.1. A directed graph for the travelling salesperson problem. Once the value of g ( j. g (3. N \ { 11) and solve the problem. We can apply (**) to calculate the function g for all the sets S that contain exactly one node (other than 1).. we can use (*) to calculate g (1. g (4..3. i =2.1: 0 10 15 20 0 9 10 6 13 0 12 8 9 0 5 L = 8 We initialize g (2. j }) is known for all the nodes j except node 1. The values of g (i.0)=Li1. 0) = 8 . Let G be the complete graph on four nodes given in Figure 5. n .6.1.. S) are therefore known when S is empty. 0) = 6.

{3. we need an extra function : J (i . g(3.{2})=18.4))=2 J(2. { 3) )) = min(29. L43 + g (3. {2.6. (Continuation of Example 5.4)) = min(L23 + g (3.13 1) =L23+g(3.3) )) = min(35.14))=20 g(4. { 2.41) = min(L 12 + g (2. { 4) ).0)= 15 g(2.) J(2. { 3) ).3 )) = min(L42 + g (2.1 has length 35.12))) = min(23.4)) = min(L 32 + g ( The Travelling Salesperson Problem 161 Using (**). (2.3. S) is the value of j chosen to minimize g at the moment when we apply (*) or (**) to calculate g(i.4)) =4 =4 J(4.4 } ). 5. L 14 + g (4.{2})= 13.1.S).6. L24 + g (4.4)) J(1. { 3.(31)=3 -1 The required computation time can be calculated as follows : In this example we find .{3.{2. 40.{41)=L24+g(4. using (**) for sets of two nodes.3.(2.4 1).{3})= 15.12.25) = 25 g (4. ( 2.4})=4 -3J(4. we have g (2. (3. 27) = 23 Finally we apply (*) to obtain g (1. 1 2. 43) = 35 The optimal circuit in Figure 5. L 34 + g (4.4)) = 2 and the optimal circuit is 1 -+J(1.31) =2 J(3.25) = 25 g (3. g(4. To know where this circuit goes. Example 5. (2.6. (4)).0)= 18 and similarly g(3. we obtain g(2.Sec. Next. L 13 + g (3. { 2 } )) = min(31.

The preceding analysis assumes that we can find in constant time a value of g (j . 5 . These operations can be used as a calculate g ( j.Dynamic Programming 162 Chap. For instance.240 15 1. as would be the case if we simply tried all the possible circuits. which is not very practical either. Time: Direct method n! Time: Dynamic programming n22" Space: Dynamic programming n 2" 5 120 800 160 10 3. whereas 20! microseconds exceeds 77 thousand years.400 10.1.520 20.6.400 491. 5. n SOLVING THE TRAVELLING SALESPERSON PROBLEM.800 102. . but it is still far from offering a practical algorithm. Since S is a set. 0) : n -1 consultations of a calculate g (1..2. 13 in Q(n 2" ).6. which data structure do you suggest to hold the values of g ? With your suggested structure.6.800 419.6.1 illustrates the dramatic increase in the time and space necessary as n goes up.31 x 1012 20 2.(n -1)k In k 21) = 19(n22") k=1 since k1 k (k1 = r 2r-1 This is considerably better than having a time in SZ(n !).971. consider . TABLE 5.628.430.n .1. S) that has already been calculated. For example.43x1018 7. S) such that 1 :5 #S = k <.7 MEMORY FUNCTIONS If we want to implement the method of Section 5. N \ (1 }) : n -1 additions. it is easy to write a function that calculates g recursively.520 Problem 5. The computation time is thus in 0(2(n -1) + .6 on a computer.. What is more . .372. how much time is needed to access one of the values of g ? Table calculate all the g (i . 202220 microseconds is less than 7 minutes. Verify that the space required to hold the values of g and J is Problem 5.2: (n -1) (n k 21 k additions in all.

5.1 ] if gtab [i. To the recursive function we add a table of the necessary size. we return the value held in the table.1)!). If not. and so on.7 Memory Functions 163 function g (i. Formulated in the following way : function g (i. we first look in the table to see whether it has already been evaluated with the same set of parameters. S ] ans F oo for each j E S do distviaj . S ) if S = 0 then return L [i . then all the sets containing just one element from N \ { I). Although it is maybe not too hard to write such a generator. the function g combines the clarity obtained from a recursive formulation and the efficiency of dynamic programming. Thereafter. then all the sets containing two elements from N \ { 1).L [i. S \ { j } ) if distviaj < ans then ans . j ] + g ( j. Unfortunately. S \ { j } ) if distviaj < ans then ans . One easy way to take advantage of the simplicity of a recursive formulation without losing the efficiency offered by dynamic programming is to use a memory function. however. it is not immediately obvious how to set about it. (In fact. For the algorithm of Section 5. we save it at the appropriate place in the table.6 let gtab be a table all of whose entries are initialized to -1 (since a distance cannot be negative). whenever we call the function. it ends up back in S2((n .oo for each j E S do distviaj F.L [i.) So how can we calculate g in the bottom-up way that characterizes dynamic programming ? We need an auxiliary program that generates first the empty set. we go ahead and calculate the function. if we calculate g in this top-down way.1] ans <. S ) if S = 0 then return L [i . j ] + g ( j. S ] > 0 then return gtab [i.distviaj return ans . we come up once more against the problem outlined at the beginning of this chapter : most values of g are recalculated many times and the program is very inefficient. In this way it is never necessary to calculate the function twice for the same values of its parameters. all the entries in this table hold a special value to indicate that they have not yet been calculated. Before returning the calculated value. Initially. If so. S ] .distviaj gtab [i. .ans return ans .Sec.

n ] and a few pointers. This is particularly desirable when in fact only a few values of the function are to be calculated. n ] and P [1 . n ] can be virtually initialized with the help of two auxiliary arrays B [1 . Show how to calculate (i) a binomial coefficient and (ii) the function series (n . We saw in Section 5. returns a default value (such as -1) otherwise } A call on any of these procedures or functions (including a call on init !) should take constant time in the worst case. the calculation takes the same amount of time but needs space in Si(nk).. however). If we are willing to use a little more space (the space needed is only multiplied by a constant factor.7. * Problem 5. We want to * Problem 5.. For instance.8 SUPPLEMENTARY PROBLEMS Let u and v be two strings of characters.2. we can transform abbac into abcbc in three stages. procedure init { virtually initializes T [1 . see Section 6. that we can calculate a binomial coefficient (k] using a time in 0 (nk) and space in 0(k)..8. add a character. it is possible to avoid the initialization time needed to set all the entries of the table to some special value. if any .Dynamic Programming 164 Chap. but we do not know in advance which ones.) Show how an array T [1 . Show that this transformation is not optimal. . abbac -* abac ababc abcbc (delete b) (add b ) (change a into c ). 5 Problem 5.1. for instance. transform u into v with the smallest possible number of operations of the following types : delete a character. 5. p) of Section 5.6.2 using a memory function. We sometimes have to pay a price for using this technique.1. change a character.2. v ) { sets T [i ] to the value v } function val (i) { returns the last value given to T [i ]. n ] } procedure store (i .. Implemented using a memory function. (For an example.7. You should write three algorithms.1.

c } . and that tells us what these operations are. Modify your algorithm from the previous problem so it returns the number of different ways of parenthesizing x to obtain a.1. Consider the alphabet E = { a. Thus ab = b. your algorithm should return "yes" because (b (bb ))(ba) = a.2. There is no extra charge if you change canoes in this way.8.8. This algorithm works correctly in a country where there are coins worth 1. In the introduction to Chapter 3 we saw a greedy algorithm for making change. (b (b (b (ba )))) = a as well. in which case you can return the first canoe at some post k between i and j and continue the journey in a second canoe. and so on. The elements of Y. the length of the string x.5. For example.Sec. Note that the multiplication defined by this table is neither commutative nor associative. and 25 units. 5. if x = bbbba.) In terms of n.4. AN ABSTRACT MULTIPLICATION TABLE Right-hand symbol Left-hand symbol a b c a b b b c b a a c a c C Find an efficient algorithm that examines a string x = x 1x2 x of characters of E and decides whether or not it is possible to parenthesize x in such a way that the value of the resulting expression is a. In terms of N. At any of these posts you can rent a canoe to be returned at any other post downstream. ba = c. How much time does your algorithm take as a function of the lengths of u and v ? Problem 5.8. Find an efficient algorithm to determine the minimum cost of a trip by canoe from each possible departure point i to each possible arrival point j.8. but it does not always find an optimal solution if there . have the multiplication table given in Table 5. TABLE 5.8 Supplementary Problems 165 Write a dynamic programming algorithm that finds the minimum number of operations needed to transform u into v. what is the computing time needed by your algorithm ? Problem 5. Problem 5. 5. However.) For each possible departure point i and each possible arrival point j the company's tariff gives the cost of a rental between i and j.8. (This expression is not unique.1. how much time does your algorithm take ? Problem 5. it can happen that the cost of renting from i to j is higher than the total cost of a series of shorter rentals. b.8. There are N Hudson's Bay Company posts on the River Koksoak. (It is next to impossible to paddle against the current. 10. For instance.3.

n ] be an array giving the value of these coins. A =B =C A =C <B C <A =B A =B <C B <A =C C <A <B A <B =C B <A <C C <B <A A <B <C A <C <B B <C <A B =C <A Give a dynamic programming algorithm that can calculate. The general problem can be solved exactly using dynamic programming. how much time does your algorithm take? iii. it is restricted to using a space in 0 (m).. A (3.Dynamic Programming 166 Chap. Your algorithm 0. Your algorithm must consist simply of two nested loops (recursion is not allowed). including the initial conditions. We suppose that an unlimited number of coins of each value is available. Let n be the number of different coins that exist. Ackermann's function is defined recursively as follows : A(0. the number of different possible orderings. Give a greedy algorithm that can make change using the minimum number of coins for any amount M <_ L once the % have been calculated. 5).i <_ n and 1 <_ j <_ L. Your algorithm should take a time in 0 (n 2) and space in 0 (n). and A (4. i. For 1 <. and let T [I . Moreover. 1 <_ j <. should take a time in 0 (n + provided You have n objects. Calculate A (2.1). A (m. 3). 13 different orderings are possible with three objects. ** Problem 5. 1) This function grows extremely rapidly. Give a recurrence for cij .n-l)) ifm. 0) = A (m . As a function of n and L.A(m. . ii.1 .L. relations "<" and "=".. as a function of n. Your algorithm may use only a single array of length L. .7.1. T[i]. 5 also exists a coin worth 12 units (see Problem 3. i. Give a dynamic programming algorithm that calculates all the c.6. For example. although some of these memory words can . or cii _+00 if the amount j cannot be obtained using just these coins.. T[2].8. Let L be a bound on the sum we wish to obtain.1.n >0. ii. n).n)=n+1 ifm >0 A(m.n)=A(m-1. 4). Give a dynamic programming algorithm to calculate A (m. let cij be the minimum number of coins required to obtain the sum j if we may only use coins of types T [1].8. which you wish to put in order using the * Problem 5.

1). We mention only Bellman 1957.3 is described in Godbole (1973) . Floyd's algorithm for calculating all shortest paths is due to Floyd (1962). 1984). the (n -1)st Catalan number (see Section 5. The algorithm in Section 5. able to solve the problem of chained matrix multiplications in a time in 0 (n log n).5.8. comes from Gilbert and Moore (1959).4. All these algorithms (with the exception of Fredman's) are unified in Tarjan (1981). 5. The algorithm of Section 5. Nemhauser (1966).8. A theoretically more efficient algorithm is known : Fredman (1976) shows how to solve the problem in a time in O (n 3(log log n / log n) 1/3 ).9. A solution to Problem 5.5. including Sloane (1973) and Purdom and Brown (1985).2 is supplied by the algorithm in Warshall (1962).11 come from Knuth (1971. ind [i ]). Bellman and Dreyfus (1962). m ] such that at every instant val [i ] = A (i.9 167 References and Further Reading 0 Figure 5. a hexagon can be cut in 14 different ways. grow quite large.9 REFERENCES AND FURTHER READING Several books are concerned with dynamic programming. The improvements suggested by Problems 5.5.10 and 5. 1973).8.1. Both Floyd's and Warshall's algorithms are essentially the same as the one in Kleene (1956) to determine the regular expression corresponding to a given finite automaton (Hopcroft and Ullman 1979). For example.. this .8.5.2 triangles using diagonal lines that do not cross is T (n .5 for constructing optimal search trees. Catalan's numbers are discussed in many places.3). including the solution to Problem 5. (Hint : use two arrays val [0..1. m ] and ind [0. The solution to Problem 5. polygon into n . Cutting a hexagon into triangles. as shown in Figure 5. a more efficient algorithm. 11 5. and Lauriere (1979). can be found in Hu and Shing (1982.Sec.) 11 Prove that the number of ways to cut an n -sided convex Problem 5.10 that is both simpler and more general is given by Yao (1980).

1 is given in Wagner and Fischer (1974).5 is discussed in Wright (1975) and Chang and Korsh (1976).2.12 in Aho.8 is discussed in Sloane (1973).5. Memory functions are introduced in Michie (1968) .7. Hopcroft and Ullman (1974). A solution to Problem 5.168 Dynamic Programming Chap.8. comes from Exercise 2.12. Problem 5.7 is based on Ackermann (1928).8.2).11: we were curious to know what proportion of all the possible answers was represented by the 69 different answers suggested by the students (see also Lemma 10. Problem 5. which suggests how to avoid initial- izing a memory function. An important dynamic programming algorithm that we have not mentioned is the one in Kasimi (1965) and Younger (1967). Problem 5.8. Problem 5. for further details see Marsh (1970). Problem 5.1.6 suggested itself to the authors one day when they set an exam including a question resembling Problem 2.1. .8. The optimal search tree for the 31 most common words in English is compared in Knuth (1973) with the tree obtained using the greedy algorithm suggested in Problem 5. 5 paper gives a sufficient condition for certain dynamic programming algorithms that run in cubic time to be transformable automatically into quadratic algorithms.8. which takes cubic time to carry out the syntactic analysis of any context-free language (Hopcroft and Ullman 1979). The algorithm for the travelling salesperson problem given in Section 5.6 comes from Held and Karp (1962).

and so on. we often use abstract graphs to represent games : each node corresponds to a particular position of the pieces on the board. all we have is a representation of the current position (that is. Up to now. We have seen. To solve such problems. the nodes are represented by a certain number of bytes. the algorithms we have seen have implicitly imposed an order on these visits : it was a case of visiting the nearest node. In this case. In this case to "mark a node" means to take any appropriate measures that enable us to recognize a position we have already seen. Sometimes. the graph exists only implicitly. the structure of the problem is such that we need only visit some of the nodes or edges. When we explore such a graph. Most of the time. we often need to look at all the nodes. and the fact that an edge exists between two nodes means that it is possible to get from the first to the second of these positions by making a single legal move. and the edges are represented by pointers. At other times. the shortest edge. or all the edges. The operations to be carried out are quite concrete : to "mark a node" means to change a bit in memory.6 Exploring Graphs 6. for instance. of the node we are in the process of visiting) and possibly representations of a few other positions. to "find a neighbouring node" means to follow a pointer. and so on. the shortest route problem and the problem of the minimal spanning tree. We shall use the word "graph" in two different ways. it does not really exist in the memory of the machine. In this chapter we introduce some general techniques that can be used when no particular order of visits is required. For instance.1 INTRODUCTION A great many problems can be formulated in terms of graphs. of a graph. A graph may be a data structure in the memory of a computer. or to avoid arriving at 169 .

Suppose that visiting a node takes a time in O(1). It is obvious how to implement any of these techniques using recursion.1. of which one node is the root. the time required is bounded above by some constant c. we are traversing the tree in inorder. and finally. g nodes are situated in the left-hand subtree. the time T (n) needed to explore a binary tree containing n nodes is in O(n). then we are visiting the tree in +c . all the nodes in the right-hand subtree. the techniques used to traverse it are essentially the +3c -d <.2 TRAVERSING TREES We shall not spend long on detailed descriptions of how to explore a tree. whether the graph is a data structure or merely an abstraction. However. Three corresponding techniques explore the tree from right to left. Preorder and postorder generalize in the obvious way to nonbinary trees. then all the nodes in the left-hand subtree. we are traversing the tree in preorder .do + c where d is a constant such that d ? 2c.2. if we visit first the left-hand subtree. We simply remind the reader that in the case of binary trees three techniques are often used. to "find a neighbouring node" means to change the current position by making a single legal move . 6. If at each node of the tree we visit first the node itself. and so on. the right-hand subtree. Suppose further that we are to explore a tree containing n nodes. Now suppose that it is true for all n . 0 <. Then T (m) <_ <_ max (T (g) + T (m -g -1) + c) max (dg +c +d(m-g-1)+c +c) 0<g <m-1 0<g <m-1 <. and n -g -1 nodes are in the right-hand subtree. By the choice of c the hypothesis is true for n = 0. 6 the same position twice . and finally. n > 0. then the node itself. These three techniques explore the tree from left to right. Lemma 6. Without loss of generality. then the right-hand subtree.T (0).n < m . For each of the six techniques mentioned. and lastly. we may suppose that c >. that is.170 Exploring Graphs Chap. the node itself. Proof. and if we visit first the left- hand subtree. Then T(n) <- max (T(g) + T(n-g -1) + c) n > 0 O<g <n -1 We prove by mathematical induction that T (n) <. In this chapter we therefore do not distinguish the two cases. for some m > 0.

Suppose that it is somehow possible to mark a node to indicate that it has already been visited. choose any node v E N as the starting point. call the procedure recursively once again. even when the nodes do not contain a pointer to their parents (otherwise the problem becomes trivial). On return from the recursive call. procedure search (G) for each v E N do mark [v] f. if there is another node adjacent to v that has not been visited. 6. 6.visited for each node w adjacent to v do if mark [w] # visited then dfs (w ) . Problem 6. Assume the trees are represented as in Figure 1.2. On the other hand.3 Depth-First Search : Undirected Graphs 171 so the hypothesis is also true for n = in. To carry out a depth-first traversal of the graph.3 DEPTH-FIRST SEARCH : UNDIRECTED GRAPHS Let G = < N. This proves that T (n) 5 do + c for every n >_ 0. Next. it is clear that T (n) is in c2(n) since each of the n nodes is visited. a recursive implementation takes memory space in S2(n) in the worst case. Show how to generalize the concepts of preorder and postorder to arbitrary (nonbinary) trees. When all the nodes adjacent to v have been marked. * Problem 6.1. Show how the preceding exploration techniques can be implemented so as to take only a time in 0(n) and space in 0(1). A > be an undirected graph all of whose nodes we wish to visit.3. no nodes are marked. and hence T (n) is in O (n). and call the procedure yet again. Therefore T (n) is in O(n).2. Prove that both these techniques still run in a time in the order of the number of nodes in the tree to be traversed. Problem 6.Sec. Here is the recursive algorithm. if there is a node adjacent to v that has not yet been visited. Initially. Continue in this way until all the nodes of G have been marked. If there remain any nodes of G that have not been visited. Mark this node to show that it has been visited.5. the search starting at v is finished. Prove that for any of the techniques mentioned. choose any one of them as a new starting point.2.2. choose this node as a new starting point and call the depth-first search procedure recursively. and so on.not-visited for each v E N do if mark [v] # visited then dfs (v ) procedure dfs (v : node ) [ node v has not been visited } mark [v] . choose this node as the next starting point.9.

and that node 1 is the first starting point.3. 6 The algorithm is called depth-first search since it tries to initiate as many recursive calls as possible before it ever returns from a call.1 if the neighbours of a given node are examined in numerical order but the initial starting point is node 6. a depth-first search of the graph in Figure 6.2). . progress is blocked there are no more nodes to visit. The execution time is thus in 0 (max(a.3. progress is blocked a neighbour of node 1 has not been visited recursive call recursive call.3. we look at the mark on each of its neighbouring nodes.3.1 progresses as follows : dfs (1) dfs (2) dfs (3) dfs (6) dfs (5) dfs (4) dfs (7) dfs (8) initial call recursive call recursive call recursive call recursive call . How much time is needed to explore a graph with n nodes and a edges ? Since each node is visited exactly once. If the graph is represented in such a way as to make the lists of adjacent nodes directly accessible (type lisgraph of Section 1.1.3. Show how a depth-first search progresses through the graph in Figure 6. When we visit a node.1. The algorithm therefore takes a time in 0 (n) for the procedure calls and a time in 0 (a) to inspect the marks. there are n calls of the procedure dfs. If we suppose that the neighbours of a given node are examined in numerical order. The recursivity is only stopped when exploration of the graph is blocked and can go no further.1. n)). 0 Problem 6. Figure 6. this work is proportional to a in total.Exploring Graphs 172 Chap. An undirected graph. Example 6. At this point the recursion "unwinds" so that alternative possibilities at higher levels can be explored.9.

7) and (7. matrix (type adjgraph of Section 1.3. the nodes of the associated tree are numbered in preorder. the second is numbered 2. Edges that are not used in the traversal of the graph have no corresponding edge in the tree.3. See Figure 6. A depth-first search also provides a way to number the nodes of the graph being visited : the first node visited (the root of the tree) is numbered 1.4.2). we need only add the following two statements at the beginning of the procedure dfs : pnum F.1 numbers the nodes as follows : node prenum 1 1 2 2 3 3 4 6 5 5 6 4 The depth-first search illus- 7 7 8 8 11 Of course.3.3. form a spanning tree for the graph in Figure 6.3.3. 12.1) The edges used in the depth-first search of Example 6. and so on. but rather a forest of trees.9. Example 6.4).3.1. a depth-first search associates to it not merely a single tree. The corresponding directed edges (1.1. Show how depth-first search can be used to find the connected components of an undirected graph. Problem 6. (Continuation of Example 6. Problem 6.pnum + 1 prenum [v] E-. A depth-first traversal of a connected graph associates a spanning tree to the graph. (2. (6.3. Exhibit the tree and the numbering generated by the search of .1 are (1.1) trated by Example 6.2.3. The root of the tree is node 1.5). the tree and the numbering generated by a depth-first search in a graph are not unique. (Continuation of Example 6.3. To implement this numbering. 14.2.3. 31.2. 2).3. 6.pnum where pnum is a global variable initialized to zero. 3). If the graph being explored is not connected.6). The edges of the tree correspond to the edges used to traverse the graph . In other words. Example 6. but depend on the chosen starting point and on the order in which neighbours are visited.Sec.3 Depth-First Search : Undirected Graphs 173 What happens if the graph is represented by an adjacency Problem 6.2) rather than by lists of adjacent nodes ? Problem 6. (3. (1. they are directed from the first node visited to the second. and so on.3.8). The initial starting point of the exploration becomes the root of the tree. one for each connected component of the graph.3.

we can be sure that all the nodes will be able to communicate with one another even if one transmission line stops working. The values of lowest are calculated in postorder.7.3) described in Example 6. there remain two connected components (2. b. say. Problem 6. if G is bicoherent.3. The root of T is an articulation point of G if and only if it has more than one child.3.3. These ideas are important in practice : if the graph G represents. for nodes 5. It is bicoherent (or isthmus free.1 generates the tree illustrated in Figure 6.4.3. and for each node v of the graph. 6. The search Example Articulation Points A node v of a connected graph is an articulation point if the subgraph obtained by deleting v and all the edges incident on v is no longer connected. prenum [v] ii. calculate lowest [v] as the minimum of i. the search at node 6. w } in G that has no corresponding edge in T W. 4. 3. Carry out a depth-first search in G. and 1 successively. For each node v visited. if we delete it. that is. starting from any node. Verify that the same articulation points are found if we start 11 . A graph G is biconnected (or unarticulated) if it is connected and has no articulation points. The following algorithm finds the articulation points of a connected graph G . H. let prenum [v ] be the number assigned by the search. For example.3. A node v other than the root of T is an articulation point of G if and only if v has a child x such that lowest [x] >_ prenum [v]. 6. lowest [x] for every child x of v in T. and 6.3. Traverse the tree T in postorder. c. Articulation points are now determined as follows: i. The articulation points of G are nodes 1 (by rule c(i)) and 4 (by rule c(ii)). and the value of lowest [v] to the right.1 .2. node 1 is an articulation point of the graph in Figure 6. The value of prenum [v] appears to the left of each node v. 8.5. or 2-edge-connected) if each articulation point is joined by at least two edges to each component of the remaining sub-graph. (Continuation of Examples 6. prenum [w] for each node w such that there exists an edge [ v. 6 6. 2.3.Exploring Graphs 174 Chap. then the fact that it is biconnected assures us that the rest of the network can continue to function even if the equipment in one of the nodes fails .3. Let T be the tree generated by the depth-first search. a.2. a telecommunications network. 8) .1. The edges of G that have no corresponding edge in T are represented by broken lines.61 and (4. 7.5.

7. Problem 6.3. 175 A depth-first search tree .3.3.3. there thus exists a chain of edges that joins x to the other nodes of the graph even if v is deleted. nected graph is bicoherent. .3. x to the parent of v if v is not the root and if Problem 6.2) necessarily joins some node v to one of its ancestors in T. Complete the proof that the algorithm is correct. 6. prenum on the left and lowest on the right.Sec.2. * Problem 6. Informally. (a broken line in Figure 6. finds a set of edges that could be added to make the graph biconnected.3.3.3 Depth-First Search : Undirected Graphs Figure 6.8.6. given an undirected graph that is connected but not biconnected. Write an efficient algorithm that.10. there is no chain joining lowest [x] >_ prenum [v]. Your algorithm should find the smallest possible set of edges. Show how to carry out the operations of steps (a) and (b) in parallel and write the corresponding algorithm. we can define lowest [v] by lowest [v] = min(prenum [w] I you can get to w from v by following down as many solid lines as you like and then going up at most one broken line) .9. Write an algorithm that decides whether or not a given con* Problem 6. If x is a child of v and if lowest [x] < prenum [v]. On the other hand. Analyse the efficiency of your algorithm. Prove that an edge of G that has no corresponding edge in T Problem 6.

6 Prove or give a counterexample : i. progress is blocked there are no more nodes to visit.13. n)). ii. DEPTH-FIRST SEARCH : DIRECTED GRAPHS The algorithm is essentially the same as the one for undirected graphs. If a graph is biconnected.3.4. then w is adjacent to v but v is not adjacent to w. then it is biconnected. Problem 6. 6. and if the starting point is node 1. node w is adjacent to node v if the directed edge (v . Consider a depth-first search of the directed graph in Figure 6.12. If the neighbours of a given node are examined in numerical order. In a directed graph. 8. If (v .3. Prove that a node v in a connected graph is an articulation point if and only if there exist two nodes a and b different from v such that every path joining a and b passes through v.Exploring Graphs 176 Problem 6.4. 5. w) exists. the edges used to visit all the nodes of a directed graph G = <N. dfs(1) 2. If a graph is bicoherent. The algorithm behaves quite differently. With this change of interpretation the procedures dfs and search from Section 6. biconnected graph. the algorithm progresses as follows : 1. however.3 shows that the time taken by this algorithm is also in 0 (max(a. initial call recursive call recursive call . A > may form a forest of several trees even . dfs (2) dfs (3) dfs (4) dfs (8) dfs (7) dfs (5) dfs (6) 4.11. however.1. Chap. w) exists and (w . the difference being in the interpretation of the word "adjacent". Illustrate the progress of the algorithm if the neighbours of a Problem 6. 6. 0 An argument identical with the one in Section 6. 3. progress is blocked new starting point recursive call . given node are examined in decreasing numerical order. In this case. v) does not.3 apply equally well in the case of a directed graph. progress is blocked a neighbour of node 1 has not been visited recursive call recursive call . 7. then it is bicoherent. there exist at least two chains of edges joining v and w that have no nodes in common (except the starting and ending nodes). Prove that for every pair of distinct nodes v and w in a Problem 6.

ii.2. and W. Prove that if (v . . those like (5.3) that join one node to another that is neither its ancestor nor its descendant.2.4. 1) or (7. 8) that lead from a node to one of its descendants .Sec. 4). Those like (3. (1.1. then prenum [v] > prenum [w]. namely (1. 8).3.2.2) or ( (8. and if v is neither an ancestor nor a descendant of w in the forest. form the forest shown by the solid lines in Figure 6. 6). In the case of a directed graph three kinds of edges can appear in A \ F (these edges are shown by the broken lines in Figure 6.4. This happens in our example : the edges used. 177 A directed graph. A depth-first search forest.2). those like (1.4. (The numbers to the left of each node are explained in Section 6. Problem 6.4) that lead from a node to one of its ancestors .4. w) is an edge of the graph that has no corresponding edge in the forest. 2).4.4 Depth-First Search : Directed Graphs Figure 6. if G is connected. where the values of prenum are attributed as in Section 6. i. In the case of an undirected graph the edges of the graph with no corresponding edge in the forest necessarily join some node to one of its ancestors (Problem 6.) Let F be the set of edges in the forest. Figure 6. 3). 6. (4. and (5. 7). (2.4. Edges of this type are necessarily directed from right to left.

Prove that G is acyclic if and only if A \ F includes no 11 edge of type (i) (that is.4. For example. Figure 6. (What is the partial ordering in question?) Finally. and the edges correspond to activities that have to be completed to pass from one state to another. directed graph G = <N.4. Such graphs also offer a natural representation for partial orderings (such as the relation "smaller than" defined on the integers and the set-inclusion relation). directed acyclic graphs are often used to represent the different stages of a complex project : the nodes are different states of the project.d) . Let F be the forest generated by a depth-first search on a Problem 6.3 represents the structure of the expression (a +b) (c +d) + (a +b) (c .4. Figure 6.5 gives an example of this type of diagram.3 A directed acyclic graph. Figure 6.1 Acyclic Graphs: Topological Sorting Directed acyclic graphs can be used to represent a number of interesting relations. This class of structures includes trees.4 illustrates part of another partial ordering defined on the integers.4. A >.4. . from a node of G to one of its ancestors in the forest).4. 6 6. Depth-first search can be used to detect whether a given directed graph is acyclic. Another directed acyclic graph.Exploring Graphs 178 Chap.4.4. from the initial state to final completion. but is less general than the class of all directed graphs. a directed acyclic graph can be used to represent the structure of an arithmetic expression that includes repeated subexpressions : thus Figure 6. Figure 6.3.

8. j ). 3.4. 2. 6.2 Strongly Connected Components A directed graph is strongly connected if there exists a path from u to v and also a path from v to u for every distinct pair of nodes u and v. The necessary modification to the procedure dfs to make it into a topological sort is immediate : if we add a supplementary line write v at the end of the procedure. E. the natural order 1. 12.8).4. 4.4) and (1. 2.4. as are several others. Each of these subgraphs is called a strongly connected component of the original graph. Problem 6. G will serve. 24 is also acceptable. 24 is adequate. D. nodes 11.5. for the graph of Figure 6. we are interested in the largest sets of nodes such that the corresponding subgraphs are strongly connected.4. a topological ordering of the states can be used to obtain a feasible schedule for the activities involved in the project. . Yet another directed acyclic graph.5. F.8}. For graphs as in Figure 6. If a directed graph is not strongly connected.4 Depth-First Search : Directed Graphs 179 C make coffee drink coffee dress D get out G bring documents Figure 6. Another component corresponds to the nodes {4. 4. 3) and the corresponding edges form a strongly connected component.4. For example. the numbers of the nodes will be printed in reverse topological order. A topological sort of the nodes of a directed acyclic graph is the operation of arranging the nodes in order in such a way that if there exists an edge (i. 6. 12.5. 2.7.1. Prove this.Sec. what is the topological order obtained if the neighbours of a node are visited in numerical order and if the depth-first search begins at node I ? 6. For the graph of Figure 6. it is not possible to merge these two strongly connected components into a single component because there exists no path from node 4 to node 1.4.4. Problem 6. 3. for instance. 6.4. B. but the order 1.4. then i precedes j in the list. 8. In the graph of Figure 6. Despite the fact that there exist edges (1. in our example. the order A. C.4.4.

4. The corresponding forest is illustrated in Figure 6.7.4. 7. (If G contains n nodes. we need only add at the end of procedure dfs the following two statements : nump F nump + 1 postnum [v] . and so on.4. Begin this search at the node w that has the highest value of postnum. this time the remaining nodes 4.1) are the subgraphs corresponding to the sets of nodes (5. iv.2. Construct a new graph G' : G' is the same as G except that the direction of every edge is reversed. we must first modify the procedure dfs. it follows that postnum [w] = n. The strongly connected components of the original graph (Fig. In other words.4. On the graph of Figure 6. Carry out a depth-first search in G'. and 2. For each node v of the graph let postnum [v] be the number assigned during the search.4. 8 1. To do this.180 Exploring Graphs Chap. the search reaches nodes 5 and 6. Prove that if two nodes u and v are in the same strongly connected component of G. 3. 7. 6 To detect the strongly connected components of a directed graph. with postnum [1] = 6. choose as the second starting point the node that has the highest value of postnum among all the unvisited nodes . Carry out a depth-first search of the graph starting from an arbitrary node. Here. We carry out a depth-first search starting from node 5. with postnum [4] = 5 . . this time the search reaches nodes 1.2 shows to the left of each node the number thus assigned. 21 and '{ 4. the nodes of the tree pro- duced are numbered in postorder. with the values of postnum shown to the left of each node. 3.4.6.) If the search starting at w does not reach all the nodes. 11. then they are in the same tree when we carry out the depthfirst search of G'. i. In Section 6. * Problem 6. we number each node at the moment when exploration of the node has been completed. The graph G' is illustrated in Figure 6. iii. Figure 6.1.6.nump where nump is a global variable initialized to zero. 6). For our second starting point. the first depth-first search assigns the values of postnum shown to the left of each node in Figure 6.1. ii.4. since postnum [5] = 8. The following algorithm finds the strongly connected components of a directed graph G . and 8 are all reached.3 we number each node at the instant when exploration of the node begins. To each tree in the resulting forest there corresponds one strongly connected component of G. For the third starting point we take node 4. Example 6. we choose node 1. 6.4.

4. In the third case it would be necessary for the same reason that r be to the right of v. and suppose v #r. Let v be a node that is in the tree whose root is r when we carry out the search of G'. Since we chose r rather than v to be the root of the tree in question. When we carried out the search in G. three possibilities seem to be open a priori: r was an ancestor of v .7.4 Depth-First Search : Directed Graphs Figure 6. It is harder to prove the result the other way.4. say) of v and r. We should have postnum [x] > postnum [r] since x is an ancestor of r. 6.Sec. since there exists a path from v to x in G.6. Since in a depth-first search the edges not used by the search never go from left to right (Problem 6.2). or r was neither an ancestor nor a descendant of v. there would exist a path from x to v in G'. The second possibility is ruled out by the fact that postnum [r] > postnum [v]. The forest of strongly connected components. we have postnum [r] > postnum [v]. 0 O O 0 O Figure 6.4. When carrying out the search of G'.1. Next. there exists at least one path from v to r in G. r was a descendant of v . as the root of a new tree) that node not yet visited with the highest value of postnum. But this is quite impossible. However.4. This implies that there exists a path from r to v in G' . Before choosing r as . we always choose as a new starting point (that is. any such path must go up the tree from v to a common ancestor (x. thus there exists a path from v to r in G. and then go down the tree to r. 181 Reversing the arrows in the graph of Figure 6.

To underline the similarities and the differences between the two methods. breadth-first search is not naturally recursive. This type represents a list of elements that are to be handled in the order "first come. Only the first possibility remains : r was an ancestor of v when we searched G. This contradicts the hypothesis that v is in the tree whose root is r when we carry out the search of G. it next tries to visit some neighbour of v.4. Here is the modified depth-first search algorithm. we would have already visited x (otherwise x rather than r would be chosen as the root of the new tree) and therefore also v. then there exist in G both a path from v to r and a path from r to v. 6 the root of a tree in the search of G'. and so on. This implies that there exists a path from r to v in G. If two nodes u and v are in the same tree when we search G'. it first visits all the neighbours of v. Here now is the breadth-first search algorithm. When a breadth-first search arrives at some node v. The function top denotes the element at the top of the stack. procedure dfs'(v : node) P F empty-stack mark [v] t.6. 6. first served". and not until this has been done does it go on to look at nodes farther away. Unlike depth-first search. Let stack be a data type allowing two operations. The function first denotes the element at the front of the queue. on the other hand. With the result of Problem 6. then a neighbour of this neighbour.Exploring Graphs 182 Chap. by contrast.5 BREADTH-FIRST SEARCH When a depth-first search arrives at some node v. they are therefore both in the same strongly connected component of G since there exist paths from u to v and from v to u in G via node r.4. this completes the proof that the algorithm works correctly. push and pop. we need a type queue that allows two operations enqueue and dequeue. Estimate the time and space needed by this algorithm. we begin by giving a nonrecursive formulation of the depth-first search algorithm. Problem 6. first served".7. . The type is intended to represent a list of elements that are to be handled in the order "last come. We have thus proved that if node v is in the tree whose root is r when we carry out the search of G'.visited push w on P { w is now top (P) } pop top (P) For the breadth-first search algorithm.visited push v on P while P is not empty do while there exists a node w adjacent to top (P) such that mark [w] # visited do mark [w] .

A >.7.empty-queue mark [v] F-. Figure 6.Sec. Problem 6. The edges of the graph that have no corresponding edge in the tree are represented by broken lines.1. and if node I is used as the starting point.not-visited for each v E N do if mark [v] # visited then { dfs' or bfs } (v ) On the graph of Figure 6.5 Breadth-First Search 183 procedure bfs (v : node ) Q .5.3. but that neither u nor v is an ancestor of the other. After a breadth-first search in an undirected graph G = <N.5.visited enqueue w into Q In both cases we need a main program to start the search.5. 8 - As for depth-first search. namely 0 (max(a. 4 5.5.visited enqueue v into Q while Q is not empty do u <--first (Q ) dequeue u from Q for each node w adjacent to u do if mark [w] # visited then mark [w] t.7.4 2.5.5. if the neighbours of a node are Example 6.8 6. visited in numerical order. Show that the edges { u . shows the tree generated by the search in Example 6. procedure search (G) for each v E N do mark [v] . 3 4. 6 7. we can associate a tree with the breadth-first search. n)). 2 3.8 7. 1 2.6 3.6 4. Node Visited Q 1. 5 6.1. If the appropriate . 6. It is easy to show that the time required by a breadth-first search is in the same order as that required by a depth-first search. breadth-first search proceeds as follows.8 5. 7 8 8. let F be the set of edges that have a corresponding edge in the forest of trees that is generated. v } E A \ F are such that u and v are in the same tree.1.6.

1. we can use the nodes of a graph to represent configurations in a game of chess and edges to represent legal moves (see Section 6. 6 Figure 6. interpretation of the word "neighbouring" is used.6 IMPLICIT GRAPHS AND TREES As mentioned at the outset of this chapter. it may be wasteful or infeasible to build it explicitly in computer memory before applying one of the search techniques we have encountered so far. If the graph contains a large number of nodes. Backtracking is a basic search technique on implicit graphs. One powerful application is in playing games of strategy by techniques known as minimax and alpha-beta . Show how a breadth-first search progresses through the graph of Figure 6.Exploring Graphs 184 Chap.5. Relevant portions of the graph can thus be built as the search progresses. 6.2.5. assuming that the neighbours of a node are always visited in numerical order. A breadth-first search tree. An implicit graph is one for which we have available a description of its nodes and edges.3 (Continuation of Problem 6. Therefore computing time is saved whenever the search succeeds before the entire graph has been constructed. The economy in memory space is even more dramatic when nodes that have already been searched can be discarded. and that necessary starting points are also chosen in numerical order. For instance. path or pattern in the associated graph.4.2) Sketch the corresponding forest and the remaining edges of the graph.1.4) Breadth-first search is most often used to carry out a partial exploration of certain infinite graphs or to find the shortest path from one point to another. the breadth-first search algorithm can be applied without modification to either directed or undirected graphs. Problem 6.6.2).5. Problem 6. Often the original problem translates to searching for a specific node. various problems can be thought of in terms of abstract graphs. How many kinds of "broken" edges are possible ? (see Section 6.5. making room for subsequent nodes to be explored.

and also two pairs of queens lie on the same diagonal.1 Backtracking Backtracking algorithms use a special technique to explore implicit directed graphs. (A queen threatens the squares in the same row. the number of cases to be considered is reduced to 88 = 16. Using this representation. the vector (3.777. in the same column.299. however. These graphs are usually trees.. Consider the classic problem of placing eight queens on a chess-board in such a way that none of them threatens any of the others. For instance. 6. we can write the algorithm very simply using eight nested loops. 8. program Queens 1 for ii-lto 8do for i2-lto 8do for i8-lto 8do try F-(iI.426. Some optimization problems can be handled by the more sophisticated branch-and-bound technique. This approach is of no practical use. The first obvious way to solve this problem consists of trying systematically all the ways of placing eight queens on a chess-board.. This might try consists of never putting more reduces the computer representation of the chess-board to a simple vector of eight elements. 7) does not represent a solution since the queens in the third and the sixth rows are in the same column. 6.. Solve this problem without using a computer. or at least they contain no cycles.368.165. looking for solutions to some problem. use a different example. 1. i8) if solution (try) then write try stop write "there is no solution" This time. each giving the position of the queen in the corresponding row.6. The first improvement we a than one queen in any given row. We now discuss these notions.Sec. 4. 2. To illustrate the general principle.852 cases. since the number of positions we have to check would be [64 J = 4. 216.) Problem 6.6 Implicit Graphs and Trees 185 pruning. . or on the same diagonals. although in fact the algorithm finds a solution and stops after considering only 1. At least one application of this technique dates back to antiquity : it allows one to find the way through a labyrinth without danger of going round and round in circles. checking each time to see whether a solution has been obtained. we shall.i2. 6. 6. A backtracking algorithm carries out a systematic search.6.1..

. perm (1). n ] is a global array initialized to [ 1. The algorithm becomes program Queens 2 try initial permutation while try # final-permutation and not solution (try) do try F. Hence we now represent the board by a vector of eight different numbers between 1 and 8. If the preceding algorithm is used to generate the permutations.3.. all the permutations of the n -1 remaining elements. easier in this case to verify whether a given position is a solution. only 2. Since we already know that two queens can neither be in the same row nor in the same column.320.6. that is. . which prevents us from ever trying to put two queens in the same if solution (try) then write try else write "there is no solution" There are several natural ways to generate systematically all the permutations of the first n integers. as a function of n . 6 Problem 6.2. For instance. n ] and the initial call is If use (T) consists simply of printing the array T on a new Problem 6. Although it is more complicated to generate permutations rather than all the possible vectors of eight integers between 1 and 8.. Assuming that use (T) takes constant time. the informa0 tion just given should be of considerable help ! Once we have realized that the chess-board can be represented by a vector. . for each of these leading values. procedure perm (i) if i = n then use (T) { T is a new permutation } else for j F. 0 line.4. to execute the call perm (1) ? Now rework the problem 0 assuming that use (T) takes a time in 0(n). show the result of calling perm (1) when n = 4. 2. it is natural to be equally systematic in our use of the columns. .i to n do exchange T [i] and T [ j ] perm (i +1) exchange T [i] and T [ j ] Here T [ 1 . we might put each value in turn in the leading position and generate recursively. on the other hand. it suffices to verify that they are not in the same diagonal. If you have not yet solved the previous problem.Exploring Graphs 186 Chap. by a permutation of the first eight integers.830 cases are in fact considered before the algorithm finds a solution.6. This approach reduces the number of possible cases to 8! = 40.6. how much time is Problem 6. it is. needed.

V [k]) threatens any of the others. This technique has two advantages over the previous algorithm that systematically tried each permutation. First.. We say that a vector V [1 . 8] is a global array. For instance. 0. For k <. it seems at first sight that we have to check each of the 28 pairs of queens on the board. if none of the k queens placed in positions (1. This check can be speeded up if we associate with each promising node the sets of columns. Although it is not easy to calculate this number theoretically.6 Implicit Graphs and Trees 187 Starting from a crude method that tried to put the queens absolutely anywhere on the chess-board. The solutions to the eight queens problem can be obtained by exploring this tree. In fact. (k. such that U is k -promising. call Queens (0.. Depth-first search is the obvious method to use. and of negative diago^. . the number of nodes in the tree is less than 8! = 40. 4. Let G = <N. any vector V is k -promising. let us reformulate the eight queens problem as a tree searching problem. nodes are generated and abandoned during the course of the exploration. Solutions to the eight queens problem correspond to vectors that are 8-promising. Secondly. j . Mathematically. k] of integers between I and 8 is k-promising. 6. 2. Its root is the empty vector (k =0). to decide if some given permutation represents a solution. 0 5 k S 8. 0. V [2]) . (2. Its leaves are either solutions (k = 8) or else they are dead ends (k < 8) such as [1.1. 0 <_ k < 8. for 0S k 5 8. where try [I .Sec.i ). in order to decide whether a vector is k-promising. To print all the solutions to the eight queens problem.11s (at 135 degrees) controlled by the queens already placed. for every i #j between 1 and k. we progressed first to a method that never puts two queens in the same row. V is (k +l )-promising. We do not generate the tree explicitly so as to explore it thereafter. This graph is a tree. and U[i]=V[i] for every i E [1 . all these algorithms share an important defect : they never test a position to see if it is a solution until all the queens have been placed on the board. 5. 8] where it is impossible to place a queen in the next row without her threatening at least one of the queens already on the board. k]. 0. however : rather. we have V[i] -V [j ] it (i . knowing that it is an extension of a (k -1)-promising vector. As a first step. where of course they threaten one another ! Backtracking allows us to do better than this. it is straightforward to count the nodes using a computer : #N = 2057. it suffices to explore 114 nodes to obtain a first solution. Let N be the set of k-promising vectors. A > be the directed graph such that (U.. .. even the best of these algorithms makes 720 useless attempts to put the last six queens on the board when it has started by putting the first two on the main diagonal.320. a vector V is k-promising if. V [1]). However. particularly if we only require one solution. of positive diagonals (at 45 degrees). and then to a better method still where the only positions considered are those where we know that two queens can neither be in the same row nor in the same column. V ) E A if and only if there exists an integer k. 0). . On the other hand. we only need to check the last queen to be added.j .

k ] do backtrack (w [I . . k ] = v [I . col. Find a more interesting case than n = 2. For example. Analyse mathematically.546.. diag 135 u { j +k)) It is clear that the problem generalizes to an arbitrary number of queens : how can we place n queens on an n x n "chess-board" in such a way that none of them threatens any of the others ? Show that the problem for n queens may have no solution. as a function of the number n of queens.600 possible permutations to be considered. 6 procedure Queens (k.1 to 8 do if j Ocol and j -k Odiag45 and j +k Odiag 135 then try [k +I] F.189 nodes. and the first solution to be found (using the generator given previously) corresponds to the 4. the number of nodes in the tree of k -promising vectors. procedure backtrack (v [ 1 . col={try [i]I 1!5 i:5k}.6.001. on the other hand.. diag45 = {try[i]-i +l I 1 !5i !5k). Problem 6. the advantage to be gained by using the backtracking algorithm instead of an exhaustive approach becomes more pronounced as n increases. As we might expect.. k ] is k -promising. for n = 12 there are 479. Here is the general scheme. col u { j }. and a solution is obtained when the 262nd node is visited. diag45.j } { try (L. k +1]) The otherwise should be present if and only if it is impossible to have two different solutions such that one is a prefix of the other. k +11 is (k +1)-promising } Queens (k +1.Exploring Graphs 188 Chap. How does this number compare to n! ? Backtracking algorithms can also be used even when the solutions sought do not necessarily all have the same length.044th position examined .6. k ]) { v is a k -promising vector) if v is a solution then write v { otherwise } for each (k +1)-promising vector w such that w [1 . and diag135={try [i]+i-1I 15i <-k} } if k = 8 then { an 8-promising vector is a solution } write try else { explore (k+1)-promising extensions of try for j F. diag45 u { j -k }.. diag 135) { try [ 1 .6. the tree explored by the backtracking algorithm contains only 856.5.. ** Problem 6.

What does your algorithm do if it is impossible to transform n into m in this way ? * Problem 6. Breadth-first search is also appropriate if we have to find a solution starting from some initial position and making as few changes as possible. red. 6. Some problems that can be formulated in terms of exploring an implicit graph have the property that they correspond to an infinite graph.6. Show how to solve this problem using backtracking.10. (This last constraint does not apply to the eight queens problem where each solution involves exactly the same number of pieces. The graph is infinite if there is no a priori limit on the number of positions possible in the game. and an edge corresponds to a legal move between two positions. or white.6. that the game is symmetric (the rules are the same for both players). your algorithm should say so rather than calculating forever. it may be necessary to use breadth-first search to avoid the interminable exploration of some fruitless infinite branch. The n-queens problem was solved using depth-first search in the corresponding tree. and that chance plays no part in the outcome (the game is deterministic). Problem 6. A node of the graph corresponds to a particular position in the game. We further suppose that no instance of the game can last forever and that no position in the game offers an infinite number of legal moves to the player whose turn it is. on one of the front faces.6.6. on one of the four bottom faces. For instance. .6 Implicit Graphs and Trees 189 Problem 6.6. In this case. The ideas we present can easily be adapted to more general contexts. For simplicity. Problem 6. some positions in the game offer no legal moves. green. The four cubes are to be placed side by side in such a way that each colour appears on one of the four top faces.8. 15 can be transformed into 4 using four function applications : 4 = gfgg (15). Give an algorithm that determines the shortest possible series of manipulations needed to change one configuration of Rubik's Cube into another.9. we assume that the game is played by two players.) The two following problems illustrate these ideas. Each of the 24 faces is coloured blue.2 Graphs and Games: An Introduction Most games of strategy can be represented in the form of directed graphs. and hence some nodes in the graph have no successors : these are the terminal positions. each of whom moves in turn. Give an algorithm capable of transforming some initial integer n into a given final integer m by the application of the smallest possible number of transformations f(i) = 3i and g(i) = Li/21. Instant Insanity is a puzzle consisting of four coloured cubes.7. 6. and on one of the rear faces.Sec. Give other applications of backtracking. If the required change is impossible. In particular.

draw. we need only attach to each node of the graph a label chosen from the set win. then you may verify that my opponent has a winning strategy. Can a player who finds himself in a winning position lose if his opponent makes an "error"? Problem 6. you may verify that a player who has the first move in a game with eight matches cannot win unless his opponent makes an error. iii. Initially. each player in turn must remove at least one match and at most twice the number of matches his opponent just took.6. The player who has the first move in a game with seven matches is therefore cer- tain to win provided that he does not make an error. The labels are assigned systematically in the following way. The label corresponds to the situation of a player about to move in the corresponding position. The first player removes as many matches as he likes. or to remove more than two.6. except that he must take at least one and he must leave at least one. Grasp intuitively how these rules arise.6. iv. at least two matches are placed on the table between two players. A nonterminal position is a losing position if all of its successors are winning positions. Example 6. There are no draws.1). In the case of an acyclic finite graph (corresponding to a game that cannot continue for an indefinite number of moves). Any remaining positions lead to a draw. but this is not necessarily the case (think of stalemate in chess). or four. On the other hand. Thereafter. three. if at the outset I choose to remove a single match. ii. I can in turn remove a single match. We illustrate these ideas with the help of a variant of Nim (also known as the Marienbad game). If I take two of them.4. If he takes more than one.1. There are seven matches on the table initially. 6 To determine a winning strategy for a game of this kind. if you find yourself in a terminal position. find a relationship between this method of labelling the nodes and topological sorting (Section 6.11. i. and he cannot prevent me from winning on my next turn.Exploring Graphs 190 Chap.12. For most games. leaving four matches on the table. The player who removes the last match wins. The labels assigned to terminal positions depend on the game in question. lose. I can remove all the matches that are left and win. On the other hand. and you have lost . Problem 6. A nonterminal position is a winning position if at least one of its successors is a losing position. two. . my opponent may take one. If he takes only one match. assuming that neither player will make an error. then there is no legal move you can make.

j. The nodes of the graph corresponding to this game are therefore pairs < i . . A position in this game is not specified merely by the number of matches that remain on the table.6.i . corresponding to the fact that such positions offer no winning move. j >.1. three. 7 >. Figure 6.2.Sec.6 Implicit Graphs and Trees 191 * Problem 6. or five matches has no winning strategy.6. < 6.13.6. Similarly. The node corresponding to the initial position in a game with n matches. i -k)>. j >. The heavy edges correspond to winning moves : in a winning position.1 shows part of the graph corresponding to this game. n -1 >. < i . 6. < 7. 1 k <. Figure 6. j > with j odd and j < i -1 cannot be reached starting from any initial position. whereas he does have such a strategy in the game with four matches. In general. 0 > for i > 0 are inaccessible. is < n . n 2. For n >. min(2k. Part of a game graph.6. The square nodes represent losing positions and the round nodes are winning positions. j > go to the j nodes < i -k.6. 5 > and their descendants to the graph of Figure 6. There are no heavy edges leaving a losing position. We observe that a player who has the first move in a game with two. Prove your answer. The edges leaving node < i . 0 > is interesting : the nodes < i. 6 >. All the nodes whose second component is zero correspond to terminal positions.14. indicates that i matches remain on the table and that any number of them between 1 and j may be removed in the next move. nodes < i. Your characterization of n should be as simple as possible. give a necessary and sufficient condition on n to ensure that the player who has the first move in a game involving n matches have a winning strategy.1. Add nodes < 8. Problem 6. but only < 0. It is also necessary to know the upper limit on the number of matches that it is permissible to remove on the next move. 1 <_ j <. choose one of the heavy edges in order to win.

not G [i -k .I < i . but rec (3. j > is a winning position. For instance. 2). as well as all the values of G [i.k + 1 G [i . k ] for 1 <. are there positions in which several different winning moves are available? Can this happen in the case of a winning initial position <n. j ] = true if and only if < i. As usual with dynamic programming.n-1>? The obvious algorithm to determine whether a position is winning is the following. rec (2. k ] for 1 <. do not read the following paragraphs yet !) The first approach consists of using dynamic programming to create a Boolean array G such that G [i. This algorithm suffers from the same defect as the algorithm fib! in Section 1. j ) { returns true if and only if the node < i. min(2k .1 to j do if not rec (i -k.false for i f. 3).1 to n do for j f.Exploring Graphs 192 Chap. 2) and rec (1. 3) also calls rec (2. (If you want to work on this problem. Find two ways to remove this inefficiency. 2) and rec (1.! while k < j and G [i -k. Problem 6.7. procedure dyn (n) {foreach1<-j <<-i n. min(2k.k <. G [i j] is set to true if and only if configuration < i. 0] E.6. Can a winning position have more than one losing position among its successors ? In other words. Modify this algorithm so that it returns an integer k such that k = 0 if the position is a losing position and 1 <. i -k)) then return true return false Problem 6. 4) returns false having called successively rec (4. i -k )] . we proceed in a bottom-up fashion. function rec (i. we assume that 0 j <. calculating all the values of G [1.17.5: it calculates the same value over and over. rec (3. j ] F.6. j > is winning } G [0. j ].k < j.1 to i do k E. 1). 1).j if it is a winning move to take away k matches. 6 Problem 6.6.k <.16. min(2k. j > is winning .i } for k F. i -k)] do k E.15. rec (5. before calculating G [i .

without explanation. but there is no "bottom-up" reason for not calculating G [12. In an initial position with n matches. we know that < 15. only 27 nodes are really useful when we calculate G [15. 1 <. initialized to false. 0. A solution that combines the advantages of both the algorithms consists of using a memory function (Section 5. 0. Using the technique suggested in Problem 5. 0] and the values of Problem 6. Because of its top-down nature. In this context dynamic programming leads us to calculate wastefully some entries of the array G that are never needed.j <.true for k . 4 > is a losing position. since these nodes are never of interest. Here. 14]. to calculate G [i. 6 > is a winning or a losing position. it never calculates an unnecessary value. although the dynamic programming algorithm looks at 121 of them. . For instance.7).. j ) if init [i. n ].6 Implicit Graphs and Trees 193 The preceding algorithm only _ises G [0. The game we have considered up to now is so simple that it can be solved without really using the associated graph.. n. The recursive algorithm given previously is inefficient because it recalculates the same value several times. 1 <. where n is an upper bound on the number of matches to be used. i -k)) then G [i. Thereafter a call on whatnow (i . j] then return G [i.. j ).6. n.2 allows us. because in any case we have to take the time to initialize the whole array init [0. This involves remembering which nodes have already been visited during the recursive computation using a global Boolean array init [0. determines in a time in O(1) the move to make in a situation where i matches remain on the table and the next player has the right to take at most j of them.Sec.1 to j do if not nim (i -k.true return true G [i.l < i . 6.6]. function nim (i.7.. k ] for 1 <. Show how to improve its efficiency by also using the values of G [i .k < j. It is no longer of any interest to know whether the next successor < 12. n ] is global. is an algorithm for determining a winning strategy that is more efficient than any of those given previously. there is no particular reason to favour this approach over dynamic programming. j ] whenever j is odd and j < i -1. G [1.18.. j] E. j] . n ]. About half this work can be avoided if we do not calculate G [i . j] false return false At first sight. to avoid this initialization and to obtain a worthwhile gain in efficiency. j]. The array T [0. k]. however.14 > is a winning position as soon as we discover that its second successor < 13. The initial call of precond (n) is an application of the preconditioning technique to be discussed in the next chapter.k <.i. In fact. min(2k. however. j] init [i. first call precond (n).

However. namely chess. the graph associated with this game contains cycles.k function whatnow (i. Once constructed. a game is declared to be a draw after 50 moves in which no irreversible action (movement of a pawn. j ) if j < T [i] then { prolong the agony ! } return 1 return T [i] * Problem 6. Estimate the number of ways in which the pieces can be * Problem 6.2k do k F k + 1 T[i] . and whether some pawn has just been moved two squares forward (to know whether a capture en passant is possible).19. 6 procedure precond (n) T [0] t-for i F 1 to n do k<-1 while T [i -k ] <. a position in chess is not defined simply by the positions of the pieces on the board. but also by an invisible item of information giving the number of matches that can be removed on the next move. which rooks and kings have moved since the beginning of the game (to know if it is legal to castle).6. Furthermore. there are no cycles in the graph corresponding to chess.Exploring Graphs 194 Chap. to win whenever it is possible and to lose only when it is inevitable. Thanks to such rules. since if two positions u and v of the pieces differ only by the legal move of a rook. say. the graph contains so many nodes that it is quite out of the question to explore it completely. Prove that this algorithm works correctly and that precond (n) takes a time in 0(n). (For simplicity we ignore exceptions to the 50-move rule. Consider now a more complex game. Remember first that in the game we just looked at. Thus we must include in our notion of position the number of moves made since the last irreversible action. Unfortunately (or perhaps fortunately for the game of chess). Similarly. then we can move equally well from u to v and from v to u. even with the fastest existing computers.20. as well as the older rule that makes a game a draw if the pieces return three times to exactly the same positions on the board. that is. a winning position for Black. For simplicity ignore the fact that certain positions are . or a draw. At first sight. we can therefore label each node as being a winning position for White.) Adapting the general rules given at the beginning of this section.6. this problem disappears on closer examination. the International Chess Federation has rules that prevent games dragging on forever : for example. or a capture) took place. placed on a chess-board. the king not being in check. We also need to know whose turn it is to move. this graph allows us to play a perfect game of chess. a position is defined not merely by the number of matches on the table.

Although it does not allow us to be certain of winning. and so on. the evaluation function should return +oo if Black has been mated. using one of several possible criteria. A compromise must be made between the accuracy of this function and the time needed to calculate it. Exploration of the graph is usually stopped before the leaves are reached. and the positions thus reached are evaluated heuristically. val F-oo for each configuration w that is a successor of u do if eval(w) >. and 0 if the game is a draw. Suppose it is White's turn to move from position u. that is. and large negative values to positions that favour Black. This technique finds a move that may reasonably be expected to be among the best moves possible while exploring only a part of the graph starting from some given position. In this situation the recursive approach comes into its own. an evaluation function that takes good account of the static aspects of the position but that is too simplistic to be of real use might be the following : for nonterminal configurations. -oc if White has been mated. When applied to a terminal position. If the static evaluation function were perfect. and 10 points for each white queen . Here we give only an outline of the technique.W It is clear that this simplistic approach would not be very successful using the evaluation function suggested earlier. and that both kings must be on the board). The best move would be to go to the position v that maximizes eval (v) among all the successors w of u. It is customary to give values not too far from zero to positions where neither side has a marked advantage. it would be easy to determine the best move to make. they can never be obtained from the initial position by a legal series of moves (but take into account the fact that each bishop moves only on either white or black squares. Since a complete search of the graph associated with the game of chess is out of the question.Sec. 5 points for each white rook. The first step is to define a static evaluation function eval that attributes some value to each possible position. 6. For example. Ideally. The minimax principle. freedom of movement. Ignore also the possibility of 11 having promoted pawns.val then val . Then we make the move that seems to cause our opponent the most trouble. control of the centre.eval(w) V f. count 1 point for each white pawn.6 Implicit Graphs and Trees 195 impossible. This evaluation function must take account of many factors : the number and the type of pieces remaining on both sides. since it would not hesitate to sacrifice a queen in order to take a pawn ! . 31/4 points for each white bishop or knight. This is in a sense merely a systematic version of the method used by some human players that consists of looking ahead a small number of moves. it underlies an important heuristic called minimax. subtract a similar number of points for each black piece. it is not practical to use a dynamic programming approach. we want the value of eval (u) to increase as the position u becomes more favourable to White.

eval(w) else valw . the better the position is supposed to be for him. on the other hand. . since the smaller the value taken by this function. and White. n) if n = 0 or w has no successor then return eval(w) else return min { White (x. tries to maximize the advantage he obtains from each move.Black (w. n -1) 1 x is a successor of w } function White (x. val --°° for each configuration w that is a successor of u do if w has no successor then valw .min{eval(x) I x is a successor of w } if valw >. n -1) 1 w is a successor of x } We see why the technique is called minimax : Black tries to minimize the advantage he allows to White. we are sure to avoid moves that would allow Black to mate immediately (provided we can avoid this).val then val f.valw v t-W There is now no question of giving away a queen to take a pawn. it is preferable to look several moves ahead. White should move to the position v given by val <. a better strategy for White is to assume that Black will reply with the move that minimizes the function eval.196 Exploring Graphs Chap.W where the functions Black and White are the following : function Black (w. n) >. he would like a large negative value. n) V f.val then val F. n) if n = 0 or x has no successor then return eval(x) else return max { Black (w. To add more dynamic aspects to the static evaluation provided by eval.-°° for each configuration w that is a successor of u do if Black (w.) We are now looking half a move ahead. (Ideally. 6 If the evaluation function is not perfect. On the other hand. To look n half-moves ahead from position U. which of course may be exactly the wrong rule to apply if it prevents White from finding the winning move : maybe if he looked further ahead the gambit would turn out to be profitable.

.Sec. Problem 6.6.6. j >.2 shows part of the graph corresponding to some game. We want to calculate the value of the root < 1. The basic minimax technique can be improved in a number of ways. the values for the other nodes can be calculated using the minimax rule. In the example we suppose that player A is trying to maximize the evaluation function and that player B is trying to minimize it. j > represent the j th node in the i th row of the tree. the exploration of certain branches can be abandoned early if the information we have about them is already sufficient to show that they cannot possibly influence the values of nodes farther up the tree. 6. Let u correspond to the initial position of the pieces. Let < i. besides the fact that it would take far too long to calculate in practice? Justify your answer.2.21. 1 > starting from the values calculated by the function eval for the leaves < 4. he will choose the second of the three possible moves.2. visiting the successors of a given node from left to right.3. Example 6.2.6 Implicit Graphs and Trees Player Rule A max Figure 6. This second type of improvement is generally known as alpha-beta pruning. Figure This assures him of a value of at least 10. it may be worthwhile to explore the most promising moves in greater depth. 12800). we carry out a bounded depth-first search in the tree. Look back at Figure 6. If the values attached to the leaves are obtained by applying the function eval to the corresponding positions. To do this. If A plays so as to maximize his advantage. 1 <_ j 5 18. Example 6. For example. Similarly. 197 The minimax principle. Alpha-beta pruning. We give just one simple example of the technique.6. What can you say about White (u.

After evaluation of the second leaf <4. we know that <4. Similarly.6.24.2>. 3 >.3>. Write a program that can beat the world backgammon champion. What modifications should be made to the principles set out in this section to take account of those games of strategy in which chance plays a certain part ? What about games with more than two players ? Player Rule A max B min A max B eva! -7 5 Figure 6.1> is 10.6. and < 2. there is no need to evaluate the other children of node < 2. we are in the situation shown in Figure 6. I> is at least 10. Since we already know that the value of < 1. . 3 > has value at most 1. < 3. we arrive after evaluation of the leaf <4. Since node <3.6.Exploring Graphs 198 Chap. 1> (a node that maximizes eval) has value at least -7. game of strategy.2> has value 5. Write a program capable of playing brilliantly your favourite 11 ** Problem 6. 3 > cannot have any influence on the value of node < 2. 6 If we want to abandon the exploration of certain branches because it is no longer useful. we know that <4. 1> is evaluated.6. we have to transmit immediately to the higher levels of the tree any information obtained by evaluating a leaf. 1 >. Continuing in this way.6.6.4> at the situation illustrated in Figure 6. -3 10 ? ? Alpha-beta pruning. 1> has value -7 and that <3.3. (This has already been done !) ** Problem 6. ** Problem 6. 1 > has value 5. Thus as soon as the first leaf <4. Node < 2. 1 > (a node that minimizes eval) has value at most 5. 3 > has value at least 10.23. we say that the corresponding branches of the tree have been pruned. To establish that the value of the root <1.22. the exact value of node < 3. 11>. 1 > has value at most -3. we visit only 19 of the 31 nodes in the tree.4. It is therefore unnecessary to evaluate the other descendants of node <3.3. whereas node < 2. after evaluation of the leaf <4.

as we have just explained. Again. to prune certain branches of a tree or to close certain paths in a graph. 3. We return to the travelling salesperson problem (see Sections . using a stack to hold those nodes that have been generated but not yet explored fully . Example 6. using this time a queue to hold those that have been generated but not yet explored (see Section 6.6. In general terms we may say that a depth-first search finishes exploring nodes in inverse order of their creation.4.2 and 5.4.6 Implicit Graphs and Trees Player Rule A max 199 It is not necessary to explore these branches B eval Figure 6.5). In the simplest version. 6.6. More alpha-beta pruning. however.4. the calculated bound is used not only to close off certain paths.6. An example illustrates the technique.6). This time. More often. so that it can be explored first. this graph is usually acyclic or even a tree. a breadth-first search finishes exploring nodes in the order of their creation. calculation of these bounds is combined with a breadthfirst or a depth-first search. and a priority list to hold those nodes that have been generated but not yet explored. Branch-and-bound uses auxiliary computations to decide at each instant which node should be explored next.Sec. At each node we calculate a bound on the possible value of any solutions that might happen to be farther on in the graph. but also to choose which of the open paths looks the most promising. then we do not need to go on exploring this part of the graph. If the bound shows that any such solution must necessarily be worse than the best solution we have found so far. 6. we are looking for the optimal solution to some problem. and serves only.3 Branch-and-Bound Like backtracking. branch-and-bound is a technique for exploring an implicit directed graph.

3). 1). (1. namely. the minimum of 14/2. The nodes in the implicit graph correspond to partially specified paths. and 18/2.2). it suffices to add elements of this kind. or 5: minimum 7/2 A visit to 3 that neither comes from I nor leaves for 2: minimum 11/2 A similar visit to 4: minimum 3 A similar visit to 5: minimum 3 A return to 1 from 3. For instance. 5. this arbitrary choice of a starting point does not alter the length of the shortest tour. 10/2. for example. 2). The successors of a given node correspond to paths in which one additional node has been specified. We have just calculated the lower bound shown for this node. For instance.) Our search begins by generating (as though for a breadth-first search) the four possible successors of the root. (This bound on the root of the implicit tree serves no purpose in the algorithm . A tour that begins with (1. 3. To obtain a bound on the length of a path. or 5: minimum 2 .5 the root of the tree specifies that the starting point for our tour is node 1. a complete tour must include a departure from node 1. and the other half when we arrive at j.6. 11/2. is calculated as follows. Similarly.5). Returning to node 1 costs at least 2.Exploring Graphs 200 Chap. 4. For instance. 4. The bound for node (1. node (1.3) corresponds to two complete tours : (1.3. 2. namely the lowest of the values 14/2. a visit to each of the nodes 2. it was computed here for the sake of illustration. 2) must include The trip 1-2 : 14 (formally. Its length is therefore at least 2+6+4+3+3+2= 20. leaving node I costs us at least 2. Obviously.4.4. Notice that this calculation does not imply the existence of a solution that costs only 20. 4. suppose that half the distance between two points i and j is counted at the moment we leave i. (1. 4/2. leaving I for 2 and arriving at 2 from 1: 7+7 ) A departure from 2 toward 3. To calculate this bound. 4/2. nodes (1. 4. At each node we calculate a lower bound on the length of the corresponding complete tours. and (1. and 5 (not necessarily in this order) and a return to 1. 6 Let G be the complete graph on five points with the following distance matrix : 014 4 10 20 14 0 7 8 7 4 5 0 7 16 11 7 9 0 2 18 7 17 4 0 We are looking for the shortest tour starting from node 1 that passes exactly once through each other node before finally returning to node 1. and 20/2. In Figure 6. 1) and (1. visiting node 2 costs us at least 6 (at least 5/2 when we arrive and at least 7/2 when we leave).

3).6 Implicit Graphs and Trees 201 1 Bound 20 1.4). The most promising node is now (1. for instance. To give just one example.2). 4.2.3. we do not need to calculate a lower bound since we may calculate immediately its length 37.2. 3. 2 :Z Bound 31 1. 3.6. Its two children (1. 2) as follows : The trip 1. 1). Next. Branch-and-bound.Sec. and (1. whose bound is 24. we do not need to continue exploration of the nodes (1. which gives a total length of at least 24.4. 5 Bound 41 Figure 6.5. The three children (1. 2). The length of such a tour is therefore at least 31. If we are only concerned to find one optimal solution.4) and (1. This time.3. corresponds to exactly one complete tour (1.3 -2: 9 A departure from 2 toward 4 or 5: minimum 7/2 A visit to 4 that comes from neither 1 nor 3 and that leaves for neither 2 nor 3: minimum 3 A similar visit to 5: minimum 3 A return to 1 from 4 or 5: minimum 11/2. 1) is 31. 2).5) of this node are therefore generated.3. we calculate the bound for node (1.2. (1.4). We find that the length of the tour (1.2. 3. 6. the most promising node seems to be (1. as node (1. 2.3. The other bounds are calculated similarly.5) are generated.3. .

Show how to solve the same problem using a backtracking algorithm that calculates a bound as shown earlier to decide whether or not a partially defined path is promising. In practice.3. we have looked at merely 15 of the 41 nodes that are present in a complete tree of the type illustrated in Figure 6. 5). 5. 4.Exploring Graphs 202 Chap. the bound at each node being obtained by solving a related problem in linear programming with continuous variables. for problems of the size encountered in applications. In the worst case it may turn out that even an excellent bound does not allow us to cut any branches off the tree. Solve the same problem using the method of Section 5.4) to explore.5. (ii) using a breadth-first search.5). How much time do your algorithms take? . we shall most likely spend more time at each one calculating the corresponding bound. (Why?) There remains only node (1. situated in all the levels of the tree and preferably sorted in order of the corresponding bounds. time in calculating the best possible bound (within reason). 1) of length 30 is optimal. 3. it almost always pays to invest the necessary. 3. The only child to offer interesting possibilities is (1. 1) and (1. 4.7. 5. we find that the tour (1.6. 3. but on the other hand. For instance.6. 6 (1. It is next to impossible to give any idea of how well the technique will perform on a given problem using a given bound. Implement this algorithm on a computer and test it on our example. Problem 6. This example illustrates the fact that although at one point (1. no elegant recursive formulation of branch-and-bound is available to the programmer. There is always a compromise to be made concerning the quality of the bound to be calculated : with a better bound we look at less nodes.4. 1).1. however.26.7 SUPPLEMENTARY PROBLEMS Write algorithms to determine whether a given undirected Problem 6. 6. which cannot possibly lead to a better solution. and all the extra work we have done is wasted. 2. the technique is sufficiently powerful that it is often used in practical applications.4) is pointless. 3. 5.6. After looking at the two complete tours (1. the optimal solution does not come from there.25.27. Unlike depth-first search and its related techniques. 2. The heap is an ideal data structure for holding this list. To obtain our answer.6. 3) was the most promising node. * Problem 6. 2. makes branch-and-bound quite hard to program. one finds applications such as integer programming handled by branch-and-bound. 5) and (1. The need to keep a list of nodes that have been generated but not yet completely explored. Even exploration of the node (1. graph is in fact a tree (i) using a depth-first search . 4. Problem 6. Nevertheless.6.

n).7. In Problem 6. n ] represents a square maze. We want to express 13 in this way. n. How much time does your algorithm take? *Problem 6.3. the edge (v . A node p of a directed graph G = <N.7 Supplementary Problems 203 Problem 6. If M [i.7. Write an algorithm that can detect the presence of a sink in G in a time in O (n).7. to find the root.1 gives an example. (This approach is used in a number of compilers.5. A > is called a sink if for every node v E N .7. v) does not exist. j ). Show how the problem of carrying out a syntactic analysis of a programming language can be solved in top-down fashion using a backtracking algorithm. Show how the problem can be expressed in terms of exploring a graph and find a minimum-length solution.7. you may use statements such as "for each point v that is a neighbour of x do "). Give a backtracking algorithm that finds a path. i. . and if so. An Euler path in a finite undirected graph is a path such that every edge appears in it exactly once. and prints the path if so.. Problem 6. The value 1 is available.7.6. starting from a given point. any resulting fraction being dropped). Repeat Problem 6.. Figure 6. your algorithm must be clear and precise.2.) A Boolean array M [ 1 .7. How much time does your algorithm take? * Problem 6.7. * Problem 6.8.4. it is permissible to go to adjacent points in the same row or in the same column. Write an algorithm that determines whether or not a given graph has an Euler path. I . if one exists. To construct other values. j] is true. then you may pass through point (i. Write an algorithm to determine whether a given directed graph is in fact a rooted tree. Without being completely formal (for instance. Operations are executed from left to right. ii. Without giving all the details of the algorithm. from (1. j] is false. v #p. 1) to (n. then you may not pass through point (i.4 for a directed graph. indicate how to solve this problem by branch-and-bound. * Problem 6.7. 6. Your algorithm should accept the graph represented by its adjacency matrix (type adjgraph of Section 1. j ). if M [i.Sec. whereas the edge (p .7. p) exists. Notice that a running time in 0 (n) for this problem is remarkable given that the instance takes a space in Q(n 2) merely to v:'rite down. Euler's problem. For instance 10= 1x2x2x2x2/3x2. general. you have available the two operations x2 (multiplication by 2) and /3 (division by 3.9.2).

A maze. 1983). Problem 6.3. Lawler (1976). consult Berliner (1980).7. Hopcroft. Other algorithms based on depth-first search appear in Aho.23) is given in Deyong (1977). The use of this technique to solve the travelling salesperson problem is described in Bellmore and Nemhauser (1968).2 is given in Robson (1973). The book by Nilsson (1971) is a gold mine of ideas concerning graphs and games. 1970). and Ullman (1974. and Deo (1977). A lively account of the first time a computer program beat the world backgammon champion (Problem 6. 6 17 Figure 6.Exploring Graphs 204 -+ 11 . We mention in chronological order Christofides (1975). Even (1980). Backtracking is described in Golomb and Baumert (1965) and techniques for analysing its efficiency are given in Knuth (1975a). The latter is analysed in Knuth (1975b).10 is solved in Rosenthal and Goldner (1977). and Tarjan (1983). A solution of problem 6.1. The branch-andbound technique is explained in Lawler and Wood (1966). Papadimitriou and Steiglitz (1982). Reingold.8 REFERENCES AND FURTHER READING There exist a number of books concerning graph algorithms or combinatorial problems that are often posed in terms of graphs. Several applications of depth-first search are taken from Tarjan (1972) and Hopcroft and Tarjan (1973). . 6.6. Gondran and Minoux (1979). -+ Chap. For a more technical description of this feat. Nievergelt. The mathematical notion of a graph is treated at length in Berge (1958. A linear time algorithm for testing the planarity of a graph is given in Hopcroft and Tarjan (1974). the minimax technique. and alpha-beta pruning. Some algorithms for playing chess appear in Good (1968).2.

Let J be a set of grammars for a family of programming languages. and so on. k > of the original problem. Suppose each instance jet can be separated into two components j E J and k E K (that is. Example 7. For example. Pascal. Even when there is only one instance to be solved. precomputation of auxiliary tables may lead to a more efficient algorithm. J might be a set of grammars in Backus-Naur form for such languages as Algol.1 PRECONDITIONING 7.1. Simula.1.7 Preconditioning and Precomputation If we know that we shall have to solve several similar instances of the same problem. In this case I is the set of instances of the type "Is kEK a valid program in the language defined by the grammar j E J ? ".1 Introduction Let I be the set of instances of a given problem. then the application of Bj on k gives the solution to the instance < j . k > E I. The general problem is to know whether a given program is syntactically correct with respect to some given language. 205 . This is preconditioning. it is sometimes worthwhile to invest some time in calculating auxiliary results that can thereafter be used to speed up the solution of each instance. Let K be a set of programs. This algorithm Bj must be such that if k E K and < j. 7.1. I c J x K ). A preconditioning algorithm for this problem is an algorithm A that accepts as input some element j of J and produces as output a new algorithm Bj .

k2>. In this case it is sometimes impractical to calculate and store ahead of time the #1 solutions to all the relevant instances.1. we are wasting our time using preconditioning if bj (k) > t ( j. else. sinon. Let a( j) = the time required to produce Bj given j b j (k) = the time required to apply Bj to k t ( j. then. on the other hand. Such an application of preconditioning may be of practical importance even if only one crucial instance is solved in the whole lifetime of the system : this may be just the instance that enables us. to know whether k E K is a program in language j.2. for example. we simply apply the O compiler Bj to k. by 1. { si. k 1 >. . We need to be able to solve any instance i E I very rapidly. We have to solve a series of instances <j.k. to stop a runaway reactor. { for. for example to ensure a sufficiently fast response time for a real-time application. k) = the time required to solve < j. k > directly It is usually the case that bj (k) <_ t( j. k) 5 a ( j) + bb (k).. finsi } . jusqu'a. one way of solving < j. k > from scratch consists of producing Bj from j and then applying it on k. a. Obviously. Preconditioning can be useful in two situations. Example 7.. 7 One possible preconditioning algorithm for this example is a compiler generator : applied to the grammar j E J . alors. . it often happens that t2 is much smaller than t I. .Preconditioning and Precomputation 206 Chap. and t2=a(j)+ n bj (k. be possible to calculate and store ahead of time #J preconditioned algorithms. 1 pour. and on the other hand. Let J be a set of sets of keywords. The time you spend studying before an exam may also be considered as an example of this kind of preconditioning. endif 1.) . pas 1 1. it generates a compiler Bj for the language in question.) if we work without preconditioning. to. Whenever n is sufficiently large. In this case the time taken to solve all the instances is ti = t(j. It may. for example J = { { if. b. k). <j.=1 with preconditioning. <j. kn > with the same j. Thereafter.

then we can subsequently solve < j. begin. if we start by sorting j (this is the preconditioning). function}. and let K be the set of pairs < v . then it is probably worthwhile to calculate the inverse of A once and for all.1.1. If we solve each instance directly. If we expect to have several systems of equations to solve with the same matrix A but different vectors b. Example 7. for example K = (Si. where nj is the number of elements in the set j. On the other hand. (By definition.Sec. Example 7. so that we can subsequently solve any particular instance in a time in 0(1).1. possible to precondition the tree in a time in O(n). Problem 5.1 Preconditioning 207 Let K be a set of keywords. recursively.1.5 suggests how to obtain an efficient greedy algorithm for making change.3.8. 7.5) is a further example of preconditioning. and to multiply this inverse by each b.1. jusqu'A. any direct solution of this instance takes a time in S2(n) in the worst case. the ancestor of all the nodes of which its children are ancestors. a (j) E O(nj log nj) for the sort bi . . every node is its own ancestor and. however.2 Ancestry in a rooted tree Let J be the set of all rooted trees.k)E8(nj) in the worst case. w > and a given rooted tree j we want to know whether node v is an ancestor of node w in tree j. For a given pair k = < v. 7.4. k > by a binary search algorithm. w > of nodes. Problem 7.1. we have t(j. then the second technique is clearly preferable. Creating an optimal search tree (Section 5. Why? It is.) If the tree j contains n nodes. where A is a non-singular square matrix and b is a column vector. Example 7. Calculating the necessary values cj is an example of preconditioning that allows us subsequently to make change quickly every time this is required. (k) E O( log nj ) for the search If there are many instances to be solved for the same j. We are to solve the system of equations Ax = b . We have to solve a large number of instances of the type "Is the keyword kEK a member of the set j E J ? ".5.

1. for example.G6 10 J 9 711 K H 8121 L 913 M 10 Figure 7. the required condition can be checked in a time in 0(1). In postorder we first number the subtrees of a node from left to right. however. and then we number the node itself. Show.1. 7 We illustrate this approach using the tree in Figure 7.9. D 1 2 B 5 4 E 3 I 2 C 6 F 4 8 12 .) 11 .2). which visits first a node and then its subtrees from right to left.5 is used. Once all the values of prenum and postnum have been calculated in a time in 0(n).1. For a node v.1. (Notice. Let v and w be two nodes in the tree. Thus prenum [v] <_ prenum [w] b v is an ancestor of w or v is to the left of w in the tree. and let postnum [v] be the number assigned during the traversal in postorder. we traverse it first in preorder and then in postorder (see Sec- tion 6.1. It contains 13 nodes. There exist several similar ways of preconditioning a tree so as to be able thereafter to verify rapidly whether one node is an ancestor of another. that traversal of a rooted tree in inverted preorder requires more work than traversal in postorder if the representation of rooted trees suggested in Figure 1. In Figure 7. let prenum [v] be the number assigned to the node when we traverse the tree in preorder.1 these two numbers appear to the left and the right of the node. Problem 7.2. respectively. numbering the nodes sequentially as we visit them.208 Preconditioning and Precomputation Chap. It follows that prenum [v] prenum [w] and postnum [v] >_ postnum [w] t* v is an ancestor of w.1. Thus postnum [v] >_ postnum [w] a v is an ancestor of w or v is to the right of w in the tree. In preorder we first number a node and then we number its subtrees from left to right. A rooted tree with preorder and postorder numberings. that this can be done using a traversal in preorder followed by a traversal in inverted preorder. To precondition the tree.

Thus . p (x) is expressed entirely in terms of polynomials of the form x` + c.13x4+3x3.r2=0. qo=-13. where i is a power of 2. we restrict ourselves even further and consider only monic polynomials (the leading coefficient is 1) of degree n = 2k .. 7.. Finally. x5. and let K be the set of values this variable can take.a =2.17. we apply the same procedure recursively to q (x) and r (x). evaluated at integer values of x. we can calculate p(x)=(x4+2)[(x2+3)(x-5)+(x+2)]+[(x2-4)x + (x +9)] using only 5 multiplications (of which two are to calculate x2 and x4) plus 9 additions.1. if p (x) is a monic polynomial of degree n = 2k -1. We use the number of integer multiplications that have to be carried out as a barometer to measure the efficiency of an algorithm. The problem consists of evaluating a given polynomial at a given point.10x2+5x . It is easy to do better. . If we evaluate p (x) as p (x) = ((((((x -5)x +4)x . lOx2. but 10 additions.Preconditioning Sec. In the preceding example we first express p (x) in the form (x4+a)(x3+82x2+qlx+qo)+(x3+r2x2+rlx+ro) . Example 7. Equating the coefficients of x6. A naive method for evaluating this polynomial is to calculate first the series of values x2. For simplicity. Consider the polynomial p(x)=x7-5x6+4x5. taking no account of the size of the operands involved nor of the number of additions and subtractions. Initially.. x7.I for some integer k ? 1. x3. Next.rt=-3.13)x +3)x . . Better still.10)x +5)x -17 we need only 6 multiplications and 7 additions. q I = 4.1. from which we can obtain 5x.andro=9.I -1. This method requires 12 multiplications and 7 additions (counting the subtractions as additions). 5x6 and finally p(x). . we restrict ourselves to polynomials with integer coefficients..6.1 209 7.3 Repeated Evaluation of a Polynomial Let J be the set of polynomials in one variable x. where a is a constant and q (x) and r (x) are monic polynomials of degree 2k . . we obtain successively q2 = -5. . 1. we first express it in the form p(x) = (x(n+t)12+a)q(x) + r(x).) In general.. . (A typical 7th degree monic polynomial would require 5 multiplications as is the case here.

6. tion M(k) = . Similarly. Problem 7. Generalize this method of preconditioning to polynomials that Problem 7. In other words.1.3x + 9 = (x2-4)x + (x +9) to arrive finally at the expression for p (x) given in Example 7. Your generalization must give an exact answer.. with no risk of rounding error due to the use of floating-point arithmetic.1.210 Preconditioning and Precomputation Chap. Consequently. Express p(x)=x7+2x6-5x4+2x3-6x2+6x -32 in preconditioned form. we find x3-5x2+4x . and hence M(k)=2k -1+k-2. This expression is the preconditioned form of the polynomial.7.1. x (n + 1)/2 We obtain the recurrence equa- 0 k=1 2M(k -1) + 1 k>2.7) method to polynomials of any degree.1. (Continuation of Problem 7. M(k) = 2k ' -1 for k >. Analysis of the method. In other words. Further generalize the .1 requires (3n .8. are not monic. x 4. 7 p(x) = (x4+2)(x3-5x2+4x -13) + (x3-3x +9).6. the time invested in preconditioning the polynomial allows us to evaluate it subsequently using essentially half the number of multiplications otherwise required. Let M(k) = M(k) .1)/2 additions in the worst case. there does not exist an algorithm that can calculate p (x) using less than n -1 multiplications in the worst case. (n -3)/2+Ig(n + 1) multiplications are sufficient to evaluate a preconditioned polynomial of degree n = 2k -1. Express p (x) ax7 in preconditioned form. Let M(k) be the number of multiplications required to evaluate p (x). Show that evaluation of a preconditioned polynomial of degree n = 2k . Problem 7. Problem 7.3. Problem 7.1.. a preconditioned monic polynomial of degree n = 2k -1. Prove that if the monic polynomial p (x) is given by its coefficients.13=(x2+3)(x-5)+(x+2) x3 . * Problem 7.1. .k + 1 be the number of multiplications required if we do not count those used in the calculation of x 2.

2 211 Problem 7. It returns r if the first occurrence of P in S begins at position r (that is. and it returns 0 if P is not a substring of S..1 =Pi . i = 1. In the worst case it makes m comparisons at each position to see whether or not P occurs there. and a pattern consisting of m characters. r is the smallest integer such that Sr+i . must occur entirely within one of these substrings (thus we exclude the possibility that P might straddle several consecutive substrings). Show using an explicit example that the method described here does not necessarily give an optimal solution (that is. .m do ifp[j]#s[i+j] then ok <--false else j F j + 1 if ok then return i + 1 return 0 The algorithm tries to find the pattern P at every position in S. information retrieval systems.1 while ok and j <. Is the method appropriate for polynomials involving real coefficients and real variables ? Justify your answer. S = S 1S 2 . m). 7. P = "aaaab". . The following naive algorithm springs immediately to mind.2 PRECOMPUTATION FOR STRING-SEARCHING PROBLEMS The following problem occurs frequently in the design of text-processing systems (editors. S = s is 2 s .m.2.1 Signatures Suppose that the target string S can be decomposed in a natural way into substrings.. 2. for example.O to n -m do ok F. . it does not necessarily minimize the required number of multiplications) even in the case of monic polynomials of degree n = 2k -1.1. Given a target string consisting of n characters. (Think of S = "aaa aab". and that the pattern P. and if so.9.1. Suppose without loss of generality that n >. whereabouts in S it occurs.true j *.) The total number of comparisons to be made is therefore in S1(m (n -m )). for i E. In the analyses that follow. 7. if it occurs in S. if the Si are the lines in a text file S and we are searching for the lines in the file that contain P.. Problem 7. S.Precomputation for String-Searching Problems Sec. which is in Q(mn) if n is much larger than in. P=p1p2 p. Can we do better ? 7.).. macroprocessors.10. This situation occurs. we use the number of comparisons between pairs of characters as a barometer to measure the efficiency of our algorithms. etc. we want to know whether P is a substring of S. .

m") = 27 x 14 + 12 mod 32 = 6. c 2) = (27 val (c 1) + val (c 2)) mod 32. cr) are set to 1 and the other bits are 0. If T (P . . Si) that can be calculated rapidly to make a preliminary test. c 3).2. . Calculating the signatures for S takes a time in O (n). then all the bits that are set to 1 in the signature of P are also set to 1 in the signature of Si . If C is the string "computers". val ("z") = 25. 7 The basic idea is to use a Boolean function T (P. * Problem 7.. For each pattern P we are given we need a further time in O (m) to calculate its signature. however. Suppose that the character set used for the strings S and P is { a. y. . s") = 27 x 17 + 18 mod 32 = 29. n = 40. B (cr_1. but we have to carry out a detailed check to verify this (for instance. This is yet another example of preconditioning. z and other are equiprobable. we calculate B ("c".. s") = 29. The improvement actually obtained in practice depends on the judicious choice of a method for calculating signatures. . b. but from then on we hope that the preliminary test will allow us to speed up the search for P. ii. o") = 27 x 2 + 14 mod 32 = 4. . Only seven bits are set to 1 in the signature because B ("e". If Si contains the pattern P. We calculate a signature for each substring Si and for the pattern P. Here is one common way of defining a signature. Si) = [(sig (P) and sig (Si )) = sig (P )J.1. c. m = 5).. Si) is false.Preconditioning and Precomputation 212 Chap. . Define the signature sig (C) of a string C = c 1c2 the bits numbered B (c 1. Signatures offer a simple way of implementing such a function.2.... cr as a 32-bit word where M. b. If signatures are calculated as described. This gives us the function T we need : T (P . then P cannot be a substring of Si . z. and if the characters a. i. Define val ("a") = 0. what is the probability that the signature of a random string of n characters contains a 1 in all the bit positions that contain a 1 in the signature of another random string of m characters ? Calculate the numerical value of this probability for some plausible values of m and n (for instance. r") = B ("r"..1. val (other) = 26. B (c 1. where we have lumped all the non-alphabetic characters together.. Si) is true.. . . using the naive algorithm given earlier). define . where the and operator represents the bitwise conjunction of two whole words. If the bits of a word are numbered from 0 (on the left) to 31 (on the right). . it is possible that P might be a substring of Si . B ("o". Suppose too that we are working on a computer with 32-bit words.. x. B ("r". val ("b") = 1. Example 7. the signature of this string is the word 0000 1110 0100 0001 0001 0000 0000 0100 . If c1 and c2 are characters. other) . B (c 2. . if T (P. c 2). T can be computed very rapidly once all the signatures have been calculated.

In the preceding example the function B takes two consecutive characters of the string as parameters. Si) is true with probability E > 0 even if Si does not contain P.2. Is this to be recommended ? If not. Is the method illustrated of interest if the target string is very long and if it cannot be divided into substrings ? Problem 7. It is easy to invent such functions based on three consecutive characters. The arrows show the comparisons carried out before we find a character that does not match.2 213 Problem 7. After this failure we try S P babcbabcabcaabcabcabcacabc abcabcacab TTTT .2. what is the order of the number of operations required in the worst case to find P in S or to confirm that it is absent ? Many variations on this theme are possible.2.2. we try the following configuration : S P babcbabcabcaabcabcabcacabc abcabcacab T We check the characters of P from left to right. what do you suggest instead ? 7. In this case there is only one comparison.2. Let S = "babcbabcabcaabcabcabcacabc" and P = "abcabcacab". 7.2. The number of bits in the signature can also be changed. which finds the occurrences of P in S in a time in 0 (n). is it useful ? Problem 7.Precomputation for String-Searching Problems Sec. If T (P. To find P in S we slide P along S from left to right. and if the computer in use has 32-bit words.2. If the character set contains the 128 characters of the ASCII code.4.2 The Knuth-Morris-Pratt Algorithm We confine ourselves to giving an informal description of this algorithm (henceforth : the KMP algorithm).5. c 2) = (128 val (c 1) + val (c 2)) mod 32. we might define B by B (c 1.3. Can we define a function B based on a single character? If this is possible. and so on. Example 7. looking at the characters that are opposite one another. Problem 7.2. Initially.

(A three-place movement is not enough: we know that the last characters examined in S are ax. two. However we now know that the last four characters examined in S are abcx where x #"a".. If next [ j ] = 0. S P babcbabcabcaabcabcabcacabc abcabcacab TTTTTTTT Following this mismatch. or three characters along : such an alignment cannot be correct. This array tells us to do when a mismatch occurs at position j in the pattern.Preconditioning and Precomputation 214 Chap. 7 This time the first three characters of P are the same as the characters opposite them in S. sliding P four places along might work. This time. however moving it three places might work. it is useless to compare further characters of the pattern to the target string at the current position. So let us try sliding P four characters along. It suffices to start checking at the current position of the pointer. We must instead line up P with the first character of S what . we have proceeded exactly as in the naive algorithm. Sliding P one or two places along cannot be right . In this case we have a second mismatch in the same position. where x is not a "b". To implement this algorithm. Without making any more comparisons with S. we need an array next[ I. S P babcbabcabcaabcabcabcacabc abcabcacab T There is no need to recheck the first four characters of P : we chose the movement of P in such a way as to ensure that they necessarily match. and this time a three-place movement is necessary. Up to now. m ]. we know that the last eight characters examined in S are abcabcax where x # "c".) S P babcbabcabcaabcabcabcacabc abcabcacab TTTTTTTT Yet again we have a mismatch. but the fourth does not match. we can conclude that it is useless to slide P one. S P babcbabcabcaabcabcabcacabc abcabcacab TTTTTT We complete the verification starting at the current position time the correspondence between the target string and the pattern is complete.

here is the algorithm for finding P in S. Precomputation of the array next [ 1 . On the other hand. function KMP j. It is correct to talk of preconditioning in this case only if the same pattern is sought in several distinct target strings. or else 0 if P is not a substring of S. Modify the KMP algorithm so that it finds all the occurrences of P in Sin a total time in 0 (n). The pointer and P can each be moved a maximum of n times. . preconditioning does not apply if several distinct patterns are sought in a given target string.2. including the search for a single pattern in a single target.. Problem 7. the execution time is thus in O (n).next [ j ] characters to the right with respect to S.2 Precomputation for String-Searching Problems 215 that has not yet been examined and start checking again at the beginning of P.8. In all cases. After each comparison of two characters. * Problem 7.2.2.k <. Follow the execution of this algorithm step by step using the strings from Example 7. 0 (M).2.2. it is correct to talk of precomputation. In the preceding example we have j 1 p[j] 2 3 4 5 8 9 10 b c a b 6 c 7 a a c a b next[ j] 0 1 1 0 1 1 0 5 0 1 Once this array has been calculated. we should align the i th character of P on the current character of S and start checking again at this position.. The time required by the algorithm is therefore in O (n). 7. Overall. or the variable k in the algorithm) or the pattern P. we move either the pointer (the arrow in the[j] k-k+1 j-j+1 if j >m then return k .m else return 0 It returns either the position of the first occurrence of P in S.6. In both cases we slide P along j . Find a way to compute the array next [ 1 . If next [ j ] = i > 0.7. which does happen in some applications.Sec. m ] in a time in Problem 7. which can be neglected since m <_ n. in ] can be carried out in a time in 0(m).1 while j <_ m and k 5 n do while j > O ands [k] # p [j ] do j'.

3 The Boyer-Moore Algorithm Like the KMP algorithm. the algorithm due to Boyer and Moore (henceforth : the BM algorithm) finds the occurrences of P in S in a time in O (n) in the worst case. however. We know that c #p [m]. Example 7. we try . the number of characters in the pattern P. The target string and the pattern are initially aligned as follows : S This is a delicate topic P cat T We examine P from right to left. Since "i" does not appear in the pattern. we align the latter just after the occurrence of c in the target string. we slide P along S from left to right. As with the KMP algorithm. The character opposite p [m] is "i". is often sublinear : it does not necessarily examine every character of S.216 Preconditioning and Precomputation Chap. on the other hand. i. If c appears elsewhere in the pattern. we slide the latter along in such a way as to align the last occurrence of c in the pattern with the character c in the target string. We use two rules to decide how far we should move P after a mismatch. increases. let c be the character opposite p [m]. checking corresponding characters. In the best case the BM algorithm finds all the occurrences of P in S in a time in O (m + n /m). ii. it makes at least n comparisons. and again there is an immediate mismatch. the BM algorithm tends to become more efficient as m. If a certain number of characters at the end of P correspond to the characters in S. Let S = "This is a delicate topic" and P = "cat".3. The BM algorithm. 7 7. since the KMP algorithm examines every character of the string S at least once in the case when P is absent. the characters of P are checked from right to left after each movement of the pattern. then we use this partial knowledge of S (just as in the KMP algorithm) to slide P along to a new position compatible with the information we possess. If c does not appear in the pattern.2. we slide the pattern just to the right of the arrow.2. There is an immediate mismatch in the position shown by the arrow. and the number of comparisons carried out can be less than n. Since the pattern does not include this character. However. This time. S P This is a delicate topic cat T Again we examine P from right to left. If we have a mismatch immediately after moving P. Furthermore.

Example 7. In our example when we start over checking P from right to left. Some unnecessary checks (corresponding to the underscored characters in P) may be made at times. this information is not contradicted.Precomputation for String-Searching Problems Sec.2 S 217 This is a delicate topic P cat T There is once again an immediate mismatch. If we slide P five places right.) S P babcbabcabcaabcabcabcacabc abcabcacab T Unlike the KMP algorithm. S This is a delicate topic P cat T After two more immediate mismatches we are in this situation. We know that starting at this position S contains the characters xcab where x #"a". The left-hand arrow shows the position of the first mismatch. We have made only 9 comparisons between a character of P and a character of S. S P Consider the same strings as in Example 7. P is correctly aligned. (Underscores show which characters were aligned. will confirm this.4.2. We slide P one place along to align the occurrences of the letter "a".2: babcbabcabcaabcabcabcacabc abcabcacab TTTT We examine P from right to left. when we slide P along one position to align the "a" in the target string with the "a" in the pattern. there is an immediate mismatch. S This is a delicate topic P cat T Now.2. always from right to left. A final check. we check all the positions of P after moving the pattern. In this example we have found P without ever using rule (ii). . 7. and start checking again (at the right-hand end of P). but this time the character "a" that appears opposite p[m] also appears in P. We slide P along to align the "c" found in S with the last "c" in P.

The interpretation of d2 is the following : after a mismatch at position i of the pattern. It is more complicated to compute d2.218 Preconditioning and Precomputation S P Chap. m ] then m elsem -max{i I p[i]=c] . Since we always examine the characters of P from right to left. Example 7. at the right-hand end) of the pattern and d2[i] characters further along S. To implement the algorithm. m -1]. the former to implement rule (i) and the latter for rule (ii). we again have a mismatch. indexed by the character set we are using. 7 babcbabcabcaabcabcabcacabc abcabcacab TTTT After four comparisons between P and S (of which one is unnecessary). carried out as usual from right to left.2. Suppose further that at some moment during our search for the string P in S we have a mismatch in position p [7]. we know that starting at the position of the mismatch the characters of S are xs. We shall not give the details here. This is the distance to move P according to rule (i) when we have an immediate mismatch. is easy to compute. but only an example.. Suppose the pattern is P = "assesses". A second application of rule (ii) gives us S P babcbabcabcaabcabcabcacabc abcabcacab T We apply rule (i) once to align the letters "a": S P babcbabcabcaabcabcabcacabc abcabcacab T and one last time to align the letters "c": S P babcbabcabcaabcabcabcacabc abcabcacab TTTTTTTTTT We have made 21 comparisons in all to find P.if c does not appear in p [ 1 .5.. where x # "e" : . The array d 1. For every character c dl[c] F. we need two arrays d1 [ { character set } ] and d2[1 . begin checking again at position m (that is.

2 S P ??????xs???????? assesses 219 x "e" TT The fact that x * "e" does not rule out the possibility of aligning the "s" in p [6] with the "s" we have found in S. 7. Starting from the position of the mismatch. suppose now that we have a mismatch at position p [6]. that is. and we must slide P all the way to the right under the characters of S that we have not yet examined : S P ?????xes???????? assesses X "s T We start checking again at the end of P. suppose we have a mismatch at position p [4]. Similarly. the characters of S are xsses. 10 characters further on in S than the previous comparison: thus d2[6] = 10.Precomputation for String-Searching Problems Sec. where x # "e": S P ???xsses???????? assesses TTTTT x "e" In this case it may be possible to align P with S by sliding it three places right : S P ???xsses???????? assesses T xe . It is therefore impossible to align P under these characters. where x # "s" : S P ?????xes???????? assesses TTT X "s" The fact that x # "s" rules out the possibility of aligning the "e" and the "s" in p [4] and p [5] with the "e" and the "s" found in S. 3 characters further on in S than the previous comparison: thus d2[7] = 3. the characters of S are xes. As a third instance. that is. Starting from the position of the mismatch. It may therefore be possible to align P as follows : S P ??????xs???????? assesses X #"e T We start checking again at the end of P.

k F. Continuing from Example 7. Calculate d 1 and d2 for P = "abracadabraaa".Preconditioning and Precomputation 220 Chap.10. d 1 ["e"] =1. Calculate d 1 and d2 for the pattern in Example 7.m while k <.2. or else 0 if P is not a substring of S. the choice between using rule (i) and rule Problem 7. For this example we find i 1 2 3 4 5 6 7 8 p[i] a s s e s s e s d2[i] 15 14 13 7 11 10 3 We also have d 1 ["s"] = 0. In this algorithm.2. However. (ii) depends on the test "j = m ? ". consider the following situation : S P ??????ts???????? assesses TT The failure of the match between "t" and "e" is of the second kind. function BM j.j . it is possible that rule (i) might allow k to advance more than rule (ii).5.k + d 1[s [k]] elsek k+d2[j] jFm if j = 0 then return k + 1 else return 0 It returns either the position of the first occurrence of P in S.k-1 j *.11.2. 7 Now we start checking at the end of P. Problem 7.1 if j # 0 then if j = m then k <-. d 1 ["a"] = 7 and d 1 [any other character ] = 8. Note that d 1 ["s"] has no significance. so d2[4] = 7.4.2. Here finally is the BM algorithm. 7 characters further on in S than the previous comparison.n and j > 0 do while j > 0 and s [k] = p [j ] do k+. Problem 7. so k is increased by d2[71 = 3 to obtain . because an immediate mismatch is impossible at a position where S contains "s". even if j < m .9.2.

How many comparisons are made altogether before the failure to match is discovered.12. provided we define d 2[m] = 1 and d 1[ p [m]] = 0. the fact that "t" does not appear in P should have allowed us to increase k directly by d 1["t"] = 8 positions. of course) for the pattern P = "assesses" in S = "I guess you possess a dress fit for a princess". Modify the BM algorithm so that it will find all the occurrences of P in S in a time in 0 (n). Problem 7. It is easy to see intuitively why the algorithm is often more efficient for longer patterns. d I [c] is equal to m for most characters c. This modification corresponds to the algorithm usually known by the name Boyer-Moore (although these authors also suggest other improvements).m by k <.13. 52 letters if we count upper.k + max(d I [s [k] ]. Show the progress of the algorithm if we search (unsuccessfully. For a character set of reasonable size (say. Prove that the total execution time of the algorithm (computation of d I and d2 and search for P) is in 0 (n). Thus we look at approximately one character out of every m in the target string. 7. and how many of these comparisons are redundant (that is.2. ??????te???????? assesses S P T Show that the algorithm is still correct if we replace if j= m then k E. Problem 7.and lowercase separately. they repeat comparisons previously made) ? * Problem 7.2 221 ??????ts???????? assesses S P T However.k+d2[j] j E.Precomputation for String-Searching Problems Sec.14. d2[ j ]) j4-m .15. ** Problem 7. As long as m stays small .2.2. ten figures and about a dozen other characters) and a pattern that is not too long.k+ d I [s [k]] else k<.2. Find a way to calculate the array d2 in a time in 0 (m).

Rytter (1980) corrects the algorithm given in Knuth. can be used to introduce the KMP algorithm in an intuitively appealing way. The KMP and BM algorithms of Sections 7. read Karp and Rabin (1987). consult Aho and Corasick (1975). Signatures are discussed in Harrison (1971).2.3 come from Knuth. only 15% of the characters in S are examined. 7. and Pratt (1977) for calculating the array d2 to be used in the Boyer-Moore algorithm. as described for instance in Hopcroft and Ullman (1979). see.2 and 7. the number of characters examined goes down as m goes up. Finite automata. For an efficient algorithm capable of finding all the occurrences of a finite set of patterns in a target string. when m = 12.222 Preconditioning and Precomputation Chap.2. Morris. about 20% of the characters are examined when m = 6. Boyer and Moore give some empirical results : if the target string S is a text in English.3 REFERENCES AND FURTHER READING Preconditioning polynomials for repeated evaluation is suggested in Belaga (1961). 7 compared to the size of the character set. for example. For a probabilistic string-searching algorithm (see Chapter 8). and Pratt (1977) and Boyer and Moore (1977). . Baase (1978). Morris.

If you were at one or the other of these two places. If you are willing to take a calculated risk. The problem is complicated by the fact that a dragon visits the treasure every night and carries part of it away to an inaccessible den in the mountains.1 INTRODUCTION Imagine that you are the hero (or the heroine) of a fairy tale. and that y is the value of the treasure carried off every night by the dragon.1. you can do better. Leaving out of consideration the possible risks and costs of setting off on a treasure-hunting expedition. It takes five days to get to either of the possible hiding-places.8 Probabilistic Algorithms 8. You estimate that it will take four more days' computation to solve the mystery of the map and thus to know with certainty where the treasure is hidden. which are. you would immediately know whether it was the right one. should you accept the elf's offer? Obviously it is preferable to give three nights' worth of treasure to the elf rather than allow the dragon four extra nights of plunder. but if you set out on a journey you will no longer have access to your computer. you can expect to come home with x .1. Remembering that it will take you five days to reach the hiding-place. or to travel from one of them to the other. Suppose further that x > 9y. however. Suppose that x is the value of the treasure remaining today. You have managed to reduce the search to two possible hiding-places. you can set out 223 . however. Problem 8. If you accept the elf's offer. a considerable distance apart.9y if you wait four days to finish deciphering the map. An elf offers to show you how to decipher the map if you pay him the equivalent of the treasure that the dragon can carry away in three nights. A treasure is hidden at a place described by a map that you cannot quite decipher.

This avoids the infinite recursion. A better strategy is to toss a coin to decide which possible hidingplace to visit first. The only algorithms they had seen were those in Sec- . It is always possible that bad luck will force the algorithm to explore many unfruitful possibilities.1. On the other hand. This gives you one chance out of two of coming home with x -5y.1. Example 8.4. of which 3y will go to pay the elf. Your expected profit is therefore x . Thus we choose a suboptimal so-called pseudomedian. but at the price of a quadratic worst case : it simply decides to use the first element of the array as the pivot.4. This fable can be translated into the context of algorithmics as follows : when an algorithm is confronted by a choice. there is no question of choosing the exact median as the pivot because this would cause an infinite recursion (see Problem 4. Despite this. compared to the time that will be saved on the average by making this optimal choice. Recall that this algorithm begins by partitioning the elements of the array on either side of a pivot. Section 4. and that it then calls itself recursively on the appropriate section of the array if need be.6.Probabilistic Algorithms 224 Chap. rather than to spend time working out which alternative is the best. for instance.8y left. 8 immediately and bring back x . you will thus have x . refers to the expected time taken by the worst possible instance of a given size. This is like buying a ticket for a lottery that has a positive expected return. and one chance out of two of coming home with x -10y . the expected execution time of a probabilistic algorithm is defined on each individual instance : it refers to the mean time that it would take to solve the same instance over and over again. but choosing the pseudomedian still takes quite some time. we saw another algorithm that is much faster on the average.7. We shall see in Section 8.6 describes an algorithm that can find the k th small- est of an array of n elements in linear time in the worst case. Clearly. without making the algorithm catastrophically bad for the worst-case instances.5y. The average execution time of a deterministic algorithm was discussed in section 1. One fundamental principle of the divide-and-conquer technique suggests that the nearer the pivot is to the median of the elements.1 that choosing the pivot randomly gives a substantial improvement in the expected execution time as compared to the algorithm using the pseudomedian. By contrast. the more efficient the algorithm will be. not the time incurred if the worst possible probabilistic choices are unfortunately taken. We once asked the students in an algorithmics course to implement the selection algorithm of their choice. This makes it meaningful to talk about the average expected time and the worst-case expected time of a probabilistic algorithm.5y. The latter. It refers to the average time taken by the algorithm when each possible instance of a given size is considered equally likely. the probabilistic algorithm can only be more efficient with respect to its expected execution time.3). it is sometimes preferable to choose a course of action at random. We make an important distinction between the words "average" and "expected". journeying on to the other if you find you have decided wrong. Such a situation arises when the time required to determine the optimal choice is prohibitive.

i <_ j. The analysis of probabilistic algorithms is often complex. where X is a nonempty finite set. Section 8. Using the same probabilistic algorithm on the same instance. Section 6. and number theory beyond the scope of this book. consult the references suggested in the last section.2. To generate random integers. b) is available. and successive calls on the generator yield independent values of x.1.6.1 describes a systematic way of exploring an implicit tree to solve the eight queens problem. however. Example 8.1 goes into this more thoroughly.1. Section 8. a number of results are cited without proof in the following sections. and independently in the interval i 5 k S j. Nevertheless.1 Introduction 225 tion 4. we may obtain different correct solutions on different occasions. Three students. uniform (X). returns an element chosen randomly. and the function returns an integer k chosen randomly. we can improve the backtracking technique by placing the first few queens at random.1. For this reason. This idea allowed them to beat their colleagues hands down : their programs took an average of 300 milliseconds to solve the trial instance.8). Similarly. but in this case the choice of algorithm determines uniquely which solution will be obtained whenever the algorithm is applied on any given instance.2 describes an efficient probabilistic algorithm to solve this problem provided that one is willing to accept an arbitrarily small probability of error. j ). thought of using a probabilistic approach. j) can be obtained if only uniform (a. a < b . and independently among the elements of X. whereas the majority of the deterministic algorithms took between 1500 and 2600 milliseconds. If we are content with finding one solution rather than all of them. For more details.6. statistics. Show how the effect of uniform (i .6.3. requiring an acquaintance with results in probability. 8. No known deterministic algorithm can decide in a reasonable Example 8. where i and j are integers. Problem 8. none of them took the risk of using a deterministic algorithm with a quadratic worst case. The distribution of x is Iniform on the interval." Of course. A call on uniform (a.1. Let a and b. such problems can also be handled by deterministic algorithms. Since the students did not know which instances would be used to test their programs (and suspecting the worst of their professors).. be real numbers. uniformly. b) returns a real number x chosen randomly in the interval a <_ x < b . For another example of the same phenomenon consider the problem : "Find a nontrivial factor of a given composite integer. time whether a given integer with several hundred decimal digits is prime or composite. This problem has important applications in cryptology (Section 4.2 raises an important consideration concerning probabilistic algorithms. uniformly. Example 8.Sec. . Throughout this chapter we suppose that we have available a random number generator that can be called at unit cost.2.5. we extend the notation to include uniform (i . They are sometimes used to solve problems that allow several correct solutions..

1. The execution time. Y2. Using the function S. This sequence is necessarily periodic. defined by yi = R (xi )..4. For example.. and even the result obtained. Most programming languages include such a generator. p ) j F. To start a sequence. The index of a modulo p is the smallest strictly positive integer i such that a i = 1 (mod p ). this seed defines a sequence : x0 = g and xi = S (xi . However. The same seed always gives rise to the same sequence. but a simple example will illustrate the general idea. independent choice of an element of X. Suggestions for further reading are given at the end of the chapter. uniform. The theory of pseudorandom generators is complex.. uniformly. the index of 2 modulo 31 is 5.Probabilistic Algorithms 226 Chap. we may choose. a seed that depends on the date or time. with a period that cannot exceed #X. the impractical hypothesis that a genuinely random generator is available is crucial when we carry out the analysis. 8 Example 8.2 CLASSIFICATION OF PROBABILISTIC ALGORITHMS By definition.uniform(1. . the theoretical results obtained in this chapter concerning the efficiency of different algorithms can generally be expected to hold. Truly random generators are not usually available in practice. Give other examples of sets in which there is an efficient way to choose an element randomly. By Fermat's theorem. an index modulo p always divides p -1 exactly. and let a be an integer such that 1 5 a < p . Using a good pseudorandom generator. Finally. and that of 5 is 3. This suggests one way of making a random. It is thus the cardinality of X = { a i mod p I j > 1 } . and independently. may vary considerably . p) { Section 4. the period can be made very long.p-1) return dexpoiter (a .1. The fundamental characteristic of these algorithms is that they may react differently if they are applied twice to the same instance. so to obtain different sequences. if S and R (and sometimes g) are chosen properly. we must supply an initial value called a seed. i >_ 0. y 1 . where X is a sufficiently large set and Y is the domain of pseudorandom values to be generated. function draw (a . although some implementations should be used with caution.1) for i > 0. and the sequence may be for most practical purposes statistically indistinguishable from a truly random sequence of elements of Y. j. 8. We shall not worry about the fact that such a concept conflicts with the definition of "algorithm" given at the beginning of the first chapter. a probabilistic algorithm leaves some of its decisions to chance.3. Most of the time pseudorandom generators are used instead : these are deterministic procedures that are able to generate long sequences of values that appear to have the properties of a random sequence.8 } Problem 8. Most generators are based on a pair of functions S : X -+ X and R : X -> Y. However. Let p be a prime number. the function R allows us to obtain the pseudorandom sequence YO. Let g r= X be a seed. for example. that of 3 is 30.

Whatever the instance to be salved. Las Vegas algorithms never return an incorrect answer. but catastrophic for a few instances. for example. Finally. Incorporating an element of randomness allows a Sherwood algorithm to reduce.) For certain real-life prob- lems. Some authors use the term "Monte Carlo" for any probabilistic algorithm. They are used when some known deterministic algorithm to solve a particular problem runs much faster on the average than in the worst case. Similarly. but its expected precision improves as the time available to the algorithm increases. for example. and Sherwood. but sometimes they do not find an answer at all. As with Monte Carlo algorithms. and the answer is always correct. or maybe because a digital computer can only handle binary or decimal values while the answer to be computed is irrational.Sec. perhaps because of uncertainties in the experimental data to be used. Probabilistic algorithms can be divided into four major classes : numerical. Monte Carlo algorithms. (The error is usually inversely proportional to the square root of the amount of work performed. are used when there is no question of accepting an approximate answer. 8. the probability of failure can be made arbitrarily small by repeating the same algorithm enough times on this instance.2 Classification of Probabilistic Algorithms 227 from one use to the next. The principal disadvantage of such algorithms is that it is not in general possible to decide efficiently whether or not the answer given is correct. Sherwood algorithms always give an answer. to estimate the mean length of a queue in a system so complex that it is impossible to get closed-form solutions or to get numerical answers by deterministic methods. but rather of breaking the link between the occurrence of such behaviour and the particular . Randomness was first used in algorithmics for the approximate solution of numerical problems. such as the simplex algorithm for linear programming. In the case of a decision problem. Sometimes the answer is given in the form of a confidence interval. It is not a case of preventing the occasional occurrence of the algorithm's worst-case behaviour. it is hard to see what an "approximation" might be. However. and only an exact solution will do. that are extremely efficient for the great majority of instances to be handled. since only two answers are possible. this difference between good and bad instances. Thus a certain doubt will always exist. of getting a correct answer) increases as the time available to the algorithm goes up. and sometimes even to eliminate. if we are trying to factorize an integer. the probability of success (that is. the probability of success increases as the time available to the algorithm goes up. it is of little interest to know that such-and-such a value is "almost a factor". For other problems. a precise answer exists but it would take too long to figure it out exactly. A way to put down seven queens on the chess-board is little help in solving the eight queens problem. Simulation can be used. on the other hand. but the answer is not necessarily right. These algorithms should not be confused with those. Monte Carlo. The answer obtained by such a probabilistic algorithm is always approximate. any answer that is obtained is necessarily correct. Las Vegas. and in particular for those we call "numerical". computation of an exact solution is not possible even in principle. A Monte Carlo algorithm always gives an answer.

A problem is well-characterized if it is always possible to Problem 8. Intuitively.3) is well-characterized. and this uncertainty is typical of probabilistic algorithms.3 NUMERICAL PROBABILISTIC ALGORITHMS Remember that it is a question of finding an approximate answer for a numerical problem. as Georges Louis Leclerc showed. . a Sherwood algorithm is less vulnerable to an unexpected probability distribution of the instances that some particular application might give it to solve (see the end of Section 1. Why 113 ? Prove it. Problem 8. Since it reacts more uniformly than the deterministic algorithm.5. this method is not used in practice since better methods of calculating the decimal expansion of it are known. each toothpick has one chance in it of falling across a crack.1 Buffon's Needle You spill a box of toothpicks onto a wooden floor.2.3. The toothpicks spread out on the ground in random positions and at random angles.1.2. Show that the problem of finding a nontrivial factor of a composite integer (Section 8.Probabilistic Algorithms 228 Chap. 8 instance to be solved. 8. show how to obtain a Monte Carlo algorithm for any problem whatsoever given that you already have a Las Vegas algorithm for the same problem. do you think ? 8. If you know that there were 355 toothpicks in the box. Contrariwise. the average number of toothpicks expected to fall across a crack can be calculated : it is almost exactly 113. and that each one is exactly half as long as the planks in the floor are wide (we realize that this gets unlikelier every minute ! ). Why Buffon ? In fact. Show how to obtain a Las Vegas algorithm to solve a wellcharacterized problem given that you already have a Monte Carlo algorithm for the same problem.2. each one independently of all the others.4). You should realize this in no way implies that the problem is easy to solve. Why "Sherwood".3. how many toothpicks will fall across a crack between two planks ? Clearly any answer between 0 and 355 is possible. Needless to say.3. Problem 8. verify efficiently the correctness of a proposed solution for any given instance. However.2. This suggests a probabilistic "algorithm" for estimating the value of it by spilling a sufficiently large number of toothpicks onto the floor.1. do you think the problem of finding the smallest nontrivial factor of a composite integer is well-characterized? Problem 8.

2.000. and return n / k as its estimate of it. how many of the latter should you drop in order to obtain with probability at least 90% an estimate of it whose absolute error does not exceed 0. At AF r M r r At Si r 4 r Figure 8.3 Numerical Probabilistic Algorithms 229 Furthermore. (It is much easier to simulate this experiment on a computer than to find a darts-player with exactly the degree of expertise-or of incompetence . using a pseudorandom generator. In our example. 8.of each toothpick that falls. so the average proportion of the darts that fall inside the circle is nr2/4r2 = it/4. except that it only throws darts into the upper right quadrant of the target.3. The following algorithm simulates this experiment.2.001 ? Supposing that you have available a random generator of the Problem 8.3. This allows us to estimate it = 4k/n. type discussed previously. . Supposing that the width of the planks is exactly twice the * Problem 8. where 28 darts have been radians . we are not surprised to find 21 of them inside the circle. whereas that of the square target is 4r2. where we expect to see on average 28n/4 = 22. give an algorithm Buffon (n) that simulates the experiment of dropping n toothpicks.1 illustrates the experiment. the precision of your estimate of it would be limited by the precision of the ratio of the length of the toothpicks to the width of the planks. What are your estimates of it ? (It is likely that you will need the value of it during the simulation to generate the random angle . length of the toothpicks.required. then its area is nr 2.) If the radius of the inscribed circle is r. Figure 8.3. Try your algorithm on a computer with n = 1000 and n = 10. But then 0 nobody said this was a practical method !) Consider next the experiment that consists of throwing n darts at a square target and counting the number k that fall inside a circle inscribed in this square. / Throwing darts to compute it. We suppose that every point in the square has exactly the same probability of being hit by a dart. Your algorithm should count the number k of toothpicks that fall across a crack.Sec.2.1.

. then Prob[ I h .uniform (0. y F x " in this algorithm ? 8. n) k+-0 for i E.) Recall that if f : [0.1) if y <-f(x)then k Fk+1 return k In Thus the algorithm using darts to estimate it is equivalent to the evaluation of 45(1-x 2)Zdx 0 by hit-or-miss Monte Carlo. 1) .S whenever the number n of iterations is at least I (1-I)/E2S.uniform (0.1) if x2+y2<-lthen k-k+1 return 4k In Problem 8. Consider two real constants a and S strictly between 0 and 1. then the area of the surface bounded by the curve y = f (x). 1] -* [0.3.230 Probabilistic Algorithms Chap.uniform (0.1) y F uniform (0.uniform (0.1 to n do x F-. we could throw a sufficient number of darts at the unit square and count how many of them fall below the curve.5.3.uniform (0.1 to n do x t. Prove that if ! is the correct value of the integral and if h is the value returned by the preceding algorithm.1] is a continuous function. (This name is unfortunate. and the line x =1 is given by if (x)dx 0 To estimate this integral. 8 function darts (n) k -0 for i F. Therefore it is sufficient to use n = 11/4c281 (because I (1-I) <-'/a) to reduce below 8 the probability of an absolute error exceeding E. because in our terminology it is not an example of a Monte Carlo algorithm.1)" by "x t. the x-axis. function hitormiss (f . the y-axis. y F. What value is estimated if we replace "x .uniform (0.1) y <.4. * Problem 8.1) .2 Numerical Integration This brings us to the best known of the numerical probabilistic algorithms : Monte Carlo integration.3.I I < E ] > 1.

b I . the number n of iterations to make and the values of c and d.3. If both are used to compute it as previously described. and b. a Your algorithm should accept as parameters. d ] be a continuous function. a. Moreover. Let a. n . b. b ) we assume n >-21 delta . Generalize the preceding algorithm to estimate h Jf(x)dx . The simplest consists of generating a number of points randomly and uniformly inside the interval concerned.b) sum E-0 for i . one of the simplest of which is the trapezoidal algorithm. 8.1 to n do x F.a. because hit-or-miss can sometimes make more iterations than crude in a given amount of time. however. The estimated value of the integral is obtained by multiplying the width of the interval by the arithmetic mean of the values of the function at these points. for instance. its variance is never worse than that of the hit-or-miss algorithm. Monte Carlo integration is of little practical use. and let f : [a. Usually more efficient probabilistic methods of estimating the value of a definite integral exist. Problem 8. and the distribution of the estimate is approximately normal when n is large. that crude always outperforms hit-or-miss. besides f . each iteration of crude requires the computation of a square root. a. b) sum . function trapezoidal (f .6.uniform (a.[c.(f (a)+f (b))/2 for x . As presented thus far.3 Numerical Probabilistic Algorithms 231 Notice that this is not very good : one more decimal digit of precision requires one hundred times more computation.sum + f (x) return (b -a) x (sum/n) fh fab Provided f (x) dx and f 2(x) dx exist. A better estimate of the integral can generally be obtained by various deterministic methods. which hit-or-miss can do without by proceeding as in darts.Sec.n. One should not immediately conclude. and d be four real numbers such that a < b and c S d. c.a +delta step delta to b -delta do sum<-sum+f(x) return sum x delta . for any fixed number of iterations. function crude(f . the variance of the estimate calculated by this algorithm is inversely proportional to the number of points generated randomly.(b -a)l(n -1) sum <-.

000 points. Compare experimentally the trapezoidal algorithm and the two probabilistic algorithms we have seen. In general. Monte Carlo integration is used to evaluate integrals of dimension four or higher. the number of sample points needed to achieve a given precision grows exponentially with the dimension of the integral to be evaluated. We would like to know the cardinality of X. that is. The same technique can also be used to estimate the value of an integer.3. Any call on trapezoidal (f . even the most sophisticated. then it will probably be necessary to use all the points of a 100 x 100 grid.4). to achieve the same precision when a double integral is evaluated. 0. but the number of elements is too large for it to be practical simply to count them one by one. If the dimension is fixed. on the other hand.7 gives a reference for further reading). numeric probabilistic algorithms are used to approximate a real number. 8 Try to grasp intuitively why this algorithm works. A classic brain-teaser helps to explain how this ability to choose random elements from X allows us to estimate its cardinality. The precision of the answer can be improved using hybrid techniques that are partly systematic and partly probabilistic. to every deterministic integration algorithm. even though the true value of this integral is z . Chap.101 returns the value zero. n . In practice.3 Probabilistic Counting In the preceding examples.3. 10. estimate the value of it by calculating fo 4(1-x2)zdx. and so on. even when f is a thoroughly ordinary function). on the other hand. If 100 points are needed to evaluate a simple integral.Probabilistic Algorithms 232 Problem 8. the trapezoidal algorithm needs many less iterations than does Monte Carlo integration to obtain a comparable degree of precision.8. there correspond continuous functions that can be constructed expressly to fool the algorithm. one million points will be needed for a triple integral. Monte Carlo integration is of interest when we have to evaluate a multiple integral. In practice. a technique not discussed here (but Section 8. it may even be preferable to use quasi Monte Carlo integration. If a deterministic integration algorithm using some systematic method to sample the function is generalized to several dimensions. uniformly. No function can play this kind of trick on the Monte Carlo integration algorithm (although there is an extremely small probability that the algorithm might manage to make a similar kind of error. the dimension of the integral generally has little effect on the precision obtained. Why is it called the trapezoidal algorithm ? Problem 8. although the amount of work for each iteration is likely to increase slightly with the dimension. This is typical of most of the natural functions that we may wish to integrate. Suppose.3. 8. and independently (see Example 8. However. .7. that we are able to choose an element from X randomly. Consider for example the function f (x) = sin2((100!)itx). In Monte Carlo integration.1) with 2 5 n <.1. In each case. Let X be a finite set.

taking into account the order in which they are chosen. uniformly and independently choose elements with replacement. This suggests the following probabilistic algorithm for estimating the number of elements in a set X . Use the more accurate formulas ..10. Problem 8.3. Since there are n k different ways of choosing k objects if repetitions are allowed. allow us to estimate this probability.253. 8. .10 does not correspond exactly to the puzzle in problem 8.Sec. When n is large. Problem 8.9. Nevertheless.12.3. there are n!/(n-k)! different ways of choosing k distinct objects from among n objects.3. The calculation in problem 8. What about leap years ? (n le )" .9. Let k be the number of choices before the occurrence of the first repetition.) The intuitive answer to the preceding question is almost invariably "of course not".3. In particular.3 Numerical Probabilistic Algorithms 233 A room contains 25 randomly chosen people.3.11. Show that n!/(n -k )!n k = e-k'/2n * Problem 8. Let X be a set of n elements from which we randomly. the probability that you would win your bet is greater than 56%. because births are not uniformly distributed through the year. it is when k z a4n-. show that the expected value of k tends to where (3 = = 1.3. More generally. Calculate 365!/340!36525 to four significant figures. the probability that k objects chosen randomly and uniformly from n (with repetition allowed) are all distinct is n!/(n -k )!n k .13. that the probability of having a repetition exceeds 50%. where a = 21n 2 z 1. n! E I (n le )" [1 + l/12n + O(n-2)] ln(l+x) Ex -x2/2+x3/3-8(x4) when -l <x < 1 to conclude that n /(n-k) nk E e-k(k-1)12n-k'/6n'±0(max(k'/n'. n! = x _X2 / 2 when x is near zero.3.177.k'/n')) provided that 1 << k << n.3. ** Problem 8. willing to bet that at least two of them share the same birthday? (Do not read the following paragraphs if you wish to think about this.14. Problem 8. Does this make it more or less likely that you would win your bet? Justify your answer intuitively. and the approximation ln(l+x) Z Stirling's approximation. Would you be Problem 8. It is harder to determine the average value of k corresponding to the first repetition.

must eventually repeat itself.t <.Probabilistic Algorithms 234 Chap.1r. we also have xq+i = xk+i for every nonnegative integer i. = x. in a time in 0 (k) and in constant space. e. We have not merely to choose an ele- ment at random from X. and thus 2k2/it is our estimate on #X. Let q be the smallest integer such that xq appears more than once in the sequence. but also to step through X in a pseudorandom. We are interested in computing the value of k.3. Does the function count provide an unbiased estimator of n? Give an algorithm count 2(X. and deduce that the smallest integer j such that xj = x1++ is precisely q. x 1. The algorithm count (X) estimates the number n of elements in X in an expected time and space that are both in O(X ).1. This defines a walk xo. function of the value of n. is such that q <. and hence deterministic.O defined by yt =X2. 6) that returns an estimate N of the number n of elements in X such that Prob[ 11. For this reason {xi } Q_o is called the tail of the walk and {xi } 4 q is its period.q +p. through X. xi = xj whenever j >_ i >. .N In I < c ] >. .. way.uniform (X) until a E S return 2k2/1t Carry out a statistical analysis of the random variable k2 as a ** Problem 8.. Let p stand for k . The space can be reduced to a constant with only a linear increase in the execution time by using a pseudorandom generator. hence in an expected time in 0 ( X ). * Problem 8. i > 0. . hence k = q +p.15. Show that the smallest integer t > 0 such that y. Incorporate all these ideas into a simple algorithm that is capable of finding the values of q and p. This is one of the rare instances where using a truly random generator would be a hindrance rather than a help.q.q and j -i = 0 (mod p ). since this corresponds precisely to the first repetition. X2. 8 function count (X :set) kF-0 S F. Let f : X -* X be a pseudorandom function and let xoE X be a randomly chosen starting point.o a <--uniform (X ) repeat k F-k+1 S <-S u (a} a . and let k be the smallest integer larger than q such that xq = xk .S. with t = q +p only possible if q = 0. Because X is a finite set. The following exercise shows how to obtain both q and p in constant space and in a time in 0 (k). Consider the sequence {y. provided operations on the set S are counted at unit cost. }1.16. and more generally. the length of the tail. This quantity of space can be prohibitive if n is large. the sequence {x. where xi = f (xi _ 1).3. Because the walk is pseudorandom. Show also that t 0 (mod p).

> 256. The first approach might be to sort the words on the tape so as to . nevertheless.3. It is clear that #Xm <_ #K if the system is closed. It suffices to choose k I and k2 randomly in K and to calculate Ek (Ek.Sec. and let n be the number of different words.(Ek. (Even 256 microseconds is more than two millennia. 8.3 Numerical Probabilistic Algorithms 235 Problem 8.) Implemented using specialized hardware.1.3.3. k 2 E K }. Let N be the total number of words on the tape. Probabilistic counting is used to estimate a lower bound on the cardinality of Xm .17. keys. possessives.(m)] . All this suggests a probabilistic approach to testing whether or not a cryptosystem is closed.) It is improbable that the system is closed if this estimate is significantly greater than the cardinality of K. it is reasonable to hope that #Xm >> #K provided that #M >> #K.(m)) in order to choose a random element from Xm . Let m be chosen randomly from M. the probabilistic algorithm was able to arrive at this conclusion in less than a day. In this application #K = 256 and #M = 264. Such a system is closed (an undesirable property) if (dki.. = { Ek (Ek. where a << b . that is. and so on) as distinct items? Two obvious solutions to this problem are based on the techniques of sorting and searching. if some elements of X are favoured to the detriment of others. which rules out any possibility of an exhaustive verification of the hypothesis that #X.4 More Probabilistic Counting You have the complete works of Shakespeare on a magnetic tape. if the system is not closed. a finite set M of messages and two permutations Ek : M -* M and Dk : M -* M for each key k E K such that Dk (Ek (m)) = m for every m E M and k E K. although this may not imply a uniform distribution on Xm .3.k2EK)(3k3EK)(b'mEM)[Ek. Show that it can. (We can only estimate a lower bound since there is no reason to believe that elements are chosen uniformly from Xm .(m)) I k I . An endomorphic cryptosystem consists of a finite set K of Example 8.3. counting different forms of the same word (plurals. The following example shows that it can nonetheless be useful if we simply need to know whether X contains less than a elements or more than b ..11) The probabilistic counting algorithm no longer works if the generation of elements in X is not uniform.3.17. On the other hand. How can you determine the number of different words he used. be used unchanged to estimate a lower bound on the number of elements in X. (Continuation of Problem 8. The variance of the estimate obtained from this algorithm is unfortunately too high for most practical applications (unless the solution to Problem 8.15 is used).(m))=Ek. For every m r= M consider the set X. 8. A similar approach was used to demonstrate that the American Data Encryption Standard is almost certainly not closed. see Problem 8.

4 % = 1.1) y[i]<-l return 7t(y. this second method requires a quantity of central memory in S2(n).) The second approach consists of making a single pass over the tape and constructing in central memory a hash table (see Section 8. (This may depend not only on the character set we are using. three. there are words x 1. ) This crude rea- .4. 1) # 4 for 16 different values of xi is (15/16)16 = 35. 0) Suppose. it is unlikely that there could be more than 16 distinct words on the tape. b ). We must first define what sequences of characters are to be considered as words. but it is in S2(Nn) in the worst case. function wordcnt { initialization } y . (The probability that n(h (xi). Consider the following algorithm. and 001. that the value returned by this algorithm is 4.string of (m + 1) bits set to zero { sequential passage through the tape } for each word x on the tape do i -7t(h(x). The required time is thus in 0 (N) on the average. Moreover. or four words. which will most likely prove prohibitive. (Such techniques are not covered in this book. or k+1 if none of the bits of y is equal to b. two. Let k be the value returned by a call on wordcnt. then there exists a probabilistic algorithm for solving this problem that is efficient with respect to both time and space.a -eh'' . Prob[ k = 41 n =16 ] = 313/4 %.6% = e-1. This means that the final y begins with 1110. "jacko'lantern". in fact. assuming h has sufficiently random behaviour . for example. If y is a string of bits of length k. Let m be a parameter somewhat larger than lg M (a more detailed analysis shows that m = 5 + Fig MI suffices).Probabilistic Algorithms 236 Chap. in fact. x2 and x3 on the tape such that h (x1) begins with 1. Prob[ k = 41 n = 4 ] = 183/4 %. 1) = 3 for at least one value of xi among 4 different values is 1-(7/8)4 = 41. This method takes a time in O(N log N) but requires a relatively modest amount of space in central memory if a suitable external sorting technique is used.) Let U be the set of such sequences. denote by 7t(y. 01. If we are willing to tolerate some imprecision in the estimate of n. Since the probability that a random binary string begins with 0001 is 2-4. (The probability that it(h (x. since the probability that a random binary string begins with 001 is 2-3. b E { 0. and "jack-in-the-box" as one. Let h : U -* {0. 8 bring identical forms together. but also whether we want to count such sequences as "jack-rabbit".4) holding a single occurrence of each form so far encountered.1 } . ) Conversely. denote by y [i] the i th bit of y. and if we already know an upper bound M on the value of n (or failing this. on the value of N ).1 }'" be a hash function that transforms a sequence from U in a pseudorandom way into a string of bits of length m. Consequently. 1 5i 5k . it is unlikely that there could be less than 4 distinct words on the tape. the smallest i such that y [i ] =b. and then to make a sequential pass over the sorted tape to count the number of different words. respectively. ). but there is no word x4 such that h (x4) begins with 0001.

5 Numerical Problems in Linear Algebra Many classic problems in linear algebra can be handled by numerical probabilistic algorithms.18. 1 }' is randomly chosen with uniform distribution among all such functions (this last assumption is not reasonable in practice).--. Unfortunately. We do not discuss any of them in detail here because it is only for very specialized applications that they perform better than the obvious deterministic algorithms. when the tape contains n different words and the function h : U ---)10. (Hint : by using t strings y 1 . the standard deviation of R shows that this estimate may be in error by a factor of 2. The reader is referred to the literature for further discussion. you can obtain a relative precision of about 0.62950 when n is sufficiently large. . where the hidden constant in 0(1) fluctuates around 0. This offers a first approach for estimating the number of different words : calculate k using the algorithm wordcnt and estimate n as 2k/1.n . Let R be the random variable returned by this algorithm ** Problem 8. Prove that the expected value of R is in Ig n + O(1). By contrast.19.54703. an n x n nonsingular matrix A.3 Numerical Probabilistic Algorithms 237 soning indicates that it is plausible to expect that the number of distinct words on the tape should lie between 2k-2 and 2"`. An intriguing feature of these probabilistic algorithms is their ability to compute independently the various entries in the result. Classic deterministic inversion algorithms compute its inverse B as a whole or perhaps column by column. Show how to obtain an arbitrarily precise estimate by using a little more space but with no appreciable increase in execution time. 8. . where I stands for the identity matrix. These algorithms are only applicable if the matrices concerned are well conditioned. Consider. in about 1/n2 of the time they would require to compute the whole inverse. matrix of m bits. provided n is sufficiently large.3.3. 8. Among those are matrix multiplication. your hash function should produce strings of m +lg t bits. which is unacceptable. for any given 1 <_ i < n and 1 < j <. for instance. there are probabilistic algorithms that are capable of estimating the value of B.Sec. Prove further that the standard deviation of R fluctuates around 1.78/-J provided t is sufficiently large (t >. It is far from obvious how to carry out a more precise analysis of the unbiased estimate of n given by k.64) . Y 22 . typically requiring that 1-A have only small eigenvalues. the solution of a system of simultaneous linear equations. ** Problem 8. and the computation of eigenvalues and eigenvectors.) 0 Notice that this approximate counting algorithm is completely insensitive to the order in which the words appear on the tape and to the number of repetitions of each of them.12127.

independently of the specific instance x to be solved. but only worst-case executions.1 Selection and Sorting We return to the problem of finding the kth smallest element in an array T of n elements (Section 4. Supposing that every instance of a given size is equiprobable. If we define Fe(n) = I te(a)l#X xex' the average expected time taken by algorithm B on a random instance of size n. it is clear that to (n) = tA(n)+s (n).6 and Example 8. Suppose. The heart of this algorithm is the choice of a .1. Let A be a deterministic algorithm and let to (x) be the time it takes to solve some instance x. be the set of instances of size n.5) is used as a subalgorithm inside a more complex algorithm.4 SHERWOOD ALGORITHMS Section 1. For every integer n let X.4 mentions that analysing the average efficiency of an algorithm may sometimes give misleading results. Thus there are no longer worst-case instances. the average time taken by the algorithm to solve an instance of size n is tA(n)= Y. Algorithm B may occasionally take more time than tA(n)+s(n) on an instance x of size n.4. Sherwood algorithms free us from the necessity of worrying about such situations by evening out the time required on different instances of a given size. that quicksort (Section 4. The Sherwood algorithm thus involves only a small increase in the average execution time if s (n) is negligible compared to to (n). 8. where tB (x) is the expected time taken by algorithm B on instance x and s (n) is the cost we have to pay for this uniformity. but this fortuitous behaviour is only due to the probabilistic choices made by the algorithm. We wish to obtain a probabilistic algorithm B such that tB (x) = 1A (n) + s (n) for every instance x of size n.1). A hypothesis that is correct for a given application of the algorithm may prove disastrously wrong for a different application. Analysis of this sorting method shows that it takes an average time in O(n log n) to sort n items provided that the instances to be sorted are chosen randomly. tA(x)/#X xEX This in no way rules out the possibility that there exists an instance x of size n such that to (x) >> FA (n). 8 8. for example.Probabilistic Algorithms 238 Chap. This analysis no longer bears any relation to reality if in fact we tend to give the algorithm only instances that are already almost sorted. The reason is that any analysis of the average case must be based on a hypothesis about the probability distribution of the instances to be handled.

(n . n ] but (3c.a) is occasionally much greater than tp (n . Using the pseudo- median as the pivot assures us of a linear execution time in the worst case. even though finding this pivot is a relatively costly operation. Suppose that the elements of T are distinct. be the set of n! permutations of the first n integers.6.. k) { finds the kth smallest element in array T .(Y)] . using the first element of the array as the pivot assures us of a linear execution time on the average. we can express it as a function of both n and a. let S. but only on their relative order.n . On the other hand. Define F.l.239 Sherwood Algorithms Sec.5 and 4. (5) be the times taken by the algorithm that uses the pseudomedian and by the simplified algorithm. (n . 8.a)/n! . n2 >> cp n >_ tp(n.6. function selectionRH (T [ 1 . (Y) for most values of a. The resulting algorithm resembles the iterative binary search of Section 4.3. it suffices to choose the pivot randomly among the n elements of the array T.ElN)(`dn?n1)(Vo (acs << cp)(3n2E lNT)(Vn ? n2)[ts(n) <_ c. Let tp (n .4 pivot around which the other elements of the array are partitioned. the permutation of the first n integers that corresponds to the relative order of the elements of the array. We have the following equations : 6ESn (3cp)(3n. t. (n) = I t . If we decide to aim for speed on the average thanks to the simpler deterministic algorithm. (n . we assume that 1 <_ k <_ n } iE. we must make sure that the instances to be solved are indeed chosen randomly and uniformly. a) and t. The execution times of the algorithms in Section 4. which forces us to distinguish between the worst case and an average case. n ]. a). On the other hand. The decision whether it is more important to have efficient execution in the worst case or on the average must be taken in the light of the particular application. respectively. with the risk that the algorithm will take quadratic time in the worst case (Problems 4. a) < tp (n .6 do not depend on the values of the elements of the array. The simplified algorithm is generally faster : for every n . The fact that we no longer calculate a pseudomedian simplifies the algorithm and avoids recursive calls. More precisely. (n . and that we are looking for the median.6).j f. For the execution time to be independent of the permutation a. Rather than express this time as a function solely of n.)(3n3E IN)(Vn >C. Despite this prohibitive worst case. the simplified algorithm is sometimes disastrous : t. the simpler algorithm has the advantage of a much smaller hidden constant on account of the time that is saved by not calculating the pseudomedian.

u -I] are less than m . Let tRH (n . i . var u. we started with an algorithm that is excellent when we consider its average execution time on all the instances of some particular size but that is very inefficient on certain specific instances. j i. 8 while i < j do m <-..k return T[i] Here partition (T.m.6.. independently of the instance to be solved. 8. Using the probabilistic approach. Its simplicity ensures that (3noEN)(f/n >.Probabilistic Algorithms 240 Chap. There are.2 Stochastic Preconditioning The modifications we made to the deterministic algorithms for sorting and for selection in order to obtain Sherwood algorithms are simple. to repeat. those of T [u .u.i. but the probability that this will happen becomes increasingly negligible as n gets larger. j ] around the value m .a)] To sum up. this probability is independent of the instance[tRH(n. if the algorithm is part of a complicated. v ] are equal to m. after this operation the elements of T [i . Problem 8.j.4. occasions when we are given a deterministic algorithm efficient on the average but that we cannot reasonably expect to modify. we have transformed this algorithm into a Sherwood algorithm that is efficient (with high probability) whatever the instance considered.(Y) < tp(n. for instance.. j .. A similar analysis to that of Problem 4.5).6. It is always possible that some particular execution of the algorithm will take quadratic time. Show how to apply the Sherwood style of probabilistic approach to quicksort (Section 4. (Y) be the average time taken by the Sherwood algorithm to determine the median of an array of n elements arranged in the order specified by a. and. The values of u and v are calculated and returned by the pivoting algorithm (see Problem 4. Stochastic preconditioning allows us to obtain a Sherwood algorithm without changing the deterministic algo- . The probabilistic nature of the algorithm ensures that tRH (n . var v) pivots the elements of T [i . (Y) is independent of a. j)] partition (T.4.4. Thus its efficiency is not affected by the peculiarities of the application in which the algorithm is used.5 shows that the expected time taken by this probabilistic selection algorithm is linear. j ] are greater than m.1). however. This happens. and those of T [v +1 .v) ifk < u <-else if k > v then i -v+1 else i.. Notice that quicksort must first be modified along 0 the lines of Problem 4. badly documented software package.T [uniform (i .5.1. m.

Simply call the following procedure before the deterministic sorting or selection algorithm.1. f(u(x. For every integer n. No posttreatment (function v) is needed to recover the solution in these cases.4. to use the given deterministic algorithm to solve this random instance.Sec. iii.Y such that i. and then to deduce the solution to the original instance. Thanks to the second property. the first property ensures that this instance is transformed into an instance y chosen randomly and uniformly from all those of the same size.uniform (A. n ] ) for i f. let X be the set of instances of size n.f (y) { solved by the deterministic algorithm } return v (r. Suppose the problem to be solved consists of the computation of some function f : X -i Y for which we already have an algorithm that is efficient on the average. Suppose for the purposes of illustration that someone were to discover an algorithm efficient on the average but prohibitively slow in the worst case. The trick is to transform the instance to be solved into a random instance. procedure shuffle (T [I . Recall that no efficient algorithm is known for calculating discrete logarithms (Section 4. The stochastic preconditioning required for selection or for Example 8. and let A be a set with the same number of elements. sorting is the same : it is simply a question of randomly shuffling the elements of the array in question.u (x.) y E.4. the functions u and v can be calculated efficiently in the worst case.r)))].4 Sherwood Algorithms 241 rithm.2. s ) Whatever the instance x to be solved. r) { random instance of size n } s .uniform (i n) interchange T [i] and T [ j ] Example 8. the solution to this random instance allows us to recover the solution of the original instance x. Assume random sampling with uniform distribution is possible efficiently within A . )[f (x) = v(r. We thus obtain the following Sherwood algorithm. 8. Let A be the union of all the A . function RH (x) { computation of f (x) by Sherwood algorithm } let n be the size of x r . and H.1 to n -1 do j . Stochastic preconditioning consists of a pair of functions u : X x A -* X and v : A x Y ..8). Denote the discrete logarithm of x modulo p to the base g by .

n ] and an integer head. Why does the algorithm dlogRH work? Point out the functions corresponding to u and v. 5.p x + logg.p (xy mod p) = (logg. p) { Section 4. perhaps for a fee.3.p(gr modp)=r for 05r Sp-2. Assume. The following equations allow us to transform our hypothetical algorithm into a Sherwood algorithm : i. 2.p a { using the assumed algorithm } return (s . 8. r) is independent of x as long as r is chosen randomly with uniform probability.p x. Assume that you would like to compute f (x) for some instance x but that you lack the computing power or the efficient algorithm to do so. . here is one way to represent the list 1.. and so on. n ] and ptr [ 1 . function dlogRH (g .r) mod (p -1) 11 Problem 8..1). logg. if val [i] is not the largest key.p y) mod (p . except for its size. 3. logs. the next smal- lest is in val [ptr [head ]]. The end of the list is marked by ptr [i] = 0. furthermore. 8. p ) r <--uniform (0. The rank of a key is the number of keys in the list that are less than or equal to the given key.4. In general. This process yields no information on your actual request. because the probability distribution of u (x. 21 .4. r .4. x . For instance. Stochastic preconditioning offers an intriguing possibility : computing with an encrypted instance. 8 logg. that some other party is capable of carrying out this computation and willing to do so for you. What should you do if you are unwilling to divulge your actual request x? The solution is easy if stochastic preconditioning applies to the computation of f : use the function u to encrypt x into some random y. Here is the Sherwood algorithm.bx mod p s f. p -2) b Fdexpoiter (g. and then use the function v to deduce f (x).Probabilistic Algorithms 242 Chap. Problem 8.2. ii.8) a F. The smallest key is in val [head ]. then ptr [i] gives the index of the following key. have f (y) computed for you.3 Searching an Ordered List A list of n keys sorted into ascending order is implemented using two arrays val [ 1 . 13.logg. Find other problems that can benefit from stochastic precondi- tioning..

and that all the elements of the list are distinct.k. but it does not have such a thing as a worst-case instance.k. wA (n) and mA (n) denote its worst-case and its mean time.4. provided that x ? val [i] and that x is indeed present.) Despite this inevitable worst case. 1 5 i <_ n .}.(Y) nxn! vES k=I Problem 8. Any instance can be characterized by a permutation a of the first n integers and by the rank k of the key we are looking for. k . Let Sn be the set of all n! permutations. any deterministic algorithm takes a time in Q (n) in the worst case to find a key in this kind of list.4. (Y) denotes the expected value of this time. (Hint : show how a worst-case instance can be constructed systematically from the probes made into the list by any given deterministic algorithm. In fact.4 implies that wA (n) E S2(n) for every deterministic algorithm A. . We want a deterministic algorithm B such that mB (n)E 0 ('/n) and a Sherwood algorithm C such that we (n) = mB (n). Prove the preceding assertion. there exists a deterministic algorithm that is capable of carrying out such a search in an average time in 0 (I ). From this we can obtain a Sherwood algorithm whose expected execution time is in 0 (') whatever the instance to be solved.(Y)I 1 <_k <_n and aeS. however. k. Given a key x.4 Sherwood Algorithms i val [i] ptr [i] 243 1 2 3 4 5 6 7 2 2 3 13 1 5 21 8 5 6 1 7 0 3 In this example head = 4 and the rank of 13 is 6.Sec. Suppose for the moment that the required key is always in fact present in the list. there is no obvious way to select the middle of the list. the problem is thus to find that index i. Thus WA(n)=max{tA(n. Whether the algorithm is deterministic or probabilistic. 8. which would correspond to the first step in binary search. As usual. respectively. and mA(n)= E EtA(n. to (n . In the case of a probabilistic algorithm. such that val [i] = x. tA (n . Here. We can use binary search to find a key efficiently in a sorted array.4. the Sherwood algorithm is no faster on the average than the corresponding deterministic algorithm. The following algorithm finds a key x starting from some position i in the list. (Y) denotes the time taken by this algorithm to find the key of rank k among the n keys in the list when the order of the latter in the array val is specified by the permutation a. If A is any deterministic algorithm. Problem 8.

w. k) be the exact number of references to the array Problem 8. and m. and m introduced in the previous probProblem 8. what values of k maximize t D (n. (The order a of the keys is irrelevant for this algorithm.5 for the definition of i .val [i] for j f.7. head ) Let iA (n. (See Problem 8. head ) x > y : return search (x .max .y return search (x. k) if and only if k > f (n). function B (x) i .4. Determine wD (n) and mD (n) for every integer n.) The quantities F. i ) while x > val [i] do i <.head max <.l to L'u J do y <--val[j] ifmax<y <_x theni F j.5.ptr [i ] return i Here is the obvious deterministic search. val made by the algorithm A to find the key of rank k in a list of n keys. Here is a first probabilistic algorithm. however. k) ? Compare wD (n) and mA (n). w. The following deterministic algorithm is efficient on the average.4. 8 function search (x. As a function of n. n) y . Show. function D (x) i .) Define WA (n) and mA (n) similarly. Determine 1A (n. k) < iA (n .6.4. k) for every integer n and for every k between 1 and n.4.4.Probabilistic Algorithms 244 Chap.val [i ] case x < y : return search (x. ptr [i ]) otherwise : return i Problem 8. Determine WA (n) and mA (n) for every integer n . lems facilitate our analysis. times in algorithm B ? Intuitively. thus apparently contradicting Problem 8. k) for every integer n and for every k between 1 and n.uniform (L. function A (x) return search (x. Give explicitly a function f (n) such that ?D (n .4. why should we choose to execute the for loop . Determine 1D (n .4. that they do not tell the whole story by exhibiting a deterministic algorithm E such that WE (n) E 0 (log n). i ) Problem 8.8.

. 0) E U(n).4. N }. however. The hash table is an array T [I .4 Universal Hashing Hash coding. besides the use of a table of lists as outlined here. .15. Starting from the deterministic algorithm B.11..4.. Problem 8. Problem 8. .11) that wC (n) E 2' + O(1). give a Sher- wood algorithm C such that wC (n) E O Show more precisely Problem 8. be the random variable that corresponds to the minimum of I integers chosen randomly. which is unavoidable from Problem 8. (Continuation of Problem 8.. N ] of lists in which T [i ] is the list of those identifiers x found in the program such that h (x) = i.14.4. Prove that m8 (n) E 0 (' ). are legion. When x * y but h (x) = h (y). (The ratio (x may well be greater than 1. A hash function is a function h : X --* { 1. Such a function is a good choice if it efficiently disperses all the probable identifiers. 2. is about n /(l + 1) when 1 is a constant and about Problem 8. give explicitly a permutation a and a rank k such that tB (n . and let N be a parameter chosen to obtain an efficient system..4. [Hint : Let M1. or simply hashing.) If we suppose that every identifier and every pointer occupies a constant amount of space.Sec. Use the structure and the algorithms we have just seen to obtain a Sherwood sorting algorithm that is able to sort n elements in a worst-case expected time in 0 (n 312). Is this better than 0 (n log n) ? Justify your answer. n }.. is used in just about every compiler to implement the symbol table. Give an efficient Sherwood algorithm that takes into account the possibility that the key we are seeking may not be in the list and that the keys may not all be distinct. k.4.. we say that there is a collision between x and y . Let X be the set of possible identifiers in the language to be compiled. Show that the when expected value of M1..4. Other ways to handle collisions. 8.4. the table takes space in O(N+n) and the average length of the lists is a. 2.9. Find a link between this random variable and the average-case analysis of algorithm B. and independently with replacement from the set ( 1. Analyse your algorithm. that w8 (n) E S2(n).12. where the meaning of w is given in Problem 8. Suggest a few. .4.4..5.13. Show. . if h (x) # h (y) for most of the pairs x * y that are likely to be found in the same program. uniformly..4 Sherwood Algorithms 245 * Problem 8.4. Problem 8.10. To do this.4. Thus we see that increasing the value of N reduces the average search time but increases the space occupied by the table.4. that is. The load factor of the table is the ratio a = n IN. Problem 8. 8. where n is the number of distinct identifiers in the table.

4.) Problem 8.. y E A such that x #y. .2... the average length of the list T [h (x)]) is less than or equal to the load factor a. without arbitrarily favouring some programs at the expense of others. Problem 8. 2.B are there if the cardinalities of A and B are a and b. . These programs will compile slowly every time they are submitted. * Problem 8. respectively ? This difficulty is solved by universal hashing. What do you think of the "solution" that consists of ignoring the problem? If we are given an a priori upper bound on the number of identifiers a program may contain. Let xEX \ S be a new identifier.18.. This technique is very efficient provided that the function h disperses the identifiers properly. A Sherwood approach allows us to retain the efficiency of hashing on the average.19.9 before answering. and let x and y be any two distinct identifiers. If we suppose.3. Show that n calls on the symbol table can take a total time in f2(n2) in the worst case. .4.246 Probabilistic Algorithms Chap. (If you have not yet solved Problem 8.4. Prove further that the probability that the number of colli- sions will be greater than t a is less than 1/t for all t >_ 1. If we choose a hash function h randomly and uniformly in H.17. Prove that the average number of collisions between x and the elements of S (that is. A program that causes a large number of collisions during one compilation will therefore probably be luckier next time it is compiled. Let S c X be a set of n identifiers already in the table. N } for it to be reasonable to choose one at random. The following problem generalizes this situation. . now is the time to think some more about it!) The basic idea is to choose the hash function randomly at the beginning of each compilation.4. 2.3. does it suffice to choose N rather larger than this bound to ensure that the probability of a collision is negligible ? (Hint : solve Problem 8. the probability of a collision between x and y is therefore at most 1/N. In a sense they are paying the price for all other programs to compile quickly. By definition a class H of functions from A to B is universal2 if # (h E H (h (x) = h (y)) < #H/#B for every X. Several efficient universal2 classes of functions are known. there are f a r too many functions from X into { 1. How many functions f :A . . 8 Problem 8.16. however. it is inevitable that certain programs will cause a large number of collisions. Unfortunately. that #X is very much greater than N. N } . We content ourselves with mentioning just one. Let H be a universa12 class of functions f r o m X to { 1.

. Las Vegas algorithms usually have a return parameter success. y .) I * Problem 8. there is no upper bound on the time that may be required to obtain a solution.4. 1. 8. Prove that H = { hm. Contrast this to a Sherwood algorithm. which is set to true if a solution is obtained and false otherwise.. .Sec. allows us to obtain an increase in efficiency.21. For example.. It is also more efficient to carry out all the computations in a Galois field whose cardinality is a power of 2. a -11. . The overall probability of success therefore increases with the amount of time we have available. 2.. In the latter case it suffices to resubmit the same instance to the same algorithm to have a second. Define hm n :X -* (0. Find applications of universal hashing that have nothing to do with compilation. A Las Vegas algorithm.4.. success). Better still is the existence of some constant 8 > 0 such that p (x) >. respectively. The distinguishing characteristic of Las Vegas algorithms is that now and again they take the risk of making a random decision that renders it impossible to find a solution. .5 LAS VEGAS ALGORITHMS Although its behaviour is more uniform.5 Las Vegas Algorithms 247 Let X be { 0. . we require that p (x) > 0 for every instance x.n < p E is a universal2 class of functions. independent chance of arriving at a solution. Thus these algorithms react by either returning a correct solution or admitting that their random decisions have led to an impasse. It may be able to solve in practice certain problems for which no efficient deterministic algorithm is known even on the average. whatever happens. not less than N. even though the expected time required for each instance may be small and the probability of encountering an excessive execution time is negligible. 8. where we are able to predict the maximum time needed to solve a given instance. the Sherwood version of quicksort (Problem 8.S for every instance x. Now consider the following algorithm. and let p be a prime number * Problem 8. (Remarks : In practice we take N to be a power of 2 so that the second mod operation can be executed efficiently.1) never takes a time in excess of 0 (n2) to sort n elements. N-1 ) by h.4. l . sometimes for every instance. . on the other hand. a Sherwood algorithm is no faster on the average than the deterministic algorithm from which it arose.. Let s (x) and e (x) be the expected times taken by the algorithm on instance x in the case of success and of failure..20. Let p (x) be the probability of success of the algorithm each time that it is asked to solve the instance x. (x) = ((mx + n) mod p) mod N. However. nor even with management of a symbol table. Let m and n be two integers.. For an algorithm to be correct. 1 5 m < p and 0 <. where y is a return parameter used to receive the solution thus obtained whenever success is set to true.. The typical call to solve instance x is LV (x .

6. y . we obtain the first solution after examining only 114 of the 2. respectively.. Recall that the backtracking technique used involves systematically exploring the nodes of the implicit tree formed by the k-promising vectors.) in the case of success and of failure. 8 function obstinate (x) repeat LV (x . taking care. This is not bad. Using this technique. With probability 1. s (x). we obtain the following recurrence : t (x) = p (x) s (x) + (1. The resulting algorithm is not recursive.s (x) ? Give your answer as a function of t. Suppose that s (x) and e (x) are not just expected times. This follows because the algorithm succeeds at the first attempt with probability p (x).1 The Eight Queens Problem Revisited The eight queens problem (Section 6. but the algorithm does not take into account one important fact : there is nothing systematic about the positions of the queens in most of the solutions. success) until success return y Let t (x) be the expected time taken by the algorithm obstinate to find an exact solution to the instance x. .1. the queens seem more to have been positioned haphazardly. For example.. for any t >. 8. The recurrence is easily solved to yield t (x) = s (x) + 1 .5. This observation suggests a greedy Las Vegas algorithm that places queens randomly on successive rows.p (x) it first makes an unsuccessful attempt to solve the instance. but that they are in fact the exact times taken by a call on LV (x . Neglecting the time taken by the control of the repeat loop. On the contrary. however. thus taking an expected time s (x). before starting all over again to solve the instance.1) provides a nice example of this kind of algorithm.057 nodes in the tree. .248 Probabilistic Algorithms Chap. What is the probability that the algorithm obstinate will find a correct solution in a time not greater than t. The algorithm ends either successfully if it manages to place all the queens on the board or in failure if there is no square in which the next queen can be added. taking an expected time e (x). s (x).( )x) WW e (x) There is a compromise to be made between p (x). e (x) and p (x). and e (x) if we want to minimize t (x).5. that the queens placed on the board do not threaten one another. Problem 8. it may be preferable to accept a smaller probability of success if this also decreases the time required to know that a failure has occurred. which still takes an expected time t (x).p (x)) (e (x) + t (x)).

.2.. We can do better still. the same probability of being chosen.k } diag 135 F. The backtracking .diag45 u { j . Using a computer we can calculate A solution is therefore obtained more than one p = 0. it is column j that has been chosen (with probability 1 / nb) } try [k +I] '. less than half the number of nodes explored by the systematic backtracking technique. 8.5.(nb > 0) To analyse the efficiency of this algorithm.j col . Clearly s = 9 (counting the 0-promising empty vector).. and e = 6.I to 8 do if i 0 col and i -k e diag45 and i +k 0 diag 135 then ( column i is available for the (k + I )st queen } nb -nb+1 if uniform (1 . then try [1 . The Las Vegas algorithm is too defeatist : as soon as it detects a failure it starts all over again from the beginning. we need to determine its probability p of success. and the average number e of nodes that it explores in the case of failure.diag 135 u { j + k } { try [ 1 . 8] (a global array) contains a solution to the eight queens problem } col. nevertheless.0 k -0 repeat { try [1 . k + 1 ] is (k + 1)-promising } k-k+1 until nb = 0 or k =8 success .971 time out of eight by proceeding in a completely random fashion ! The expected number of nodes explored if we repeat the algorithm until a success is finally obtained is given by the general formula s + (1-p )e/p = 55. When there is more than one position open for the (k + I )st Problem 8. k ] is k-promising } nb .colu{j} diag45 . queen.927 . the average number s of nodes that it explores in the case of success. diag45.Sec. diag 135 f.1293 .0 for i E. nb) = 1 then ( maybe try column i } j4-i if nb > 0 then ( amongst all nb possibilities for the (k + 1)st queen.5 Las Vegas Algorithms 249 procedure QueensLV (var success) { if success = true at the end. Show that each position has. the algorithm QueensLV chooses one at random without first counting the number nb of possibilities..

if the first two queens are placed in positions 1 and 3.8750 0. on the other hand. the expected number s of nodes explored in the case of success.1293 114. diag45.93 55. The case stopVegas = 0 corresponds to using the deterministic algorithm directly. This happens.00 e t 39. The more queens we place randomly. but the greater is the probability of a failure.0000 0. except that the last two lines are replaced by until nb = 0 or k = stopVegas if nb > 0 then backtrack (k .6.92 53.2618 0. The original greedy algorithm QueensLV. The pure backtracking algorithm finds the first solution in 40 milliseconds.63 28.00 39.79 7.33 9.1357 0.97 6.31 9. reconsidering the positions of the queens that were placed randomly.1624 0. The following table gives for each value of stopVegas the probability p of suc- cess.00 9. diag 135. which places all the queens in a random way. whereas an average of 10 milliseconds is all that is needed if the first two or three queens are placed at random.63 22.67 15. and the expected number t = s +(I -p)e/p of nodes explored if the algorithm is repeated until it eventually finds a solution.29 6.93 -- We tried these different algorithms on a CYBER 835.97 114.4931 0. success) else success *.1 except that it has an extra parameter success and that it returns immediately after finding the first solution if there is one.53 13. and then uses backtracking to try and add the remaining queens.00 39. the expected number e of nodes explored in the case of failure. col. The latter looks like the algorithm Queens of Section 6. takes on the average 23 milliseconds to find a solution.50 55.10 46. the smaller is the average time needed by the subsequent backtracking stage.48 1 2 3 4 5 6 7 8 10.10 8. stopVegas p s 0 1.20 29. where 1 < stopVegas S 8 indicates how many queens are to be placed randomly before moving on to the backtracking phase. makes a systematic search for a solution that we know has nothing systematic about it.01 35. 8 algorithm.0000 1.98 6.05 9. for instance. however.false . without. An unfortunate random choice of the positions of the first few queens can make it impossible to add the others.Probabilistic Algorithms 250 Chap. This is a fraction more than .1293 0. The resulting algorithm is similar to QueensLV. respectively. A judicious combination of these two algorithms first places a number of queens on the board in a random way.

If you are still not convinced of the value of this technique. we suggest you try to solve the twelve queens problem without using a computer. 8.5039 0. This is one reason why it is more efficient to place the first queen at random rather than to begin the systematic search immediately.4] are explored to no effect.23 10.Sec.20 262. Here are the values of p.5.11 On the CYBER 835 the Las Vegas algorithm that places the first five queens randomly before starting backtracking requires only 37 milliseconds on the average to find a solution.000 times faster per solution than the deterministic algorithm. we waste time with [1. and then try again. a systematic search for a solution beginning with the first queen in the first column takes quite some time.5] begins. a systematic search that begins with the first queen in the fifth column is astonishingly quick. ** Problem 8. and t for a few values of stopVegas in the case of the twelve queens problem. First. is that a solution can be obtained more rapidly on the average if several queens are positioned randomly before embarking on the backtracking phase. it wastes so much time making its pseudorandom choices of position that it requires essentially the same amount of time as the pure backtracking algorithm.39 222.00 33.5. this time placing the first five queens randomly.0465 262.21 and [1. An empirical study of the twenty queens problem was also carried out using an Apple II personal computer. 36 different solutions were found in about five and a half minutes. Thus the probabilistic algorithm turned out to be almost 1. it is obviously silly to analyse exhaustively all the possibilities so as to discover the optimal .3] and [1. Even when the search starting from node [1. On the other hand. Once again. s. t stopVegas p s e 0 5 12 1. For the eight queens problem. this can be understood intuitively in terms of the lack of regularity in the solutions (at least when the number of queens is not 4k + 2 for some integer k). (Try it!) This unlucky characteristic of the upper left-hand corner is nothing more than a meaningless accident. try to solve the problem systematically.0000 0.5.4. As for the greedy Las Vegas algorithm.00 80.3. If we want a solution to the general n queens problem. the same corner is a better than average starting point for the problems with five or twelve queens.88 13. What is significant.5.5 Las Vegas Algorithms 251 half the time taken by the backtracking algorithm because we must also take into account the time required to make the necessary pseudorandom choices of position.00 - 47. Using the probabilistic approach and placing the first ten queens at random. Problem 8. The deterministic backtracking algorithm took more than 2 hours to find a first solution. For instance. however. whereas the pure backtracking algorithm takes 125 milliseconds.71. e. First the trees below the 2-promising nodes [1.

* Problem 8.y # y and that 1 <_ p .) The preceding problem suggests an efficient algorithm for testing whether x is a quadratic residue modulo p : it suffices to use the fast exponentiation algorithm of Section 4. Prove that p . Prove further that x is a quadratic residue modulo p if and only if x(P. Any quadratic residue has at least two distinct square roots since (p _Y)2 = p 2 . Prove that x(P-1)/2 =± 1 (mod p) for every integer 1<. (Hint : assuming that a 2 ° h 2 (mod p).5.5. can you find a constant S > 0 such that the probability of success of the Las Vegas algorithm to solve the n queens problem is at least S for every n ? 8.5. Prove.9. Prove or disprove : the n queens problem can be solved for every n >.I and z is not a quadratic residue modulo p.p . that no quadratic residue has more than two distinct square roots.p -l and every odd prime p.p .5.x <. value of stopVegas to be determined rapidly as a function of n.4. but not necessarily optimal.2py + y 2 = y 2 (mod p).7.1.Probabilistic Algorithms 252 Chap.y <. the obstinate proba- bilistic algorithm will loop forever without realizing what is happening. In fact. This is the case if and only if there exists at least one solution. An integer z is a quadratic nonresidue modulo p if 1 < z <.1. Problem 8. Combining this with Problem 8.5. Given an odd prime p and a quadratic residue x .1)/2 = + 1 (mod p). For instance. Such a y is a square root o x modulo p provided 1 :. An integer x is a quadratic residue modulo p if 1 <. 8 value of stopVegas.8 to calculate x(p .1)/2 mod p. 63 is a square root of 55 modulo 103. Technically. (We needed more than 50 minutes computation on the CYBER to establish that stopVegas = 5 is the optimal choice for the twelve queens problem !) Find an analytic method that enables a good.x < p -1 and if there exists an integer y such that x = y2 (mod p). the general algorithm obtained using the previous problem (first determine stopVegas as a function of n. Conclude from the preceding results that exactly half the integers between 1 and p -1 are quadratic residues modulo p.) Problem 8.5.6. the other direction requires some 0 knowledge of group theory.y <. and then try to place the queens on the board) can only be considered to be a Las Vegas algorithm if its probability of success is strictly positive for every n.5. consider a 2 . If no solution exists. Problem 8. the number of queens. (Hint: one direction follows immediately from Fermat's theorem : x n = 1 (mod p ).p .5. on the other hand. ** Problem 8.8. and then to apply the corresponding Las Vegas algorithm.4. determining the optimal value of stopVegas takes longer than a straightforward search for a solution using backtracking.2 Square Roots Modulo p Let p be an odd prime.b 2.

There exists.1 (mod 53) and c =. Prove that ±x(p+ 1)/4 mod p are the two square roots of x modulo p. .11.1)/2. Even if the value of is unknown.5.5. Example 8.2 (mod 53).8. and hence d / .1 (mod 53). and d product is ((ac +bdx) mod p) + ((ad + bc) mod p Note the similarity to a product of complex numbers.5. culation (2+I7)26 = 0+41J (mod 53) in detail. but no efficient deterministic algorithm is known to solve this problem when p =. c -d Using Example 8.1 as a guide. does there exist an efficient algorithm for calculating the two square roots of x modulo p ? The problem is easy when p . that c +d 1 _. A preliminary computation shows that x is a quadratic residue modulo p since 726 = 1 (mod 53). Let us decide arbitrarily to denote by the smaller of the two square roots of x. (All the following calculations are modulo 53.F7-) =0+421 (1+ 41 )26 = -1 (mod 53). Calculate 5526 mod 103 and verify that its square modulo 103 is indeed 55. it is possible to carry out the symbolic multiplication of a +b-6. and the symbolic calculation that we just carried out is valid regardless of which of them we choose to call '. carry out the symbolic calProblem 8.F7_ =(22+101)(22+101)=18+16I (1+J)12=(18+161)(18+l61)=49+461 (1+x)13=(1+x)(49+46-. b.3 (mod 4). we obtain 2c = 0 (mod 53) and hence c = 0 since 0 <. Suppose that p .I (mod 53). What happens if we calculate symbolically (a + / )26 == c +dI (mod 53) in a case when one of (a +') mod 53 and (a -i7) mod 53 is a quadratic residue modulo p and the other is not? for instance.and c + d are integers between 0 and p . however. But 7 has two square roots modulo p. Adding these two equations.Sec.)" can be calculated efficiently by adapting the algorithms of Section 4.1 (mod 4).) (1 + 7)2 = (1 + -0) (1 + 17) =8+ (1+V)3 =(1+1)(8+2J) =22+10. Consequently. modulo p.1. we conclude that (1+') mod 53 is a quadratic nonresidue modulo 53 (Problem 8.1. Since 26 = (p . an efficient Las Vegas algorithm to solve this problem when p = 1 (mod 4). c. where a.5.3 (mod 4) and let x be a quadratic residue Problem 8. 8. Let p = 53 = 1 (mod 4) and x = 7. Subtracting them.9).c < 52. we find 2d/ . This modulo p.10. The symbolic exponentiation (a +b-6. (1-1) mod 53 is also a quadratic nonresidue modulo 53.5. Suppose.5 Las Vegas Algorithms 253 modulo p. Let us calculate symbolically (1 mod 53.

y <. I (mod p).a p -1. If v = d.1. * Problem 8. modulo p. 8 To obtain a square root of 7. we find y = 22 because 41 x 22 .7. An integer a.p .5. let w = u mod v.true y Fa else compute c and d such that 0 <-c p-1.1 (mod 53). Give an efficient algorithm for calculating y In our example (following Problem 8.52 and 41y . 1 <. Your algorithm should not calculate d before starting to work on a and b. gives the key to ' if (a2-x) mod p is not a quadratic residue modulo P. If p is prime and 1 a <. This is indeed a square root of 7 modulo 53 since 222 = 7 (mod 53). now let a' and b' be such that a'v + b'w = d. a.1 (mod 4). Prove that there exist integers a and b such that au + by = d.uniform (I.y < p -1 and dy = l (mod p ) It remains to determine the probability of success of this algorithm. var y.12. and let d be their greatest common divisor. Otherwise. This can be done efficiently using a modification of Euclid's algorithm for calculating the greatest common divisor (Section 1.Probabilistic Algorithms 254 Chap.true compute y such that 1 <. p . i. the proof is trivial (a = 0 and b = 1).] ii.13. Give an efficient iterative algorithm for calculating d. [Hint : Suppose without loss of generality that u >. prove that there exists a unique y such that 1 <. The other square root is 53 .p -1 and ay given p and a. First show that d is also the greatest common divisor of v and w. and x is a quadratic residue modulo p } a F.y <.22 = 31.) By mathematical induction. This suggests the following Las Vegas algorithm for calculating square roots.1 (mod 53). var success) { may find some y such that y 2 = x (modp ) assuming p is a prime. the preceding problem shows that we need only find the unique integer y such that 1 <.v. Prove that . Let p = 1 (mod 4) be prime.5.5.. and let x be a quadratic residue * Problem 8.4).11). procedure rootLV (x . and b from u and v. p. Then we need only take a = b' and b = a' -b' Lu l v j. Let u and v be two positive integers. iii.0 <d p-1 and (a +')(r-1)12-c+dL (modp) if d = 0 then success F false else { c = 0) success f. (This is the heart of Euclid's algorithm. p -1) if a 2 = x (modp) { very unlikely } then success f.

Sec. 8.5

Las Vegas Algorithms


i. The Las Vegas algorithm finds a square root of x if and only if it randomly
chooses an a that gives the key to ' ; and
ii. Exactly (p + 3)/2 of the p -1 possible random choices for a give the key to x_.
[Hint : Consider the function








, p ,p-


defined by the equation (a - NFx_ ) f (a) = a +L (mod p ). Prove that this function is one-to-one and that f (a) is a quadratic residue modulo p if and only if a


does not give the key to '.]

This shows that the Las Vegas algorithm succeeds with probability somewhat
greater than one half, so that on the average it suffices to call it twice to obtain a
square root of x. In view of the high proportion of integers that give a key to ', it is
curious that no known efficient deterministic algorithm is capable of finding even one
of them with certainty.

The previous problem suggests a modification to the algoProblem 8.5.14.
rithm rootLV : only carry out the symbolic calculation of (a +' )(p- 1)12 if
(a2-x) mod p is a quadratic nonresidue. This allows us to detect a failure more
rapidly, but it takes longer in the case of a success. Give the modified algorithm explicitly. Is it to be preferred to the original algorithm ? Justify your answer.
* Problem 8.5.15.
if p

1 (mod 8)

The following algorithm increases the probability of success


procedure rootLV 2(x, p, var y, var success)
I assume that p is a prime and p = 1 (mod 4) }

a F-uniform(1..p-1)
if a 2 = -x (mod p)

I very unlikely and unfortunate }

then success - false
else let odd t and k >- 2 be such that p = 2k t + 1

compute c and d such that 0 <c 5p-1,0<-d <p-1
)t =-c + d

and (a +

(mod p)

ifc =0ord =0
then success - false
else while c 2

d 2x (mod p) do

b F-(c2-d2x)modp
d E-- 2cd mod p

c -h

compute y such that 1 5 y 5 p- 1 and yd = 1 (mod p)

y - cy mod p
success F- true
(1/2)k-1 and that the
Prove that the probability of failure of this algorithm is exactly
while loop is executed at most k - 2 times, where k is specified in the algorithm.

Probabilistic Algorithms


Chap. 8

Problem 8.5.16.
An even more elementary problem for which no efficient
deterministic algorithm is known is to find a quadratic nonresidue modulo p where
p = 1 (mod 4) is a prime.
i. Give an efficient Las Vegas algorithm to solve this problem.

ii. Show that the problem is not more difficult than the problem of finding an
efficient deterministic algorithm to calculate a square root. To do this, suppose
that there exists a deterministic algorithm root 2(x , p) that is able to calculate
efficiently ' mod p, where p =- 1 (mod 4) is prime and x is a quadratic residue
modulo p. Show that it suffices to call this algorithm less than L19 P I times to
be certain of obtaining, in a way that is both deterministic and efficient, a quadratic nonresidue modulo p. (Hint : Let k be the largest integer such that 2k

divides p -1 exactly. Consider the sequence x i = p - 1, xi = x7 mod p for
2<- i <- k. Prove that xi is a quadratic residue modulo p for 1 <- i < k , but that
xk is not.)

** Problem 8.5.17.
The converse of Problem 8.5.16. Give an efficient deterministic algorithm rootDET (x , p , z) to calculate a square root of x modulo p, provided

p is an odd prime, x is a quadratic residue modulo p, and z is an arbitrary quadratic
nonresidue modulo p.
The two preceding problems show the computational equivalence between the
efficient deterministic calculation of square roots modulo p and the efficient deterministic discovery of a quadratic nonresidue modulo p. This is an example of the
technique called reduction, which we study in Chapter 10.

8.5.3 Factorizing Integers
Let n be an integer greater than 1. The problem of factorizing n consists of finding the
unique decomposition n = p 1 ' p 2
such that m I , M2, ... , Mk are positive
integers and p I < P 2 <
< p k are prime numbers. If n is composite, a nontrivial
factor is an integer x, 1 < x < n , that divides n exactly. Given composite n, the
problem of splitting n consists of finding some nontrivial factor of n.

Problem 8.5.18.
Suppose you have available an algorithm prime (n), which
tests whether or not n is prime, and an algorithm split(n), which finds a nontrivial
factor of n provided n is composite. Using these two algorithms as primitives, give an
algorithm to factorize any integer.

Section 8.6.2 concerns an efficient Monte Carlo algorithm for determining primality. Thus the preceding problem shows that the problem of factorization reduces to
the problem of splitting. Here is the naive algorithm for the latter problem.

Sec. 8.5

Las Vegas Algorithms


function split (n)
{ finds the smallest nontrivial factor of n if n is composite
or returns 1 if n is prime }

for i F- 2 to LJ j do
if (n mod i) = 0 then return i
return I
Problem 8.5.19.

Why is it sufficient to loop no further than ' ?

The preceding algorithm takes a time in Q(J) in the worst case to split n. It is
therefore of no practical use even on medium-sized integers : it could take more than 3

million years in the worst case to split a number with forty or so decimal digits,
counting just
microsecond for each trip round the loop. No known algorithm,
whether deterministic or probabilistic, can split n in a time in 0 (p (m)) in the worst
case, where p is a polynomial and m = [log(l +n)l is the size of n. Notice that
Z 10"'12, which is not a polynomial in m. Dixon's probabilistic algorithm is

nevertheless able to split n in a time in 0 (20( m '09 - ))

Problem 8.5.20.
Prove that 0 (mk ) c O (2014"' kg n )) C O
the values of the positive constants k and b.


The notion of quadratic residue modulo a prime number (Section 8.5.2) generalLet n be any positive integer. An integer x,
1 5 x S n -1, is a quadratic residue modulo n if it is relatively prime to n (they have
no nontrivial common factor) and if there exists an integer y, 1 <- y < n - 1, such that
izes to composite numbers.

x = y 2 (mod n). Such a y is a square root of x modulo n. We saw that a quadratic
residue modulo p has exactly two distinct square roots modulo p when p is prime.
This is no longer true modulo n if n has at least two distinct odd prime factors. For
instance, 82 = 132 = 222 = 272 = 29 (mod 35).

* Problem 8.5.21.
Prove that if n = pq, where p and q are distinct odd primes,
then each quadratic residue modulo n has exactly four square roots. Prove further that

exactly one quarter of the integers x that are relatively prime to n and such that

<- x <- n - 1 are quadratic residues modulo n.

Section 8.5.2 gave efficient algorithms for testing whether x is a quadratic residue

modulo p, and if so for finding its square roots. These two problems can also be
solved efficiently modulo a composite number n provided the factorization of n is
given. If the factorization of n is not given, no efficient algorithm is known for either

of these problems. The essential step in Dixon's factorization algorithm is to find two
integers a and h relatively prime to n such that a2 = b2 (mod n) but a ±b (mod n).
This implies that a2 - h2 = (a - b)(a + b) = 0 (mod n). Given that n is a divisor nei-

ther of a +h nor of a -b, it follows that some nontrivial factor x of n must divide
a + b while nix divides a -b. The greatest common divisor of n and a + b is thus a

Probabilistic Algorithms


Chap. 8

nontrivial factor of n. In the previous example, a = 8, b = 13, and n = 35, and the
greatest common divisor of a +b = 21 and n = 35 is x = 7, a nontrivial factor of 35.
Here is an outline of Dixon's algorithm.

procedure Dixon (n, var x, var success)
{ tries to find some nontrivial factor x of composite number n }

if n is even then x <- 2, success - true
else for i - 2 to Llog3n J do
if n 11' is an integer then x <- n
success - true
{ since n is assumed composite, we now know that it has
at least two distinct odd prime factors }
a, b <- two integers such that a 2 = b 2 (mod n)

if a °±b (mod n) then success <- false
else x <- gcd(a+b, n) { using Euclid's algorithm }
success F- true

So how we can find a and b such that a2 = b2 (mod n)? Let k be an integer to
be specified later. An integer is k -smooth if all its prime factors are among the first k
prime numbers. For instance, 120 = 23 x 3 x 5 is 3-smooth, but 35 = 5 x 7 is not.
When k is small, k-smooth integers can be factorized efficiently by an adaptation of
the naive algorithm split (n) given earlier. In its first phase, Dixon's algorithm chooses
integers x randomly between 1 and n - 1. A nontrivial factor of n is already found if
by a lucky fluke x is not relatively prime to n . Otherwise, let y = x 2 mod n . If y is
k-smooth, both x and the factorization of y are kept in a table. The process is repeated
until we have k + 1 different integers for which we know the factorization of their
squares modulo n.

Let n = 2,537 and k = 7. We are thus concerned only with
Example 8.5.2.
the primes 2, 3, 5, 7, 11, 13, and 17. A first integer x = 1,769 is chosen randomly.
We calculate its square modulo n : y = 1,240. An attempt to factorize
1,240 = 23 x 5 x 31 fails since 31 is not divisible by any of the admissible primes.
A second attempt with x = 2,455 is more successful : its square modulo n is
y = 1,650 = 2 x 3 x 52 x 11. Continuing thus, we obtain
x,= 2 , 455 y I= 1 ,650 = 2 x 3 x 52 x 11

y2=2,210 =2x5x 13x 17
X3=1,105 y3= 728 =23x7x13

x2= 970

X4 = 1,458

y4 = 2,295 = 33 x 5 x 17

x5= 216

y5= 990 =2x32x5x11

X6 =


y6= 1,326 = 2 x 3 x 13 x 17

X7 = 1,844

y7= 756 =22x33x7

xs =

ys= 2,288 = 24 x 11 x 13


Sec. 8.5

Las Vegas Algorithms


Problem 8.5.22. Given that there are 512 integers x between 1 and 2,536 such
that x2 mod 2537 is 7-smooth, what is the average number of trials needed to obtain
eight successes like those in Example 8.5.2 ?

The second phase of Dixon's algorithm finds a nonempty subset of the k + I
equations such that the product of the corresponding factorizations includes each of the
k admissible prime numbers to an even power (including zero).

Example 8.5.3.

There are seven possible ways of doing this in Example 8.5.2,


y1y2y4y8=26x34x54x70 x112x132x172
Y1Y3Y4Y5Y6Y7=28 x310x54x72x 112x 132x 172.

Problem 8.5.23.

Find the other five possibilities.

Problem 8.5.24. Why is there always at least one solution ? Give an efficient
algorithm for finding one. [Hint : Form a (k + 1) x k binary matrix containing the parities of the exponents. The rows of this matrix cannot be independent (in arithmetic
modulo 2) because there are more rows than columns. In Example 8.5.3, the first
dependence corresponds to
(1,1,0,0,1,0,0) + (1,0,1,0,0,1,1) + (0,1,1,0,0,0,1) + (0,0,0,0,1,1,0)

(0,0,0,0,0,0,0) (mod 2) .

Use Gauss-Jordan elimination to find a linear dependence between the rows.]

This gives us two integers a and b such that a2 = b2 (mod n). The integer a is
obtained by multiplying the appropriate xi and the integer b by halving the powers of
the primes in the product of the y; If a 4t±b (mod n), it only remains to calculate the
greatest common divisor of a + b and n to obtain a nontrivial factor. This occurs with
probability at least one half.

Example 8.5.4.
The first solution of Example 8.5.3 gives
a = x1x2x4x8 mod n = 2,455 x 970 x 1,458 x 433 mod 2,537 = 1,127 and

b = 23 x 32 x 52 x 11 x 13 x 17 mod 2,537 = 2,012 *±a (mod n).
The greatest common divisor of a +b = 3,139 and n = 2,537 is 43, a nontrivial factor
of n. On the other hand, the second solution gives
a =xlx3x4x5x6x7 mod n = 564 and

b=24x35x52x7x11x13x17mod 2,537=1,973=-a (mod n),
which does not reveal a factor.

Problem 8.5.25.
Why is there at least one chance in two that
a t ± b (mod n)? In the case when a
b (mod n), can we do better than simply
starting all over again ?

Probabilistic Algorithms


Chap. 8

It remains to be seen how we choose the best value for the parameter k. The
larger this parameter, the higher the probability that x 2 mod n will be k -smooth when
x is chosen randomly. On the other hand, the smaller this parameter, the faster we can
carry out a test of k -smoothness and factorize the k -smooth values y; , and the fewer
such values we need to be sure of having a linear dependence. Set L = e
let h E IR+. It can be shown that if k = L b , there are about L 112h failures for every success when we try to factorize x2 mod n. Since each unsuccessful attempt requires k
divisions and since it takes k + 1 successes to end the first phase, the latter takes an
average time that is approximately in 0 (L 2h +1/2h ), which is minimized by b = z . The
second phase takes a time in O (k 3) = 0(L31) by Problem 8.5.24 (it is possible to do
better than this), which is negligible when compared to the first phase. The third phase
can also be neglected. Thus, if we take k = E , Dixon's algorithm splits n
with probability at least one half in an approximate expected time in 0(L2)
) and in a space in O (L).
= O (e 2 I
Several improvements make the algorithm more practical. For example, the proI

bability that y will be k-smooth is improved if x is chosen near [I 1, rather than
being chosen randomly between I and n - 1. A generalization of this approach,
known as the continued fraction algorithm, has been used successfully. Unlike
Dixon's algorithm, however, its rigorous analysis is unknown. It is therefore more
properly called a heuristic. Another heuristic, the quadratic sieve, operates in a time in
0 (L 9is) and space in 0 (LD). In practice, we would never implement Dixon's algorithm because the heuristics perform so much better. More recently, H. W. Lenstra Jr.
has proposed a factorization algorithm based on the theory of elliptic curves.

Problem 8.5.26.

Let n = 1040 and L = e 1° n In Inn


Compare L 9/8, L 2, and

microseconds. Repeat the problem with n = 10s0p
8.5.4 Choosing a Leader

A number of identical synchronous processors are linked into a network in the shape of

a ring, as in Figure 8.5.1. Each processor can communicate directly with its two
immediate neighbours. Each processor starts in the same state with the same program
and the same data in its memory. Such a network is of little use so long as all the processors do exactly the same thing at exactly the same time, for in this case a single
processor would suffice (unless such duplication is intended to catch erratic behaviour
from faulty processors in a sensitive real-time application). We seek a protocol that

allows the network to choose a leader in such a way that all the processors are in
agreement on its identity. The processor that is elected leader can thereafter break the
symmetry of the network in whatever way it pleases by giving different tasks to different processors.
No deterministic algorithm can solve this problem, no matter how much time is
available. Whatever happens, the processors continue to do the same thing at the same
time. If one of them decides that it wants to be the leader, for example, then so do all

Sec. 8.5

Las Vegas Algorithms

Figure 8.5.1.


A ring of identical processors.

the others simultaneously. We can compare the situation to the deadlock that arises
when two people whose degree of courtesy is exactly equal try to pass simultaneously
through the same narrow door. However, if each processor knows in advance how
many others are connected in the network, there exists a Las Vegas algorithm that is
able to solve this problem in linear expected time. The symmetry can be broken on the
condition that the random generators used by the processors are independent. If the
generators are in fact pseudorandom and not genuinely random, and if each processor
starts from the same seed, then the technique will not work.

Suppose there are n processors in the ring. During phase zero, each processor
initializes its local variable m to the known value n, and its Boolean indicator active to
the value true. During phase k, k > 0, each active processor chooses an integer randomly, uniformly, and independently between 1 and m. Those processors that chose 1
inform the others by sending a one-bit message round the ring. (The inactive processors continue, nevertheless, to pass on messages.) After n -1 clock pulses, each processor knows the number I of processors that chose 1. There are three possibilities. If
/ = 0, phase k produces no change in the situation. If l > 1, only those processors that
chose 1 remain active, and they set their local variable m to the value 1. In either case
phase k + I is now begun. The protocol ends when l = 1 with the election of the
single processor that just chose 1.
This protocol is classed as a Las Vegas algorithm, despite the fact that it never
ends by admitting a failure, because there is no upper bound on the time required for it

to succeed. However, it never gives an incorrect solution: it can neither end after
electing more than one leader nor after electing none.

Let 1(n) be the expected number of phases needed to choose a leader among n
processors using this algorithm (not counting phase zero, the initialization). Let
p (n , j) _ [ ] n-i (1- lln)" -i be the probability that j processors out of n randomly
choose the value I during the first stage. With probability p (n, 1) only a single phase
is needed ; with probability p (n , 0) we have to start all over again ; and with

6.5. (It performs much better on small integers. Consequently.27.1.Probabilistic Algorithms 262 Chap. j ). ** Problem 8. n>_2. l (j) subsequent phases will be necessary on the average. Prove that the following algorithm decides correctly whether or not an integer is prime in more than 80% of the cases. Since each phase thus takes linear time.5. whether it be deterministic or probabilistic.j)l(j). that it is not a Monte Carlo algorithm by exhibiting an instance on which the algorithm systematically gives a wrong answer.n)) i=2 * Problem 8. only failing now and again in some special cases. 30030) = 1 then return true else return false ( using Euclid's algorithm } . This is not the same as saying that it works correctly on a majority of instances.442. n l(n)=1+p(n.5. No warning is usually given when the algorithm makes a mistake. 2 <_ j <_ n.6 MONTE CARLO ALGORITHMS There exist problems for which no efficient algorithm is known that is able to obtain a correct solution every time. n-)oo 11 One phase of this protocol consists of single bit messages passed round the ring. 8 probability p (n .0)l(n)+J:p(n. Show that l (n) < e = 2.718 for every n >_ 2. Problem 8. but it finds a correct solution with high probability whatever the instance considered.28. on the other hand. give a Monte Carlo algorithm that is able to determine n exactly with a probability of error less than p whatever the number of processors.) Show.p(n. the preceding problems show that the choice of a leader also takes expected linear time. Nevertheless. A Monte Carlo algorithm occasionally makes a mistake. Prove that no protocol (not even a Las Vegas protocol) can solve the problem of choosing a leader in a ring of n identical processors if they are not given the value n . function wrong (n) if gcd(n . 8.0)-p(n. j=2 A little manipulation gives n-I I(n)=(I+ F.j)l(j))/(1-p(n.29. given an arbitrary parameter p > 0. Show that lim I (n) < 2. * Problem 8.

It suffices to call MC (x) at least [cElg 1/81 times and to return the most frequent answer (ties can be broken arbi- trarily) to obtain an algorithm that is consistent and (1.6 263 What constant should be used instead of 30. This allows us to amplify the advantage of the algorithm. p = and q = I -p = z -e. Consider the following algorithm. Some Monte Carlo algorithms take as a parameter not only the instance to be solved but also an upper bound on the probability of error that is acceptable. To prove the preceding claim. . u . A Monte Carlo algorithm is p-correct if it returns a correct solution with probability not less than p. 8.MC (x) . The repetitive algorithm finds the correct answer if the latter is that the ('+F-)-correct algorithm is called. whatever the instance considered. The advantage of such an algorithm is p -'/2.2. To increase the probability of success of a consistent.Monte Carlo Algorithms Sec. Show on the other hand that it might not even be 71%-correct if MC.cE lg 1/8 be the number of times Let m = Ln / 2j+ 1. p-correct algorithm.S)-correct. The algorithm is consistent if it never gives two different correct solutions to the same instance. This increases our confidence in the result in a way similar to the world series calculation of Section 5. v . Let c. hence 84%-correct. 75%-correct Monte Carlo algo- rithm. were not consistent. Let MC(x) be a consistent. Problem 8. The time taken by such algorithms is then expressed as a function of both the size of the instance and the reciprocal of the acceptable error probability. let a and S be two positive real numbers such that e + S < -1. Let x be some instance to be solved. = -2 / Ig (1. More generally. let n >. though 75%-correct. Let MC (x) be a Monte Carlo algorithm that is consistent and (z + e)-correct. Its error probability is therefore at most m-1 E Prob[ i correct answers in n tries ] i=0 < M-1 1: i=0 n []piqnl i z+e.6. function MC 3(x) MC (x) . so as to obtain a new algorithm whose error probability is as small as we choose.030 to bring the proportion of successes above 85% even if very large integers are considered? Let p be a real number such that 1/2 < p < 1. however small it may be.2. we need only call it several times and choose the most frequent answer. obtained at least m times.4c2 ).MC (x) t if t = u or t = v then return t return v Prove that this algorithm is consistent and 27/32-correct.

4E2)( = 2-1g 1/6 12) lg 1/8 since 0 < 1. For example. where m-1 S= 2-E v 2i i =0 i (4 -£2)i < 1 (1-4E2)m 4E nm The first part of this formula can be used efficiently to find the exact number of repetitions required to reduce the probability of error below any desired threshold S. and that repeating it 600 times yields an algorithm that is better that 99%-correct. n i=0 (pq )n12 (qlp) Chap. Alternatively. errors being possible only when it returns the answer false.Probabilistic Algorithms 264 =(Pq)ni2 I. A more precise calculation shows that it is enough to repeat the original algorithm 269 times. The probability of success of the repetitive algorithm is therefore at least 1. 8 -i m-1 since q /p < 1 and 2. the resulting algorithm is (1-S)-correct. Prove that cE < (In 2)/2E2. Assume for simplicity that we are dealing with a decision problem and that the original Monte Carlo algorithm is biased in the sense that it is always correct whenever it returns the answer true. most Monte Carlo algorithms that occur in practice are such that we can increase our confidence in the result obtained much more rapidly.3. The preceding theorem tells us that this can be achieved by calling the original algorithm about 600 times on the given instance. suppose we have a consistent Monte Carlo algorithm whose advantage is 5% and we wish to obtain an algorithm whose probability of error is less than 5% (that is.6. it would be silly to return the most frequent answer : a single true outweighs any number of falses. [This is because some of the inequalities used in the proof are rather crude.S. Problem 8. Fortunately. it suffices to repeat such an . As we shall see shortly. A more complicated argument shows that if a consistent (Z + E)-correct Monte Carlo algorithm is repeated 2m -1 times.] Repeating an algorithm several hundred times to obtain a reasonably small probability of error is not attractive.0 i=0 (pq)n/2 ` (nl (Pq)ni22n = 2)n i2 _ (4pq )n i2 = (1-4F (1. a good upper bound on this number of repetitions is quickly obtained from the second part: find x such that e'x .i i >.4E2 < 1 =6 since aliisa = 2 for every a > 0. If we repeat such an algorithm several times to increase our confidence in the final result.1/(2SI) and then set m = Fx/4E21. we wish to go from a 55%-correct algorithm to a 95%-correct algorithm).

Monte Carlo Algorithms Sec. or 6 times to obtain a 99%-correct algorithm. . the restriction that the original algorithm be p-correct for some p > i no longer applies : arbitrarily high confidence can be obtained by repeating a biased p-correct algorithm enough times. In both cases. Yk- If there exists an i such that y. 8. the probability of such an error occurring is not greater than 1-p given that the algorithm is p-correct. = y # yo for all the i. and if x E X. the correct solution to all the instances that belong to X is yo. . the correct solution is necessarily yo. the algorithm always returns the correct solution. More formally. the solution returned by the algorithm is always correct whenever the instance to be solved is not in X. the preceding argument shows that this is indeed the correct solution . = yo. A Monte Carlo algorithm is yo-biased if there exists a subset X of the instances such that i. Let x be an instance. even if p < z (as long as p > 0). What can we conclude if y =yo? If x e X. y is indeed correct .p )k . so yo is indeed correct . the only possible explanation is that x EX (because the algorithm is consistent by assumption). and let yo be some distinguished answer. let us return to an arbitrary problem (not necessarily a decision problem). and if x EX. we may conclude that yo is a correct solution. What happens if on the other handy # y o ? If x e X. yo-biased and p-correct. and ii. but the algorithm may not always return the correct solution to these instances. it is not required that an efficient test be known for membership in X. but the probability of such an occurrence is at most (1.6 265 algorithm 4 times to improve it from 55%-correct to 95%-correct. if there exist i # j such that y. Let MC be a Monte Carlo algorithm that is consistent. and let y be the solution returned by MC (x). the algorithm has made a mistake since the correct solution is y o . and therefore the correct solution is yo. and if y. it is still possible that the correct solution is yo and that the algorithm has made a mistake k times in succession on x E X. Although the distinguished answer yo is known explicitly. The following paragraph shows that this definition is tuned precisely to make sure that the algorithm is always correct when it answers yo. Moreover. :t.yj . Now suppose that we call MC (x) k times and that the answers obtained are y.

p-correct. given any x E I. is averaged over the entire sequence of answers given by the algorithm. Indeed. This last remark may appear trivial. we may be interested in generating a random prime of a given length . Assume your consistent. It may be tempting to conclude that "the probability that y is an incorrect answer is at most (1-p )k ". despite the fact that we cannot tell for sure which it is. p-correct Monte Carlo algorithm to decide. yo-biased Monte Carlo algorithm has yielded k times in a row the same answer y # yo on some instance x. if you systematically quiz the algorithm with instances for which the correct solution is yo. that p = s (once again. Consider the following algorithms. it will always be wrong whenever it "believes" otherwise. Prob[ MC (x) = true ] = I for each instance x E S and Prob[ MC (x) = true ] <_ q for each instance x 0 S. consider a nonempty finite set I of instances.see Section 8. It is important to understand how to interpret such behaviour correctly. but it is in fact crucial if the probabilistic algorithm is used to generate with high probability some random instance x on which the correct answer is a specific Y #Yo To illustrate this situation. this is not allowed for general Monte Carlo algorithms.6. my proportion of errors should not significantly exceed (1-p )k ". 8 Suppose.Probabilistic Algorithms 266 Chap. The correct interpretation is as follows : "I believe that y is the correct answer and if you quiz me enough times on different instances. The "proportion of errors" in question. or not. The probability in question is therefore either 0 or 1. By definition of a false-biased algorithm. however. for example. function repeatMC (x . We are interested in generating a random member of some subset S S 1. whether x E S.2. yo-biased algorithm yield an algorithm that is (1 . but it causes no problems with a biased algorithm). k) i-0 ans E. In general. k repetitions of a consistent. Let q =1-p . (As a practical example. not only over those occurrences in which the algorithm actually provides such probabilistic answers.true while ans and i < k do if-i+1 ans F. or extremely confident that the solution obtained on every one of the trials is correct (since otherwise the probability of obtaining the results observed is less than one chance in a million). It suffices to repeat the algorithm at most 20 times to be either sure that the correct solution is yo (if either of the first two cases previously described is observed). Such a conclusion of course makes no sense because either the correct answer is indeed y. p-correct.MC (x) return ans .) Let MC be a false-biased.(1-p )k )-correct and still consistent and yo-biased.

we need an a priori probability that a call on uniform (I) returns a member of S. Give an efficient Las Vegas algorithm LV (x .6 267 Monte Carlo Algorithms function genrand (k) repeat x . whereas algorithm B is q-correct and false-biased. 8. solving the same decision problem.Sec. then the probability that a call on genrand (k) returns some x 0 S is about q k /r if q k<< r<< 1. Prove that the probability that a call on genrand (k) erroneously returns some x 0 S is at most 1 1+ 1r q-' r k In particular. if the error probability of MC on instances not in S is exactly q (rather than at most q). The problem is that we must not confuse the conditional probabilities Prob[ X I Y ] and Prob[ Y I X ]. and nearly 1 if r << g k This can be significant when the confidence one gets in the belief that x belongs to S from running MC several times must be weighed against the a priori likelihood that a randomly selected x does not belong to S. This is wrong in general. * Problem 8. We are not aware of any unbiased Monte Carlo algorithm sufficiently simple to feature in this introduction.6. that is. but we are in fact interested in Prob[ Y I X ]. var y. It is correct that Prob[X I Y ] <_ q k. What is the best value of r you can deduce so that your Las Vegas algorithm succeeds with probability at least r on each instance ? El .4. var success) to solve the same problem. about z2 if q k= r << 1. They all involve the solution of a decision problem. k) return x It is tempting to think that genrand(k) returns a random member of S with a probability of failure at most q k . To calculate this pro- bability. Thus the section continues with some examples of biased Monte Carlo algorithms.uniform (I) until repeatMC (x. Let A and B be two efficient Monte Carlo algorithms for Problem 8.6. Algorithm A is p-correct and true-biased.5. where X stands for "repeatMC (x . This situation illustrates dramatically the difference between the error probability of genrand and that of repeatMC. k) returns true" and Y stands for "x 0 S ". the only possible answers that they can return are true and false. Let r denote the probability that x E S given that x is returned by a call on uniform (I).

but 3/4-correct. if the first call of maj (T) returns false. 4.. the element chosen is a majority element.11. each call of maj(T) is certain to return false. with certainty . with probability > z T has no majority element = maj (T) = false. the probability of choosing an element that is in a minority is less than one-half. If the array does indeed contain a majority element. n ]) i F uniform (1 .0 forj f. it is nonetheless possible that T contains a majority element. and 4.11.7). Summing up. function maj (T [1.11. The probability of error decreases because the successive calls of maj (T) are independent : the fact that .Probabilistic Algorithms 268 Chap. hence so does maj 2(T ). which happens with probability 1-p .6..1 Majority Element in an Array The problem is to determine whether an array T [I .ltondoifT[jI=x thenk '-k +1 return (k > n / 2) We see that maj (T) chooses an element of the array at random. If the array does have a majority element. z that is T has a majority element = maj (T) = true. n ] has a majority element (see Problems 4.6. The general technique for biased Monte Carlo algorithms allows us to reduce this probability efficiently to any arbitrary value. on the other hand. If the answer returned is true.5. and then checks whether this element forms a majority in T. On the other hand. and hence trivially there is a majority element in T. and if one of its elements is chosen at random. the answer returned is false. n) x -T[i] k F. if the answer returned by maj (T) is false. An error probability of 50% is intolerable in practice. although in this case the element chosen randomly is in a minority. the probability that maj 2(T) will return true if the array T has a majority element is p+(I-p)p=I-(1-p)2>3/4. the second call of maj (T) may still with probability p return true. The algorithm maj2 is therefore also true-biased. If. in which case maj 2(T) also returns true. since majority elements occupy more than half the array. the probability that the first call of maj (T) will return true is p > z . First. In sum. too. this algorithm is true-biased and 4correct.. we may reasonably suspect that T does indeed have no majority element. consider function maj 2(T) if maj (T) then return true else return maj (T) If the array does not have a majority element. Consider the following algorithm. Therefore. 8 8. and in this case maj 2(T) returns true.

return false is less than 2-k if T contains a majority element.) A first approach to finding a probabilistic algorithm might be function prime (n) d . 8. The problem is to decide whether a given integer is prime or composite. we can be certain that T contains a majority element. and we can be certain that n is composite. (It is currently possible to establish with certainty the primality of numbers up to 213 decimal digits within approximately 10 minutes of computing time on a CDC CYBER 170/750. For larger values of n the situation gets worse. Consider for example n = 2. function majMC (T. 8.6). the algorithm has been lucky enough to find a nontrivial factor of n. but it is still unsatisfactory.6. E) k rlg(1/E)1 for i <-. . This is interesting only as an illustration of a Monte Carlo algorithm since a linear time deterministic algorithm is known (Problems 4. No deterministic or Las Vegas algorithm is known that can solve this problem in a reasonable time when the number to be tested has more than a few hundred decimal digits. the answer true is returned with high probability even if n is in fact composite. The algorithm chooses an integer randomly between 2 and 51.6. where n is the number of elements in the array and E is the acceptable probability of error.623 = 43 x 61.uniform (2 . On the other hand. The algorithm can be improved slightly by testing whether n and d are relatively prime. In 98% of calls the algorithm will inform us incorrectly that n is prime.2 Probabilistic Primality Testing This classic Monte Carlo algorithm recalls the algorithm used to determine whether or not an array has a majority element. Show that the probability that k successive calls of maj(T) all Problem 8.6. Thus there is only a meagre 2% probability that it will happen on d = 43 and hence return false. LJ J) return ((n mod d) # 0) If the answer returned is false. El The following Monte Carlo algorithm solves the problem of detecting the presence of a majority element with a probability of error less than E for every E > 0. as soon as any call returns true.11.Sec.. Unfortunately.6 Monte Carlo Algorithms 269 maj (T) has returned false on an array with a majority element does not change the probability that it will return true on the following call on the same instance.I to k do if maj (T) then return true return false The algorithm takes a time in 0 (n log(1/E)).5 and 4.11. using Euclid's algorithm.

If n is composite. 8 To obtain an efficient Monte Carlo algorithm for the primality problem. The situation is even better if n is composed of a large number r of distinct prime factors : in this case it cannot be a strong pseu- doprime to more than 4(n)/2r-' . For example. ** Problem 8.2 different bases. Let a be an integer such that 2 5 a <.5. If n is prime.8.i < s and art _ -1 (mod n). 289 has only 14 false witnesses of primality. Let n be an odd integer greater than 4.2).2. This suggests the following algorithm. 1582"9 = 1312 = 110 (mod 289). in which case we obtain the correct answer with cer- . This certainty is not accompanied however by any indication of what are the nontrivial factors of n.n . Your algorithm should not take significantly longer (and sometimes even be faster) than simply calculating a (" . it is a strong pseudoprime to any base.1)/2 mod n with dexpoiter from Section 4. n -2) if n is strongly pseudoprime to the base a then return true else return false For an odd integer n > 4 the theorem ensures that n is composite if Rabin (n) returns false. where 4(n) < n -1 is Euler's totient function. For instance. 158 is a false witness of primality for 289 because 288 = 9 x 25. In both cases. function Rabin (n) { this algorithm is only called if n > 4 is odd } a <--uniform (2. and 1588 9 = 2512 =_ I (mod 289). Contrariwise. 1589 = 131 (mod 289). This Monte Carlo algorithm for deciding primality is falsebiased and 3/4-correct. Give an efficient algorithm for testing whether n is a strong pseudoprime to the base a.8. This test of primality has several points in common with the Las Vegas algorithm for finding square roots modulo p (Section 8. 1584"9 = 1102 = 251 (mod 289).Probabilistic Algorithms 270 Chap. and let s and t be positive integers such that n -1 = 2s t. an integer a is chosen at random. The theorem is generally pessimistic. Prove this theorem.. There exist however composite numbers that are strong pseudoprimes to some bases. Problem 8. there is at least a 75% chance that n will not be a strong pseudoprime to the base a. finally.9)/4 different bases. Such a base is then a false witness of primality for this composite number.6. The theorem assures us that if n is composite. it cannot be a strong pseudoprime to more than (n . whereas 737 does not even have one.6.7. we need a theorem whose proof lies beyond the scope of this book. we may begin to suspect that n is prime if Rabin (n) returns true. where t is odd. We say that n is a strong pseudoprime to the base a if a' _ 1 (mod n) or if there exists an integer i such that 0 <.

2.6. 8. program print primes print 2. The difference can also be explained using Problem 8. At any given moment the question "Does Si = Si ?" may be asked. since every integer larger than I is either prime or composite.6. Notice that this combines with the false-biased algorithm described previously to yield a Las Vegas algorithm (by Problem 8.10.6 271 tainty.) ** Problem 8. the probability of error can be made arbitrarily small by repeating the algorithm. Consider the following nonterminating program. . regardless of how long the program is allowed to run. but rather. Llg n j) then print n n -n + 2 ad museum Clearly. 8. if n is a strong pseudoprime to the base a.2. The first reply would be nonsense. Prove that this is unlikely to happen. Find a true-biased Monte Carlo algorithm for primality testing whose running time is polynomial in the logarithm of the integer being tester. otherwise I have observed a natural phenomenon whose probability of occurrence was not greater than E". where xE U \ Si and I S i _< n. this can be due either to the fact that n is indeed prime or to the fact that a is a false witness of primality for the composite number n. A philosophical remark is once again in order: the algorithm does not reply "this number is prime with probability 1-E".6. all of which are empty at the outset.Monte Carlo Algorithms Sec. not necessarily disjoint. We suppose that N is quite large while n is small. More precisely. * Problem 8. The basic operation consists of adding x to the set Si . As usual. This difference is explained by the fact that the Las Vegas algorithm is able to detect when it has been unlucky : the fact that a does not provide a key for is easy to test. (Note : this figure of 99% is very conservative as it would still hold even if Rabin (n) had a flat 25% chance of failure on each composite integer. "I believe this number to be prime . Nevertheless the algorithm for testing primality is only a Monte Carlo algorithm whereas the one for finding square roots is Las Vegas.5). prove that the probability is better than 99% that not even a single composite number larger than 100 will ever be produced. and a collection of n sets. every prime number will eventually be printed by this program. On the other hand.9.6. One might also expect composite numbers to be produced erroneously once in a while.3 n 5 repeat if repeatRahin (n.3 A Probabilistic Test for Set Equality We have a universe U of N elements. Similarly there is better than a 50% chance that a will provide a key for finding .

x) v[i] F.4. each call of rand (x) takes constant expected time. Each call of rand (x) returns a random string chosen with equal probability among all the strings of length k. For any e > 0 fixed in advance. each test of equality will take a time in S2(k). This algorithm provides an interesting application of universal hashing (Section 8. if indeed it is not in S2(k log k ). Let k = Ilg(max(m. Whatever structure is chosen.some random k -bit string add x to the table and associate y to it return y Notice that this is a memory function in the sense of Section 5. 8 The naive way to solve this problem is to keep the sets in arrays. Let e > 0 be the error probability that can be tolerated for each request to test the equality of two sets. To each set Si we associate a variable v [i] initialized to the binary string composed of k zeros. 1/E))]. Here is the algorithm for adding an element x to the set Si .Probabilistic Algorithms 272 Chap. What is the probability that v [i ] = v [ j ] when Si # Sj ? Suppose without loss of generality that there exists an xo a Si such that xa 9 S/ . lists. For a set S c U.4). The table is used to implement a random function rand : U -+ (0. let . there exists a Monte Carlo algorithm that is able to handle a sequence of m questions in an average total time in 0 (m). j ) ifv[i]=v[jI then return true else return false It is obvious that Si # Sj if v [i] # v [ j ]. Let Si = Si \ (x0}.1 }k . The Monte Carlo algorithm first chooses a function at random in this class and then initializes a hash table that has U for its domain.1 }k as follows. Let H be a universal2 class of functions from U into (0. procedure add (i . We suppose that x is not already a member of Si . The algorithm never makes an error when Si = S/ . in the opposite case its probability of error does not exceed E. the set of k -bit strings. search trees.v[i]®rand(x) The notation t ® u stands for the bit-by-bit exclusive-or of the binary strings t and u. where k is the cardinality of the larger of the two sets concerned. and two calls with different arguments are independent.7. Thanks to the use of universal hashing. function rand (x) if x is in the table then return its associated value y E. or hash tables. The algorithm to test the equality of Si and S1 is: function test (i . Two different calls with the same argument return the same value.

Si = Sj u {x }. v [i] = XOR (Si) = rand (xo) ®XOR (S.6. Problem 8. Notice the similarity to the use of signatures in Section 7.. N 1. which does not worry us when we want to test set equality.6 Monte Carlo Algorithms 273 XOR (S) be the exclusive-or of the rand (x) for every x E S . the different tests of equality are not independent. may be troublesome for other applications.6. Your implementation should satisfy each request in constant time in the worst case. Let yo = XOR (S. 2. Two such calls separated by a call of init should on the other hand be independent.4.14. A call of elim (i.) and v[j] = XOR (SS ). which decides without ever making an error whether x E Si . x 9 Si u SS . Universal hashing allows us to implement a random function ** Problem 8.12. What happens with this algorithm if by mistake a call of Problem 8. and if an application of the algorithm replies incorrectly that Si = Sj . x). . x). Two calls p (i) and p Q) that are not separated by a call of init should therefore return two different answers if and only if i # j. x) is only permitted when x is already in Si . x) is made when x is already in Si ? Problem 8. The fact that v [i ] = v [ j ] implies that rand (xo) = yo. ) ®XOR (Si ). . The possibility that rand (x 1) = rand (X2) even though x I * X2. Sk = Si u {x }. the probability of this happening is only 2-k since the value of rand (xo) is chosen independently of those values that contribute to yo.13. if Si # Sj . Modify the algorithm so that it will work correctly (with probability of error c) even if a call of add (i. It is only possible to increase our confidence in the set of answers obtained to a sequence of requests by repeating the application of the algorithm to the entire sequence.1. You must accept two kinds of request : init and p (i) for 1 <_ i <_ N. rand : U -*{O. let N be an integer and let U = { 1. This Monte Carlo algorithm differs from those in the two previous sections in that our confidence in an answer "Si = Si " cannot be increased by repeating the call of test (i .1). A call of p (i) returns the value 7t(i) for the current permutation. j ). Show how you could also implement a procedure elim (i. v) takes constant time for 1 <_ u <_ v <_ N. . then it will also reply incorrectly that Sk=S1. For instance. A call of init initializes a new permutation tt : U -* U.1}". By definition. More precisely..Sec. x) is made when xE Si . A sequence of m requests must still be handled in an expected time in O (m). 8. whatever happens. You may use memory space in O (N).6. which removes the element x from the set Si . Moreover.7. Also implement a request member (i . Suppose that a call of uniform (u .2 and Example 8. but no request may consult or modify more than a constant number of memory locations : thus it is not possible to create the whole permutation when init is called.6.11.2.. (Hint: reread Problem 5. add (i. Show how to implement a random permutation.

5. decide probabilistically whether B is the inverse of A. which only computes the product approximately. but no such algorithm is known that only takes a time in 0 (n).) 8. and with the probabilistic algorithm mentioned in Section 8. we must compute ABX as A times BX.16..6. see for instance Hall (1873). which takes a time in 0 (n 2. and C and you would like to decide whether AB = C. B . (Hint: consider the columns of the matrix AB -C and show that at least half the ways to add and subtract them yield a nonzero column vector. Your algorithm should run in a time in 0 (n ).3. function goodproduct (A. but it was already in use in the secret world of atomic research during World War 11. and r (x) of degrees n. For the solution to Problem 8.6. (1495). (In the next chapter we shall see a deterministic algorithm that is capable of computing the symbolic product of two polynomials of degree n in a time in 0 (n log n ). provided 11 AB #C. B . C .11) if ABX = CX then return true else return false In order to take a time in 0 (n 2). in particular in Los Alamos. B. whenever AB = C. Here is an intriguing false-biased.17.1 ton do X [i ] <.Probabilistic Algorithms 274 Chap. correct Monte Carlo algorithm to decide whether r (x) is the symbolic product of p z (x) and q (x). n and 2n.7 REFERENCES AND FURTHER READING The experiment devised by Leclerc (1777) was carried out several times in the nineteenth century.376). Compare this with the fastest i known deterministic algorithm to compute the product AB (Section 4.2. 8 8. . n) array X [ 1 . The term "Las Vegas" was introduced by Babai' (1979) to distinguish probabilistic algorithms that occasionally make a mistake from those that reply correctly if they reply at all.uniform ({ -1. give a false-biased.6.15.9). New Mexico. The term "Monte Carlo" was introduced into the literature by Metropolis and Ulam (1949). providing a dramatic example of the topic discussed in Section 5.4 Matrix Multiplication Revisited You have three n x n matrices A. respectively. Given three polynomials p (x). Recall that it is often used to describe any probabilistic algorithm. n ] { to be considered as a column vector) for i F.6. adapt this algorithm to Problem 8.3. q (x). n) returns true Problem 8. The term "Sherwood" is our own. correct Monte Carlo algorithm that is capable of solving this problem in a time in d (n2).) Given two n x n matrices A and B. Prove that it returns false with probability at least z whenever AB # C. It is obvious that goodproduct (A . Problem 8. see Anon. It is no doubt the earliest recorded probabilistic algorithm.3. C.

The former includes tests for trying to distinguish a pseudorandom sequence from one that is truly random. this article and the one by Yao (1982) introduce the notion of an unpredictable generator.3. For more information on numeric probabilistic algorithms.12. the same reference gives the statement of Problem 8. is due to Peralta (1986). A more interesting generator from a cryptographic point of view is given by Blum and Micali (1984).14 is solved in Klamkin and Newman (1967). As a generalization. and Bennett. An early (1970) linear expected time probabilistic median finding algorithm is attributed to Floyd : see Exercise 5. Brassard. and Steele (1981). Numeric probabilistic algorithms designed to solve problems from linear algebra are discussed in Curtiss (1956). The point is made in Fox (1986) that pure Monte Carlo methods are not specially good for numerical integration with a fixed dimension : it is preferable to choose your points systemat- ically so that they are well spaced. and Shub (1986). Hopcroft.5. a more efficient unpredictable pseudorandom generator is proposed in Blum.21.4. Hammersley and Handscomb (1965).7 References and Further Reading 275 Two encyclopaedic sources of techniques for generating pseudorandom numbers are Knuth (1969) and Devroye (1986).4 follows Flajolet and Martin (1985).13 in Knuth (1973). Rabin (1980a) gives an efficient probabilistic algorithm for com- puting roots of arbitrary polynomials over any finite field.4.20. The experiments on the twenty queens problem were carried out by Pierre Beauchemin.4. Blum. The probabilistic approach to the eight queens problem was suggested to the authors by Manuel Blum. including Problem 8.15.Sec. The algorithm of Section 8. see Wegman and Carter (1981).4.3.15 appear there.4.14) come from Janko (1976).4. Feigenbaum. A guide to simulation is provided by Bratley. Section 8.4. Stanat. Computation with encrypted instances (end of Section 8. For a solution to Problem 8. several universal2 classes are described there. An analysis of this technique (Problem 8. consult Aho. Rivest. For solutions to Problem 8. which can pass any statistical test that can be carried out in polynomial time. It predates the classic worst-case linear time deterministic algorithm described in Section 4.1) is described in Kaliski. General techniques are given in Vazirani (1986. Universal hashing was invented by Carter and Wegman (1979). A probabilistic algorithm that is capable of finding the i th smallest among n elements in an expected number of comparisons in n +i +O(J) is given in Rivest and Floyd (1973). More references on this subject can be found in Brassard (1988). The solution to . and Robert (1988). 1987) to cope with generators that are only "semirandom".3. and Kilian (1987).3. 8.12) is given in Bentley. Fox. including the one from Problem 8. Problem 8. and Sherman (1988). The technique for searching in an ordered list and its application to sorting (Problem 8.3. Under the assumption that it is infeasible to factorize large numbers. consult Sobol' (1974). a technique known as quasi Monte Carlo.5.4. and Ullman (1974).6.2) is an idea originating in Feigenbaum (1986) and developed further in Abadi. Early algorithms to solve this problem are given in Lehmer (1969) and Berlekamp (1970). The application of probabilistic counting to the Data Encryption Standard (Example 8.5. Classic hash coding is described in Knuth (1968). and Schrage (1983). Vickery (1956). and Carasso (1971).2 for finding square roots modulo a prime number. many solutions to Problem 8.

The probabilistic test for set equality comes from Wegman and Carter (1981). The algorithm based on elliptic curves is discussed in Lenstra (1986). The expected number of false witnesses of primality for a random composite integer is investigated in Erdos and Pomerance (1986) . and Shallit (1986). Miller. The probabilistic integer factorization algorithm discussed in Section 8. and Miller (1977). We close by mentioning a few of them. Several interesting probabilistic algorithms have not been discussed in this chapter.14). Adleman. also read Freivalds (1977). see also Monier (1980).3 originated with Dixon (1981) . Goutier. Amplification of the advantage of an unbiased Monte Carlo algorithm is used to serve cryptographic ends in Goldwasser and Micali (1984). they also give a cryptographic application of universal hashing. consult Montgomery (1987). For more information on tests of primality and their implementation. More information on number theory can be found in the classic Hardy and Wright (1938).276 Probabilistic Algorithms Chap.6. Pomerance. . is described in Bach. read Valois (1987). 1980b). which allows us to decide efficiently whether a given integer is a perfect number and whether a pair of integers is amicable. The implication of Problem 8.6. Rabin (1976) gives an algorithm that is capable of finding the closest pair in expected linear time (contrast this with Problem 4.29. for a comparison with other methods. and Cohen and Lenstra (1987). which also gives a fast probabilistic splitting algorithm whose probability of success on any given composite integer is at least as large as the probability of failure of Rabin's test on the same integer. consult Williams (1978). The integer factorization algorithm of Pollard (1975) has a probabilistic flavour. Lenstra (1982).14 is in Brassard and Kannan (1988). The probabilistic test of primality presented here is equivalent to the one in Rabin (1976.5. including Problem 8. Consult Zippel (1979) for sparse polynomial interpolation probabilistic algorithms. Brassard. refer to Pomerance (1982). and for finding irreducible polynomials.4 for the generation of random numbers that are probably prime is explained in Beauchemin. Rabin (1980a) gives an efficient probabilistic algorithm for factorizing polynomials over arbitrary finite fields.5.6. Manders. For an anthology of probabilistic algorithms.6. A Monte Carlo algorithm is given in Schwartz (1978) to decide whether a multivariate polynomial over an infinite domain is identically zero and to test whether two such polynomials are identical.17 are given in Freivalds (1979). Kranakis (1986). An efficient probabilistic algorithm is given in Karp and Rabin (1987) to solve the stringsearching problem discussed in Section 7. Given the cartesian coordinates of points in the plane. The test of Solovay and Strassen (1977) was discovered independently.10 is given in Goldwasser and Kilian (1986) and Adleman and Huang (1987). A theoretical solution to Problem 8. Our favourite unbiased Monte Carlo algorithm for a decision problem.11. 8 Problem 8. comes from Itai and Rodeh (1981).6.4) and the solution to Problem 8. The algorithm for electing a leader in a network. For efficiency considerations in factorization algorithms. and Rumely (1983).17 is given by the algorithm of Shanks (1972) and Adleman. and Pomerance (1988). Cre peau.2.5. The Monte Carlo algorithm to verify matrix multiplication (Section 8. The solution to Problem 8.

(You would thus use an algorism.1 illustrates this principle. this idea is only of interest when tables of logarithms are . x 2 .1. This allows a multiplication to be replaced by the calculation of a logarithm. In this case.--- Example 9.1. Since the computation of a and a1 would take more time than the original multiplication. The most important transformation used before the advent of computers resulted from the invention of logarithms by Napier in 1614. y) =x +y. to multiply two large numbers given in Roman figures. Let f :Dt -4 D be a function to be calculated.1 INTRODUCTION It is sometimes useful to reformulate a problem before trying to solve it. an addition... for all x 1 . Such a transformation is of interest if g can be calculated in the transformed domain more rapidly than f can be calculated in the original domain. and an exponentiation. xt) = a' ' (g (a(x 1). let D be the domain of objects to be manipulated in order to solve a given problem. a(xt ))) xt in the domain D .1. If you were asked. Kepler found this discovery so useful that he dedicated his Tabulae Rudolphinae to Napier. you would probably begin by translating them into Arabic notation. an invertible transformation function a : D -+ R and a transformed function g : R' -> R such that PX1 .--C02).9 Transformations of the Domain 9. v) = u x v . x22 . Figure 9. a(u) = In u and g (x . . for example. An algebraic transformation consists of a transformed domain R. and if the transformations a and 6-1 can also be computed efficiently. R = R. D = IN+ or R+. with this word's original meaning!) More generally. f (u . .

Most computers that handle numerical data read these data Example 9. Using the transformed representation. 3. 7.2.4. taking p = (1. Such tables. 2. this does not allow us to recover the coefficients of r (x) because a polynomial of degree 6 is not uniquely defined by its value at only four points. 53. Transformation of the domain. 238).2. 106). We would have needed to use seven points from the outset. The polynomials in our example would be represented by p = (1. 1. 34) and q = ( . In this example. 2. f o r instance. 3. 22.3. Example 9. 34. and print the results in decimal but carry out their computations in binary. However. 9 I D Dt o` Rt Figure 9. .6.Transformations of the Domain 278 Chap. 7). 1.d.2. computed beforehand.1. . 246. . 1. calculating the transformed function (carrying out the pointwise multiplication) is much quicker than straightforwardly calculating the original function . but its meaning has changed. These polynomials are represented by their coefficients.2. Example 9. 0.2. Thus we obtain r = (.1.. it takes a time in O(d 2) if the scalar operations are taken to be elementary.2. The original domain of the problem is therefore Zd + 1. 2.1. thus furnish a historical example of preconditioning (Chapter 7).. d. The transformed Zd+I. calculated once and for all.109. p(x) = 3x3-5x2-x +I and q(x) =x3-4x2+6x -2. The naive algorithm for multiplying these polynomials resembles the classic algorithm for integer multiplication . An alternative way to represent the same polynomials is to give their values at d+ 1 distinct points. . The new representation suffices to domain is still define the original polynomials because there is exactly one polynomial of degree at most 3 that passes through any four given points. 463) and q = (.2.. we can carry out the multiplication in a time in O (d) since r (i) is calculated straightforwardly as p (i) q (i) for 0 5 i <. where d is the degree of the polynomials involved. for example.1. You want to perform a symbolic multiplication of two poly- nomials. if the computation of r were to be carried out correctly using this representation. Let r(x) = 3x6-17x5+37x4-31x3+8x -2 be the product of p(x) and q(x).1. It is often useful to transform between Cartesian and polar coordinates.

a naive implementation of Lagrange's algorithm to carry out the final interpolation would take a time in S2(d 3). C [0. a`12 = (m2)t/2 = m' = wn/2 = -1.1. co) : array [0. it seems that the computation of the values p (i) and q (i) for 0 <_ i <. an_3. . As in Example 9. The t-tuples b and c defined by b = (ao. so it is legitimate to talk about Fa.... Consider an n-tuple a = (ao. and w= (1+i )/J is a possible value in the field of complex numbers. this is only useful if the transformation function (evaluation of the polynomials) and its inverse (interpolation) can be calculated rapidly. We denote by co some constant such that 012 =-1. function FFT (a [0.a [0] else t . where a=w2. Suppose that n > 2 and set t = n / 2. However. If n = 8. B. this is not in fact the case. . This algorithm is vitally important for a variety of applications. Let n > I be a power of 2. it appears at first glance that the number of scalar operations needed to calculate this transform is in S2(n2). pa (wn .Sec. The discrete Fourier transform of a with respect to m is the n-tuple Fu. . 9.2d must take a time in S2(d2). a 2 . provided that the points at which we evaluate the polynomials are chosen judiciously. t -1 ] ( intermediate arrays ) ( creation of the sub-instances ) fori -0tot-1 dob[i] 4-a[2i] c[i] E-a[2i+1] . 9. which appears to remove any interest this approach may have had. an-4. n -1 ] (n is a power of2andwn'2=-1 ) array A [0. However. an _ 1).6). c. a' = 1 and co' _ -1..1)). Clearly.. .w' pC (c'). This defines in the natural way a polynomial pa(x)=an_1Xn-I+an_2Xn-2+ "' +alx+ao of degree less than n. a i . n -1 ]. ... so that pa (w`+') = Pb ((X') .7.. . . then w= 4 is a possible value if we are using arithmetic modulo 257. a3. In particular.4. hence a`+' = a` and w`+` d.2 The Discrete Fourier Transform 279 (performing a symbolic multiplication of two polynomials given by their coefficients). We shall see in Section 9. thanks to an algorithm known as the Fast Fourier Transform (FFT). pa (w).4 that this is not in fact the case. Furthermore. The Fourier transform Fu. .n / 2 arrays b. At first glance. Worse still. n -1] [ the answer is computed in this array ) if n = 1 then A [0] <.. an-2) and c = (al ...or more generally in any commutative ring.(b) and Fa(c ). particularly in the area of signal processing (Section 1.2 THE DISCRETE FOURIER TRANSFORM For the rest of this chapter all the arithmetic operations involved are carried out either modulo some integer m to be determined or in the field of complex numbers .(a) = (pa (1). Pa (w')=pb((X')+w'pC((X'). They are assumed to be executed at unit cost unless it is explicitly stated otherwise. pa ((02). for example. an_j) are such that pa(X)=ph(x2)+xpc(x2)..(a) is calculated using the divide-and-conquer technique.

2. let q (x) = (x-x1 )(x-x2) .3 THE INVERSE TRANSFORM Despite its importance in signal processing. (x x1) be a polynomial of degree t such that x1 are arbitrary distinct values. 75). Despite the conceptual simplicity of this recursive algorithm. yield B = F 16(b) = (38.-1 (mod 257). 9) and C = F 16(c) = (217. A[ 11 E-.9+7w3 = 200 32 .127. and an iterative algorithm. 240.1. which use t = 4 and w2 = = 75 The final result is therefore A = (255. 200. In particular. 170. 85.1. w. ** Problem 9. 78..0. the remainder of the symbolic division of p (x) by the monomial (x-x.] 9. 7). Show that p (xi) = r (x.I for i F.37.aC [i] a F-ato return A Show that the execution time of this algorithm Problem 9.) is the constant polynomial whose value is p (x1). 22. We combine these results to obtain A. .217 = 78 A [5] F 170-43w = 255 A [2] A [6] A [0] 38 + 217 = 255 32 + 22002 = 127 A [3] F.240. where x 1 .8.3) and c = (8. 226.(o 2) { Fourier transform computation of the original instance } af.2. 43.w2) C -FFT(c.0 to t-1 do (a= w' I A[i] F-B[i]+aC[i] A [t+i ] E. & -1 in a time in 0 (n log n). Using the fast Fourier transform allows us to evaluate the polynomials p (x) and q (x) at the points 1. Our principal aim is to save the idea proposed in Example 9. where w = 4.38 . 194. .0. This can be done because 44 =.-- Let r (x) be the remainder of the symbolic division of p (x) by q (x)..2.0).. 32.Transformations of the Domain 280 Chap. Let n = 8 and a = (255. First a is split into b = (255.37.t. Example 9. 0).170+43co = 85 A [4] <. cue. our concern here is to use the discrete Fourier transform as a tool for transforming the domain of a problem. Let us calculate FW(a) in arithmetic modulo m = 257. 9 { recursive Fourier transform computation of the sub-instances } B t-FFT(b. is in O(n log n).4.) for 15 i <. it is preferable in practice to use an iterative version. where n is a power of 2 greater than the degree of the product .2. Give such [Hint: Let p (x) be an arbitrary polynomial. 255.B [i] .3. The recursive calls. q(x1) = q (x 2) = = q (xr) = 0. X2.22to2 = 194 A [7] .

1. and use the existence of n-1. and let w be an element of the ring such that con/2 = . As a slight abuse of notation. v. Notice that (w 1)n i2 = -1. wn -' are all distinct . iii. The pointwise multiplication can be carried out in a time in O (n). Assuming this n is not zero. we shall denote it by col. E or" = 0 for every I <. With this in view we say that w is a principal n th root of unity if it fulfills three conditions : 1.i. * Problem 9. iii. Let n > 1 be a power of 2.1. Let s =2k-1.1. Obvious. Theorem 9. .i < j < n and let p = j .1.3. Problem 9.3.2. is a principal n th root of unity in the field of . w' . Show that co-" = .1. Assume co' = w j for 0 <. w 1 is also a principal n th root of unity. wn = 1. Then wP = 1 and 1 <. j=0 When n is a power of 2. iv.p < n . Use p = n/2 in condition (3). To show condition (3).they are called the n th roots of unity. Prove that e 2i 1/. iv. ii.Sec. wn 12 = -1 is a consequence of w being a principal n th root of unity. it turns out that these conditions are automatically satisfied whenever con/2 = .3. let us use " n " also to denote the element in our ring obtained by adding n ones together (the "1" from our ring). wn -1 is the multiplicative inverse of co in the ring in question . Thus we have in fact to invert the Fourier transformation. Prove Theorem 9.p < n . each summing to zero. To obtain the final result.1. w is a principal n th root of unity. w# 1 2.3 The Inverse Transform 281 polynomial. v. 1 = w0 .3. and n-I 3. complex numbers. Hints : i. Conditions (1) and (2) are obviously fulfilled. and v is odd.(dj+s )P for every integer j. 9. we still have to interpolate the unique polynomial r (x) of degree less than n that passes through these points. Assuming the existence of a multiplicative inverse n-1 for n in our ring. Conclude by splitting y o' wjP into 2n sub sums of 2s elements. let n = 2k and decompose p = 2n v where u and v are integers. w2 .. as we have assumed already. ii. More generally we use W-' to denote (0-1)' for any integer i. Consider any commutative ring. Use condition (3) and n * 0 to obtain a contradiction. Then i.

andsoC11=n-1 nxn-1=1. More generally.j <. Let C = AB. n -'Pa ((02 ).2".j = n-1 Y. then co('-j)k=co0=1.2 justifies the following definition. n . set d =i-j. Prove that w is a principal n th root of unity in arithmetic modulo m.3. assume that w is a principal n th root of unity and that n-1 exists in the algebraic structure considered. let p = i -j.3.(m -1)/n .b < m . Proof.I iii. If i > j. Let n and w be positive powers of 2.3. Prove further that n-1 exists modulo m.d < m and ab = d (mod m). since 1 <. Prove that 0 <. If i < j.n-I ik -o w(i-j)k There are three cases to consider. Cu = Jk _o A.3. .p < n . When m is of the form 2" + 1. Theorem 9.. show how x mod m can be calculated in a time in p 0 (u) when m = 2" + 1.1) be an n-tuple. Now C. an .' (a) = aB = (n -'pa (1). By definition.j =n(o)1'P = 0 because co 1 is also a k=0 principal n th root of unity by Theorem 9..5.1(iii).k Bki . let p = j . Now C.i... . The main theorem asserts that A has an inverse. Let A and B be the matrices just defined.2.3. by showing that n-1 = m . where 0 <. and let m = w"12 + 1.(a)) = Fw(FW' (a)) = a for every a.j = n'o'.(a) = aA. n -1pa (W-').i < 2' and 0 <. provided that . Then AB = I. Let c = ab be the ordinary product obtained from a multiplication.. The inverse Fourier transform of a with respect to w is the n-tuple F. provided that n-1 is either known or easily calculable.Ifi=j.Transformations of the Domain 282 Chap. . Decompose c into two blocks of u bits : c = 2" j + i.3.3. n i. Problem 9. Let a = (a o.i < n and 0 <. From now on.3. we wish to obtain their product modulo m. Theorem 9. namely the matrix B defined by B. a I . (The only possibility for j = 2" is when a = b = m -1. 9 Problem 9. Let a and b be two integers such that 0 <..(22u + 2u) < x < 22u +2u+'. and otherwise set d = m + i .j = di for 0 <.j < n . . the n x n identity matrix. This matrix A provides us with another equivalent definition of the discrete Fourier transform : F". multiplications modulo m can be carried out on a binary computer without using division operations. OP = 0 by property (3) of a principal k=0 n th root of unity.3. as in Problem 9.) If i > j. n -'Pa (w (n -1))) Problem 9. The inverse Fourier transform can be calculated efficiently by the following algorithm. k=0 n-1 H. .4. Let A be the n x n matrix defined by A.j . Prove that Fw' (Fu.a < m and 0 <.

133) and C =FFT(c.1. n -1 ] F <-FFT(a.B ± C T Q <. thanks to the particular form chosen for m.. Let n = 8 and a = (255. The calls with yield B =FFT(b. n . Secondly. There remains the mul- tiplication by n-' = m -(m -1)/n = 225 (Problem 9. Combined.143. denoted by y. The heart of the algorithm consists of executing instructions of the form A .12.4. 0).3.w" -' + w" '2.64.3 The Inverse Transform 283 function FFTinv (a [0.4) that reductions modulo m can be carried out in a time in O (log m).(B ± C T (3) mod m. First.. 200.3. w is indeed a principal n th root of unity. For the rest of this section we no longer suppose that arithmetic operations can be performed at unit cost : the addition of two numbers of size 1 takes a time in O (1).1) Ig w.4.65.226. n -1 ] { n is a power of 2 and w" = 1 } array F [0. First we calculate FFT (a . the complete computation of the Fourier .B < m and 0 <. By Problem 9. the recursive calls are made with 2y rather than w2 as the second argument. 24. instead of giving w as the second argument. we supply the base 2 logarithm of w..0. If the Fourier transform is calculated in the field of complex numbers (Problem 9. a is decomposed into b=(255.3. even when the recursive calls are taken into account.0). x x 2Y.3. On the other hand.3.75). if the transform is calculated modulo m (Problem 9.1.3).B[i]-C[i]TP RF-R+y.2. 127.C < m .31).3.w2)_(101. All the arithmetic is carried out modulo m = w"'2+ I using Problem 9. these results give A = (241. 9. where x Ty denotes the value of x shifted left y binary places.11. For this it is convenient to modify the algorithm slightly.w" -' <.240.8. 194.3.255. 121. The value of the shift never exceeds (z . rounding errors may occur on the computer. which is consistent with Example 9. 255. The final result is thus F = (255.2). that is. Furthermore.Sec.0 for i f-0tot-1 do I R= iy} A [i] <-B[i]+C[i]TR A[t+i]<-. 78. of ' ).3. The final loop becomes R <-. which means that it can be reduced modulo m in a time in O (log m) = O (n log (o) by Problem 9.w"-') for i <-Oton-1 do F[i]Fn-'F[i] return F Example 9. where w = 4.3).3. it may be necessary to handle large integers.194) 0-2 = 241 recursive and c = (85.127. Consequently . 85.9. where 0 <.78. 75).3. Since the number of operations of this type is in 0 (n log n).0.221.w2)_(140. the fact that w is a power of 2 means that multiplications in the FFT algorithm can be replaced by shifts. To do this. We already know (Problem 9.37. co) : array [0. Let us calculate Fj' (a) in arithmetic modulo m = 257.200. 39. where w' = co7 = 193.

.. B = F". .5. and C = Fu..6.1. 0 .(b). bt.) 9. Show that the inverse transform modulo m = co"t2+ I can also be computed in a time in 0 (n 2 log n log co).O. Therefore C is the pointwise product of A and B. since -n-1 = (o"t2/n is a power of 2. . Otherwise a direct call on the new FFT with y = (n .3.4 can no longer be applied. Notice that this reasoning made no use of the classic unique interpolation theorem.(c).) Problem 9. and this is fortunate because unique interpolation does not always hold when the arithmetic is performed in a ring rather than in a field. = r (co') = p (w`) q (co`) = 0 .0. C.4 SYMBOLIC OPERATIONS ON POLYNOMIALS We now have available the tools that are necessary to finish Example 9. . . cd . 0). the coefficients of the product polynomial r (x) are given by the first d + 1 entries in c = F.1. we need to be able to calculate efficiently a principal n th root of unity.3. b. for instance. . which means that Problem 9. By Problem 9.) Problem 9. (Consider. (Padding c with zeros is unnecessary if d -1 is a power of 2.(a). 0).284 Chap. if m is sufficiently small that arithmetic modulo m can be considered to be elementary. Let p(x) =a.. Let n be the smallest power of 2 greater than d. Both of these degree 1 polynomials evaluate to 1 and 7 at the points 0 and 3. B. We want to calculate symbolically the product polynomial r(x)=cdxd+cd_IXd-1+ +clx+co=p(x)q(x) of degree d = s + t. and let co be a principal n th root of unity. respectively. 1(Fw(a) x Fjb)). respectively. 9 Transformations of the Domain transform modulo m can be carried out in a time in 0 (n 2 log n log co).. ..2. It is somewhat surprising that efficient symbolic multiplication of polynomials with integer . Show that it can be used to multiply two polynomials whose product is of degree d with a number of scalar operations in 0 (d log d). respectively. the algorithm takes a time in 0 (n log n).a1. (From a practical point of view. . Putting all this together.4.xs +aS_. The easiest approach is to use the complex number field and Problem 9.. . .. provided that a principal n th root of unity and the multiplicative inverse of n are both easily obtainable.xs-1+ +alx+ao and q(x) = btxt+bt_lxt-1+ +b1x+bo be two polynomials of degrees s and t. causes shifts that can go up to (2 -1) (n . In order to implement this algorithm for the symbolic multiplication of polynomials. (The algorithm FFTinv has to be modified. corresponding to the use of w = co' as a principal root of unity. the final multiplication by n-1 can profitably be replaced by a multiplication by -n-1 followed by a change of sign.) Let A = Fu.0.(x) = 2x + 1 and p2(x) = 5x + 1 in the ring of integers modulo 9. b =(bo.. where n is the smallest power of 2 greater than d. By definition of the Fourier transform..3.4.. 1 1 Similarly. Give explicitly the algorithm we have just sketched. Let a. c = Fw 1(C ).. c 1.1) lg co. and c = (c0 . as.3.1) lg co. p. and c be the n-tuples defined by a =(ao. 0).0.

. -31. thus it suffices to calculate them modulo in = 257. The pointwise product of these two transforms. 82.1. w.0).3.1.(a) = (255. respectively.4. the algorithm multiplies p (x) and q (x) symbolically in a time in 0 (d log d). (Use Problems 9. The final answer is therefore r(x) = 3x6-17x5+37x4-31x3+8x -2 . the integers 255. Since the product is of degree 6.3.(c) = C is c = (255. a = 5.251. so no coefficient of r(x) can exceed 120 in absolute value. is C =(255. 226. . -1. Prove that no coefficient of the product polynomial p(x)q(x) exceeds ab (u+l ) in absolute value.-4. s+t ] that carries out symbolic multiplication of polynomials with integer coefficients. 0).4) We wish to multiply sym- bolically the polynomials p(x) = 3x3-5x2-x +l and q(x) =x3-4x2+6x -2. respectively. 3. and -17. 37.2. 0.4.1. it suffices to take n = 8. Two applications of the algorithm FFT yield A = F.194.247. This may require the use of multipleprecision arithmetic. (In Example 9. your algorithm should determine suitable values for n.Sec. Problem 9.6. By Example 9.0. it becomes necessary to take precautions against the possibility of rounding errors on the computer. 3.) Q The analysis of the algorithm of Problem 9.29. Generalize this idea : give explicitly an algorithm mul (a [0 . all the coefficients of the product polynomial r (x) = p (x) q (x) lie between -120 and 120. 179.O.3.5. b [0 . 244.4.2 for this.3. If an exact answer is required. For this reason it may be more attractive to carry out the arithmetic modulo a sufficiently large number (Problem 9. 8. 240. 0. and n -' = 225. 188) . 103.(b) _ (1. the vector c such that F. If the latter are sufficiently small that it is reasonable to consider operations modulo m to be elementary..127. 0.3 and 9. Let a and b be the maxima of the absolute values of the coefficients of p (x) and q (x).4.2). w = 4 is a principal n th root of unity in arithmetic modulo 257. Let u be the maximum of the degrees of the two polynomials.200. Let a = (1. Problem 9. Among other things. Since all the coefficients of r (x) lie between -120 and 120. and m. 226.3 to obtain a principal n th root of unity.2. By Problem 9.0.4. s ].75). 22. and 240 correspond to -2.85.4 285 Symbolic Operations on Polynomials coefficients should require operations on complex numbers. b = 6.133) and B = F. (Continuation of Example 9. In this case a more thorough analysis of the possible build-up of rounding errors is needed..4.) Example 9. 193.3. Let p (x) and q (x) be two polynomials with integer coefficients.255. t ]) : array[0 .4.78. 0) and b =(-2.199.3 depends on the degrees s and t of the polynomials to be multiplied and on the size of their coefficients. and u = 3. and to use Problem 9. By Problem 9.109. 0. 9.4. still working modulo 257..

59). . The naive algorithm would have taken a time in 0 (st ). On the other hand. Suppose for simplicity that n is a power of 2 (nonsignificant leading zeros are added at the left of the operands if necessary).4. . Let x 1 .11. Let p (x) be a polynomial of degree n . (Hint : and let yn be n values.5. the naive algorithm takes a time in 0 (s t M (1)). where M (1) is the time required to multiply two integers of size 1. y 1. = p (x1) for 1 <_ i <_ n. where 1 is the size of the largest coefficient in the polynomials to be multiplied. 9.x a time be n in 0 (n log2n). Let x 1 . t )) in absolute value. distinct points. then the symbolic product of these polynomials is obtained using the discrete Fourier transform. Give an efficient algorithm to calculate the coefficients of the unique monic polynomial p (x) of degree n such that p (xi) = 0 for every 1 <_ i <_ n. not necessarily distinct. x 22 . Your algorithm should take the hint to Problem 9. and the intermediate pointwise multiplication of the transforms takes a time in 0 (d M (d log co)). (Hint : see Problem 4. The original integer domain is first transformed into the domain of polynomials represented by their coefficients .2.2. and 4.--. . (The polynomial is monic if the coefficient of x" is 1.Transformations of the Domain 286 Chap.2. if arithmetic can be carried out on integers of size l at unit cost.. if we are obliged to use multiple-precision arithmetic.4.5 MULTIPLICATION OF LARGE INTEGERS We return once more to the problem of multiplying large integers (Sections 1.. the algorithm that uses 0) = e 2fi /" can multiply approximately the two polynomials in a time in 0 (d log d). x 2. x2 2 .8).5). Y2.7).x be n distinct points.4. In every case. Your algorithm should take a time in 0 (n log2n ).) Your algorithm should take a time in 0 (n log2n) provided all the necessary operations are taken to be elementary. The total time is therefore in 0 (d M (d log co)). the initial computation of the Fourier transforms and the final calculation of the inverse take a time in 0 (d 2 log d log (o).1.7. Give an efficient algorithm to calculate the coefficients of the unique polynomial p (x) of degree less than n such that p (xi) = yi for every 1 <_ i <_ n. It is possible for this time to be in 0 (st) in practice.4. Give an efficient algorithm to calculate each y. . x be n distinct points. the first term in this analysis can be neglected. and let x 1 . The naive algorithm is therefore preferable to the "fast" algorithm if d is very large and I is reasonably small.) * Problem 9. 9 where d = s + t.2 is relevant here too.7. Let a and b be two n-bit integers whose product we wish to calculate.6. The classic algorithm takes a time in f (n2).--. We can do better than this thanks to a double transformation of the domain. 1.) ** Problem 9. whereas the algorithm using divide-and-conquer requires only a time in 0 (n 1. Problem 9. Since M (1) n O(1 log l log log 1) with the best-known algorithm for integer multiplication (Section 9.) By comparison. (Remember that n is the smallest power of 2 greater than d. or even in 0 (n") for any a > 1 (Problem 4. where Co = 2 suffices if none of the coefficients of the polynomials to be multiplied exceeds 2' 14/ 2(1 + max(s .

so that pa (10) = a. The algorithm is recursive because one of the stages in the symbolic multiplication of polynomials consists of a pointwise multiplication of Fourier transforms. in order that using the discrete Fourier transform should be attractive.519. Clearly. (The second term is added in case certain operands are exactly w" 2. This polynomial is thus of degree less than n / 2.7 consists of representing each integer by a polynomial of degree 1 whose two coefficients lie between 0 and 2'12 . and pa (4) = a. The original multiplication of two integers of size n therefore requires 2n multiplications of slightly larger integers ! To correct this.1. since this integer of length 1 + i n Ig (o has to be treated specially. the algorithm of Section 4. even if we take co = 2. As before. pa (2) = a for every integer a. we need only calculate symbolically the polynomial r (x) = p (x) q (x) using the fast Fourier transform (Section 9.) Taking into account also the time spent in computing the two initial Fourier transforms and the final inverse transform.3. where n is a power of 2.4). .1. and then evaluate r (2). and the final answer is obtained by evaluating r (4). where d = 2n is a power of 2 greater than the degree of the product polynomial. This can be carried out in a time in n M (2 n Ig w) + 0 (n 2). these integers are of silt. However. For the purpose of this example only.5. 0 The recursive nature of this algorithm obliges us to refine the analysis of symbolic multiplication of polynomials given following Problem 9. Let M (n) be the time required to multiply two n-bit integers. To make the illustration simpler. p53(x) = X5+X4+X2 + I because 53 in binary is 00110101. here in decimal rather than in binary. we perform the computation Example 9. Unfortunately. 9.Sec. The central step in the symbolic multiplication of Pa (x) and Pb (x) therefore only requires n multiplications of integers less than m = w"i2+ 1. we must reduce the degree of the polynomials used to represent the integers to be multiplied.4.5 Multiplication of Large Integers 287 We denote by pa (x) the polynomial of degree less than n whose coefficients are given by the successive bits of the integer a. For instance. The symbolic product is r (x) = pa (X)pb (x) = 2x6+3x5 + 18x4+38x3 + 15x2+9x +5 and we obtain the desired product ab as r (10) = 2. 1 + d Ig w = n + 1. The central step in the symbolic multiplication of two polynomials of degree less than n consists of d multiplications f integers less than wd12+ 1. the polynomial r (x) = Pa (x)pb (x) is calculated symbolically using the Fourier transform. the symbolic multiplication takes a time in n M (-'n Ig (o) + 0 (n 2log n logo)). Let a = 2301 and b = 1095. To obtain the product of the integers a and b. As an extreme case.595. This time the degree of the product polynomial r (x) is less than n. For instance. Thus pa (x) = 2x3 +3 X2 + 1 and Pb (x) = x3+9x +5. even if this means increasing the size of their coefficients. its coefficients lie between 0 and 3. suppose we redefine pa (x) to be the polynomial whose coefficients are given by the successive figures of the representation of a in base 4. let pa (x) therefore denote the polynomial whose coefficients are given by the successive digits of a. the polynomials considered must have a sufficiently high degree.

Let d = 2k. denote by pa (x) the polynomial of degree less than k whose coefficients correspond to the k blocks of I successive bits in the binary representation of a.5.288 Transformations of the Domain Chap. This time. which takes a negligible time in 0 (n log n).260. Since the coefficients of the polynomials pa (x) and pb (x) lie between 0 and 3. The symbolic product is Example 9.260. The first of these polynomials.1. whatever The preceding problem shows that the modified algorithm is still bad news.2).2. namely the evaluation of r (4). To calculate the product of the integers a and b. and since these polynomials are of degree less than n / 2. and k = 4. As the computations are carried out modulo m = o)'12 + 1. comes from the decomposition into four blocks of the binary representation of a 0010 0110 1001 1101. r(x) =pa(x)pb(x)=10x6+36x5+63x4+116x3+111x2+108x + 156 and the final evaluation yields r (16) = 210. The choice to = 2 is adequate provided n >: 16. even if we do not take into account the time required to compute the Fourier transforms ! This is explained by the fact that we used the "fast" algorithm for multiplying two polynomials in exactly the circumstances when it should be avoided : the polynomials are of high degree and their coefficients are small. We form the polynomials pa (x) = 2x3+6x2+9x + 13 and Pb (x) = 5x3 + 3x2 + 12. the largest coefficient possible in r (x) is 9n/2 (Problem 9.3. we must choose a principal n th root of unity tv. The symbolic computation of r (x) therefore takes a time in n M (n / 2) + 0 (n 2 log n).5. Let k = n /I. t(1) = 1 n =2 k . for instance. it is thus sufficient that 9n / 2 < (0/2 . The last stage of the multiplication of a and b. Problem 9. a power of 2 greater than the degree of the product polynomial r (x). Show that t(n) 0 (nk ). consists of n shifts and n additions of integers whose size is not more than lg (9n / 2).155. Thus we have that pa (21) = a. This time we need to choose a principal d th root of unity t. we must lower the degree of the polynomials still further. 9 Before analysing the new multiplication algorithm. and then evaluate r (21). k E lN+ . Note that 1 and k are powers of 2.4. We thus obtain the asymptotic recurrence M (n) e n M (n / 2) + 0 (n 2 log n). that is. Let a = 9885 and b = 21.1. depending on whether lg n is even or odd. When n is a power of 2. / = 4.100 = 9885 x 21. 1! Ign1 Let l = 2 2 . so that n = 16. Since the coefficients of the polynomials pa (x) and Pb (x) lie between 0 and 21 . Consider the recurrence t (n) = nt (n / 2). using Fourier transforms. To correct this. prove that t(n) = the value of the constant k. we need only calculate symbolically the polynomial r (x) = Pa (X)Pb (x). and the degree of these poly- . l = I or l = 2n .

yt. this can easily be carried out in a time in 0 (d 2 log w). which takes a time in d M (2 d lg w) + O (d 2 log d log w). For the third case prove by constructive induction that t (n) 5 S [(lg n)1gT . provided n is sufficiently large. which is negligible.(21 + Ig k)/(dl 2). we obtain Ig w ? 4 + (Ig' )/ . It suffices therefore that k 221 < m = wd 12 + 1.Sec. this algorithm can multiply two n-bit integers in a time in M (n) = nt (n) E 0 (n (log n) n)(197). Simi- larly. Problem 9. hence. d = 2J and w = 8.p Ig n . [Hints: For the second case use the fact that Iglg(1J) (Iglgn) . and w = 32 is sufficient when Ig n is odd.5. w = 8 suffices to guarantee that the computation of the coefficients of r (x) will be correct when Ig n is even. and let t(n) be a function satisfying the asymptotic recurrence t(n) E yt(O (' )) + 0 (log n). and t(n)E5t(52 )+O(logn) when n is odd. This algorithm therefore outperforms all those we have seen previously. This is possible provided we increase slightly the size of the coefficients of the polynomials used in order to decrease their degree.3. Consequently. Problem 9. which gives M (n) e 24W M (3') + O (n log n). Prove that 0 (n (log n)2. where . * Problem 9. for some constants S. and p that you must determine and for n sufficiently large.5. that is Ig co >.59 ).)/ = 2 + (Ig J )/W W. When n is odd. when n is odd. By Problem 9. When n is even. In the case when Ig n is even I = k = dl 2 = I and Ig w >. for all real constants Q ? 1 and y > 2.11 . we split the n-bit integers to be multiplied into k blocks of I bits. The equations obtained earlier for M (n) lead to t(n)E6t(3N/n-)+O(logn) when lgn is even. Also use the fact that (lg1')Ig1< T(lgn)IST+2lgylg1(lg1`r)(IgY)-1 provided n y21g R. d = 2n and w = 32.2. Is this algorithm optimal ? To go still faster using a similar approach.59) c n°C whatever the value of the real constant a > 1. As far as the final evaluation of r(21) is concerned.] Let t(n) = M (n)/n .lg(5/3) provided that n >_ (310 for every real constant 0 > 1. Let y > 0 be a real constant.(2' + IgFn. More precisely. Consequently. the largest coefficient possible in r (x) is k (21 -1)2.5.2. t (n) E O ((log n)1 g 6) c 0 ((log n)2-59). Prove that t(n)E if y < 2 O (log n) O (log n log log n) if y= 2 O ((log n)19 Y) if y > 2 . which arises here as the max- imum of 2 x 3 and ' x i .2 suggests that we should reduce the constant y = 6.5. 9.5 Multiplication of Large Integers 289 nomials is less than k. The multiplication of two n-bit integers is thus carried out using a symbolic multiplication of two polynomials. M (n) E 2n M (2 I) + 0 (n log n).

To go faster still. where a=2+Ig(1+2-1-21) <2+2-1-2'/ln2 can be reduced arbitrarily close to 2. 9. In view of the great practical importance of Fourier transforms. they used a variant involving operations in the complex number field. For a more complete account of the history of the fast Fourier transform. This approach allowed Schonhage and Strassen to obtain y = 2. The outcome is an algorithm that is capable of multiplying two n-bit integers in a time in 0 (n log2n). Detailed analysis shows that recursive calls on integers of size (2' + 1 + 2-' if Ig n is this gives rise to 21-' even. it is then possible to obtain the coefficients of r (x) by the Chinese remainder theorem. ignl and k = n/I for an arbitrary constant i >_ 0. Lewis. we have to redefine the notion of the "product" of two polynomials so as to avoid doubling the degree of the result obtained. The corresponding y is thus 1 y=max(21-'(2'+1+2-'). It also allows the computation to be carried out directly in decimal. Needless to say. Because 221 + 1 and k are coprime. The complexity of this algorithm is such that it is of theoretical interest only. . These authors mention that the source of their method goes back to Runge and Konig (1924). 9 I _ 2r+f. and finally to evaluate r(2 21 ). an algorithm that takes a time in 0 (n log n log log n). that is. An efficient implementation and numerous applications are suggested in Gentleman and Sande (1966). and Welch (1967). An integer multiplication algorithm based on the fast Fourier transform has been used by the Japanese to calculate it to 10 million decimal places.Transformations of the Domain 290 Chap. Even more decimals of it have been calculated since. 2-' f(2'++2-i`1)J)=4+21 2r The algorithm obtained takes a time in O(n(log n)a). at least asymptotically. Rather than resorting to modulo computations in a finite ring.6 REFERENCES AND FURTHER READING The first published algorithm for calculating discrete Fourier transforms in a time in O (n log n) is by Danielson and Lanczos (1942). read Cooley.7). We mention simply that it is possible to obtain y=4 by calculating the coefficients of the polynomial r (x) modulo 221 + 1 (using the Fourier transform and proceeding recursively) and modulo k (using the algorithm for integer multiplication of Section 4. and 2-' 2n recursive calls on integers of size (2' + + 2-' if Ig n is odd provided n is sufficiently large. Their approach requires care to avoid problems due to rounding errors but gives rise to a simpler algorithm. increasing the parameter i reduces the exponent a at the expense of increasing the hidden constant in the asymptotic notation. This is still not optimal. thus avoiding a costly conversion when printing out the result. it is astonishing that the existence of a fast algorithm remained almost entirely unknown until its rediscovery nearly a quarter of a century later by Cooley and Tukey (1965). but the algorithms that are even faster are too complicated to be described in detail here.

although the solution given there to Problem 9. but does not explain the algorithm used. read Gleick (1987).6 is given in Aho.3.4. Hopcroft. The algorithm used by the Japanese to com- pute it to 10 million decimal places is described in Kanada. The empire struck back shortly thereafter when the Japanese computed 134 million decimals. The book by Brigham (1974) is also worth mentioning. Tamura. The algorithm that is able to multiply two integers of size n in a time in O (n log2n) is attributed to Karp and described in Borodin and Munro (1975).6 References and Further Reading 291 and Rabiner and Gold (1974).3. and interpolation can be found in Borodin and Munro (1971. Further ideas concerning the symbolic manipulation of polynomials. which is the world record at the time of this writing. Also read Turk (1982). and Turk (1982). Monet.2 is described in several references. Hopcroft. The nonrecursive algorithm suggested in Problem 9. The details of the algorithm by Schonhage and Strassen (1971) are spelled out in Brassard. and Ullman (1974). 9. Cray Research (1986) mentions an even more precise computation of the decimals of it.Sec. The second edition of Knuth (1969) includes a survey of algorithms for integer multiplication.4. Yoshino and Ushiro (1986). Pollard (1971) studies the computation of Fourier transforms in a finite field. and Ullman (1974).3 is unnecessarily complicated in the light of Theorem 9.1. The solution to Problems 9. A practical algorithm for the rapid multiplication of integers with up to 10 thousand decimal digits is given in Pollard (1971).2. for example Aho. . evalua- tion. 1975). And the saga goes ever on. Horowitz and Sahni (1978). and Zuffellato (1986).5 and 9.

a field of study that runs in parallel with algorithmics. Using complexity. since then we know that we have found the most efficient algorithm possible (except perhaps for changes in the hidden multiplicative constant).10 Introduction to Complexity Up to now. each more efficient than its predecessors. Using algorithmics.1 DECISION TREES This technique applies to a variety of problems that use the concept of comparisons between elements. we try to find a function g(n) as large as possible and to prove that any algorithm that is capable of solving our problem correctly on all of its instances must necessarily take a time in 92(g(n)). to solve some given problem. We illustrate it with the sorting problem. this does not happen often. considers globally the class of all algorithms that are able to solve a given problem. ignoring those that may be made to control the loops in our program. 10. we can prove. Computational complexity. In this case we say that the complexity of the problem is known exactly . Thus we ask the following question : what is the minimum number of comparisons that are necessary to sort n elements? For simplicity we count only comparisons between the elements to be sorted. unfortunately. we have been interested in the systematic development and analysis of specific algorithms. Consider first the following algorithm. Our satisfaction is complete when f (n) e O(g(n)). that a certain problem can be solved in a time in 0 (f (n)) for some function f (n) that we aim to reduce as much as possible. In this chapter we introduce only a few of the principal techniques and concepts used in the study of computational complexity. 292 . by giving an explicit algorithm.

Problem 10.Sec. which is the greater. it becomes impractical. 5.. only in rare applications will these algorithms prove preferable to quicksort or heapsort. 2. a trip through the tree consists of starting from the root and asking oneself the question that is found there.1 Decision Trees 293 procedure countsort(T [1. 3.min(T) = #T . For example.. However.min(T). if max(T) . the algorithm provides an efficient and practical way of sorting an array in linear time. However. directed binary tree. As a function of n. On the other hand. j F. A decision tree for sorting n elements is valid if to each possible order relation between the elements it associates a verdict that is . Show exactly how countsort can be said to carry out arithmetic operations on the elements to be sorted. In this book we pay no further attention to algorithms for sorting by transformation. n ]) i <. This difference resembles that between binary search and hash coding. Each internal node contains a comparison between two of the elements to be sorted. the trip continues recursively in the left-hand subtree . 1. on account of both the memory and the time it requires.1 to ndo C[T[k]]*-C[T[k]]+1 k F--1 for p F. 9.2. when the difference between the elements to be sorted is large. if not. this leaf contains the verdict associated with the order relation used. 4.j]F-0 fork F. The most important characteristic of countsort and its variations is that they work using transformations : arithmetic operations are carried out on the elements to be sorted. Simulate the operation of T [ 1 . how many comparisons between elements are made? Coming back to the question we asked at the beginning of this section : what is the minimum number of comparisons that are necessary in any algorithm for sorting n elements by comparison ? Although the theorems set out in this section still hold even if we consider probabilistic sorting algorithms (Section 8.max(T) array C[i. 5.1). otherwise it continues recursively in the right-hand subtree.1.1 to C [p ] do T[k]*-p k -k+l Problem 10.1.4. 10.. Each leaf contains an ordering of the elements. 1. all the sorting algorithms considered in the preceding chapters work using comparisons : the only operation allowed on the elements to be sorted consists of comparing them pairwise to determine whether they are equal and. the number of elements to be sorted. In this case. Given a total order relation between the elements. 10] containing the values 3.1. variants such as radix sort or lexicographic sort (not discussed here) can sometimes be used to advantage. this algorithm on an array This algorithm is very efficient if the difference between the largest and the smallest values in the array to be sorted is not too large. A decision tree is a labelled. If the answer is "yes". we shall for simplicity confine our discussion to deterministic algorithms. The trip ends when it reaches a leaf .i to j do for q <. 6.

and C.1.C. respectively. a decision tree is pruned if all its leaves are accessible from the root by making some consistent sequence of decisions. B.B else if B < C then if A <C then T .B. C. 3]) A.C else T .1. The following problem will help you grasp these notions.3 give the trees corresponding to the insertion sorting algorithm (Section 1.B.1.<A<B B<A < C B<C<A A valid decision tree for sorting three elements.C..C.T [1].1 is valid for sorting three elements A. to the decision tree of Figure 10.) Notice that heapsort C<B<A A<B<C 1 A<C<B Figure 10. Problem 10.1. (The annotations on the trees are intended to help follow the progress of the corresponding algorithms.B.A.A. For example. Finally.A.B else T .T [3] if A < B then if B < C then { already sorted } else if A < C then T . to every deterministic algorithm for sorting by comparison there corresponds. when three elements are to be sorted. B.Introduction to Complexity 294 Chap.C.1.A Similarly.4 and Problem 2. a decision tree that is valid for sorting n elements. Figures 10. 10 compatible with this relation. .3).9.A else T <.T [2].1. Every valid decision tree for sorting n elements gives rise to an ad hoc sorting algorithm for the same number of elements. C <.3. Verify that the decision tree given in Figure 10.2.1 there corresponds the following algorithm.1. procedure adhocsort3(T [l .4) and to heapsort (Section 1. for each value of n.2 and 10.

5) for the case of three elements. The three element insertion sort decision tree.4).1.1. asks again whether B > A before reaching its conclusion. Give the pruned decision trees corresponding to the algorithms for sorting by selection (Section 1. and then whether C > A (answer: yes). It would now be possible to establish the correct verdict. Problem 10. the tree is pruned : the leaf that would correspond to a contradictory answer "yes" to the third question has been removed.<C<B ABC A<B. but it.5. gives the number of comparisons carried . (Despite this. so that every leaf can be reached by some consistent sequence of decisions.3 first tests whether B >A (answer : no). nonetheless.1. In the two latter cases do not stop the recursive calls until there remains only a single element to be "sorted". that is. the distance from the root to the most distant leaf. if B <A A < C.1 295 ABC BAC ABB BAA x=C CBA I C<B<A x=C BAC BCA B. Give the pruned decision trees corresponding to the insertion sorting algorithm and to heapsort for the case of four elements.2. This situation does not occur with the decision tree of Figure 10.1. the decision tree of Figure 10. For instance. (You will need a big 0 piece of paper !) The following observation is crucial: the height of the pruned decision tree corresponding to any algorithm for sorting n elements by comparison.4.Decision Trees Sec.4) and by merging (Section 4. and to quicksort (Section 4.) Thus heapsort is not optimal insofar as the number of comparisons is concerned.2. 0 Problem 10.1. 10.<C<A I ACB A.C Figure 10. but beware of appearances : it occurs even more frequently with the insertion sorting algorithm than with heapsort when the number of elements to be sorted increases. sometimes makes unnecessary comparisons.

and hence it has at most 2" leaves.Introduction to Complexity 296 Chap.3.1. this time on the height of the tree). The three element heapsort decision tree. Can we find a valid decision tree for sorting three elements whose height is less ? If so. we shall have an ad hoc algorithm for sorting three elements that is more efficient in the worst case.1. Lemma 10. The lemma follows immediately.8. Any valid decision tree for sorting n elements contains at least Lemma 10. and C < B ? situated on the path from the root to the appropriate verdict in the decision tree all have to be made.1. It is easy to show (by mathematical induction on the total number of nodes in the tree) that any binary tree with k leaves must have at least k -1 internal nodes. The decision trees we have seen for sorting three elements are all of height 3.) . in this case the three comparisons B < A ?. The upper limit on the number of leaves of any pruned decision tree can be computed with Problem 5. (It may have more than n! leaves if it is not pruned or if some of the leaves can only be reached when some keys are equal. To say the same thing differently. n! leaves. a possible worst case for sorting three elements by insertion is encountered if the array is already sorted into descending order (C < B < A ).6. Proof. 10 yes C> B C A rBA BC C A BC A C BC AB BA A C BC A C A B BA BC AB CB AC C B BA CA A<B<C C<A<B A< C<B B<A<C C < B < A B<C<A I Figure 10. We now prove more generally that such a tree is impossible. Try it : you will soon see that this cannot be done. a binary tree with t nodes in all cannot have more than It /21 leaves. For example. C < A ?.1. Now a binary tree of height h can have at most 2h+1-1 nodes in all (by another simple argument using mathematical induction. out by this algorithm in the worst case. Any binary tree with k leaves has a height of at least [lg k 1.2.

By the crucial observation that precedes Lemma 10. and that sorting by merging almost attains the lower bound.6. 0 Prove that the number of comparisons carried out by heap- sort on n elements. This proof shows that any deterministic algorithm for sorting by comparison must make at least Iig(n!)l comparisons in the worst case when sorting n elements. This tree contains at least n! leaves by Lemma 10.17). Suppose we ask our sorting algorithm not merely to determine the order of the elements but also to determine which ones.1. Rework this problem assuming that there are three possible outcomes of a comparison between A and B : A < B . if any.7.1. A valid tree must be able to produce at least one verdict corresponding to each of the n! possible orderings of the n elements to be sorted. Prove further that if n is a power of 2. Give a lower bound on the number of comparisons required in the worst case to handle n elements.1. Problem 10. In the worst case.1. the algorithm takes a time in S2(n log n) in the worst case.1.C is not acceptable : the algorithm must answer either A < B < C or A < B = C.n +I comparisons in the worst case when sorting n elements.1. and lg(n!)eS2(n logn) (Problem 2. a verdict such as A< B <.2. What can you say about sorting by merging in the general case ? More precise analysis shows that [lg(n!)1 E n lgn -O(n). . the insertion sorting algorithm makes 66 comparisons when sorting 12 elements. Its height is therefore at least [lg(n!)1 by Lemma 10. the algorithm thus needs at least [lg(n!)l comparisons in the worst case to sort n elements. To every deterministic algorithm for sorting by comparison there corresponds a pruned valid decision tree for sorting n elements. orA >B.8.1. A = B.1. is never greater than 2n ign.) Problem 10. Proof. For example. a time in S2(n log n) to sort n elements in the worst case.1.Sec. then mergesort makes n Ig n . Since each comparison takes a time in 82(1). whereas heapsort makes 59 (of which the first 18 are made during construction of the heap). Any deterministic algorithm for sorting by comparison takes Theorem 10. are equal. (Some modifications of heapsort come very close to being optimal for the worst-case number of comparisons. In fact. 10. and yet rlg(12!)] = 29. it has been proved that 30 comparisons are necessary and sufficient in the worst case for sorting 12 elements.1. How well do these algorithms do when compared to the lower bound Fig (n!)] for n = 50 ? ** Problem 10.1 Decision Trees 297 Proof. The previous problem therefore shows that heapsort is optimal to within a factor of 2 as far as the number of comparisons needed in the worst case is concerned. n ? 2.1. This certainly does not mean that it is always possible to sort n elements with as few as [lg(n!)1 comparisons in the worst case. Give exact formulas for the number of comparisons carried out in the worst case by the insertion sorting algorithm and by the selection sorting algorithm when sorting n elements.

i leaves.1. and let x be some element. Lemma 10.1. with the logical convention that T [0] = .x <. which is ..i S n and T [i ] <. Let T be a binary tree. At first sight this recurrence is not well founded since it defines h (k) in terms of itself (when we take i =0 or i =k in the minimum. Since the dis- tance from each leaf to the root of A is one less than the distance from the same leaf to the root of T. If each verdict is equally likely. or 2 children. Define the average height of T as the sum of the depths of all the leaves divided by the number of leaves.k -1. for some 1 <. rather than to 0 determine its position ? Decision trees can also be used to analyse the complexity of a problem on the average rather than in the worst case. The root of T can have 0. the single child is the root of a subtree A. H (T) = 16 for the tree in Figure 10. where x e R is such that 1 <. 10 Problem 10.x). Suppose for simplicity that the n elements are all distinct.x) lg(k . the average height of T is H (T )/k .1.1. Any binary tree with k leaves has an average height of at least lgk. consider the function g (x) = x Igx + (k .) Proof. respectively.Ig (k -x).o0 and T [n + 1 ] = + oo. In the first case the root is the only leaf in the tree and H (T) = 0. define h (k) as the smallest value possible for H (X) for all the binary trees X with k leaves. Define H (T) as the sum of the depths of the leaves.1. the problem is to find an index i such that 0 <.1. the decision tree of Figure 10. we see that there is little difference between the worst case and the average. How does binary search compare to this lower bound ? What lower bound on the number of comparisons do you obtain using the decision tree technique if the problem is simply to determine whether x is in the array. h (1) = 0. By a similar argument we obtain this time H (T) = H (B) +H (C) + k. n ] be an array sorted into ascending order.298 Introduction to Complexity Chap. then 8/3 is the average number of comparisons made by the sorting algorithm associated with this tree.9. If we define h (0) = 0. This difficulty disappears because it is impossible that h (k) = h (k) + k.3.3.i ) + k 10<-i <-k } for every k > 1. which also has k leaves. Let T [I . 1.1. By definition. In particular. we have H (T) = H (A) + k. How many comparisons between elements are needed in the worst case to locate x in the array ? As in Section 4. (By comparison with Lemma 10. For example. 0 if k<-1 h(k) = k +min[h(i)+h(k-i) 11 <-i <-k-1 } ifk > 1 Now. the preceding discussion and the principle of optimality used in dynamic programming lead to h (k) = min I h (i) + h (k . In the second case.x < T [i + 11.1 has an average height (2+3+3+3+3+2)/6=8/3. We can thus reformulate the recurrence that defines h (k). Let T be a binary tree with k leaves. In the third case the tree T is composed of a root and of two subtrees B and C with i and k . which corresponds to the root having only one child). Calculating the derivative gives g' (x) =1g x . For example. For k >-1.i < k .

x E 1R } . . where h (k) is the function used in the proof of Lemma 10. give an algorithm able to solve this problem in a time in 0 (n log k) and a space in O (k) in the worst case. it follows that h (k) > k + min {g (x) I < x 5 k -1.11.k lg k for every integer k ? 1 now follows by mathematical induction. Your algorithm should make no more than one sequential pass through the array T . This shows that H (T) >.3.k.k lg k. Since the second derivative is positive. 10. Proof. * Problem 10. Prove that h (k) = k t + 21 . Let t = Llg k and 1 =k-2. The problem consists of returning in descending order the k largest elements of T. Because min X ? min Y for any two nonempty sets X and Y such that X c Y. Suppose by the induction hypothesis that h (j) >.1. if x = k12. Justify your analysis of the time and space used by your algorithm. Problem 10. Determine the number of comparisons performed on the average by the insertion sorting algorithm and by the selection sorting algorithm when sorting n elements. This minimum is g (k / 2) = (k lg k) .) Theorem 10.1. Any deterministic algorithm for sorting by comparison makes at least lg(n!) comparisons on the average to sort n elements.Sec.1. that is. Using the result obtained previously.3. it is therefore efficient even if n is large and if the array is supplied on a magnetic tape.10. h(k)=k +min{h(i)+h(k-i) I 1 <i <-k-1 }. The average height of T being H (T )lk . It therefore takes an average time in Q (n log n). both in the worst case and on the average. The base k = 1 is immediate. Follows immediately from Lemmas 10.2 and 10.. h (k) ? k + min { g (i) I 1 < i 5 k -1. Let T [1 . Let k > 1. Prove that any deterministic algorithm that solves this problem using comparisons between the elements must make at least (k / 2) lg (n / 2) comparisons. On the other hand.1. The proof that h (k) >. How do these values compare to the number of comparisons performed by these algorithms in the worst case? * Problem 10. (Optionally : give an intuitive interpretation of this formula in the context of the average height of a tree with k leaves. Conclude that it must take a time in Q(k log n). i E IN }.1.j lg j for every strictly positive integer j < k -1.n an integer. By the induction hypothesis.1. g (x) attains its minimum at x = k / 2. it is therefore at least Ig k.1.1 Decision Trees 299 zero if and only if x = k -x. By definition.2.k lg k for every tree T with k leaves. we have h (k) ? k + g (k / 2) = k lg k. n ] be an array and k <. Prove that this also implies that h (k) >.12.

10 10. From a negative point of view. when A =' B we can be sure that they are the same. Prove that the relations <_ 1 and = 1 are transitive.2. For the purposes of this problem only. if these problems have all been studied independently in the past. Unfortunately. write A <g B if the existence of an algorithm for B that works in a time in O(t(n)).Introduction to Complexity 300 Chap. In the remainder of this section we shall see a number of examples of reduction from a variety of application areas. implies that there exists an algorithm for A that works in a time in 0 (t(0 (n))). where p (n) is a polynomial. cible to B . Section 10. both on the average and in the worst case. Suppose we are able to prove that a certain number of problems are equivalent in the sense that they have about the same complexity. at least in theory. if the existence of an algorithm for B that works in a time in 0(t(n)). then the fact that the problems are equivalent makes it even more unlikely that such an algorithm exists. denoted A <_ 1 B. for any function t(n). A is linearly reduDefinition 10. A less restrictive definition of linear reduction is obtained if * Problem 10. Even if we are not able to determine the complexities of A and B exactly. Let A and B be two solvable problems. Except for the value of the multiplicative constant.2. .1. for any function t(n). There are two reasons for doing this.2. and if all the efforts to find an efficient algorithm for any one of them have failed. a more efficient algorithm for all the others. When A 51 B and B S ' A both hold. we content ourselves with comparing the efficiency of algorithms for A on instances of size n with the efficiency of algorithms for B on instances of size in O (n). On the other hand. we often have to be content to compare the relative difficulty of different problems. Any algorithmic improvement in the method of solution of one of these problems now automatically yields. Because it is so difficult to determine the exact complexity of most of the problems we meet in practice.1.2 REDUCTION We have just shown that any algorithm for sorting by comparison takes a minimum time in SZ(n log n) to sort n elements. Show with the help of an explicit example that the notions A <_! B and A <g B are not equivalent even if there exists an algorithm for B that works in a time in O (p (n )). it does not often happen in the present state of our knowledge that the bounds derived from algorithmics and complexity meet so satisfactorily.2. implies that there exists an algorithm for A that also works in a time in 0 (t (n)). the question of the complexity of sorting by comparison is therefore settled : a time in O(n log n) is both necessary and sufficient for sorting n elements. A and B are linearly equivalent. Problem 10. we know that heapsort and mergesort both solve the problem in a time in O (n log n).3 goes into this second motivation in more detail. denoted A = 1 B.

8. implies that there exists an algorithm for A that also works in a time in 0 (t(n)). which cannot be smooth. Finally.Sec. counting the multiplications at unit cost.1. it takes longer to compute a 31st power than a 32nd. 0 iii.8. An algorithm is smooth if it takes a time in O(t(n)) for some smooth function t. it is supra quadratic if it is eventually nondecreasing and if there exists an CEIR+ such that t(an) >_ a2+et(n) for every positive integer a and every sufficiently large integer n.2 Reduction 301 Let us extend the notion of smooth functions (introduced in Section 2. strongly and supra linear functions are defined similarly. At least. (Hint: apply Problem 2. . It is strongly at least quadratic (strongly quadratic for short) if it is eventually nondecreasing and if t(an) >_ a2t(n) for every positive integer a and every sufficiently large integer n. this does not imply that the actual time taken by a specific implementation of the algorithm must also be given by an eventually nondecreasing function. By "reasonable". 10. assuming B is smooth". In particular. Consider for instance the modular exponentiation algorithm dexpo of Section 4. Show that n 2log n is strongly quadratic but not supra quadratic.3. such as "A <_ ' B. i.1. No problem could be smooth without this restriction to reasonable algorithms because any problem that can be solved at all can be solved by an algorithm that takes a time in Q(2'). Nonetheless. The actual time taken by any reasonable implementation of this algorithm is not an eventually nondecreasing function of the exponent.5) that the time it takes to compute a"mod m is a linear function both of lg n and the number of Is in the binary representation of n . Even though a smooth function must be eventually nondecreasing by definition.5) to algorithms and problems. we mean an algorithm that does not purposely waste time. Problem 10. A problem is smooth if any reasonable algorithm that solves it is smooth. Moreover. Prove that any strongly quadratic function is at least quadratic.20. This can be interpreted literally as meaning that A <_ 1 B under the assumption that B is smooth. and logn is smooth. These notions extend to algorithms and problems as in the case of smooth functions.) ii. this algorithm is smooth because it takes a time in O(log n ).2. We have seen (Problem 4. Give an explicit example of an eventually nondecreasing function that is at least quadratic but not strongly quadratic. Most theorems in this section are stated conditionally on a "reasonable" assumption. all these theorems are constructive : the algorithm for B follows from the algorithm for A and the proof of the corresponding theorem. for any smooth function t(n). From a more practical point of view it also means that the existence of an algorithm for B that works in a time in 0 (t(n)). A function t : IN -* It* is at least quadratic if t(n) E c (n2).

MT. 10 10. more efficient algorithm for inverting arbitrary nonsingular matrices (at least in theory). No confusion should arise from this. and inversion of nonsingular upper triangular matrices. so that a time in O(n2) is really linear. Is it possible that multiplication of upper triangular matrices could be carried out significantly faster than the multiplication of two arbitrary square matrices ? From another point of view. M. In what follows we measure the complexity of algorithms that manipulate n x n matrices in terms of n.) Once again this means that any new algorithm that allows us to multiply upper triangular matrices more efficiently will also provide us with a new. it implies that we can invert any nonsingular n x n matrix in a time in 0 (n 2.2. We shall show under reasonable assumptions that MQ I MT =_1 IT.81) (or even 0 (n 2-376)) is sufficient to multiply two arbitrary n x n matrices. We denote these three problems. Theorem 10. contrary to the intuition that may suggest that this problem will inevitably require a time in S2(n3). this is incorrect because the running time should be given as a function of the size of the instance. MQ <_ ' MT. Proof. that is.2. Any algorithm that can multiply two arbitrary square matrices can be used directly for multiplying upper triangular matrices. (The problem of inverting an arbitrary nonsingular matrix is also linearly equivalent to the three preceding problems (Problem 10. assuming MT is smooth. Proof. Notice that the problems considered are at least quadratic in the worst case because any algorithm that solves them must look at each entry of the matrix or matrices concerned.2. Let A and B be two arbitrary n x n matrices to be multiplied. where t (n) is a smooth function.2. = 0 when i > j. multiplication of arbitrary square matrices. that is. Suppose there exists an algorithm that is able to multiply two n x n upper triangular matrices in a time in 0 (t (n)).1 Reductions Among Matrix Problems An upper triangular matrix is a square matrix M whose entries below the diagonal are all zero. multiplication of upper triangular matrices.2. Theorem 10. experience might well lead us to believe that inverting nonsingular upper triangular matrices should be an operation inherently more difficult than multiplying them.1. by MQ. and the resulting algorithm is numerically unstable. respectively. In particular. it requires a slightly stronger assumption. Formally speaking. We saw in Section 4.9 that a time in O (n 2. MT <_ I MQ. referring to an algorithm that runs in a time in O(n 2) as quadratic. Consider the following matrix product: .9).Introduction to Complexity 302 Chap.376). and IT. but the proof of this is much more difficult.

10. IT <' MQ. The time required for this operation is in 0 (n 2) for the preparation of the two big matrices and the extraction of AB from their product. t (3n) E O (t (n)).2. Theorem 10.Sec. Suppose there exists an algorithm that is able to invert a nonsingular n x n upper triangular matrix in a time in O (t(n)). Suppose there exists an algorithm that is able to multiply two arbitrary n x n matrices in a time in 0 (t (n)). Proof. weaker but less natural hypothesis suffices : it is enough that MQ be supra linear. By the smoothness of t(n).2. and D each of size n / 2 x n / 2 such that Cl A= I B0DJ where B and D are upper triangular and C is arbitrary.) Proof. C. Let A be a nonsingular n x n upper triangular matrix to be inverted. Now consider the following product : BC 0D X B-1 -B-'CD-' 0 D-' 10 01 This product shows us how to obtain A-' by first calculating B-1.3. n 2 E O (t (n)).4 for the general case. . Because t(n) is at least quadratic. inversion is trivial.2 Reduction 303 0 A 0 000 0 0 AB 0 0 0 x 0 0 0 00B = 0 0 0 00 0 0 0 0 where the "0" are n x n matrices all of whose entries are zero. Let A and B be two arbitrary n x n matrices to be multiplied. and then multiplying the matrices B-'. this operation takes a time in 0 (t(n)). and D-'. Consequently. the matrices B and D are nonsingular. Consider the following matrix product : I AO 0IB x 0 0/ 1 -A AB 0 1 -B 00 1 100 = 010 001 where I is the n x n identity matrix. assuming IT is smooth. a Theorem 10. and D-'.' IT. This product shows us how to obtain the desired result AB by inverting the first of the 3n x 3n upper triangular matrices. C.) If n = 1. (In fact. Suppose for simplicity that n is a power of 2. (See Problem 10. Otherwise decompose A into three submatrices B.5. where t (n) is strongly quadratic.2. where t (n) is a smooth function. MQ <. By Problem 10.2. assuming MQ is strongly quadratic. the total time required to obtain the product AB is in 0 (t(n)). As in the proof of the previous theorem.4. This product shows us how to obtain the desired result AB by multiplying two upper triangular 3n x 3n matrices. plus 0 (t (3n)) for the multiplication of the two upper triangular matrices. The upper triangular matrices B and D.

5. are smaller than the original matrix A. Prove that MS =' MQ under suitable assumptions. with the natural conventions that x +(+oo) =+oo and min(x.2.+oo) = x for all x e R-. Prove that if A is a nonsingular upper triangular matrix whose size is even. and if B and D are defined as in the proof of theorem 10. diagonal are 1. Prove that SU need ? A matrix A is symmetric if Aid = Aji for all i and j. Problem 10.3(iii). Using the divide-and-conquer technique suggests a recursive algorithm for inverting A in a time in O(g(n)) where g(n) a 2g(nl2)+2t(nl2)+O(n2).) 10.13(iv). (Hint : apply Problem 2. What assumptions do you need? Denote by IQ the problem of inverting an arbitrary non** Problem 10.3. 2.2 Reductions Among Graph Problems In this section IR°° denotes IR*u I+-}.2.2. Denote Problem and Z be three sets of nodes. All that the proof of theorem 10.Introduction to Complexity 304 Chap. Let f : X x Y -4 IR°° and g : Yx Z -4 IR°° be two functions representing the cost of going directly from one node to another.7. An infinite cost represents the absence of a direct route. Y.9.2. Denote by SU the problem of squaring a unitary upper triangular MQ under suitable assumptions. What assumptions do you matrix. Complete the proof that IT <_' MQ.4.6. this implies that g(n) e 0 (t (n) I n is a power of 2).2. Let IT2 be the problem of inverting nonsingular upper triangular matrices whose size is a power of 2. The fact that t(n)ES2(n2) and the assumption that t(n) is eventually nondecreasing (since it is strongly quadratic) yield g(n) e 2g(n / 2) +O (t(n)) when n is a power of 2. Prove that if g(n) E 2g(n / 2) +O (t (n)) when n is a power of Problem 10. then g(n) E O (t(n) I n is a power of 2). 10 which we now have to invert.) An upper triangular matrix is unitary if all the entries on its Problem 10. then B and D are nonsingular.6.8. singular matrix.4 really shows is that IT2 <_' MQ. Prove that IQ =' MQ. By Problem 10.2. Problem 10. using the assumption that t(n) is strongly quadratic (or at least supra linear). Assume that both IQ and MQ are smooth and supra quadratic. note that it is enough to assume that t(n) is supra linear. by MS the problem of multiplying symmetric matrices.2. and if t(n) is strongly quadratic. (Note : this reduction would not go through should an algorithm that is capable of multiplying n x n matrices in a time in 0 (n 2 log n) exist .4.2. Denote by .see Problem t7 10. Let X.

The meaning of f' is similar for any i > 0. there is no obvious way of adapting to this problem Strassen's algorithm for ordinary matrix multiplication (Section 4. (The intuitive reason is that Strassen's algorithm does subtractions. and also the functions f and g. it suffices to consider only those paths whose length is less than the number of nodes in X. Notice the analogy between this definition and ordinary matrix multiplication (where addition and multiplication are replaced by the minimum operation and addition. and Z. We thus have that f * = min ( f' I 0 <_ i < n . is therefore f * = min { f' I i >_ 0 1. f2) gives the minimum cost of going from one node of X to another either directly or by passing through exactly one intermediate node. Because they are quite complicated and have only theoretical advantages. min(f . The existence of algorithms asymptotically more .Sec. This calculation is nothing other than the calculation of f * . Thus it is possible to get away with a time in O(n 3) after all. Y.4 a dynamic programming algorithm for calculating shortest paths in a graph. However. namely Floyd's algorithm.) So the definition of f * can be taken to give us a direct algorithm for calculating its value in a time in O(n4). y) + g (y. This represents the minimum cost of going from x to z passing through exactly one node in Y. At first sight. there exist more efficient algorithms for this problem. In this case ff . gives the minimum cost of going from one node of X to another (possibly the same) while passing through exactly one intermediate node (possibly the same. we do not discuss them here. computing f * for a given function f seems to need more time than calculating a simple product fg. Similarly.9). z) = min { f (x. which we shall write f 2. By analogy. too). This definition is not practical because it apparently implies an infinite computation . f ° represents the cost of going from one node to another while staying in the same place. we saw in Section 5. without increasing the cost of the resulting path. respectively). The preceding notation becomes particularly interesting when the sets X. Consequently. f never takes negative values. Let this number be n.2 305 Reduction fg the function h : X x Z -* IR°° defined for every x E X and z (=Z by h (x . Unfortunately. } There is no equivalent to this operation in the present context since taking the minimum is not a reversible operation. The minimum cost of going from one node to another without restrictions on the number of nodes on the path. Nevertheless. so that ° f (x' y) 0 ifx =y +00 otherwise. which we write f*. 10. Could it be that the problems of calculating fg and f * are of the same complexity ? The following two theorems show that this is indeed the case : these two problems are linearly equivalent. The straightforward algorithm for calculating fg takes a time in O(n3) if the three sets of nodes concerned are of cardinality n. it is not even immediately clear that f * is well defined. However. but do not confuse this operation with the composition of functions. z) I y E Y }. Any path that passes twice through the same node can therefore be shortened by taking out the loop thus formed. coincide.

for instance. min ( h (u . w) + h 2(w . MUL <_ ' TRC. Therefore h (u. Y. v) = hh 2(u . for a smooth function t(n). Proof. v) I w E W I. would be considered quadratic even though it is linear in the number of edges (for dense graphs). . y) :. y) and h (y. w) _ + oo when w E X whereas h 2(W. v) I y c: Y ). h 3(u . Let X. v) for all i>3. and Z be three sets of nodes of cardinality n . But h (u . v) I W E W 1. Now.v)= g(u. By definition. we have 2 h (u. h 2) is given by the following equation. n 2 and n 3.& + oo when y E Y is to have u EX. Define the function h : W x W. v) = min ( h (u. v) = min { f (u . and v e Z it suffices to note that h (u .v) ifuEY andvEZ +00 otherwise . v) = g (y. v) _ + oo for all u. An algorithm such as Dijkstra's.5. The calculation of h 3(u . The conclusion from all this is that h* = min(h °. y) + h (y.= v) fg(u. let us find the value of h 2(u . time complexities will be measured as a function of the number of nodes in the graphs concerned. respectively. y) + g (y. v) to conclude that h 2(U. v). The same holds for h ` (u . As in the previous section. and the only way to have h (y. But the only way to have h (u. z) _ + oo when x e X and z E Z.v) ifuEX andvEY h(u. Notice in particular that h (x. it is impossible that h (u. v) =min { h (u . which is precisely the definition of f g (u. Let W = X u Y u Z.1R°°. v) = h 2(w .Introduction to Complexity 306 Chap. v) = +. h . and let f : X x Y . v) = +o. v) # +oo when y E Y is to have v EZ. Summing up. v E W. If u 0 X or if v 0 Z. Denote by MUL and TRC the problems consisting of calculating fg and f*. respectively. Consequently. h 2(U. w) + h (w. Y. y) = f (u . w) # +oo and h (w. By the definition of h. Theorem 10.1R°° be two functions for which we wish to calculate fg. v) = + 00 when w e X. v).1R°° as follows : f(u. it therefore follows that h 2(u . v) # +oo simultaneously unless w c= Y. By definition. w) + which implies that h 3(U. Again. y E Y. and Z are disjoint. where n is the cardinality of the set W such that h : W x W . v) I y E Y 1. at least in theory. assuming TRC is smooth. 10 efficient than O(n3) for solving the problem of calculating fg therefore implies that Floyd's algorithm for calculating shortest routes is not optimal. the problems considered are at least quadratic in the worst case because any algorithm that solves them must look at each edge concerned.v) if u EX and veZ + otherwise . Suppose there exists an algorithm that is able to calculate h* in a time in O(t(n)).1R°° and g : Y x Z . v) is easier. h 2(U.0 for every WE W. In the case when u EX. Suppose without loss of generality that X.2.

In other words. c*de*bc* )) (u . n 2. v) if u E K and V E J (min(c*. Notice that e (u .v) = < if ueJ and veJ e*bc* (u. the nodes in K can be used as often as necessary.1R°° be the function given by e = min(a . v) ifu E K and v E K To calculate h* using divide-and-conquer. e*(u. we therefore solve recursively two instances of the TRC problem.Sec. b : J x K . Let n = n I +n 2+n 3 be the cardinality of W. Theorem 10. each containing half the nodes.v) = if uEX andvEY ifuEY andvEZ fg(u. Using the algorithm for calculating h* thus allows us to compute fg in a time in t (n) + O (n 2)C O (t(3 max(n i . Y. n 2. (In fact. Assume t (n) is strongly quadratic. assuming MUL is strongly quadratic.v) h*(u. TRC <_' MUL. v) if u E J and v E K c*de* (u. The minimum cost for going from u to v with no restrictions on the path is therefore e* (u. e* is the restriction to J x J of the h* that we wish to calculate.2 Reduction 307 0 ifu =v f(u.2.4. n 3))) because t(n) is smooth and at least quadratic. As in theorem 10.and g : Y x Z -* 1R°°. be*d ).6. however. v) when u and v are both in J. The desired result is then obtained after a number of instances of the MUL problem. If n = 1. or not used at all if this is a cost function for which we wish to calculate h*.11 for the general case). v ). n 3))) + O (n 2)g 0 (t (max(n I . Let e : J x J . each of size n / 2. Define a : J x J -> IR°°. Suppose for simplicity that n is a power of 2 (see Problem 10. Suppose there exists an algorithm that is able to calculate fg in a time in 0 (t (max(n I . Let H be a set of cardinality n and let h : H x H .as the restrictions of h to the corresponding subdomains. in order to obtain c* and e*.) Proof.v) h*(u. 10. n 2. a weaker but less natural hypothesis suffices again : it is enough that MUL be supra linear. and n 3 are the cardinalities of the sets X.1R.v) +oo otherwise Therefore the restriction of h* to X x Z is precisely the product fg we wished to calculate. represents the minimum cost for going from u to v without passing through any node in J (not counting the endpoints) . The other restrictions are obtained similarly. and d : KxJ -* R. n 3))).2. which implies that g(n) E 0 (t(n) I n is a power of 2). n 2. it is obvious that h* (u . for u and v in J.2. where n I . . c : K x K -* 1R°°.1R°°. Otherwise split H into two disjoint subsets J and K. and Z such that f : X x Y -+ 1R. u) = 0 for the unique u EH.v) if uEX and vEZ g(u. the time g(n) required by this approach is characterized by the equation g(n) E 2g(n / 2)+0 (t(n)).

2. * Problem 10. regardless of the cost of the path.7. and TRCBS. Prove that MULBS=1 MULB.7. 10. We saw that Warshall's algorithm (Problem 5. Problem 10. Furthermore. where MQ is the problem of multiplying arbitrary arithmetic square matrices (Problem 10. thus showing that Warshall's algorithm is not optimal. when the cost functions are restricted in this way. Strassen's algorithm can therefore be used to solve the problems MULB and TRCB in a time in 0 (n 2. Unlike the case of arbitrary cost functions.+-l be two Problem 10. Let TRC2 be the problem of calculating h* for h : X x X .2.+-) and g: Y x Z -* (0. however. No algorithm is known that can solve MULB faster than MQ. Let MULB and TRCB be the problems consisting of calculating fg and h*. Each of the four problems discussed earlier has a symmetric version that arises when the cost functions involved are symmetric. When the range of the cost functions is restricted to 10.Introduction to Complexity 308 * Problem 10. v E X . It is clear that MULB <_ 1 MUL and TRCB <_ 1 TRC since the general algorithms can also be used to solve instances of the restricted problems. TRCS.2) solves this problem in a time in 0(n3). v) = f (v . A cost function f : X x X -4R. u) for every u . Chap. calculating f * comes down to determining for each pair of nodes whether or not there is a path joining them.5). This is interesting because MULB <_ 1 MQ. and 9.4. Do you believe that MULBS=1 TRCBS? If not. Assuming we count arithmetic operations at unit cost.12. 10 Prove formally that the preceding formula for h* is correct. does one of these two problems appear strictly more difficult than the other? Which one? Justify your answer. +00 }. Complete the proof that TRC 5 ' MUL.81).2. All the proof of theorem 10. restricted cost functions. Note. Show that the arithmetic can be done modulo p.2. Call these four problems MULS. respectively.11.10.6 really shows is that TRC2 5 1 MUL. show how to transform the problem of calculating fg into the computation of an ordinary arithmetic matrix multiplication.2. that using Strassen's algorithm requires a number of arithmetic operations in 0 (n 2-81 ) . Let f : X x Y -* 10. 4.12).59) and even in 0 (n log n log log n) symmetric if f (u . the proof that MUL =/ TRC can easily be adapted to show that MULB =' TRCB. where p is any prime number larger than n. the time in 0 (n 3) taken by Warshall's algorithm counts only Boolean operations as elementary.1R°° when the cardinality of X is a power of 2.2. MULBS. We saw that it is possible to multiply two integers of size n in a time in 0 (n Reductions Among Arithmetic and Polynomial Problems We return to the problems posed by the arithmetic of large integers (sections 1. Conclude that MULB <_ 1 MQ. What can we say about integer division and taking square roots ? Our everyday experience leads us to believe that the second of .

and DIV be the problems consisting of squaring an integer of size n. these problems are at least linear because any algorithm that solves them must take into account every bit of the operands involved. Notice that a direct approach using discrete Fourier transforms. We have seen in Section 9. Once again this turns out not to be true. we take a moment to prove that SQR MLT. assuming SQR is smooth (a weaker assumption would do). Clearly. Let x and y be two integers of size n to be multiplied. a few additions. We now show that the problem of polynomial division is linearly equivalent to that of multiplication.Sec.2.22). respectively.4 how to multiply two polynomials of degree n in a time in 0 (n log n). Clearly. As mentioned in Section 1. suppose there exists an algorithm that is able to square an integer of size n in a time in 0 (t(n)). SQR =' MLT =' DIV. let p(x)=x3+3x2+x+2 and d(x) = x2+x +2. Let SQR. we can solve MLT in a time in 2t(n +1)+O(n)c O(t(n)) because t(n) is smooth and t (n) E Q(n).7. Nonetheless. Assume without loss of generality that x >. provided that the necessary arithmetic operations on the coefficients can be counted at unit cost. which works so well for multiplying two polynomials.2 Reduction 309 these problems. of multiplying two integers of size n. This is the case precisely because we assume all these algorithms to be smooth. MLT. In this .2. The following formula enables us to obtain their product by carrying out two squaring operations of integers of size at most n + 1. For this reason we content ourselves in the rest of this section with showing the equivalence of these operations in the "cleaner" domain of polynomial arithmetic. and of determining the quotient when an integer of size 2n is divided by an integer of size n. is genuinely more difficult than multiplication. Proof outline.) Theorem 10. (For simplicity we measure the size of integers in bits. Since the additions and the division by 4 can be carried out in a time in O (n). For example. since squaring is only a special case of multiplication. SQR 51 MLT. 10. The full proof of this theorem is long and technical. and probably the first one. To show that MLT <_' SQR. however.y . Its conceptual beauty is also defaced in places by the necessity of using an inordinate number of ad hoc tricks to circumvent the problems caused by integer truncation (see Problem 10. is inapplicable in the case of division unless the two polynomials concerned divide one another exactly with no remainder. The quotient of the division of p (x) by d (x) is q (x) =x+2. where t(n) is smooth (it is enough to assume that t (n + 1) E O (t (n)) ). assuming these three problems are smooth and MLT is strongly linear (weaker but more complicated assumptions would suffice). and a division by 4: xy =((x+y)2-(x-y)2)/4. this choice is not critical : the time taken by the various algorithms would be in the same order if given as a function of the size of their operands in decimal digits or computer words. too.7.2.

We even have that p (1) = 7 is not divisible by d (1) = 4.Introduction to Complexity 310 Chap. the remainder. p I (x) and p 2(x) are three arbitrary poly* Problem 10.m if n > m -1 .23.2. nomials. Let p (x) be a nonzero polynomial of degree n. but q (2) = 4 # 24/8. if p(x)=x3+3x2+x+2.) Prove that if p (x). Prove the existence and the uniqueness of the quotient and Problem 10. We call q (x) and r (x). and d(x)=x-2. is defined by p*(x)= Lx 2nIp(x)]. then p*(x)= x 3 . INVP of determining the inverse of a polynomial of degree n and DIVP of calculating the . Then there exists a unique polynomial r (x) of degree strictly less than m and a unique polynomial q (x) such that p (x) = q (x) d (x) + r (x). provided that the degree of p1(x) P I(x) d i(x) is not more than twice the degree of LP 2(X)ld (x)j. the quotient and the remainder of the division of p (x) by d (x).2. Let p(x) = x 3 +x 2 + 5 x + 1 Problem 10. and iii. which we denote p* (x). By analogy with the integers. and d* (x). respectively. 11 Consider the four following problems : SQRP consists of squaring a polynomial of degree n. and let d (x) be a nonzero polynomial of degree m. The inverse of p (x). 10 case p (2) = 24 and d (2) = 8.3 x 2 + 8x .'_ 0 ai x' is a polynomial of degree n provided that an # 0. Calculate Prove that if p (x) is a nonzero polynomial and if Problem 10. The polynomial q (x) is of degree n . it is possible to determine the quotient and the remainder produced when a polynomial of degree 2n is divided by a polynomial of degree n in a time in 0 (n log n) by reducing these problems to a certain number of polynomial multiplications calculated using the Fourier transform.2. and if d (x). By convention the polynomial p (x) = 0 is of degree -1. otherwise q (x) = 0.15.3x . the quotient is denoted by q(x) = Lp(x)Id(x)J. Notice that p (x) and p* (x) are always of the same degree.14. p* (x). Let p (x) be a polynomial of degree n. (There is a very simple proof. We also need the notion of an inverse. L p (x)/d (x)]. then p2(x) _ p i(x)d 2(x) ± p2(x)d 1(x) d i(x)d2(x) ± d2(x) in particular. For example. Despite this difficulty.2. Lp1(x)/Lp2(x)/d(x)Jj = L(p1(x)d(x))/p2(x)]. MLTP of multiplying two polynomials of degree at most n. Lp1(x)/d(x)] ± Lp2(x)/d(x)] = L(p1(x)±p2(x))/d(x)].2. ii. Recall that p (x) = E. Show that if both p (x) and d (x) are monic polynomials (the coefficient of highest degree is 1) with integer coefficients then both q (x) and r (x) have integer coefficients and q (x) is monic (unless q (x) = 0). d i(x) and d2(x) are three nonzero polynomials. LLp(x)/di(x)j/d2(x)J = Lp(x)/(di(x)d2(x))]. q (x) = p* (x) then q* (x) = p (x).16. This is all due to the remainder of the division r (x) = .17.

Sec. 10.2



quotient of the division of a polynomial of degree at most 2n by a polynomial
of degree n. We now prove under suitable assumptions that these four problems
linearly equivalent using the following chain of reductions : MLTP <-'
SQRP <' INVP <-' MLTP and INVP:-' DIVP <-' INVP. Again, all these problems
are at least linear. We assume arithmetic operations can be carried out at unit cost on

the coefficients.

MLTP <-' SQRP, assuming SQRP is eventually nonde-

Theorem 10.2.8.

Proof. Essentially the same as the proof that MLT <-1 SQR given in theorem
10.2.7. There is no need for a smoothness assumption this time because the sum or

difference of two polynomials of degree n cannot exceed degree n.

Theorem 10.2.9.


SQRP <- 1 INVP, assuming INVP is smooth.

Proof. The intuitive idea is given by the following formula, where x is a
nonzero real number :



A direct attempt to calculate the square of a polynomial p (x) using the analogous formula (p* (x) - (p (x) + 1)* )* -p (x) has no chance of working : the degree of this
expression cannot be greater than the degree of p (x). This failure is caused by truncation errors, which we can, nevertheless, eliminate using an appropriate scaling factor.

Suppose there exists an algorithm that is able to calculate the inverse of a polynomial of degree n in a time in 0 (t(n)), where t(n) is a smooth function. Let p (x) be a
polynomial of degree n > 1 whose square we wish to calculate. The polynomial
x In p (x) is of degree 3n, so
[X2np(x)]* = Lx6n/x2np(x)J = LX4n/p(x)J


[x2n (p(x)+1)]* = Lx4n/(p(x)+l)J
By Problem 10.2.17
[x2n p(x)]* -[x2n (p(x)+l)]* = Lx4n/p(x)]- Lx4n/(p(x)+1)]


(p(x)+1)-x 4np(x))l(p(x)(p(x)+1))j

= Lx4nl(p2(x)+p(x))j

= [p2(x)+p(x)]*


The last equality follows from the fact that p 2(x) + p (x) is of degree 2n. By Problem
10.2.16, we conclude finally that
p2(x) _

[[x2n p(x)]* -[x2n (p(x)+l)]*]* -p(x)


Introduction to Complexity

Chap. 10

This gives us an algorithm for calculating p2(x) by performing two inversions of polynomials of degree 3n, one inversion of a polynomial of degree 2n, and a few operations (additions, subtractions, multiplications by powers of x) that take a time in O (n).
This algorithm can therefore solve SQRP in a time in

2t(3n)+t(2n)+0(n)c 0(t(n))
because t(n) is smooth and at least linear.

Theorem 10.2.10.


Proof. To calculate p* (x), where p (x) is a polynomial of degree n, we evaluate
Lx2n /p (x)j, an instance of size n of the problem of polynomial division.
Theorem 10.2.11.

DIVP :- I INVP, assuming INVP is smooth.

Proof. The intuitive idea is given by the following formula, where x and y are
real numbers and y # 0:

x/y = xy-I
If we try to calculate the quotient of a polynomial p (x) divided by a nonzero polynomial d (x) using directly the analogous formula p (x) d* (x), the degree of the result is
too high. To solve this problem we divide the result by an appropriate scaling factor.
Suppose there exists an algorithm that is able to calculate the inverse of a polynomial of degree n in a time in 0 (t(n)), where t(n) is a smooth function. Let p (x) be

a polynomial of degree less than or equal to 2n, and let d (x) be a polynomial of
degree n. We wish to calculate LP (x)/d (x)j. Let r (x) be the remainder of the division
of x 2n by d (x), which is to say that d* (x) = Lx 2" /d (x) j = (x 2n - r (x)) /d (x) and that
the degree of r (x) is strictly less than n . Now consider
d* (x)p (x)
X 2"


x 2n p (x) - r (x)p (x)
x2n d(x)
x 2n p (x)



r (x) p (x)


by Problem 10.2.17(i). But the degree of r (x) p (x) is strictly less than 3n , whereas the
degree of x 2" d (x) is equal to 3n , and so L(r (x) p (x))/(x 2" d (x))] = 0. Consequently,
L(d* (x)p (x ))/x2n] = L p (x) Id (x)], which allows us to obtain the desired quotient by
performing an inversion of a polynomial of degree n, the multiplication of two polynomials of degree at most 2n, and the calculation of the quotient from a division by a

power of x. This last operation corresponds to a simple shift and can be carried
out in a time in O(n). The multiplication can be performed in a time in O(t(2n))
thanks to theorems 10.2.8 and 10.2.9 (using the assumption that t(n) is smooth).

Sec. 10.2



The calculation of Lp(x)/d(x)] can therefore be carried out in a time in
t(n)+O(t(2n))+O(n)c O(t(n)) because t(n) is smooth and at least linear.
Theorem 10.2.12.
INVP 5' MLTP, assuming MLTP is strongly linear
(a weaker assumption will do - see the hint of Problem 10.2.19).
Proof. This reduction is more difficult than the previous ones. Once again, we
appeal to an analogy with the domain of real numbers, namely, Newton's method for
finding the zero of f (w) = I - xw. Let x be a positive real number for which we want
to calculate x-1. Let y be an approximation to x-1 in the sense that xy = I -S, for
-1 < S < 1. We can improve the approximation y by calculating z = 2y -y2x.
Indeed, xz = x (2y - y 2x) = xy (2 -xy) _ (1- S) (1 + S) = l _ S2. From our assumption
on S, S2 is smaller than 6 in absolute value, so that z is closer than y to x-1. To calculate the inverse of a polynomial, we proceed similarly, first finding a good approximation to this inverse and then correcting the error.
Suppose there exists an algorithm able to multiply two polynomials of degrees
less than or equal to n in a time in 0 (t (n)) where t(n) is strongly linear. Let p (x) be a
nonzero polynomial of degree n whose inverse we wish to calculate. Suppose for simplicity that n + 1 is a power of 2 (see Problem 10.2.18 for the general case).
If n =0, the inverse p* (x) is easy to calculate. Otherwise let k = (n + 1)/2.
During the first stage of the polynomial inversion algorithm we find an approximation h (x) to p* (x) such that the degree of x In - p (x) h (x) is less than 3k -1. (Note
that the degree of x In - p (x) p* (x) can be as high as n -1 = 2k - 2.) The idea is to rid
ourselves provisionally of the k coefficients of lowest degree in the polynomial p (x) by
dividing the latter by x'. Let h (x) = x k LP (x)/x k J * . Note first that the degree of
LP(x)/xk] is n -k = k -1, so Lp(x)/xk j* = Lx2k-2/LP(x)/xk]] = LX 3k -21p (x)] by
Problem 10.2.17(iii). Let r (x) be the polynomial of degree less than n such that
Lx 3k -2/p (x) J = (x 3k-2 - r (x)) /p (x). Then we have
x2n-p(x)h(x)=x4k-2 _p(x)xk(x3k-2-r(x))/p(x)=xkr(x)

which is indeed a polynomial of degree less than 3k - 1.
During the second stage, we improve the approximation h (x) in order to obtain
p* (x) exactly. Taking into account the appropriate scaling factor, the analogy introduced at the beginning of this proof suggests that we should calculate
q (x) = 2h (x) - Lh2(x)p (x)1x2n j. Let s (x) be the polynomial of degree less that 2n

such that Lh2(x)p(x)/x2n ] _ (h2(x)p(x)-s(x))/x2". Now calculate
p (x) q (x) = 2p (x) h (x) _ (p 2(x) h 2(x) _ P (x) s (x)) /x


= [(P(x)h(x))(2x2n -p(x)h(x))+P(x)s(x)]Ix In
_xkr(x))(xIn +xkr(x))+p(x)s(x)]/xIn
= [(x2n
= [xa" _x2kr2(x)+P(x)s(x)]lx2"

=x2n +(P(x)s(x)-x2kr'2(x))/x2"

Introduction to Complexity


Chap. 10

It remains to remark that the polynomials p (x) s (x) and x 2k r 2(x) are of degree at most
3n -1 to conclude that the degree of x 2n - p (x) q (x) is less than n , hence
q (x) = p* (x), which is what we set out to calculate.
Combining these two stages, we obtain the following recursive formula :

p*(x) = 2xk LP(x)lxkJ* - Lv(x)[Lp(x)lxk J*]2/xn-1 J
Let g(n) be the time taken to calculate the inverse of a polynomial of degree n by the
divide-and-conquer algorithm suggested by this formula. Taking into account the
recursive evaluation of the inverse of Lp(x)/xk J, the two polynomial multiplications
that allow us to improve our approximation, the subtractions, and the multiplications

and divisions by powers of x, we see that

g(n)Eg((n -1)/2)+t((n -1)/2)+t(n)+O(n)c g((n -1)/2)+O(t(n))
because t(n) is strongly
g(n) E O (t (n))


Using Problem 10.2.19, we conclude that

Problem 10.2.18.
Let INVP2 be the problem of calculating p* (x) when p (x)
is a polynomial of degree n such that n + 1 is a power of 2. All that the proof of
theorem 10.2.12 really shows is that INVP2 <-' MLTP. Complete the proof that

Problem 10.2.19.
Prove that if g(n)Eg((n -1)/2)+O(t(n)) when n +1 is a
power of 2, and if t(n) is strongly linear, then g(n) E O (t (n) I n +1 is a power of 2).
(Hint : apply Problem 2.3.13(iv) with the change of variable T (n) = g(n -1) ; note
that it is enough to assume the existence of a real constant a> 1 such that
t (2n) ? a t (n) for all sufficiently large n - strong linearity of t (n) would unnecessarily impose a=2.)
Problem 10.2.20.
Let p (x) = x 3 +x 2 + 5 x + 1. Calculate p* (x) using the
approach described in the proof of theorem 10.2.12. You may carry out directly the
intermediate calculation of the inverse of Lp (x)/x2J rather than doing so recursively.
Compare your answer to the one obtained as a solution to Problem 10.2.15.
* Problem 10.2.21.
We saw in Section 9.4 how Fourier transforms can be used
to perform the multiplication of two polynomials of degrees not greater than n in a
time in O (n log n). Theorems 10.2.11 and 10.2.12 allow us to conclude that this time
is also sufficient to determine the quotient obtained when a polynomial of degree at
most 2n is divided by a polynomial of degree n. However, the proof of theorem
10.2.11 depends crucially on the fact that the degree of the dividend is not more than
double the degree of the divisor. Generalize this result by showing how we can divide

a polynomial of degree m by a polynomial of degree n in a time in 0 (m log n).
** Problem 10.2.22.

Following the general style of theorems 10.2.9 to 10.2.12,

complete the proof of theorem 10.2.7. You will have to define the notion of

Sec. 10.3

Introduction to NP-Completeness


inverse for an integer : if i is an n-bit integer (that is, 2n-1 < i <- 2" -1), define
i* = L22n - '/i J. Notice that i* is also an n-bit integer, unless i is a power of 2. The
problem INV is defined on the integers in the same way as INVP on the polynomials.
The difficulties start with the fact that (i* )* is not always equal to i, contrary to
Problem 10.2.16. (For example, 13* = 9 but 9* = 14.) This hinders all the proofs.
For example, consider how we prove that DIV <- I INV. Let i be an integer of size 2n

and let j be an integer of size n ; we want to calculate

Li I j J.

If we define

z = Li j * / 22n - 1 ] by analogy with the calculation of L(p (x) d* (x )) /x 2n j in the proof

of theorem 10.2.11, we no longer obtain automatically the desired result z = Li / j].
Detailed analysis shows, however, that z <- Li / j J < z + 2. The exact value of Li / j J
can therefore be obtained by a correction loop that goes around at most three times.
z E- Li j* /22n


t - (z +1)x j
while t

<- i

t Ft +j


return z

The other proofs have to be adapted similarly.
** Problem 10.2.23.
Let SQRT be the problem of computing the largest integer
less than or equal to the square root of a given integer of size n. Prove under suitable

assumptions that SQRT=/ MLT. What assumptions do you need? (Hint: for the
reduction SQRT <-' MLT, follow the general lines of theorem 10.2.12 but use
Newton's method to find the positive zero of f (w) = w 2 -x ; for the inverse reduction,
use the fact that

x++- x l- x- x+l+ + 1=1/x.)
Let MOD be the problem of computing the remainder when
Problem 10.2.24.
an integer of size 2n is divided by an integer of size n . Prove that MOD <_ 1 MLT.
Let GCD be the problem of computing the greatest common
** Problem 10.2.25.
divisor of two integers of size at most n . Prove or disprove GCD = MLT. (Warning:
at the time of writing, this is an open problem.)


There exist many real-life, practical problems for which no efficient algorithm is
known, but whose intrinsic difficulty no one has yet managed to prove. Among these
are such different problems as the travelling salesperson (Sections 3.4.2, 5.6, and
6.6.3), optimal graph colouring (Section 3.4.1), the knapsack problem, Hamiltonian cir-

cuits (Example 10.3.2), integer programming, finding the longest simple path in a


Introduction to Complexity

Chap. 10

graph (Problem 5.1.3), and the problem of satisfying a Boolean expression. (Some of
these problems are described later.) Should we blame algorithmics or complexity ?
Maybe there do in fact exist efficient algorithms for these problems. After all, computer science is a relative newcomer: it is certain that new algorithmic techniques
remain to be discovered.
This section presents a remarkable result : an efficient algorithm to solve any one
of the problems we have listed in the previous paragraph would automatically provide
us with efficient algorithms for all of them. We do not know whether these problems
are easy or hard to solve, but we do know that they are all of similar complexity. The
practical importance of these problems ensured that each of them separately has been
the object of sustained efforts to find an efficient method of solution. For this reason it
is widely conjectured that such algorithms do not exist. If you have a problem to solve
and you are able to show that it is equivalent (see Definition 10.3.1) to one of those

mentioned previously, you may take this result as convincing evidence that your
problem is hard (but evidence is not a proof). At the very least you will be certain that
nobody else claims to be able to solve your problem efficiently at the moment.

10.3.1 The Classes P and NP

Before going further it will help to define what we mean by an efficient algorithm.
Does this mean it takes a time in O (n log n) ? O (n 2) ? O (n'-") ? It all depends on
the problem to be solved. A sorting algorithm taking a time in O(n2) is inefficient,
whereas an algorithm for matrix multiplication taking a time in 0 (n 2 log n) would be
an astonishing breakthrough. So we might be tempted to say that an algorithm is
efficient if it is better than the obvious straightforward algorithm, or maybe if it is the
best possible algorithm to solve our problem. But then what should we say about the
dynamic programming algorithm for the travelling salesperson problem (Section 5.6)
or the branch-and-bound algorithm (Section 6.6.3) ? Although more efficient than an
exhaustive search, in practice these algorithms are only good enough to solve instances
of moderate size. If there exists no significantly more efficient algorithm to solve this
problem, might it not be reasonable to decide that the problem is inherently intractable ?

For our present purposes we answer this question by stipulating that an algorithm
is efficient (or polynomial-time) if there exists a polynomial p (n) such that the algorithm can solve any instance of size n in a time in 0 (p (n)). This definition is
motivated by the comparison in Section 1.6 between an algorithm that takes a time in
O(2') and one that only requires a time in O (n 3 ), and also by sections 1.7.3, 1.7.4,
and 1.7.5. An exponential-time algorithm becomes rapidly useless in practice, whereas

generally speaking a polynomial-time algorithm allows us to solve much larger
instances. The definition should, nevertheless, be taken with a grain of salt. Given
two algorithms requiring a time in O(ntg'g" ) and in O(n 10), respectively, the first, not
being polynomial, is "inefficient". However, it will beat the polynomial algorithm on
all instances of size less than 10300 assuming that the hidden constants are similar. In
fact, it is not reasonable to assert that an algorithm requiring a time in O(n 10) is

Sec. 10.3

Introduction to NP-Completeness


efficient in practice. Nonetheless, to decree that O(n3) is efficient whereas S2(n4) is
not, for example, seems rather too arbitrary.

In this section, it is crucial to avoid pathological algorithms and analyses such as
those suggested in Problem 1.5.1. Hence no algorithm is allowed to perform arithmetic operations at unit cost on operands whose size exceeds some fixed polynomial in
the size of the instance being solved. (The polynomial may depend on the algorithm
but not of course on the instance.) If the algorithm needs larger operands (as would be
the case in the solution of Problem 1.5.1), it must break them into sections, keep them

in an array, and spend the required time to carry out multiprecision arithmetic.
Without loss of generality, we also restrict all arrays to contain a number of elements
at most polynomial in the size of the instance considered.
The notion of linear reduction and of linear equivalence considered in Section
10.2 is interesting for problems that can be solved in quadratic or cubic time. It is,
however, too restrictive when we consider problems for which the best-known algorithms take exponential time. For this reason we introduce a different kind of reduction.

Definition 10.3.1.
Let X and Y be two problems. Problem X is polynomially
reducible to problem Y in the sense of Turing, denoted X <_ T Y, if there exists an algo-

rithm for solving X in a time that would be polynomial if we took no account of the
time needed to solve arbitrary instances of problem Y. In other words, the algorithm
for solving problem X may make whatever use it chooses of an imaginary procedure
that can somehow magically solve problem Y at no cost. When X <_T Y and Y <_T X
simultaneously, then X and Y are polynomially equivalent in the sense of Turing,
denoted X -T Y. (This notion applies in a natural way to unsolvable problems - see
Problem 10.3.32.)

Example 10.3.1.
Let SMALLFACT(n) be the problem of finding the smallest
integer x > 2 such that x divides n (for n >_ 2), let PRIME(n) be the problem of determining whether n >_ 2 is a prime number, and let NBFACT(n) be the problem of counting
the number of distinct primes that divide n. Then both PRIME <_ T SMALLFACT and
NBFACT <_ T SMALLFACT. Indeed, imagine solutions to the problem SMALLFACT can be

obtained at no cost by a call on SolveSF ; then the following procedures solve the
other two problems in a time polynomial in the size of their operand.

function DecidePRiME(n)
we assume n _ 2}
if n = SolveSF (n) then return true
else return false
function SolveNBF (n)
nb - 0

while n > I do nh - nh + I
x - SolveSF (n)
while x divides n don f- n /x
return nh

the equivalence mentioned in the introduction to this section implies that either all the problems listed there can be solved in polynomial time. When no confusion can arise. Suppose there exists an algorithm that is able to solve problem Y in a time in O (t(n)). We also assume that the instances can be coded efficiently in the form of strings of bits. Let X and Y be two decision problems. given some x e 1. Prove that there exist a polynomial p (n) and an algorithm that is able to solve problem X in a time in O (p (n) t(p (n))). "Is n a prime number?" is a decision problem. A decision problem can be thought of as defining a subset X of the set I of all its instances. then X E P.J computable in polynomial time.3. . if there exists a function f :I .2. denoted by X <m Y. Problem Definition 10. Prove that if X <_ T Y and Y E P. denoted X = m Y. 10 Notice that SolveNBF works in polynomial time (counting calls of SolveSF at no cost) because no integer n can have more than Llg n ] prime factors.Introduction to Complexity 318 Chap. whereas "find the smallest prime factor of n " is not. For example. Prove that Problem 10. or none of them can.3. or "the set of all possible graphs". then X and Y are many-one polynomially equivalent.3. P is the class of decision problems that can be solved by a 0 polynomial-time algorithm.3.2. Problem 10. Problem 10. we may sometimes omit to state explicitly the set of instances for the decision problem under consideration.° Y and Y <_ m X both hold. whether or not x EX.3. We generally assume that the set of all instances is easy to recognize. where t(n) is a nonzero. such as IN. Then the problem consists of deciding. Let X and Y be two problems such that X <T Y. The restriction to decision problems allows us to introduce a simplified notion of polynomial reduction. known as the reduction function between X and Y. the existence of an algorithm to solve problem Y in polynomial time implies that there also exists a polynomial-time algorithm to solve problem X. such that (VXEI)[xEX f (x)EY] . In particular.3. even taking repetitions into account. Let X c I and Y c J be two decision problems.1.3. X is many-one polynomially reducible to problem Y. The usefulness of this definition is brought out by the two following exercises. nondecreasing function. Definition 10. For technical reasons we confine ourselves from now on to the study of decision problems. Let X and Y be two problems such that X <_ T Y Y. When X <_.

1 does not necessarily hold by giving explicitly two decision problems X and Y for which you can prove that X <_ T Y whereas it is not the case that X <_ m Y. Then the following procedure solves X in polynomial time. v) _ I if (u.6 and 6. then xSTY. Prove that the relations <_ T . If X and Y are two decision problems such that X <_m Y.3. function DecideX (x) y -f (X) if DecideY (y) then return true else return false * Problem 10.2 shows that the restriction to decision problems is not a severe constraint.3. the cost function c(u . Lemma 10.3. together with some bound L used to turn the travelling salesperson optimization problem (as in Sections into a decision problem : the question is to decide whether there exists a tour in the graph that begins and ends at some node. respectively.3. A > be a directed graph for which you would like to decide if it has a Hamiltonian circuit. G E HAM if and only if < H. most optimization problems are polynomially equivalent in the sense of Turing to an analogous decision problem. <_ m . Problem 10. Prove that the converse of Lemma 10. El T . c. and the question is to decide whether there exists a circuit in the graph passing exactly once through each node n (with no optimality constraint). The introduction of TSPD in Example 10. L > E TSPD. An instance of HAM is a directed graph. but this is significantly harder to prove. NxN >. 10. let G = < N.5. after having visited each of the other nodes exactly once.3. Proof. . as the following exercise illustrates. and whose cost does not exceed L. Let TSPD and HAM be the travelling salesperson decision problem and the Hamiltonian circuit problem.1. and = R.v)EA 2 otherwise and the bound L = #N.Sec.3 Introduction to NP-Completeness 319 Example 10. the number of nodes in G. An instance of TSPD consists of a directed graph with costs on the edges. are transi- tive. To prove that HAM 5 m TSPD. In fact.4.2. It is also the case that TSPD m HAM. Define f (G) as the instance for TSPD consisting of the complete graph H = <N. Clearly. Imagine solutions to problem Y can be obtained at no cost by a call on DecideY and let f be the polynomial-time computable reduction function between X and Y.

Introduction to Complexity


Chap. 10

* Problem 10.3.6.
Let G = < N, A > be an undirected graph, let k be an
integer, and c : N -* (1, 2, ... , k } a function. This function is a valid colouring of G
if there do not exist nodes u , v E N such that (u , v ) E A and c (u) = c (v) (Section
3.4.1). The graph G can be coloured with k colours if there exists such a valid

colouring. The smallest integer k such that G can be coloured with k colours is called
the chromatic number of G, and in this case a colouring with k colours is called an
optimal colouring. Consider the three following problems.
COLD: Given a graph G and an integer k, can G be coloured with k colours ?
COLO: Given a graph G, find the chromatic number of G.
COLC: Given a graph G, find an optimal colouring of G.

Conclude that there exists a polynomial-time
algorithm to determine the chromatic number of a graph, and even to find an optimal
Prove that COLD =T COLO =T COLC.

colouring, if and only if COLD E P.

These graph colouring problems have the characteristic that although it is perhaps

difficult to decide whether or not a graph can be coloured with a given number of
colours, it is easy to check whether a suggested colouring is valid.

Let X be a decision problem. Let Q be a set, arbitrary for
Definition 10.3.4.
the time being, which we call the proof space for X. A proof system for X is a subset

FcXxQ such that (`dxEX)(3gEQ)[<x,q>EF]. Any q such that <x,q >EF
is known as a proof or a certificate that x E X. Intuitively, each true statement of the
type x E X has a proof in F, whereas no false statement of this type has one (because if
x E X, there does not exist a q E Q such that < x, q > (=- F ).
Example 10.3.3.
Let I = N and COMP = { n I n is a composite integer }. We
can take Q = 1N as the proof space and F = ( < n, q > I 1 < q < n and q divides n
exactly } as the proof system. Notice that some problems may have more than one
natural proof system. In this example we could also use the ideas of Section 8.6.2 to


< n, q > I (n is even and n > 2) or
(1 < q < n and n is not a strong pseudoprime to the base q

which offers a large number of proofs for all odd composite numbers.

Example 10.3.4.
Consider the set of instances I = { < G , k > I G is an
undirected graph and k is an integer } and the problem COLD = { < G, k > E 1 I G
coloured with k colours }. As proof space we may take Q =
(c : N -* (1, 2, ... , k} I N is a set of nodes and k is an integer). Then a proof

can be

system is given by

Sec. 10.3


Introduction to NP-Completeness


< < G, k >, c > I G = < N, A > is an undirected graph,
k is an integer,
c : N -4 { 1, 2, ... , k } is a function and

(`du,vEN)[{u,v}EA =c(u)#c(v)] }


Problem 10.3.7. Let G = < N, A > be an undirected graph. A clique in G is
a set of nodes K E_ N such that { u, v } E A for every pair of nodes u, v K. Given a
graph G and an integer k, the CLIQUE problem consists of determining whether there
exists a clique of k nodes in G. Give a proof space and a proof system for this decision problem.

Definition 10.3.5.
NP is the class of decision problems for which there exists
a proof system such that the proofs are succinct and easy to check. More precisely, a
decision problem X is in NP if and only if there exist a proof space Q, a proof system
F c X x Q, and a polynomial p (n) such that

i. (VxEX)(agEQ)[<x,q>eFand IqI <p(Ix1)],
where I q I and I x I denote the sizes of q and x, respectively ; and
ii. F E P.

We do not require that there should exist an efficient way to find a proof of x when
x c X, only that there should exist an efficient way to check the validity of a proposed
short proof.

The conceptual distinction between P and NP is best grasped
Example 10.3.5.
with an example. Let comp be the problem in Example 10.3.3. In order to have
comp E P, we would need an algorithm
function DecideCOMP (n)
{ decides whether n is a composite number or not }

return true

return false

whose running time is polynomial in the size of n. No such algorithm is currently
known. However, to show that coMP E NP, we need only exhibit the following
(obvious) polynomial-time algorithm.

Introduction to Complexity


Chap. 10

function VerifyCOMP (n, q)

if 1 < q < n and q divides n then return true
else return false
By definition of NP, any run of VerifyCOMP (n, q) that returns true is a proof that n is

composite, and every composite number has at least one such proof (but prime
numbers have none). However, the situation is not the same as for a probabilistic
algorithm (Chapter 8) : we are content even if there exist very few q (for some
composite n) such that VerifyCOMP (n , q) is true and if our chance of hitting one at
random would be staggeringly low.

Problem 10.3.8.
Let X be a decision problem for which there exists a
polynomial-time true-biased Monte Carlo algorithm (section 8.6). Prove that X E NP.
(Hint : the proof space is the set of all sequences of random decisions possibly taken
by the Monte Carlo algorithm.)
Problem 10.3.9.
Prove that P E_ NP. (Hint : Let X be a decision problem
in P. It suffices to take Q = { 0 } and F = (< x, 0 > I x e X } to obtain a system of
"proofs" that are succinct and easy to check. This example provides an extreme illustration of the fact that the same proof may serve for more than one instance of the
same problem.)

Example 10.3.6.
The problems COLD and CLIQUE considered in example
10.3.4 and Problem 10.3.7 are in NP.
Although COLD is in NP and COLO =T COLD, it does not appear that NP contains
the problem of deciding, given a graph G and an integer k, whether k is the chromatic
number of G. Indeed, although it suffices to exhibit a valid colouring to prove that a
graph can be coloured with a given number of colours (Example 10.3.4), no one has
yet been able to invent an efficient proof system to demonstrate that a graph cannot be
coloured with less than k colours.

Definition 10.3.6.
Let X S I be a decision problem. Its complementary
problem consists of answering "Yes" for an instance x E I if and only if x 0 X. The
class co-NP is the class of decision problems whose complementary problem is in NP.
For instance, the preceding remark indicates that we do not know whether COLD E
co-NP. Nonetheless, we know that COLD E co-NP if and only if NP = co-NP (Problems 10.3.27 and 10.3.16). The current conjecture is that NP # co-NP, and therefore
that COLD


Problem 10.3.10.

Let A and B be two decision problems. Prove that if

A 5 m B and B E NP, then A E NP.

Problem 10.3.11.
Let A and B be two decision problems. Do you believe that
if A 5 T B and B E NP, then A E NP ? Justify your answer.

Sec. 10.3

Introduction to NP-Completeness


Show that HAM, the Hamiltonian circuit problem defined in
Problem 10.3.12.
Example 10.3.2, is in NP.


In 1903, two centuries after Mersenne claimed without proof
Example 10.3.7.
is a prime number, Frank Cole showed that
267_ 1

267 -1 = 193,707,721 x 761,838,257, 287


It took him "three years of Sundays" to discover this factorization. He was lucky that
the number he chose to attack is indeed composite, since this enabled him to offer a
proof of his result that is both short and easy to check. (This was not all luck : Lucas
had already shown in the nineteenth century that 267-1 is composite, but without
finding the factors.)
The story would have had quite a different ending if this number had been prime.
In this case the only "proof" of his discovery that Cole would have been able to produce would have been a thick bundle of papers covered in calculations. The proof
would be far too long to have any practical value, since it would take just as long to
check as it did to produce in the first place. (A similar argument may be advanced
concerning the "proof' by computer of the famous four colour theorem.) This results
from a phenomenon like the one mentioned in connection with the chromatic number
of a graph : the problem of recognizing composite numbers is in NP (Example 10.3.3),

but it seems certain at first sight not to be in co-NP, that is, the complementary
problem of recognizing prime numbers seems not to be in NP.

However, nothing is certain in this world except death and taxes : this problem
too is in NP, although the notion of a proof (or certificate) of primality is rather more
subtle than that of a proof of nonprimality. A result from the theory of numbers shows
that n, an odd integer greater than 2, is prime if and only if there exists an integer x
such that

0<x <n
x"- I = 1 (mod n ), and
xI" - I)lP 4P 1 (mod n) for each prime factor p of n -1

A proof of primality for n therefore consists of a suitable x, the decomposition of n -1
into prime factors, and a collection of (recursive) proofs that each of these factors is
indeed prime. (More succinct proof systems are known.)

Complete the proof sketched in Example 10.3.7 that the
* Problem 10.3.13.
problem of primality is in NP. It remains to show that the length of a recursive proof
of primality is bounded above by a polynomial in the size (that is, the logarithm) of the
integer n concerned, and that the validity of such a proof can be checked in polynomial

Introduction to Complexity


Chap. 10

Problem 10.3.14.
Let F = { < x , y > I x , y E N and x has a prime factor less
than y }. Let FACT be the problem of decomposing an integer into prime factors.
Prove that

i. F E NP n co-NP; and
ii. F = T FACT.

If we accept the conjecture that no polynomial-time factorization algorithm exists, we
can therefore conclude that F E (NP n co-NP) 1 P.

10.3.1 NP-Complete Problems

The fundamental question concerning the classes P and NP is whether the inclusion
P9 NP is strict. Does there exist a problem that allows an efficient proof system but
for which it is inherently difficult to discover such proofs in the worst case ? Our intuition and experience lead us to believe that it is generally more difficult to discover a
proof than to check it: progress in mathematics would be much faster were this not so.
In our context this intuition translates into the hypothesis that P # NP. It is a cause of
considerable chagrin to workers in the theory of complexity that they can neither prove
nor disprove this hypothesis. If indeed there exists a simple proof that P # NP, it has
certainly not been easy to find !

On the other hand, one of the great successes of this theory is the demonstration

that there exist a large number of practical problems in NP such that if any one of
them were in P then NP would be equal to P. The evidence that supports the
hypothesis P # NP therefore also lends credence to the view that none of these problems can be solved by a polynomial-time algorithm in the worst case. Such problems
are called NP-complete.
Definition 10.3.7.

A decision problem X is NP-complete if

I. X E NP ; and
ii. for every problem Y E NP, Y S T X.

Some authors replace the second condition by Y

n, X or by other (usually stronger)

kinds of reduction.

Prove that there exists an NP-complete problem X such that
Problem 10.3.15.
X E P if and only if P = NP.

Prove that if there exists an NP-complete problem X such
* Problem 10.3.16.
that X E co-NP, then NP = co-NP.

Sec. 10.3

Introduction to NP-Completeness


Prove that if the problem X is NP-complete and the

Problem 10.3.17.
problem Z E NP, then

i. Z is NP-complete if and only if X <T Z ;
ii. if X n, Z, then Z is NP-complete.

Be sure to work this important problem. It provides the fundamental tool for
proving NP-completeness. Suppose we have a pool of problems that have already
been shown to be NP-complete. To prove that Z is NP-complete, we can choose an
appropriate problem X from the pool and show that X is polynomially reducible to Z
(either many-one or in the sense of Turing). We must also show that Z E NP by exhibiting an efficient proof system for Z. Several thousand NP-complete problems have
been enumerated in this way.
This is all well and good once the process is under way, since the more problems

there are in the pool, the more likely it is that we can find one that can be reduced
without too much difficulty to some new problem. The trick, of course, is to get the
ball rolling. What should we do at the outset when the pool is empty to prove for the
very first time that some particular problem is NP-complete? (Problem 10.3.17 is then
powerless.) This is the tour de force that Steve Cook managed to perform in 1971,
opening the way to the whole theory of NP-completeness. (A similar theorem was
discovered independently by Leonid Levin.)

10.3.2 Cook's Theorem
Definition 10.3.8.
A Boolean variable takes its values in the set
B = { true, false }. Boolean variables are combined using logical operators (not, and,
or, t:* , r , and so on) and parentheses to form Boolean expressions. It is customary
to represent disjunction (or) in such expressions by the symbol "+" and conjunction
(and) by simply juxtaposing the operands (as for arithmetic multiplication). Negation
is often denoted by a horizontal bar above the variable or the expression concerned. A
Boolean expression is satisfiable if there exists at least one way of assigning values to
its variables so as to make the expression true. A Boolean expression is a tautology if
it remains true whatever values are assigned to its variables. A Boolean expression is a
contradiction if it is not satisfiable, that is, if its negation is a tautology. We denote by

SAT, TAUT and CONT, respectively, the problems of deciding, given a Boolean
expression, whether it is satisfiable, whether it is a tautology, and whether it is a con-



Example 10.3.8.
ables p and q

Here are three Boolean expressions using the Boolean vari-


i. (p+q)=pq
ii. (p b q) (p +q)(p +q )
iii. p (p +q)q

Introduction to Complexity


Chap. 10

Expression (i) is satisfiable because it is true if p = true and q = true, but it is
not a tautology because it is false if p = true and q = false . Verify that expression
(ii) is a tautology and that expression (iii) is a contradiction.
To prove that a Boolean expression is satisfiable, it suffices to produce an assignment that satisfies it. Moreover, such a proof is easy to check. This shows that
SATE NP. It is not apparent that the same is true of the two other problems : what
short and easy proof can one give in general of the fact that a Boolean expression is a
tautology or a contradiction? These three problems are, nevertheless, polynomially
equivalent in the sense of Turing.
Prove that

Problem 10.3.18.

i. SAT = T TAUT - T CONT ; and even

ii. TAUT -m CONT.
It is possible in principle to decide whether a Boolean expression is satisfiable by
working out its value for every possible assignment to its Boolean variables. However
this approach is impractical when the number n of Boolean variables involved is large,
since there are 2" possible assignments. No efficient algorithm to solve this problem is

Definition 10.3.9.
A literal is either a Boolean variable or its negation.
A clause is a literal or a disjunction of literals. A Boolean expression is in conjunctive
normal form (CNF) if it is a clause or a conjunction of clauses. It is in k-CNF for
some positive integer k if it is composed of clauses, each of which contains at most k
literals (some authors say : exactly k literals).

Example 10.3.9.

Consider the following expressions.

L (p +q +r)(p +q +r)q r
ii. (p +qr)(p +q (p +r))

W. (p =* q)b(p+q)
Expression (i) is composed of four clauses. It is in 3-CNF (and therefore in CNF), but

not in 2-CNF. Expression (ii) is not in CNF since neither p +qr nor p +q (p +r) is a
clause. Expression (iii) is also not in CNF since it contains operators other than conjunction, disjunction and negation.

*Problem 10.3.19.

i. Show that to every Boolean expression there corresponds an equivalent expression in CNF.

ii. Show on the other hand that the shortest equivalent expression in CNF can be
exponentially longer than the original Boolean expression.

for each bit i of memory that algorithm A may need to use when solving an instance of size n. More interestingly.Sec. Theorem 10.1.3.--. We already know that SAT-CNF is in NP. Thus it remains to prove that X <_ T SAT-CNF for every problem X E NP.20. Let Q be a proof space and F an efficient proof system for X. -The Boolean expression is constructed so that there exists a way to satisfy it by choosing the values of its other Boolean variables if and only if algorithm A accepts the instance corresponding to the Boolean value of the x variables.3. 10. El The interest of Boolean expressions in the context of NP-completeness arises from their ability to simulate algorithms.) We are finally in a position to state and prove the fundamental theorem of the theory of NP-completeness. the size of 'Y (A) is also polynomial in n). The proof that this Boolean expression exists and that it can be constructed efficiently poses difficult technical problems. It usually requires a formal model of computation beyond the scope of this book. x2 2 . algorithm A accepts at least one instance of size 5 if and only if T5(A) is satisfiable. X2. For example. SAT-CNF is the restriction of the SAT problem to Boolean expressions in CNF. SAT-k-CNF is the restriction of SAT-CNF to Boolean expressions in k-CNF. For every integer n there exists a Boolean expression 'P (A) in CNF that can be obtained efficiently (in a time polynomial in n.10. Clearly. Polynomial-time algorithms are known for a few of them. where the polynomial may depend on the algorithm A . We content ourselves with mentioning that the expression 'P (A) contains among other things a distinct Boolean variable b. (The number of additional Boolean variables is polynomial in the size of the instance because the algorithm runs in polynomial time and because we have assumed without loss of generality that none of its variables or arrays can ever occupy more than a polynomial number of bits of memory. The problems TAUT-(k-)CNF and 11 CONT-(k-)CNF are defined similarly. Once the variables x I .3. among which x correspond in a natural way to the bits of instances of size n for A. For any positive integer k.. polynomial time. This Boolean expression contains a large number of variables. x i . Consider an arbitrary decision problem that can be solved by a polynomial-time algorithm A. the clauses of 'P (A) force these Boolean variables to simulate the step-by-step execution of the algorithm on the corresponding instance. and for each unit t of time taken by this computation.x are fixed. SAT-CNF is NP-complete. Prove that SAT-2-CNF and TAUT-CNF can be solved in * Problem 10. all these problems are in NP or in co-NP. Suppose that the size of the instances is measured in bits. Proof. algorithm A accepts the instance 10010 if and only if the expression xix2x3xax5`P5(A) is satisfiable.3 Introduction to NP-Completeness 327 Definition 10. Let p (n) be the polynomial (given by the definition of NP) such . such as the Turing machine.

and let q e Q be a proof of size i that x E X.) Problem 10. function Ax. (q ) if A (< x. there corresponds a two variable polynomial r such that the Boolean formula Ti (Ax) can be constructed in a time in 0 (r (I x I. The reduction is immediate because Boolean expressions in CNF . We already know that SAT E NP. This is an instance of size i. which is part of the definition of X E NP. q > belongs to F ). given < x. if we imagine that answers concerning the satisfiability of certain Boolean expressions in CNF can be obtained at no cost by a call on DecideSATCNF.3 Some Reductions We have just seen that SAT-CNF is NP-complete. Prove that in fact X <_ m SAT-CNF for any decision problem 10. q > implies that algorithm Ax accepts q. Let A be a polynomial-time algorithm able to decide. That such an algorithm exists is ensured by the fact that F E P. 10 that to every x E X there corresponds a q E Q whose length is bounded above by p (I x I) such that < x. To show that X too is NP-complete.21. q > as input. (To be precise.Introduction to Complexity 328 Chap. The fact that algorithm A accepts < x. SAT is NP-complete. and therefore Ax accepts no inputs.3. let Ax be an algorithm whose purpose is to verify whether a given q E Q is a valid proof that x E X.3. then there exists no q e Q such that < x.3. Example 10. as DecideX (x) will discover. It therefore remains to show that SAT-CNF <_ T SAT. For each x.O to p (n) do if DecideSATCNF (Ti (Ax)) then return true return false Let X E X be of size n.3. whether < x.10. XENP. Thereafter. one technical detail is worth mentioning : to each algorithm A. whether q is a proof that x E X (that is. q >) then return true else return false Here finally is an algorithm to solve problem X in polynomial time. q > E F. where 0 <_ i <_ p (n). i )) and such that its size is bounded similarly. if x 0 X. Conversely. q > E F. we need only prove that SAT CNF <_ T X (Problem 10. Let X E NP be some other decision problem. which implies that DecideX (x) will find no integer i such that Ti (Ax) is satisfiable.17). We illustrate this principle with several examples. function DecideX (x) let n be the size of x (in bits) for i . we have the choice of proving SAT-CNF <_ T Y or X <_ T Y. to show that Y E NP is NP-complete. This completes the proof that X <_ T SAT-CNF. hence the Boolean expression Ti (Ax) is satisfiable.

treat each of them independently (using different new variables for each clause) and form the conjunction of all the expressions in 3-CNF thus obtained. If k = P. 3. More generally. resist the temptation to use Problem 10. let 1 . 1 1 =(11+12+u1)(u1+13+u2)(u2+14+u3) (uk_3+lk-1+lk). can be exponentially longer than expression T. which is therefore a disjunction of k literals. .3.19(i) to obtain the algorithm function DecideSAT(Y) let E be a Boolean expression in CNF equivalent to Y if DecideSATCNF (E. in 3-CNF that is satisfiable if and only if P is satisfiable.3.11) If `P=(p+q+r+s)(r+s)(p+s+x+v+u) we obtain E=(p+q+u1)(u1+r+s)('+s)(p+s+u2)(u2+x+u3)(u3+v+w) . Example 10.. More precisely. Let P be a Boolean expression in CNF.22.12. SAT-3-CNF is NP-complete. . lk be the literals such that `P is +lk . (Hint : This problem has a very simple solution.3. Using Example 10. Our problem is to construct efficiently a Boolean expression i. Problem 10.10. 12. if k >_ 4.3. (Continuation of Example 10.3 Introduction to NP-Completeness 329 are simply a special case of general Boolean expressions. . conclude that SAT= T SAT-CNF. . if we imagine that the satisfiability of Boolean expressions can be decided at no cost by a call on DecideSAT. 10.Sec. If the expression `P consists of several clauses.19(ii) shows that expression i. l3 ..3. function DecideSATCNF(`Y) if P is not in CNF then return false if DecideSAT (P) then return true else return false Prove that SAT <_ T SAT-CNF. u2 . It remains to show that SAT CNF <_ T SAT-3-CNF. This time let us show that SAT-CNF <_ m SAT-3-CNF. set If k = 4. which is already in 3-CNF. Let u be a new Boolean variable. Consider first how to proceed if `P contains only one clause. Take = (l 1 + 12 + u) (i + 13 +14)iii.3. 12. i. Take l 1 +12+ ii. so it cannot be computed in polynomial time in the worst case.3..) then return true else return false because Problem 10.) Example 10.. here is a polynomial-time algorithm for solving SAT-CNF.11. However. and 14 be the literals such that `P is l 1 + 12 + 13 + l4 . let l . . uk_3 be new Boolean variables. Let u 1 . We have already seen that SAT-3-CNF is in NP.

13.-y. xi is false if it is zi that is the same colour as T. .Introduction to Complexity 330 Problem 10. we shall prove this time that SAT-3-CNF <_ m 3-COL.1. One copy of this widget is added to the graph for each clause in T. For each Boolean variable x. Suppose for simplicity that every clause of the expression `P contains exactly three literals (see Problem 10.24. 3-COL is NP-complete.3. It is easy to see that 3-COL E NP. corresponds to an assignment of Boolean values to x t . . The problem k-COL consists of determining whether G can be coloured with k colours (see Problem 10.3. Given a Boolean expression `P in 3-CNF. imagine that the colours assigned to T and F represent the Boolean values true and false. respectively. Prove that the construction of Example 10. The control triangle. The graph G that we are about to build contains 3 + 2t + 6k nodes and x2 2 .3. is the same colour as T. Y2.3.2 shows the part of the graph that we have constructed up to now. and C.3. Figure 10. See Figure 10.1. . of `P the graph contains two nodes yj and zi that are linked to each other and to the control node C. Figure 10.3. The colour used for node C will be a control colour. -x.11. Example 10.6). Three distinguished nodes of this graph are linked in a triangle : call them T.--3 + 3t + 12k edges. for example. To show that 3-COL is NP-complete. corresponds to the literal x. In any colouring of G in three colours. Contrariwise. we have to construct efficiently a graph G that can be coloured in three colours if and only if T is satisfiable. that satisfies every clause. F. This can be accomplished thanks to the widget illustrated in Figure 10. These are added in such a way that the graph will be colourable with three colours if and only if the choice of colours f o r y l . x.3.3.3. This reduction is considerably more complex than those we have seen so far. If y. Prove that SAT-CNF <_ m SAT-3-CNF still holds even if in the definition of SAT-3-CNF we insist that each clause should contain exactly three literals. X2 . If t = 3. 10 is satisfiable if and only if P is satisfiable in Problem 10. and node zi corresponds to the literal x. When the time comes to colour G in three colours. think of this intuitively as an assignment of the value true to the Boolean variable x. Suppose further without loss of generality that the Boolean variables appearing in `P are x t .24).3. this forces yj to be the same colour as either T or F and zi to be the complementary colour. Let k be the number of clauses in T. . Chap. In every case node y.3. We still have to add 6 nodes and 12 edges for each clause in T. Let G be an undirected graph and k an integer constant.23.

3. A widget. and F if and only if at least one of the nodes 1. since the colour assigned to node T represents true. 2 and 3 cannot be the same colour as C.3. and 3 is coloured with the same colour as node T. Because these input nodes 1. shows that the widget can be coloured with the colours assigned to C. In other words.Sec. the widget simulates the disjunction of the three literals represented by the nodes to which it is joined. 10. and three nodes chosen from the y. T. so as to correspond to the three literals of the clause concerned.3 Introduction to NP-Completeness Figure 10. 331 Graph representation of three Boolean variables. T Figure 10.3. and z. Problem 10. . Each widget is linked to five other nodes of the graph : nodes C and T of the control triangle.

Give a simple proof that 4-COL is NP-complete. It is clear that the graph can be constructed efficiently starting from the Boolean expression `I' in 3-CNF. Prove that CLIQUE (Problem 10. that 2-COL is in P. 10 This ends the description of the graph G.3. (Hint : prove that SAT-3-CNF <_ m CLIQUE.3. * Problem 10. Problem 10.30. 10. Prove however that there exist decision problems that are not polynomially reducible to the halting problem in the sense of Turing.3. whether the latter will ever halt when started. We conclude that SAT-3-CNF <_ m 3-COL. if and only if at least one of the input nodes is coloured with the same colour as node T (knowing that the input nodes cannot be coloured with the same colour as node C).7) is NP-complete. we do not do here). . Show.4 Non-determinism The class NP is usually defined quite differently. Prove that COLD (Problem 10. given any program as instance. The halting problem consists of deciding.) ** Problem 10.28. Problem 10. which we only sketch here. ** Problem 10. although the definitions are equivalent. Verify that the colours attributed to nodes C.31.3. Problem 10. T. The name NP arose from this other definition: it represents the class of problems that can be solved by a Non-deterministic algorithm in Polynomial time.3. Prove that the problem of the Hamiltonian circuit (Example 10.3. Prove that 3-COL is still NP-complete even if we restrict ourselves to planar graphs of degree not greater than 4.12) is NP-complete. on the other hand. * Problem 10. * Problem 10.6) is NP-complete. which can be coloured with three colours if and only if `i is satisfiable.3.332 Introduction to Complexity Chap. and F suffice to colour the widget of Figure 10.26. The notion of reducibility extends in a natural way to unsolvable problems such as the halting problem (although it is usual to drop the polynomial-time aspect of the reductions .2 and Problem 10.3.3. A function f : IN -* IN is polynomially bounded if the size of its value on any argument is polynomially bounded by the size of its argument. The classic definition involves the notion of non-deterministic algorithms. Prove that the problem of computing any polynomially bounded computable function is polynomially reducible to the halting problem in the sense of Turing.27. and therefore that 3-COL is NP-complete.

but instead. The domain of the algorithm is the set of instances on which it has at least one computation. hence to returning a solution. although nondeterministic algorithms can solve NP-complete problems in polynomial time. j) as in Chapter 8. 10.bool return . then y is a correct solution to instance x . it computes a well-defined function on the instances of its domain.11. A computation of the algorithm is a sequence of nondeterministic choices that leads to setting success to true. we write return success . (This explains why. a non-deterministic algorithm resembles a Las Vegas probabilistic algorithm (Section 8.Sec. Notice that there is no limit on how long a polynomial-time non-deterministic algorithm can run if the "wrong" non-deterministic ..bool as an abbreviation for success . The algorithm is total if its domain is the set of all possible instances. where y and success are return parameters. For this reason non-deterministic algorithms are only a mathematical abstraction that cannot be used directly in practice : we would not program such an algorithm in the hope of running it successfully and efficiently on a computer. nor is it subject to the laws of probability.3. Definition 10. The time taken by a non-deterministic algorithm on a given instance of its domain is defined as the shortest possible time that any computation can cause it to run on this instance . success). otherwise no solution is obtained. The difference between non-deterministic and Las Vegas algorithms is that for the former we do not set any bound on the probability of success.3 Introduction to NP-Completeness 333 On the surface. we do not denote nondeterministic choices by calls on uniform (i . we use a special instruction choose n between i and j . this does not imply that P = NP. The actual value assigned to n is not specified by the algorithm. The effect of the algorithm is determined by the existence or the nonexistence of sequences of non-deterministic choices that lead to the production of a result.5). We are not concerned with how such sequences could be determined efficiently or how their nonexistence could be established. The algorithm is consistent if two different computations on any given instance always lead to the same result. For simplicity. If the algorithm sets success to true. we call ND (x . It is even allowable that for some instances the algorithm will never set success to true.) To avoid confusion with probabilistic algorithms. the time is undefined on instances that are not in the domain of the algorithm. To solve instance x of some problem X. whose effect is to set n to some value between i and j inclusive. y . A non-deterministic algorithm runs in polynomial time if the time it takes is bounded by some polynomial in the size of its instance provided that the instance is in the domain of the algorithm. When a non-deterministic algorithm is consistent.

Consider the following total consistent non-deterministic primality testing algorithm (recall from Section 4.7 } if dexpo (x. procedure primeND (n . var success) { non-deterministically determines whether n is prime } if n < 3 or n is even then prime F. Notice again that it would be pointless to attempt implementing this algorithm on the computer by replacing the choose instructions by random choices in the same interval : the probability of success would be infinitesimal. (n -1)/p . Clearly.Introduction to Complexity 334 Chap.true choose guess between 0 and 1 if guess = 0 then { the guess is that n is composite . var prime. pr. n >. and guessing successively each prime divisor of n -1. n) computes x' mod n efficiently).3 or when n is even.true else success .false else { the guess is that n is prime . It is also consistent by the same theorem from Example 10. the algorithm also has a computation when n <.7. the algorithm has a computation that consists of choosing guess =0 and non-deterministically setting m to some nontrivial factor of n. When n is prime.false choose m between 2 and n -1 if m divides n then success .let's guess a proof ! } prime . choosing x in accordance with the theorem mentioned in example 10.3. n >.3.true 1 When n is an odd composite number.1. n .true choose x between 1 and n -1 { the guess is that x is as in Example 10. i .false success f. the algorithm also has a computation that consists of choosing guess = 1. Example 10.8 that dexpo (x .false m f-n -1 while m > 1 do choose p between 2 and m { the guess is that p is a new prime divisor of n -1 } primeND (p. It is even possible for a computation to be arbitrarily long.(2<. The algorithm primeND is therefore total. 10 choices are made. suc) if suc and pr and p divides m and dexpo (x.5.m /p else return success f.14. n) then while p divides m do m .3.9.let's guess a proof ! } prime .n < 3) return success f. provided that the same instance also admits at least one polynomially bounded computation. . n) # 1 then return success f.7.3.

3 335 * Problem 10. q > E F then ans . Although the authors prefer the definition based on proof systems. 10. In this case.empty binary string for i . let Q be its proof space. (Hint: use the set of all possible computations as proof space. Show. it makes Problems 10.1 to l do choose b between 0 and 1 append bit b to the right of q if < x.3.3. and the corresponding parameter may be ignored altogether (there is no point in algorithm XND setting ans to true when it finds a q such that <x. this problem belongs to NP. it is sometimes easier to show that a problem belongs to NP with this other definition.q > EF ). var ans. var success) n .3.true. however.35 Prove the converse of Theorem 10.2 Every decision problem in NP is the domain of some polynomial-time non-deterministic algorithm. .9 completely obvious. let F c X x Q be its efficient proof system. Let X E NP be a decision problem.3.8 and 10. procedure XND (x.) Proof.2: whenever a decision problem is the domain of some polynomial-time non-deterministic algorithm.33.Introduction to NP-Completeness Sec.3.) The preceding theorem and problem suggest the alternative (and more usual) definition for NP: it is the class of decision problems that are the domain of some polynomial-time non-deterministic algorithm. success .false 11 Problem 10.3. * Problem 10. The following polynomialtime non-deterministic algorithm has a computation on instance x if and only if x E X . For instance. we are only concerned with the existence or nonexistence of computations (usually called accepting computations in this context) . Prove that a decision problem can be solved by a total con- sistent polynomial-time non-deterministic algorithm if and only if it belongs to NP c co-NP.3. (The algorithm can easily be made consistent but it will not in general be total. that there are sequences of non-deterministic choices that can cause the algorithm to run for a time exponential in the size of its instance. Theorem 10. and let p be the polynomial that bounds the length of a proof as a function of the length of the corresponding positive instance. the actual result returned by the algorithm in case of success is irrelevant.size of x choose l between 0 and p (n) q . Prove that the non-deterministic algorithm primeND of Example 10.true else success .3.14 runs in a time polynomial in the size of its instance. Assume for simplicity that Q is the set of binary strings.34.

11. the original statement from Cook (1971) is that X <_ T TAUT-DNF for every X E NP.2. and 10. where m is the sum of the sizes of the elements to be sorted. and Faradzev (1970) present an algorithm to calculate fg using a number of Boolean operations in 0 (n 3/log n).5 is due to Fischer and Meyer (1971) and Theorem 10.28 can be found in Stockmeyer (1973).12 is solved in Fischer and Meyer (1971).16 are from Brassard (1979). The theory of NP-completeness originated with two fundamental papers : Cook (1971) proves that SAT-CNF is NP-complete.3.9 and 4.16). The uncontested authority in matters of NP-completeness is Garey and Johnson (1979). Borodin and .4 REFERENCES AND FURTHER READING For an overview of algorithms for sorting by transformation consult Aho.3.2.8 to 4.+-). 10 10. This chapter has done little more than sketch the theory of computational complexity. comes from Cook and Aanderaa (1969). more succinct primality certificates can be found in Pomerance (1987). and Ullman (1974). Hopcroft.2.22). The reduction INV <_/ MLT (Problem 10. Dinic. The reduction IQ <_/ MQ (Problem 10.2.12. For further information concerning the topics of Section 10. A good introduction can also be found in Hopcroft and Ullman (1979). and Ullman (1974). For modifications of heapsort that come close to being optimal for the worstcase number of comparisons. the lexicographic sorting algorithm can sort n elements in a time in O(n +m).3. Pippenger (1978) describes a method similar to decision trees for determining a lower bound on the size of logic circuits such as those presented in Problems 4.2. and Karp (1972) underlines the importance of this notion by presenting a large number of NP-complete problems. In practice.11. Problems 10.14 and 10. In the case of cost functions whose range is restricted to (0.3.336 Introduction to Complexity Chap.9) comes from Bunch and Hopcroft (1974). this technique shows that the required solutions for Problems 4. 1987). Problem 10. Sahni and Horowitz (1978). the fact that a problem is NP-complete does not make it go away.3. Part of the solution to Problem 10.14.7. Kronrod. however.3. In particular.7 and 10. Theorem 10. where TAUT-DNF is concerned with tautologies in disjunctive normal form .2.33. and Ullman (1974). see Hopcroft and Ullman (1979). A similar theory was developed independently by Levin (1973). To find out more about non-determinism. An algebraic approach to lower bounds is described in Aho.12 are optimal. Arlazarov. which is crucial in the proof of Theorem 10.2. To be historically exact.3. However in this case we have to be content with heuristics and approximations as described in Garey and Johnson (1976).3.11.6 is due to Furman (1970). consult Aho.2. In his tutorial article. then an algorithm that is asymptotically more efficient than the naive algorithm for calculating fg is given in Fredman (1976). and Horowitz and Sahni (1978). Hopcroft. The fact that the set of prime numbers is in NP (Examples 10. In particular. 10. it should be noted that TAUT-DNF is probably not NP-complete since otherwise NP = co-NP (Problem 10.34) was discovered by Pratt (1975). If f and g are two cost functions as in section 10.3. consult Gonnet and Munro (1986) and Carlsson (1986.13. Several important techniques have gone unmentioned.11. Hopcroft.2. and Problems 10.

Gardner and Bennett (1979). For an introduction to adversary arguments (Problem 8. These can be solved in theory. as described in Aho. 10.4).3. As mentioned in Problem 10.Sec. Hopcroft and Ullman (1979). and Winograd (1980).4. consult Horowitz and Sahni (1978). whatever the resources available. read Turing (1936). and Brassard and Monet (1982). there exist problems that are intrinsically difficult.32. . and Ullman (1974). but it can be proved that no algorithm can solve them in practice when the instances are of moderate size.4 References and Further Reading 337 Munro (1975). Although we do not know how to prove that there are no efficient algorithms for NP-complete problems. there also exist problems that cannot be solved by any algorithm. Hopcroft. even if it is allowed to take a time comparable to the age of the universe and as many bits of memory as there are elementary particles in the known universe (Stockmeyer and Chandra 1979).

2.3 arithmetic and matrix multiplication . and b.7182818. Cartesian product pointer assignment return parameter of a procedure or function dynamic end of a procedure dynamic end of a function with value v returned size of instance x .2. e. respectively basis of the natural logarithm : 2.. In. extended to polynomials in Section 10.. n! > 1) number of combinations of k elements chosen among n 338 . [lg(1+x)l if x is an integer. interval of integers : { k E IN I i <_ k <_ j } arithmetic quotient and modulo . mod x T E- var x return return v IxI number of elements in array T . n factorial (0! = 1 and n! =n x (n -1)!. logb e n! absolute value of x logarithm in basis 10. log.Table of Notation #T i j div. lg. cardinality of set T.

b > (in particular : edge of a directed graph) . j) x =y (mod n) xO+y floor of x : largest integer less than or equal to x extended to polynomials in Section 10.1. and Booleans (see Section 2. b) 339 ordered pair consisting of elements a and b same as < a.b > (a. set of elements such that set inclusion (allowing equality) strict set inclusion set union: AuB={x IxEA or xEB { set intersection : A nB = { x I x E A and x E B I set membership set nonmembership set difference : A\ B={ x IxEA and x 0 B empty set sets of integers. open interval {x E IR I a <x < b [ closed interval {xeIR I a <-x <_b } set of real numbers larger than or equal to a denotes comments in algorithms . bit-by-bit exclusive-or for bit strings .Table of Notation <a.. P (x) asymptotic notation (see Section 2.1) f is a function from A to B there exists an x such that P (x) there exists one and only one for all x.1) implies if and only if summation integral plus or minus derivative of the function f (x) 00 infinity limit of f (x) when x goes to infinity rx lg* uniform (i .2.3 ceiling of x : smallest integer larger than or equal to x iterated logarithm (see page 63) randomly and uniformly selected integer between i and j x is congruent to y modulo n (n divides x .y exactly) exclusive-or of x and y for bits or Booleans . reals.

Table of Notation 340 and.(a) Fw' (a) A B A B A .B A is many-one polynomially reducible to B A PB A is many-one polynomially equivalent to B P NP co-NP choose class of decision problems that can be solved in polynomial time class of decision problems that have an efficient proof system class of decision problems whose complementary problem is in NP instruction for non-deterministic choice . or A <_ T B Boolean conjunction and disjunction Boolean complement of x Fourier transform of vector a with respect to co inverse Fourier transform A is linearly reducible to B A is linearly equivalent to B A is polynomially reducible to B in the sense of Turing A =T B A is polynomially equivalent to B in the sense of Turing x F(.

J.. AHO. A. Communications of the ACM. Proceedings of 19th Annual ACM Symposium on the Theory of Computing. L. and E.. MANDERS. and R. ADLEMAN. 18(6). D. KOMLOS. 118-133. 173-206. 263-266. The Design and Analysis of Computer Algorithms. and J. Proceedings of 15th Annual ACM Symposium on the Theory of Computing. 115-132. POMERANCE. M. L. AHO. HOPCROFT.. 117. 175-178. M. Proceedings of 18th Annual IEEE Symposium on the Foundations of Computer Science. E. M. D. 462-469. SIAM Journal on Computing. J. KILIAN (1987). "On hiding information from an oracle". M. M. "An O(n logn) sorting network". Wynkyn de Worde. Data Structures and Algorithms.. A. Reading. and E. 333-340. Addison-Wesley. W. A. "On finding lowest common ancestors in trees". Doklady Akademii Nauk SSSR. and M. A. ACKERMANN. S. pp. ULLMAN (1976). ADLEMAN... ADEL'SON-VEL'SKII. E. "On distinguishing prime numbers from composite numbers". ULLMAN (1974). V. J. CORASICK (1975). and J. and J. Reading. AHO. "Recognizing primes in random polynomial time". LANDIS (1962). AJTAI. MA.Bibliography ABADI. ADLEMAN. V. 1-9. and G. J. ANON.. pp. 146. K. MILLER (1977). 195-203. A. (1928).-D. M. V. HUANG (1987). "Zum Hilbertschen Aufbau der reellen Zahlen". pp. M. and M. "Efficient string matching : An aid to bibliographic search". "On taking roots in finite fields". "An algorithm for the organization of information" (in Russian). 1495) Lytell Geste of Robyn Hode. pp. J. (c. ULLMAN (1983). RUMELY (1983). FEIGENBAUM. V. 5(1). Mathematische Annalen. G. D. and J. L. HOPCROFT. SZEMEREDI (1983). 99.E. London. Proceedings of 19th Annual ACM Symposium on the Theory of Computing. Addison-Wesley. C. J. AHO. MA. HOPCROFr. Annals of Mathematics. 341 .

Research report. BERLINER. Doklady Akademii Nauk SSSR. BLUM. R. (1968). M.H. P. BRASSARD. BERGE. C. Princeton. Universite de Montreal. "A simple unpredictable pseudo-random number generator". SIAM Journal on Computing. Communications of the ACM. "The traveling salesman problem: A survey". L. C. J. (1984). and I. D. GOUTIER. E. (1894). "Analysis of a randomized data structure for representing ordered sets". Graphes et hypergraphes. E. "On computing polynomials in one variable with initial preconditioning of the coefficients". Dynamic Programming. 194. and Computing. "Programming pearls : Algorithm design techniques". J. (1978). M. B. L. 364-372. 1(1). 364-383.. no. Addison-Wesley. "Privacy amplification through public discussion". (1979). "Divide-and-conquer in multidimensional space". BELLMAN. MA. and I. translated as: The Theory of Graphs and Its Applications (1962).G. 79-10. C. Departement de mathematiques et de statistique. ROBERT (1988). 12(3). . 15(2). "The generation of random numbers that are probably prime". D. CR£PEAU. 27(9). HAKEN.. D. 7-15. 1143-1154. L. J. "A general method for solving divide-andconquer recurrences".G. Control. E. C. (1957). second edition.S. J. and J. NJ. SHAMOS (1976). 2: Die Analytische Zahlentheorie. (1980). E. 713-735. BACH. and G. Leipzig. B.. Dunod. Journal of Cryptology. NEMHAUSER (1968). SAXE (1980). and J. 205-220. DINIC. "On economical construction of the transitive closure of a directed graph" (in Russian). KRONROD. NJ. North Holland. A. BENTLEY. BATCHER. H. BABAI. SHUB (1986). (1970). translated as : Graphs and Hypergraphs (1973). Dunod. P. Princeton University Press. BERLEKAMP. DREYFUS (1962).-M. pp.. Zahlentheorie.. 24(111). Princeton. BENNETT. BENTLEY. E.342 Bibliography ARLAZAROV. "Sorting networks and their applications". and J. BELLMAN. Proceedings of 19th Annual Allerton Conference on Communication. F.M. SIAM Journal on Computing. second edition. (1970). Methuen & Co. and M. perfect numbers and factoring". vol. 1967. 307-314. London. M. and J. M. Problemi Kibernetiki. "Backgammon computer program beats world champion". pp. BEAUCHEMIN. 5. E. BENTLEY. L. SHALLIT (1986). 538-558. "Factoring polynomials over large finite fields". Proceedings of 8th Annual ACM Symposium on the Theory of Computing. 16(3). L. SIAM Journal on Computing. "Monte Carlo algorithms in graph isomorphism techniques". BACHMANN. 1987. Computer Algorithms: Introduction to Design and Analysis. Applied Dynamic Programming. L. SIGACT News. Reading. S. Paris . C. Proceedings of AFIPS 32nd Spring Joint Computer Conference. R. BERGE. J. FARADZEV (1970).. and C. 865-871. C. Paris. BELLMORE.. A. MILLER. (1958). BRASSARD. G. (1961). A. BENTLEY. Operations Research. Teubner. G. pp. Mathematics of Computation. POMERANCE (1988). BLUM. Artificial Intelligence. 14. H. 15(4). in press. ACM. K. 36-44. Amsterdam. BAASE. L. STANAT. V. Princeton University Press. in press. R. and S. STEELE (1981). 220-230. "Sums of divisors. Theorie des graphes et ses applications. BELAGA. 487-488. G. E..

J. V. and J. G. CHANG. 2-17. (1975). L. 418-422. Lund. 231-236. Information Processing Letters. New York. MICAU (1984). "A note on the complexity of cryptography". HOPCROFT (1974). L. R. MONET. MUNRO (1971).. The Computational Complexity of Algebraic and Numeric Problems. Publica- tion no. 37-58. Academic Press. IEEE Transactions on Information Theory. J. Mathematics of Computation. G. (1987). BORODIN. 28(125). Canada. 1987. S. and M. 27. M.5). and J. Graph Theory: An Algorithmic Approach. FLOYD. Department of Computer Science. and R. C. MOORE (1977). Sweden. 850-864. (1971). "Triangular factorization and inversion by fast matrix multiplication". R. 143-154. CHRISTOFIDES. W. WEGMAN (1979). "Crusade for a better notation". . S. and J.R. Springer-Verlag. L. 18(2). "The generation of random permutations on the fly". SIGACT News. "How to generate cryptographically strong sequences of pseudo-random bits". "The towers of Hanoi problem". The Fast Fourier Transform. NY. Lecture Notes in Computer Science. E. Journal of Computer and System Sciences. R. and S. and L. Praca Morarske Prirodovedecke Spolecnosti. Analyse numerique. CARLSSON. D. BRASSARD. and D. E. second edition. LEVY (1980). S. BRASSARD. 10(4. RIVEST. PRATT. Doctoral dissertation. and J. 445. "Universal classes of hash functions". 448-461. and S. (1985). 66-68. BRATLEY. 89-102. Bit. "L'arithmetique des tres grands entiers". "Evaluating polynomials at many points". 5(4). (1926). Lidec. and S.Bibliography 343 BLUM. BUNCH. NY. BRASSARD. S. Fox. NY.. G. SIAM Journal on Computing. 60-64. BORUVKA. CODEN: LUNFD6/(NFCS-I(X)3)/(1-70)/(1986). 724-742. E. Montreal. 23(3). NJ. TSI: Technique et Science Informatiques. IT-25(2). 762-772. CARLSSON. G. "Canonical coin changing and greedy solutions". SIAM Journal on Computing. BUNEMAN. (1979).. B. ZUFFELLATO (1986). Heaps. Quebec. BORODIN. KORSH (1976). (1988). and R. NY.O. BOYER. P. Universite de Montreal. E. S. 1. KANNAN (1988). 7(4). "Time bounds for selection". 20(10). Information Processing Letters. G. (1974). New York. BRASSARD. L. Springer-Verlag. 243-244. TARJAN (1976). 5(2). CHERITON. (1986). A. E. Modern Cr_yptology: A Tutorial. Journal of Computer and System Sciences. B. O. "Finding minimum spanning trees". New York. and L. Information Processing Letters (in press). 3. TARJAN (1972). ACM. G. 1. MONET (1982). CARASSO. "L'indecidabilite sans larme (ni diagonalisation)". P. Departement d'informatique et de recherche operationnelle. 232-233. "Average case results on heapsort". SCHRAGE (1983). "O jistem problemu minimalnim". and J. New York. CARTER. MUNRO (1975). Prentice-Hall. BLUM. 13(4). B. M. BRASSARD. 17(1). BRIGHAM. Communications of theACM. A. 1(2). "A fast string searching algorithm". A Guide to Simulation. American Elsevier. Journal of the ACM. N. BRASSARD. N. Englewood Cliffs. Lund University.

K. AANDERAA (1969). (1959). Carnegie-Mellon University. DEMARS. North Holland. Numerische Mathematik. N. J. New York. (1971). Scientific American. (1980). Graph Algorithms. 259-279. J. 1-6. 1. in Symposium on Monte Carlo Methods. J. Proceedings of the IEEE. M. NJ. 48(177). "On the minimum complexity of functions". Mathematics of Computation. E. Proceedings of 19th Annual ACM Symposium on the Theory of Computing. E. "New directions in cryptography". and C. L. Management Sciences Research Report no. W. DROMEY. POMERANCE (1986). 142. (1984). L. "Transformee de Fourier rapide". 155-159. WINOGRAD (1987). A. W. ed. (1961). (1981). "The complexity of theorem-proving procedures". and C. H. MD. S. J. A. 251(5). and P. NY. and S. "Some improvements in practical Fourier analysis and their application to X-ray scattering from liquids". (1983). Addison-Wesley. 291-314. (1977). LANCZOS (1942). Proceedings of 3rd Annual ACM Symposium on the Theory of Computing. 435-452. "CRAY-2 computer system takes a slice out of pi". 36(153). Pittsburgh. IEEE Transactions on Information Theory. Asymptotic Methods in Analysis. "A note on two problems in connexion with graphs". Non-Uniform Random Variate Generation. . (1956). 19-28. Amsterdam. 19(90). John Wiley & Sons. TUKEY (1965). "On the number of false witnesses for a composite number". DEVROYE. Springer-Verlag. Transactions of the American Mathematical Society. COOK. EVEN. Mathematics of Computation. (1981). 46(173). 8(2). and J. (1982). "Worst-case analysis of a new heuristic for the traveling salesman problem". pp. H. G. (1976). N. COHEN. 644-654. (1986). DE BRUIJN. G. Chicago.344 Bibliography CHRISTOFIDES. COPPERSMITH. "A theoretical comparison of the efficiencies of two classical methods and a Monte Carlo method for computing one component of the solution of a set of linear algebraic equations". 297-301. Meyer. 255-260. CRAY RESEARCH (1986). the tower of Hanoi and the Chinese rings". COOK. A. and A. 151-158. DIJKSTRA. D. "Matrix multiplication via arithmetic progressions". Journal of the Franklin Institute. Playboy's Book of Backgammon. DIXON. CURTISS. MA. pp. LENSTRA. S. DEYONG. "Computer recreations : Yin and yang : recursion and iteration. Mathematics of Computation. R. (1987).E. Prentice-Hall. P. D. P. 103-121. Mathematics of Computation. K. Cray Channels. Micro-Systemes". and M. COOLEY. Computer Science Press. NY. DANIELSON. 39. S. 269-271.R. A. 365-380. WELCH (1967). C. 55. Reading. IT-22(6). G. 233. 388. "Asymptotically fast factorization of integers". H. Cryptography and Data Security. 1675-1679. 191-233. LEWIS. Playboy Press. A. How to Solve It by Computer. New York. ERDOS.. "Implementation of a new primality test". DENNING. O. and S. PA. Rockville. COOLEY. "History of the fast Fourier transform". DIFFIE. "An algorithm for the machine calculation of complex Fourier series". D.. pp. C. DEWDNEY. W. IL. HELLMAN (1976). D. M. Englewood Cliffs.

GODBOLE. 237(2). F. L. H. MOORE (1959). FREIVALDS. Freeman and Co. Bell System Technical Journal. pp.. GILBERT. M. c. KILIAN (1986). Computers and Intractability: A Guide to the Theory of NP-Completeness. FLOYD. Journal of Computer and System Sciences. pp. GAREY. J. (1962). R. R. Scientific American. "Approximation algorithms for combinatorial problems : An annotated bibliography". SIAM Journal on Computing. "Fibonacci heaps and their uses in improved network optimization algorithms". and C. H. "Application of a method of fast multiplication of matrices in the problem of finding the transitive closure of a graph" (in Russian). (1977). GOLDWASSER. can you take advantage of someone without having to trust him?". J. FREDMAN. "Almost all primes can be quickly certified". "Boolean matrix multiplication and transitive closure". Proceedings of AFIPS Fall Joint Computer Conference. 524. 345 (1986). CA. 933-968.. J. and A. 270-299. R. Doklady Akademii Nauk SSSR. "Encrypting problem instances. 31(2). Proceedings of Information Processing 77. MEYER (1971). E. Washington. M. (1986). pp. ACM Transactions on Mathematical Software. 839-842. N. B. "Probabilistic encryption". 241(5). 194. 29. Proceedings of 25th Annual IEEE Symposium on the Foundations of Computer Science. JOHNSON (1976). "Probabilistic counting algorithms for data base applications". Berlin. (1979). pp. San Francisco. 120-124. 477-488. and G. "Probabilistic machines can use less running time". 5(1). GAREY. TARJAN (1984). 57-69. FISCHER. "Mathematical games : The random number omega bids fair to hold the mysteries of the universe". E. FLAJOLET. 864-866. and G. GOLDWASSER. pp. R. . P. GARDNER. and D. Lecture Notes in Computer Science. 38(4). S. (1973). MARTIN (1985). "New bounds on the complexity of the shortest path problem". SANDE (1966). M. Fox. MICALI (1984). 83-89. IEEE Transactions on Computers. 20-34. in Traub (1976). M. and E.. BENNETT (1979). "Algorithm 647: Implementation and relative efficiency of quasirandom sequence generators". C-22(9). DC. Communications of the ACM. 563-578. M. M. FREIVALDS. pp. "Fast probabilistic algorithms". S. 316-329. and J. pp. W. or . 12(4). M. M. M. GLEICK. 338-346. 362-376. Berlin. FURMAN. pp. R. 74. N. Proceedings of 8th Symposium on the Mathematical Foundations of Computer Science. Journal of Computer and System Sciences. and S. "Algorithm 97: Shortest path". March 14. Scientific American. (1987). (1970). R. 345. "Variable length encodings". S.. 28(2). GENTLEMAN. "Fast Fourier transforms-for fun and profit". (1976). 41-52. and R. Springer-Verlag. S. W. "Mathematical games : A new kind of cipher that would take millions of years to break". Proceedings of IEEE 12th Annual Symposium on Switching and Automata Theory. 129-131. Proceedings of CRYPTO 85. W. L.Bibliography FEIGENBAUM. JOHNSON (1979). Spartan. "On efficient computation of matrix chain products". 5(6). and D. 182-209. "Calculating pi to 134 million digits hailed as great test for computer". Proceedings of 18th Annual ACM Symposium on the Theory of Computing. FREDMAN. (1977).L.E. Springer-Verlag. S. GARDNER. New York Times.

113-114. J. NY. MA. NY. and E. GRAHAM. D. and D. Paris . HALL. pp. An Introduction to the Theory of Numbers. eds. GONNET. "Efficient planarity testing". 549-568. Introduction to Automata Theory. (1968). Ithaca. 5(1). LEVIN (1980). 20(1). John Wiley & Sons. 146-157. reprinted in 1979 by Chapman and Hall. HANDSCOMB (1965). 15(4). H. and J. WRIGHT (1938). D. HAREL. vol. J. 11(2). Handbook of Algorithms and Data Structures. Communications of the ACM. 2. BAUMERT (1965). 1979. New York. 21(4). fifth edition. (1987). Michie. "Implementation of the substring test by hashing". Reading. Journal of the ACM. M. and R. (1873). 294-303. 372-378. 89-118. GREENE. H. Algorithmics : The Spirit of Computing. Technical report TR-71-114. D. MA. The Science of Programming. 10-15. Graphes et algorithmes. E. GRIES. Reading.A. and D. D. and P. NY. 516-524. Springer-Verlag. E. E. SIAM Journal on Com- puting. HOPCROFT. "On minimizing the number of multiplications necessary for matrix multiplication". Addison-Wesley. D. 10(1). MA.C. Languages. and J. "A dynamic programming approach to sequencing problems". "Backtrack programming". E. HAMMERSLEY. M. MA. 964-971. E. "On an experimental determination of it ". GONDRAN. L. (1971). England . HELD. 43-57. 12(4). Oxford Science Publications. 16(6). "Quicksort". Addison-Wesley. J. HOARE. G. 777-779. 30-36. J. Messenger of Mathematics. (1984). KARP (1962). Reading. GONNET. Department of Computer Science. (1980). HOPCROFT. HELLMAN. 2. C.E. Birkhauser. "Heaps on heaps". Oxford. E. S. "Set merging algorithms". R. SIAM Journal on Computing. HoPCROFT. GRIES. "An algorithm for testing the equivalence of finite automata". HOPCRoFT. J. MUNRO (1986). GOOD. 241(2). Computer Journal. New York. 14(12). and R. ULLMAN (1979). and M. Dale and D. KNUTH (1981). HARRISON. TARTAN (1973). . E. Addison-Wesley. 196-210. HOPCROFT. (1979). and J. H. London. in Machine Intelligence. J. J. "A five-year plan for automatic chess". (1962). M.346 Bibliography GOLOMB. I. G. NY. Eyrolles. KERR (1971). and G. J. 68-69. Cornell University. SIAM Journal on Applied Mathematics. SIAM Journal on Applied Mathematics. M. 2(4). Information Processing Letters. New York. and R. E. A. and Computation. G. KARP (1971). translated as : Graphs and Algorithms (1984). TARJAN (1974).. H. M. M. C. Communications of the ACM. Boston. (1981). I. Journal of the ACM. and R. R. "The mathematics of public-key cryptography".R. Mathematics for the Analysis of Algorithms. HARDY. E. ULLMAN (1973). "Computing Fibonacci numbers (and similarly defined functions) in log time". 7(1). HOPCROFT. American Elsevier. and L. Annals of the History of Computing. HELL (1985). Monte Carlo Methods. "On the history of the minimum spanning tree problem". and L. D. "Efficient algorithms for graph manipulation". E. Scientific American.

(1965). 1973. "Multiplication of multidigit numbers on automata" (in Russian). and S. SHING (1982). SAHNI (1978). Proceedings of 22nd Annual IEEE Symposium on the Foundations of Computer Science. SHERMAN (1988). and M. KNUTH. YOSHINO. "0 jistem problemu minimalnim". IBM Journal of Research and Development. "Is the Data Encryption Standard a group?". second edition. KARATSUBA. Computer Programming. Air Force Cambridge Research Laboratory. "Priority queues with update and finding minimum spanning trees". Y. 293-294. B. SAHNI (1976). R. Bedford. Hu. New York. "Computations of matrix chain products". V. Thatcher. NJ. "Reducibility among combinatorial problems".C.013. NEWMAN (1967). RABIN (1987). Doklady Akademii Nauk SSSR. (1969). JENSEN. T. and N. A. Computer Science Press. R. O..E. KLAMKIN. Part I. D.. 2: Seminumerical Algorithms. Addison-Wesley. KLEENE. JANKO. eds. Miller and J. The Art of. Miner. second edition. 228-251. 1-13. "Extensions of the birthday surprise". Fundamentals of Data Structures. S. KANADA. A. T. in Complexity of Computer Computations. Rockville. WIRTH (1985). "An efficient recognition and syntax algorithm for context-free languages". KASIMI. KARP. Plenum Press. and Y. pp. and A. third edition revised by A. NY. eds. Y. KALISKI. and M. C. (1956).395 decimal places based on the Gauss-Legendre algorithm and Gauss arctangent relation". RIVEST. Journal of Combinatorial Theory. D. Addison-Wesley. B. 11(2). JOHNSON.S. Princeton University Press. 150-158. B. MA. NY. in press. Rockville. (1930). Part II. TAMURA. 143-153. Computer Science Press. D. Springer-Verlag. 57-63.T. Scientific Report AFCRL-65-758. Princeton. KARP. "Representation of events in nerve nets and finite automata". The Codebreakers: The Story of Secret Writing. 53-57. USHIRO (1986). Journal of Cryptology. (1). HoRowrrz. (1975). B. McCarthy.S. Reading. 6. ACM Transactions on Mathematical Software. 85-104. MA. "Efficient randomized pattern-matching algorithms". (1968). MD. Hu. 31(2). . The Art of Computer Programming. 4(3). Macmillan. ITAI. E. SIAM Journal on Computing. in Automata Studies. 145. and S. JARNIK. and M. pp. NY. M. KAHN. 279-282. T. C. R. Pascal User Manual and Report. 3(3).C. (1972). MD. W. R. 249-260. manuscript. 1 : Fundamental Algorithms. RODEH (1981).J. and Y. Shannon and J.L. OFMAN (1962). S. Journal of the ACM. "A list insertion sort for keys with arbitrary key distribution". K. 2(2). "Computations of matrix chain products". E. MA. E. (1977). 1981. and M. E. Praca Moravske Prirodovedecke Spolecnosti. (1976). JOHNSON. and D.Bibliography 347 HoRowiTz. 362-373. E. 13(2). KNUTH. New York. "Symmetry breaking in distributive networks". "Efficient algorithms for shortest paths in sparse networks". SIAM Journal on Computing. D. W. R. SHING (1984). Reading.. New York. (1967). F. Michel and J. pp. Information Processing Letters. 24(1). Fundamentals of Computer Algorithms. D. "Calculation of it to 10. R. 3-40.

(1956). 12(4). to appear in Annals of Mathematics.. Mathematisch Instituut. New York. L. (1976). "Building heaps fast". LeVeque. (1982).. Mathematical Centre Tracts 154. "Algorithms". "The efficiency of algorithms". and B. "Optimal binary search trees". H. LENSTRA. E. D. E. 55-97. Papadimitriou (1978). Mathematical Association of America. "Estimating the efficiency of backtrack programs". American Elsevier. Elements de programmation dynamique. Proceedings of the American Mathematical Society. . (1979). eds. E. (1777). The Art of Computer Programming. (1982). KNUTH. (1973). TIJDEMAN. (1976). "Computer technology applied to the theory of numbers". (1980). NY. 419-436. 2: Graph Algorithms and NPCompleteness. Pascal pour programmeurs. (1971). the Graph Traverser. L. Jr. "Fast pattern matching in strings". NEBUT (1985). (1984a). J.348 Bibliography KNUTH. "An analysis of alpha-beta cutoffs". 115-116. NY. Mathematics of Computation. 281-300. H. Scientific American. "Branch-and-bound methods : A survey". Berlin.. O. Berlin. MARSH. D. Acta Informatica. Computing Surveys. 96-109. Paris. S. 1. Reading. Jr. KNUTH.. H. E. H. D. LAWLER. D. LAWLER. "Primality testing". and V. RI. 1 : Sorting and Searching. AddisonWesley. 699-719. Rinehart and Winston. Jr. "Memo functions. MELHORN. J.. E. J. LAURIERE .W. (1973). L. C.W. W. 236(4). in Studies in Number Theory. E. Essai d'arithmetique morale.-L. J. 29. G. pp. Springer-Verlag. 240-267. report 86-18. D. MELHORN. KNUTH. R. Problemy Peredaci Informacii. and a simple control situation".. and J. submitted to the Journal of Algorithms. New York. and C. 5. in Lenstra and Tijdeman (1982). D. "On the shortest spanning subtree of a graph and the traveling salesman problem". 48-50. Jr. W. Paris.. KNUTH. 8(2). 238(1). (1986). Primality and Cryptography. Amsterdam. and R. McGraw-Hill. 63-80. REED (1987). 117. 14-25. PRATT (1977). Computational Methods in Number Theory. L. "Big Omicron and big Omega and big Theta". 293-326. Michie. (1970). and Edinburgh University Press. 14(4). "Some techniques for solving recurrences". "Factoring integers with elliptic curves". E. B. H. D. E. Holt. Data Structures and Algorithms. E. D. H. Scientific American. 3: Sorting and Searching. SIAM Journal on Computing. Combinatorial Optimization : Networks and Matroids. Wiley-Teubner Series in Computer Science. WOOD (1966). LEWIS. Mathematisch Centrum. SIGACT News. Providence. Operations Research. H. (1986). R. D. B. Data Structures and Algorithms. 6. MORRIS. ACM. Bordas. Meltzer and D. W. LEVIN. G. eds. KRUSKAL. E. KNUTH. LENSTRA. 9. A. H. LENSTRA. LEHMER. Universiteit van Amsterdam. Part I. in Machine Intelligence. (1975a). (1984b). K. MCDIARMID. Artificial Intelligence. MA. (1977). KRANAKIS. (1975b). ed. 7(1).-L. (1969). LUEKER. K. "Universal search problems" (in Russian). pp. 6(2). SpringerVerlag. p. LECLERC. KNUTH. 121-136. J. LECARME. and D. 18-24.

BROWN (1985). NY. PERALTA. E.R. Journal of the American Statistical Association. NY. P. J. SIAM Journal on Computing. MONTGOMERY. Springer-Verlag. pp. Mathematics of Computation. 4(3). Bit. (1975). NJ. "Every prime has a succinct certificate".. (1986).Bibliography 349 MELHORN. DEO (1977). 114-124. RABIN. (1971). and C. (1980). RABINER. and N.. "The fast Fourier transform in a finite field". L. POLLARD. New York. The Analysis of Algorithms. POMERANCE. M. A. MICHIE. Prentice-Hall. 89-139. C. GOLD (1974). I. 128-138. Nature. Englewood Cliffs. NJ. (1978). N. 48(177). 12. PAN. POHL. 238(6). "Evaluation and comparison of two efficient probabilistic primality testing algorithms". (1980b). 335-341. Problem Solving Methods in Artificial Intelligence. Combinatorial Optimization : Algorithms and Complexity. pp. and B. pp. 243-264. 15(6). "The Monte Carlo method". "Probabilistic algorithm for primality testing". R. 36. PAPADIMITRIOU. and S. "Speeding the Pollard and elliptic curve methods of factorization". Communications of the ACM. Proceedings of 19th Annual IEEE Symposium on the Foundations of Computer Science. (1976). . MONIER. 12. "Complexity theory". I. K. O. PRIM. NJ. REINGOLD. C. R. RABIN. 3: Multi-Dimensional Searching and Computational Geometry. Holt. (1980a). SIAM Journal on Computing. 365-374. V. McGraw-Hill. NY. 331-334. "Shortest connection networks and some generalizations". G. 48(177). (1966). W. Prentice-Hall. 218. (1978). Jr. NEMHAUSER. (1982). RABIN. "A sorting problem and its complexity". Combinatorial Algorithms : Theory and Practice. New York. NIEVERGELT. Introduction to Dynamic Programming. Prentice-Hall. NILSSON. New York. "Probabilistic algorithms in finite fields". (1968). "Analysis and comparison of some integer factoring algorithms". 44(247). 214-220. POLLARD. (1971). ULAM (1949).C. Inc. 19-22. Rinehart and Winston. Englewood Cliffs. (1987). 9(2). 15. Berlin. Scientific American. C. "Strassen's algorithm is not optimal". H. in Traub (1976). R. N. M. J. O. (1975). Theoretical Computer Science. "Very short primality proofs". 462-463. Mathematics of Computation. O. 166-176. N. M. Journal of Number Theory. M. V. IEEE Transactions on Information Theory. P. METROPOLIS. "A simple and fast probabilistic algorithm for computing square roots modulo a prime number". POMERANCE. 273-280. L. 315-322. Mathematics of Computation. "Probabilistic algorithms". (1984c). J. 21-39. "'Memo' functions and machine learning". (1972). Englewood Cliffs. M. John Wiley & Sons. PIPPENGER. PURDOM. D. 1389-1401. STEIGLITZ (1982). M. in Lenstra and Tijdeman (1982). L. (1957). PRATT. IT-32(6). and K.. C. Data Structures and Algorithms. 97-108. (1987). Digital Signal Processing. 25(114). 846-847. Bell System Technical Journal. "A Monte Carlo method of factorization".

SIAM Journal on Computing..M. NY. Communications of the ACM. MA. (1980). H. University of Chicago Press. New York. SOBOL'. L. "Planar 3-colorability is polynomial complete". Pierre. "A method for obtaining digital signatures and public-key cryptosystems". Berlin. Information Processing Letters. "Five number-theoretic algorithms". NY. A. STOCKMEYER. (1953). 240(5). S. A. The Monte Carlo Method. Algorithmics Press. STRASSEN. 146-160. FLOYD (1973). (1973). K. Introduction to Computer Organization and Data Structures. T. A. An Introduction to the Design and Analysis of Algorithms. Numerische Mathematik. (1973). "Intrinsically difficult problems". (1983). SHAMIR. IL. 13. W. N. Addison-Wesley. Courant Institute. (1972). "An optimal encoding with minimum longest code and total number of digits".J. Reading. R. New York University. Scientific American. McGraw-Hill. Die Grundlehren der Mathematischen Wissenschaften.350 Bibliography RIVEST. S. in Combinatorial Algorithms. CHANDRA (1979). E. KoNIG (1924). pp. (1978). 354-356. Computer Science Department.M. (1974). TARJAN. SCHWARTZ. MA.J. 9(3). D. 604. Information and Control. RIVEST. NY. 84-85. R. (1985). E. "A fast Monte-Carlo test for primality". "Gaussian elimination is not optimal". ROBSON. "Combinatorial problems: reducibility and approximation". 69-76. STOCKMEYER. A. SCHWARTZ. McGraw-Hill. SAHNI. SEDGEWICK. SIAM Journal on Computing. and V. 37-44. V. R. SOLOVAY. 26(4). Addison-Wesley. Manitoba. 7(1). (1969). R. Data Structure Techniques. ed. ibid. 21(2). J. SCHONHAGE. I. 51-70. New York. (1964). Rustin. SIGACT News. 120-126. 6(1). "An improved algorithm for traversing binary trees without auxiliary stack". Chicago. 28-31. ROSENTHAL. SHAMIR. "A correct preprocessing algorithm for Boyer-Moore string searching". "Factoring numbers in O (log n) arithmetic steps". 281-292. (1979). (1972). . HOROWITZ (1978). 19-25.J.. 5(3). SHANKS. M. 8(1). R. STINSON. St. 1(2). GOLDNER (1977). Information Processing Letters. RYTTER. (1980). Computing. and R. (1972). 2(1). and A. STANDISH. (1973). erratum (1978). STRASSEN (1971). "Bounds on the expected time for median computations". NY. Springer. W. and L. (1978). The Charles Babbage Research Centre. L. SIAM Journal on Computing. SLOANE. 6(1). Operations Research. S. "Smallest augmentation to biconnect a graph". and A. A. second edition. A. and V. "Depth-first search and linear graph algorithms". R. 7. Reading. New York. 118. C. W. L. STRASSEN (1977). STONE. RUNGE. 7. and E. Technical Report no. 509-512. SIAM Journal on Computing. A Handbook of Integer Sequences. "Probabilistic algorithms for verification of polynomial identities". and H. "Schnelle Multiplikation grosser Zahlen". D. 11. RUDIN. R. 140-159. 12-14. Principles of Mathematical Analysis. ADLEMAN. 718-759. pp. New York. Algorithms. Proceedings of the Second Manitoba Conference on Numerical Mathematics. Academic Press. J. L. 55-66.

Doctoral dissertation. VALOIS. and J. Ars Combinatoria. F. Journal of'the ACM. Probabilistic Algorithms for Sparse Polynomials. R. S.. NY. "An O (log n) algorithm for computing the n th element of the solution of a difference equation". C. U. YOUNGER. VAZIRANI. New York. Cambridge. A. 347-348. VAZIRANI. 22(2). J. (1936). A. J. Randomness. (1986). (1961). Philadelphia.E. (1956). 66-67. H. 80-91. "New hash functions and their use in authentication and set equality". 28(3). Massachusetts Institute of Technology. W. YAO. FISCHER (1974). "On computable numbers with an application to the Entscheidungsproblem". VICKERY. 4(1). D. J. E. WARSHALL.E. 11-12. 265-279. 5. Communications of the ACM. Algorithms and Complexity : Recent Results and New Directions. in Lenstra and Tijdeman (1982). YAO. and M. "Efficient dynamic programming using quadrangle inequalities". WILLIAMS. (1975). 21-23. pp. (1979). R. A. 7(6). Berkeley. . TARJAN. ed. pp. (1964). 189-208. ZIPPED.Bibliography 351 TARJAN. Editions du Seuil. CARTER (1981).N. (1975). pp. M. 145-146. Journal of the ACM. 2(42).W. NY. TARJAN. WRIGHT. PA. J. R. University of California. M. "On the efficiency of a good but not linear set merging algorithm". ed. "Recognition of context-free languages in time n 3 ". TURING. PA. F. Adversaries and Computation. (1980). Proceedings of 12th Annual ACM Symposium on the Theory of Computing. "Primality testing on a computer". H. Journal of the ACM. 215-225. "Theory and applications of trapdoor functions". 21(1). Academic Press. 577-593. A. Algorithmes prohabilistes: une anthologie. Arithmetic Complexity of Computations. 168-173. John Wiley & Sons.J. (1983). Information Processing Letters. Information and Control. "Algorithm 232: Heapsort". New York. 22(3). (1980). W. (1982). A. J. pp. Les nombres et leurs mysteres. (1976). "Fast arithmetic operations on numbers and polynomials". Symposium on Monte Carlo Methods. 429-435. R. R. Paris. (1987). V. Journal of the ACM. (1975). V. J. "The string-to-string correction problem". 22(1). 11(2). Information Processing Letters. Data Structures and Network Algorithms. (1962). 160-168. WARUSFEL. Proceedings of 19th Annual ACM Symposium on the Theory of Computing. F. 127-185. Proceedings of the London Mathematical Society. (1967). YAO. MA. "An O (JE Ilog logI V 1) algorithm for finding minimum spanning trees". (1987). A. "A unified approach to path problems". 125-128. U. "Experimental determination of eigenvalues and dynamic influence coefficients for complex structures such as airplanes". 230-265. Computer Science. CA. TRAUB. E. M. "The change-making problem". Universite de Montreal. W.. F. WAGNER. C. (1981). SIAM. Journal of Computer and System Sciences. C. 43-54. Proceedings of 23rd Annual IEEE Symposium on the Foundations of Computer Science. TURK. pp. 9(1). S. L. D. Meyer. "A theorem on Boolean matrices". WINOGRAD. "Efficiency considerations in using semi-random sources". H. WILLIAMS. WEGMAN. (1978). 10(2). Philadelphia. (1980). (1982). URBANEK. Masters Thesis. Journal of the ACM. SIAM. Doctoral dissertation. Departement d'informatique et de recherche operationnelle.


138 Baumert. G. 336 Arlazarov. 35. 141.H. 63. 36. 276 Analysis of algorithms: on average. V. 4 Apple II. 184. 204 alter-heap. 296 Adder. M. 207 Ancestry in a rooted tree.L.E. 122. 222. 275. 37-51 with several arguments. 342 353 . 22. 341 adjgraph. E.. 342 Bellmore. 168. 140. 5. 7-9 Ancestor. 336. 167. 137 Adel'son-Vel'skii. 276. 205 Algorithm. 2.. P. 185. 5 in worst case. 18 theoretical. M. 207-208 APL. 342 Backgammon.. Complexity AVL tree. 342 Belaga. 337 Aho. 198. 204 backtrack... 52 Basic subalgorithm. R. 6. 23. 2-3 tree Barometer. 197-199. 341 Ajtai. 341. P. 204. I Algorithmics. 166 Acyclic graph. 274 Average height. 342 Bach.O.. 203 Adleman. xiii..G. 185-189. 342 Babaf. 205 Balanced tree. 78... 294. 248-252 Backus-Naur form.C. L. K. M. 168. 35. 227. A. 222.. 35 Alpha-beta pruning. 27 Amplification of stochastic advantage. 35 Baase. 291. Nested asymptotic notation. 222. 350 Adversary argument. 341 Algebraic transformation. 301 Atomic research. 298 Average.. 5 hybrid.. Operations on asymptotic notation Asymptotic recurrence. 342 Batcher's sorting circuit. 313.V.. 276. S. 277 Algol. 20 Articulation point.. L. 238 empirical. 178-179. 106 Batcher.. 199 Ad hoc sorting. 251 Approximation. 276. 341 Ackermann.. 47 At least quadratic. 190. 342 Array. 336. 25.. 78. 274. L. 344 Abadi. 377. 7-9. 346 Beauchemin. See Analysis of algorithms. 141.. S. 342 Bellman. W. 43 See also Conditional asymptotic notation. 263-266. 204. 3. See AVL tree.Index Aanderaa. 336. 174-176 Asymptotic notation. 224. 54. 204. 188 Backtracking.M. 341 Ackermann's function. 173. 342 Bachmann. 35. 22. 275. 275. 204.M. E. 118.

106 Conditional asymptotic notation.. 48-51 Context-free language. 35. 276. 276. 235.. 155.M. 345 Bentley. 339 Certificate of primality. 275. 35. 71 Cheriton. 4.A. 196 Blum. 72-75 Characteristic equation. 337. 342 Berliner. 325 Boolean variable. 290.. Merge circuit. C.. 342. 293 Compiler generator. 65. Hamiltonian circuit.. 142 Boyer. 104. D.H. 234 . 343 Chess. 194-198. See Fibonacci heap Birthday surprise. 344 Coppersmith. 79 Canonical object. C. 304-308 of large integer arithmetic. 343 Change making. 342 Blum. J. 25 Binary search.. C. 19.. M. A. 336. 344 Cole. 168 Continued fraction algorithm. 342 Berlekamp. 336.. 332 CNF. 351 Catalan number. 314-315 of matrix problems. Tally circuit. 104. 235 Cryptology.. 234 countsort. 109-115. 275. 275. See Conjunctive normal form Cohen. H.J. 322 Consistent probabilistic algorithm. 342 Berge. 291 Crepeau. 36. 298-299 of circuits.. 148. H. 275. 350 Branch-and-bound. See p-correct probabilistic algorithm correction loop. 343 Candidate.R. 275. 309-314 of sorting. 23 Chinese remainder theorem. 344 Cook's theorem.. D. 35. 265 Bicoherent.. 31 Carasso. 263 Constructive induction. 343 Brown.L. 2 Comparison sort. C. 65. 138 of graph problems. L. S. 167. 167 Ceiling. 140. Batcher's sorting circuit.. A. 350 Chang. See Multiplication Clause. J. 291.. 275. 279-290 Complexity. 164 Binomial queue. 293 CRAY. 291. 233.L. 315-335 of polynomial arithmetic. 343 Carlsson.O. N. 337. 69. 222. 220 Boolean expression. 78. P. 342. 78.B. 344 CYBER... 10. 78. 167 Chandra. 78 Conditional probability. 325-328 Cooley. L. 343 Bordvka. 231 Cryptographic key.Index 354 Bennett. J. 343 BM. 349 Buffon's needle. 52. 260 Contradiction. 225. 276. 184. 326 Connected component. 343 Boyer-Moore algorithm. 276 choose. 336. 337.. 269 Cycle detection. 143. 323. 14-17. 274 Bunch. 275. 292-337 on average. 227 Conjunctive normal form. 216. 199-204 Brassard. J. 174-176 Biconnected. 124. E. 308-309. 320 Circuit. 325 Borodin.S.. 45-47.. 168. G. 343 Breadth-first search. 35. 292-299 Compound interest. 80. M.. 104.. 315 count. E. J. 276. F.. 182-184 Brigham. 140. 344 Coprime. 0. 140. 78. 343 Bottom-up technique. 204. 333 Christofides.. 146-150. 243. 275.. 250-252. 128-132. 267 Confidence interval. See Graph colouring Comment. See Adder. 204. 298 Binary tree. 293. 245 Colouring. 6 Curtiss. 24 Binomial coefficient.K. 343 Buneman. 290 Choice of leader. 174-176 binary-node. 141. 336 Chained matrix multiplication. 275. 206 Complementary problem.J. S. 104... 133. 325. 341 Correct. 35. 204 Child. 343 Carter.H. Telephone switching Classic. 216-222.. 336. 321. 325 Cook. R. 343. 344 Chromatic number. 173 Co-NP. 204. 260-262. 323 Collision. 15 Corasick. 276 Cubic algorithm. 336. 291. 342 crude.. 35. 302-304 of NP-complete problems.. 140. 342. 222. 322 Complex number. 342 bfs. 228-230. 207 Change of variable. 343 Bratley. 183 Biased probabilistic algorithm. 275.A. 128. 326 Clique. C. 343. 35. P.. 246 Black. 275.. 165.

275 Data Strutures. 275. 140 Execution time. 124. D.J. 270 Evaluation of polynomial: with fast Fourier transform.. 204. E. 204. 138 Depth of node. 185-188. 56-57 Devroye. 349 Depth of circuit. 210 Floor.Index 355 Danielson. 142-168. 302-315 Division. 275 Elementary operation. 176-182 Discrete Fourier transform. 345. 35. 167. 67 Deadline.. 342 Directed graph. 291 with preconditioning. B. 35. 79 Feigenbaum. 300 many-one. I.E. 350 Floyd's algorithm. 143. 275 Endomorphic cryptosystem.. 344 dexpo. 258. 336. L. 78. 344 De Moivre's formula. 235 Data Encryption Standard. 78.A. 344 Eventually nondecreasing function. 225. J. C. 36. 66. 192-196. 19. 104. 336.. 270. 298 Efficient algorithm. 16. 345 . 344 dfs. 143 See also De Moivre's formula fib 1. 342 Fast Fourier transform. 87-92.. 339 Floyd. 46 pruned.L.. 17. 10. 58. 345 Fredman. M. 256-260 dlogRH. 351 Flajolet.. 131. 276. 242. 275.. 242 Double ended heap.D. 203 Euler's function. 78. 60-63. 167. 342 Dromey. 167. 151. 267. 19.A. G. 19. 242. 167.L. 127. 11. 204 Determinant of matrix. 222 Fischer.. 45.. 167. R. 9-11 Exact order of (e). 293 valid. 252 FFT. See Nontrivial factor Factorization. 343.. 16. 35. 192 fib2. 344 Dense graph. 279. 345 Fermat's theorem. 59. 13 Four colour theorem. 21. 34 See also Disjoint sets find-max. 343. 19. 135. 270 Deyong. See Dicrete Fourier transform Fox.K. 10 Fibonacci heap. 17-18. 222 Even. 305. See Large integer arithmetic. N. 345 Floating point. 344 darts.. 336. 275. 2 delete-max. 98-100 Disjunctive normal form. 275. See Double ended heap Decision problem.G. 36 Dreyfus. 142. Polynomial arithmetic Dixon. 18. 136 find. See Large integer arithmetic Factor. 316 Eigenvalue. 248-252. 76. A. 87. 235. 58-59 fib3... 318 Decision tree. R.. 16. P.. L. 129. 344 Denning. 35. 104 Fibonacci sequence. 84. 309 Discrete logarithm. 35.W. 323 Fourier transform. 344 Euclid. 269 extended. 20-34 de Bruijn. 17-18. 237 Eight queens problem.. 262. 294 trip. S. 17. 30. 317. 344 Dewdney. 57. 30-34. 230 Encrypted instance.W. J.C. 18. 345. 256-260. E. 28 Finite state automaton. W. 150-153. 344 Dijkstra. 36. 30.. 293 Declaration.R. M. 341. 58. 78. 344 Dijkstra's algorithm. 151 Dinic. 279.. 105-141. 134-136. 276. 182 Diffie. 130. 226..E. 104. 11. 103. 209-211. 276 False witness of primality. 344 Dixon's algorithm. See Fast Fourier transform Fibonacci. 293 verdict. 171-182. 261 Deap. 28 Demars. 91. 306 Deo. 16.G. 35. 237 Eigenvector. 104 Euler path. I1-12. 6 Exponentiation. 336 div. 57 Euclid's algorithm. 140. 276 Faradzev. 95-100 Deadlock. 204. 168. 275. 14-15. 1. 19. 6 Expected time. 226. 3 Divide-and-conquer. 104. 301. N. 136.. 318 polynomial. 306 FORTRAN. 292-299 Equivalence: linear. 224 Exponential algorithm. 24 Depth-first search. P. 171 dfs'. 140. 334 dexpoiler. 42 Exchange of two sections in an array. 336. 286. S. 279-280 Feasible. 241 Disjoint sets.. 290. 318 sense of Turing. 16.. 317 Erdo s. 254 Euclidean travelling salesperson. 128. 140. 344 Dynamic programming.

G. 170 insert. 346 Hashing. D. 291. 347 Johnson. 168. 276. 336.. 341. 104. R. 291.. 119.N. 84. G.. 237. 345 Gleick.H. 78. S. R. 276. 332 Hamiltonian path. D. D. 336. 91-92. M. 336 heapsort. See Backgammon. 167. M. 336.. 186. xiii. 274. 54 Harrison. 64 Godbole.. 230 Hoare. 295-299 insert-node. 315 Greedy algorithm. 16.. 128.. 13. 35. See Greatest common divisor Generation of permutation. 78. 128. 166. 79-104. 345 Gauss-Jordan elimination. 20.. 346 Good. 104 Implicit graph. 25. 53-54.. 21-22. 280-284 modulo p. 115. R. 346 Hell.C.. 345 Gilbert.B. 13.. 78.R. 291. 346 Halting problem. C. 30. 346 Heuristic.E. 275. 332 Hamiltonian circuit. M.C. 276. S. 347 Jensen. 346 See also Quicksort Hode. 202 See also Double ended heap. D. 275.. xiv. 204 Games. 341 Hudson's Bay. 35. 168. 320.Index 356 Freivalds. 347 Kasimi. 346 Hellman. 254 of a matrix. 257. 65-68 Hopcroft.S. 346 Harel. 104. xiv. 260. I.. J. 346 Hanoi. 35. 336... M. 349 Golden ratio. 275. 337. 330-332 Greatest common divisor. 345 Full adder.J. 291. J.. E. 3 Infinite graph. 115.. A. 204. 140. 13. See Bicoherent Itai. 115.L. 36... T. 104 Hammersley. 323. 104. E.. See Towers of Hanoi Hardy. W. M. 315. 165 Huffman code. 316. 342 Graham. 140. 15. 36. M.. T. 116. 345 Game graph. 277 . 101-104. 222. 291. 290. 347 Jarm'k. 195. D. 35-36. 128 genrand. 204. 310 of an integer. 100 Janko. 56 Height.H. 342 Hall. W.. 24 See also Average height Held. 350 Goldwasser. 35.. 101-102.. 151. 345 God.. 140.C. 336. 25-30. 100-104. 347 Kepler. 337. 20. 347 Kaliski. 159. 141. 23 Invariance. 346 Gonnet. 7..-D. 204.. A. 35. 276. 346 Handscomb. 343. J.. 346 Homer's rule. Fibonacci heap Heap property.E. B. 302-304 of a polynomial... 104. 315 Isthmus-free. Chess. 276. B. 304-308 Graph colouring. 26.R. 347 Huang. Nim Gardner. 336. 7. Y. 140. V. 319. 337. 6 hitormiss. 222. 345 Kahn. 168. See Adder Furman. 68-72 Inorder. 36. S. 168. 267 Gentleman. 167. 104. 346 Harmonic series. 204. 167. 67 Goldner. 275.. D.. P... 35.M. 167. 169-204. 345 Golomb. 347 Karp. 42. 140. 274. 140. 15. 43.. 346 Gries. 274 Goutier. 336.E. 248-252 Greene. 85.. 140. A. 346 goodproduct.A. 344. 291.. 345 Gold. See Robyn Hode Homogeneous recurrence. 28 Instance. 120. 347 Iterated logarithm. 273 Generator of cyclic group. C. 209 Horowitz. 347. 63. 346 Gondran. See Principle of invariance Inverse: Fourier transform.M.. 345 Garey. S. 347 Johnson. 204. 293-298.H. 135.. 117 Insertion sort. See Numeric integration Internal node.. 185-202 Indentation. 36. 4 Instant Insanity. A. 276. 276.. 346 Haken. 85 Heapsort. 275. 336 Hidden constant. M. M. 189 Integration. J..A. 189 Inherently difficult problem. 56.S.. 347 Kannan. 259 gcd. K. 245-247 Heap. D... 276. 346 Graph. 337 Inhomogeneous recurrence. 55-56. 184. 150-153. 35. 347 Kanada.. 189-198. 184. 81-92... 346. D. 350 Hu. 343 Karatsuba.

168. 268 majMC. 308. 348 Lewis. 140. D. 35. 323 Lueker. 204 Minimization of waiting time. 276. P. 175 Lucas. 7 Merge circuit. 275. 275.. 115 Mersenne. 204. 252-253. 40. 274. 268 maj 2. 137. 254. 228. 276. 286-290. 276.Index Kerr. 204. 13. 341. D. 343 Lewis. 348 Knuth-Morris-Pratt's algorithm. 168. 323 Metropolis. 0.L.. 346 Levin. 35. J. 23 Lecarme.. 7. 92-94 Minimum spanning tree. 348 Levy.R. 213. 138. J-L. 269 Majority element. 346. G. 341. 167. 336. 276..C. 333 LauriBre. Triangular matrix.. 35.. 258 k th smallest element. 24 Lehmer. H. 347 KMP. 185. A.S. M. Inverse of a matrix. 81-87. 351 Level. 222 Koksoak. See Atomic research Lower bound theory. 119. 348 Leclerc. M.. 104. 202. 237 equivalence. Unitary matrix McDiarmid.R. 346 Key for' . 19. 276.. 120. 36. 213-215. 40. 227. 144. 216. 121-124. 344. 172 List. 345. 128-132. 348 Leader. 33 See also Disjoint sets Merge sort. 275 Melhom. L.. 348. G. 203-204 Lanczos. C. K. 139 merge. 315 Los Alamos. 141. 140. 140. G. Strassen's algorithm. 286-290. 309. 92. 275 algorithm. Reductions among martix problems. 168. 215 Knapsack. See Equivalence. 343. 347. 315 multiplication. 336 lowest. 274... H. 227 reduction.. 271 Kilian.. 19. 35. 276. 35.. 247-262.E. 308-315 addition. 22.. S. 78.S. 140. 188. 349 Memory function. E.. 276. Jr. 78 357 Linear: algebra. 238-240. Iterated logarithm Longest simple path.. 92. 300 programming. G. 301 equations. 104. 128. 35. 342 Kruskal.J.A. 348 Kruskal's algorithm. 224. 315 Knight's tour problem. 169 Minoux. 222. S. 140. 300 lisgraph. 91. A.. 308-315 square root. 291...... 341 Konig. 347 Kleene. 187. L. 345 Klamkin.. 204.. 275. 124-128.. 343 k-promising. 184. 195-197. 348. 345 Matrix. 342. 3 Modular arithmetic. 308. 168. 351 Micali. 248-251 Kranakis. 346. 336.M. Determinant of matrix. 78 Limit.B. J. 128.W.. 141. 13-14. IN. 78. 252-256. 29. 341 Many-one. 104 k-smooth integer. 241 Label.. 293..A. 6. Symmetric matrix. 87. 290 ... 268-269 make-heap. G.. Jr. D. 325. 238-240.E. 245 Logarithm.R. 35. 5. 260. K. 115-116. 344 Lexicographic sorting. 309 Las Vegas algorithm. 237. J. 290. 346 mod. 24 Levin. 140. 315 exponentiation.N. 78. See Chained matrix multiplication. 140. 348. 275. 120-124. 349 Meyer. See Nim Marsh. 308 reductions. 348 Macroprocessor. 274. 297 mergesort. 315 squaring. 348 Kronrod. 55 Manders. J. 168. 124 division. F.L. 272 Memory space. 128-131. L.. 275. 348 Lawler. See Canonical object Labyrinth. 140.H. 290. 349 Miller. M. H. 13-14. 124-132.. 348 Lenstra. 193.H.. M. 341 Large integer arithmetic. 348 Median. 345 Michie.. 275. 336. 211 maj. C. 349. 326 Load factor. 348 Martin. D. 15. 347 Minimax principle. 82-85.. 117. Reduction Marienbad. 167. 277 See also Discrete logarithm. 20-21 Literal. 224. 128. See: Choice of leader Leaf. 103. 344 Landis.K. E. 162-164. 165 Koml6s. 350 Korsh. 290. Multiplication of matrices. 35.... 348 Left-hand child. 336 L'Hopital's rule. 167. 1-4. 301 greatest common divisor.A.. 104 Knuth. E. 344 Lenstra. 228.

276. 215. 167. 173. 28-30. 154-159. Monte Carlo algorithm. 277 n-ary-node. D. 349 Nested asymptotic notation. 313. 315-335. R. 62 Pattern. 286. 342. 34. 180. 349 Nim. 104.S. 144-159. V. 120. 231 Numerical probabilistic algorithm... 278 prenum.I. J. 222. 35. 3.Index 358 Monet. See Principle of optimality Order of (0). 204. 322 See also Numeric integration.J. 275 multiple. 216. 27. 170. 274. 334 Principal root of unity. 262-274. J. 256. 320 . 24 Nebut. 343. 35. 284-286. 38 Principle of optimality. 291. 348. Reduction Pomerance. 348.. 207 Optimality.. 227. 1. S. 174. 204. 9.. Numerical probabilistic algorithm. 136. 167-168. 346 Napier. 20 Pollard. See Biased probabilistic algorithm. 140. 298 Priority list. 209-211. 309-314 reductions.. 143. 237. 316 Polynomial arithmetic: division. 20. 26. 208 Pratt. 208 Postorder. 153. 30. Consistent probabilistic algorithm.R. 5. 124 classic. 348 Nemhauser. 349 Moore. Unbiased probabilistic algorithm Probabilistic counting. 35. 211-222 Preconditioning. 330-332 Optimal search tree. 349 Percolate. 286 multiplication. 21 node. 349 Prim's algorithm. 208 Preorder. 291. 342.-L. 2-4... 315. 291. 102. 256. 222. 79 See also k-promising Proof space. J.F. 278. 334 NP. 263. 6.H. See Eventually nondecreasing function Non-determinism. 4 Program. 275. 344. 336-337 Number theory. 332-335. 290-291 Pippenger. 343 Morris. 205 Pascal's triangle. Random permutation Pi (it). Sherwood algorithm.. 232-237. 205-211. 117 Planar graph. 321. 23 Pascal (computer language). 212-222 p-correct probabilistic algorithm. 51. 318 Pan. 349 Nilsson.. 349 postnum. 143. 116. 154-159. See also Large integer arithmetic. 275. 348 Multiplication. 21 Nondecreasing. 166. 227. 37 P. 124-132. 276.. J..C. 343 Monic polynomial. See Equivalence. 309-314 evaluation. 239-240 pivot. 228-237.. 35. 248 Ofman. 91 percolate. 275 Probability of success. 124. J. 45 Newman. 333 Problem. 204. Polynomial arithmetic a la russe.. 269-271. 332-335 NP-complete. E. 27.. 78 Operations on asymptotic notation. 315 Nievergelt. 247. 211. 349 parent. 337. Y.. 336 Non-deterministic algorithm. 2.190-194 Node. 13-14. 205. 140. 263 Peralta. 41 One-way equality. 136. Quasi Monte Carlo Montgomery. 341. R. 336. 349 Precomputation. 349 Pivot. 92 Primality testing. 349 Polynomial algorithm. 275 Objective function.. 43-45 Optimal graph colouring. 6. 35. L. 232. 132-133. 308-314 Polynomial. 2 Promising. 336. 281. 230-232. V. 275 trapezoidal. 333 Nontrivial factor. 1. 35. 276. 209-211. J. 228. 349 Papadimitriou. 228-234. 222. 140. G. 193. P. 8-17..C. N. 167. 281 Principle of invariance. 225. 349 Pointer. 33. 204. 140. 240-242.L. 222. See Generation of Permutation. 13-14. 332 Pohl. 199 Probabilistic algorithm. 334 See also Certificate of primality primeND.M. 170. 146 Path compression. 141. 133. 85-87. Las Vegas algorithm... 345 Moore. p-correct probabilistic algorithm. 343. 347 Newton's binomial. 278 of matrices. N. 146-150.. 136. 79-80 obstinate. 320. 140. 208 Prim. 36.H. 302-308 Munro. 6.. 35. 10 Newton's method. C. 336. 25. 336. 2 Programming language. 276. 27 Permutation. 106. 276. 3. 286. See Theory of numbers Numeric integration: Monte Carlo. 204. 224. 337. 347 Omega (a).. C. 324-332 NP-completeness. 103. 349 Monte Carlo algorithm. J. 291 interpolation. 310 Monier.

146 Set difference.. 242-245 in a tree. 290.S. 115. J. 336 Saxe. 204. M. C. 247. A. W. 15. 3 RH. 25. 277 rootLV. E. 350 Sherman. 80 selectionRH. 326-332. 260-262 Rivest. 276 Rabiner. E. 110 series. 35.. 35. 350 Rytter. See coprime repeatMC. 116-119. 350 Rubik's Cube. 44 Set equality. 260 Quasi Monte Carlo. 65-78 return. 171-184 in a sorted array. 167.. 350 Robyn Hode.. 227 See also Binomial queue Quicksort. 204.. Jr.. 276. M. 35. 153. 290 Schrage. 276. 276. 291... 342 Shanks.... 117 Rabin. 6. A. See Binary search in a sorted list. 140. 350 Rumely. 240. 238. 271-273. 19. 276 Shakespeare. See Decision tree Pruning. R. 275.. 317 Reductions among: arithmetic problems. 337. 122-124. 308-309. 275. 13. W. 228 Rodeh. 140. 139-140 Schonhage.. AT.O. 250 QueensLV. 290. 211-222 in a graph. 350 Shamos. 350 Seed of a pseudorandom generator. 154-159. 350 Sahni. 304-308 Regular expression. 295-299 Sequential search. 232.W. 343 Schwartz. 301 nonresidue. 276. 78. 270. L. S. 3 Recurrence. 309-314 Reed. 266 Resolution of recurrences.. 167... 349 Radix sort. 238-247. 242-245 record. 317. 342 Robson. 347.S. 207 Sedgewick. 23-25 Rosenthal.-M. 182. I. 342 Scaling factor. 104. 13. 207 Searching: for a string.. 104. 293-295 quicksort. See Satisfiability Satisfiability. 106 Reduction: linear. 25. 272 Random permutation. See Asymptotic recurrence.M. 275 Queens.Index Proof system. Resolution of recurrences Recursivity. 252.. M... R. J. 36.. See kth smallest element selection. 78. 23 Rooted tree. 315-335 polynomial problems. 189 Rudin. 293 Random function. 52. 120 Selection function. 226.R.A.. 54.L. B.. 140.M. 261 select. 222. 304-308 matrix problems. 341 Runge. 255 Root of a tree. 60-63. 347 Roman figures.. 291. 252 residue. 34. 249 Queue. J. 254 rootLV2. 290. 320 Pseudorandom generation. 347. 275. 300 many-one. 347 Sherwood algorithm. Inhomogeneous recurrence.R. 141. 350 Schonhage-Strassen algorithm. 275 Pseudorandom walk. 234 Purdom P. 349 Rabin's algorithm. 188. 239 Selection sort. L. 311-313 Scheduling. 78. 276. 349 Relatively prime. See Alpha-beta pruning pseudomed. 228. 302-304 NP-complete problems. G. 226. 204.. 270. 343. 318 polynomial. 122 Pseudomedian. 235 Shallit. 349 Quadratic: algorithm. R. 350 Schwartz. 7. 154-159. 318 sense of Turing. 75 Rank. 347. J. 92-100. 274 Shing. A. 273 Range transformation... 316. 350 Robert. 35.. 342 Shamir. Homogeneous recurrence. 347 . 241 Right-hand child. 345 SAT. J. 348 Reflexive transitive closure. 24 359 Ring of processors. 350 Search tree. 7 Selection. 167 Reingold. 314-315 graph problems. 320 Pruned.. 224. 239 Pseudoprime. 291. 199.E.B. 79. 227. 275. D. 350 Sande. 257 sieve. 275. 276. 336. 8. 325. 170-171.

107-109 ultimate. 241 Sibling. 137 Tamura. Reduction Turing machine. Selection sort. 141. 350 Solovay. 102. Topological sorting. 35 Tukey. 199-202. 347 Target string. 121. 28. 140. See Percolate Signal processing. 204. K. 256. 87. 346. 227. 304-308 Shortest simple path. L. See Amplification of stochastic advantage Stochastic preconditioning. 182. 140. 55-56.. 342 Standish. J. See Bicoherent 2-3 tree. 151 Special path. 132. 336. 277 sort.. 78.F. 104. 275 Steele. 275.. 290.W. 104. 301 Supra quadratic. 336 Telephone switching.R. 86. 87-92. 342 shuffle. See also Large integer arithmetic modulo n. Comparison sort.Index 360 Shortest path. 153. A. 275. 87. 204. Lexicographic sorting. Traversal of tree treenode. Y. 36. 91 Switch. 56 Smooth algorithm. 150 Sparse graph. 240-242 Stockmeyer. 137. 22.. 276. 134-136 Simula. See Polynomial arithmetic Symmetric matrix. 276 Square root. Radix sort. R.A. 270. countsort. Transformation sort Source. See Decision tree TRS-80. 37. 64. 343. 211-213. 277 Trapezoidal algorithm. 22 See also Ancestry in a rooted tree. 27. 35. 320 Strongly connected. 203.J. I.. 167. D. 139-140 Top-down technique. 351 Timetable. 153. 348. 25. 28-30. 275. 35. 291. 279. 69. See k-smooth integer Smooth problem. 275 Sink.. 319 See also Euclidean travelling salesperson Traversal of tree. 19.. See Ad hoc sorting. 350. 350 Stone.. 315 Simplex. 337. 78 Transformation: of the domain. Balanced tree.M.S.. 291. R. 87. 277-291 function. Batcher's sorting circuit. Decision tree. 23 Triangle inequality. 27 sift-up. 144-146 Towers of Hanoi. 349. 103-104. M. 132-133. 199 Stanat. 106. 35. 290. 336 Text editor. Optimal search tree. 205 Szemeredi.. 305. 91.A. 351 2-edge-connected. 203 Size of circuit. 337. 351 See also Equivalence. 345. 178-179. Insertion sort. E. 20 Tally circuit. 109-115. V. 23 Sift-down. 291. 88-90 Splitting. D. 351 Tautology. 350 Strassen. 315. 144. 74. 179-182 Strongly quadratic. 341 sift-down. 293. 35. 21. 168. 35. 301 Sobol'. 141. 327 Turk. Postorder. Minimum spanning tree. 211 Theory of numbers... 106 determination. 30.. 280 Signature. 28. 211. 138 Size of instance. 304 Syntactic analysis. 276. 153 Shub. 326-327. 167.M. 233 Stochastic advantage.. H. 213-222 Tarjan.E. Quicksort. 35 .J. 159-162. N. 275. 342 Steiglitz. 350 Statistical test. 19. xiv. Searching in a tree. 301 Smooth integer.. 107. 301 tablist.. Rooted tree. 168. 273 Simple path. 349 Stinson. See Searching Strong pseudoprime.M.M. 35. Binary tree. 204. 344 Turing. 350 Sort. 227 Simplification. 350 Strassen's algorithm. J. 39. J. 277 Transformed function. 168. 103. Preorder. 46. 103 Triangular matrix. 226.. 252-256 Stack. Searching Tree. 302 Trip. 349. 257 modulo p. 205 Simulation. 15. J. T. 128-132. 336 Transformed domain.. See Numeric integration Traub. 150-153..W. Heapsort. 345. 5 Sloane. 276 Threshold for asymptotic notation. See Inorder. 140. 325. See Telephone switching Symbolic manipulation of polynomials.F. 139-140. 301 Smooth function. 308 String. 108. 351 Travelling salesperson. 43 Threshold for divide-and-conquer. 350 Stirling's formula. 350 slow-make-heap. 30. 123 Tijdeman. 222. 190 Tournament. 142 Topological sorting. 211 Text processing. 167. R.. 140.

D. 140... U. 78. 343 . 349 Ullman. D. 204. 222. 351 Warshall. 275. 274. A. 291. 351 361 Warusfel.. 351 Yoshino. Heapsort Wilson's theorem.. 346 Ultimate.N. 35.. 351 Wirth.. 168. 193. R. 343. 348 wordcnt. See Biconnected Unbiased probabilistic algorithm.F. 308 Yao. J.. 276. S. 276 Undecidable problem. 275.E.J. J. F. 275. J.. S..D. 291. 275.. 10 Winograd. 35. 144-146. F. 167.. 275-276 Unpredictable pseudorandom generator. See Decision tree Valois. 276..W. D. 272. 245-247. 13 Vazirani. 351 See also Heap. Y. 168.W. 351 Warshall's algorithm. R.. 342 Urbanek. 341.C. 351 Zippel. 351 VAX. 351 Yao. 35. 78. 351 Verdict. 167. 168. 164.Index Ulam. 13. D. 168. 228 White. 332. 317. 35.M. 141. 347 Valid.. N.. 346 Wright. 290. M. 236 World Series.W. 204. 351 Ushiro. 351 Wegman. 351 Zuffellato. 275... H.V. 196 Widget. 8. 276. 331 Williams. 21. E. 36. 140. 36.. 344 Well-characterized problem. 225 Unit cost. 351 Virtual initialization. 291. 133. See Analysis of algorithms Wright.. S.. 336. A.. See Threshold for divide-and-conquer Unarticulated.. 351 Williams. 276.J. 304 Universal hashing. 171-176 Uniform random generation. 56. 167. 263 Worst case. 337. 104. 337 Undirected graph.A. 273 Wagner.. 337. 347 Younger. 344. C. 276.H. See Decision tree Vickery. See Elementary operation Unitary matrix. 168. S. 104.. 347 Wood. 351 Welch. P.W. 9.. 275. D. 153. 291.

artificial intelligence. However. operations research. Details each technique in full. cryptography. PRENTICE HALL. computing in the humanities. Now.11fl Mcpo)[PT TIPMT0T( GILLES BRASSARD PAUL BRATLEY The computer explosion has made calculations once thought impossible seem routine. Contains approximately 500 exercises-many of which call for an algorithm to be implemented on a computer so that its efficiency may be measured experimentally and compared to the efficiency of alternative solutions. this innovative new book gives readers the basic tools they need to develop their own algorithms--in whatever field of application they may be required! CONTENT HIGHLIGHTS: Concentrates on the techniques needed to design and analyze algorithms. symbolic computation. Englewood Cliffs. linear algebra. among others. another factor has had an even more important effect in extending the frontiers of feasible computation: the use of efficient algorithms. Illustrates each technique with concrete examples of algorithms taken from such different applications as optimization. numerical analysis. Presents real-life applications for most algorithms. NJ 07632 ISBN 0-13-023243-2 .

","doc_promotions_enabled":false,"static_promo_banner_cta_url":""},"eligible_for_exclusive_trial_roadblock":false,"eligible_for_seo_roadblock":false,"exclusive_free_trial_roadblock_props_path":"/doc-page/exclusive-free-trial-props/328690982","flashes":[],"footer_props":{"urls":{"about":"/about","press":"/press","blog":"","careers":"/careers","contact":"/contact","plans_landing":"/subscribe","referrals":"/referrals?source=footer","giftcards":"/giftcards","faq":"/faq","accessibility":"/accessibility-policy","faq_paths":{"accounts":"","announcements":"","copyright":"","downloading":"","publishing":"","reading":"","selling":"","store":"","status":"","terms":"","writing":"","adchoices":"","paid_features":"","failed_uploads":"","copyright_infringement":"","end_user_license":"","terms_of_use":""},"publishers":"/publishers","static_terms":"/terms","static_privacy":"/privacy","copyright":"/copyright","ios_app":"","android_app":"","books":"/books","sitemap":"/directory"}},"global_nav_props":{"header_props":{"logo_src":"","root_url":"","search_term":"","small_logo_src":"","uploads_url":"/upload-document","search_props":{"redirect_to_app":true,"search_url":"/search","query":"","search_page":false}},"user_menu_props":null,"sidebar_props":{"urls":{"bestsellers":"","home":"","saved":"/saved","subscribe":"/archive/pmp_checkout?doc=328690982&metadata=%7B%22context%22%3A%22pmp%22%2C%22action%22%3A%22start_trial%22%2C%22logged_in%22%3Afalse%2C%22platform%22%3A%22web%22%7D","top_charts":"/bestsellers","upload":""},"categories":{"book":{"icon":"icon-ic_book","icon_filled":"icon-ic_book_fill","url":"","name":"Books","type":"book"},"news":{"icon":"icon-ic_articles","icon_filled":"icon-ic_articles_fill","url":"","name":"News","type":"news"},"audiobook":{"icon":"icon-ic_audiobook","icon_filled":"icon-ic_audiobook_fill","url":"","name":"Audiobooks","type":"audiobook"},"magazine":{"icon":"icon-ic_magazine","icon_filled":"icon-ic_magazine_fill","url":"","name":"Magazines","type":"magazine"},"document":{"icon":"icon-ic_document","icon_filled":"icon-ic_document_fill","url":"","name":"Documents","type":"document"},"sheet_music":{"icon":"icon-ic_songbook","icon_filled":"icon-ic_songbook_fill","url":"","name":"Sheet Music","type":"sheet_music"},"summary":{"icon":"icon-ic_globalnav_snapshot","icon_filled":"icon-ic_globalnav_snapshot_fill","url":"","name":"Snapshots","type":"summary"}},"nav_categories":["mixed","book","audiobook","magazine","document","sheet_music"],"selected_content_type":"mixed","username":"","search_overlay_props":{"search_input_props":{"focused":false,"keep_suggestions_on_blur":false}}}},"recommenders":{"related_titles_recommender":{"ids":[106303356,79231337,23280955,44756594,47769724,258040,39809348,59344566,17158103,23293984,55797032,39971884,45826591,37162291,124632256,48952795,48641567,48828415,49047827,50080038,30079211,49243611,61363785,12410508,49171291,53421562,50920787,51588433,50917268,51163936,253264900,244157917,212863738,239488191,263504218,282766939,224419023,163646054,224258670,238704340,224355300,225916486,224369806,224410295,293461549,239588474,211302755,202691564,182553141,234028503,224349281,224326250,224426877,182565115,254039238,249309502,337536061,224306619,273440534,273582508,249308781,182546874,231689346,224248577,250006950,235583696,224285255,224252178,235411767,224270540,163580087,249308236,224248520,163564256,163579056,182560283,246897514,163603389,163647832,216442529,182522032,224266633,224420008],"title_link":null,"title":null,"track_opts":{"compilation_id":"jj3OGmZvNtiFFRePCWUvCCtt08o=","module_id":"V/zx79o5gXFxpKsnnpWun4W7TSQ=","widget_name":"right sidebar","track_id":"flattened_recommender"}},"footer_recommenders":{"recommenders":[{"ids":[106303356,79231337,23280955,44756594,47769724,258040,39809348,59344566,17158103,23293984,55797032,39971884,45826591,37162291,124632256,48952795,48641567,48828415,49047827,50080038,30079211,49243611,61363785,12410508,49171291,53421562,50920787,51588433,50917268,51163936],"title_link":null,"title":"Documents Similar To 0130232432Algorithmics","track_opts":{"compilation_id":"jj3OGmZvNtiFFRePCWUvCCtt08o=","module_id":"1NjUEESUKB4DZ2IEPgfeNe02kew=","widget_name":"document_carousel"}}]},"seo_new_docs_recommenders":{"recommenders":[]},"documents":{"106303356":{"type":"document","id":106303356,"thumb_url":"","retina_thumb_url":"","title":"Mathematics and Algorithms","short_title":"Mathematics and Algorithms","author":"jlva_lkpb","tracking":{"object_type":"document","object_id":106303356,"track":"flattened_recommender","doc_uuid":"f9X1n51sd4FsjrxD+UwU9duBEfc="},"url":"","top_badge":null},"79231337":{"type":"document","id":79231337,"thumb_url":"","retina_thumb_url":"","title":"CS502_all","short_title":"CS502_all","author":"Faisal Dost Ali","tracking":{"object_type":"document","object_id":79231337,"track":"flattened_recommender","doc_uuid":"UvfEy/+L0eOFTlPhdfI13Q+KNrc="},"url":"","top_badge":null},"23280955":{"type":"document","id":23280955,"thumb_url":"","retina_thumb_url":"","title":"Algo Notes Tirupattur","short_title":"Algo Notes Tirupattur","author":"api-26368914","tracking":{"object_type":"document","object_id":23280955,"track":"flattened_recommender","doc_uuid":"gaxu/lWXFz19HPUosrkv5nlBJuM="},"url":"","top_badge":null},"44756594":{"type":"document","id":44756594,"thumb_url":"","retina_thumb_url":"","title":"Introduction to Lisp","short_title":"Introduction to Lisp","author":"lalitha","tracking":{"object_type":"document","object_id":44756594,"track":"flattened_recommender","doc_uuid":"KjurGHs1gFPM9Z9BUFwI4bSO4LI="},"url":"","top_badge":null},"47769724":{"type":"document","id":47769724,"thumb_url":"","retina_thumb_url":"","title":"Design and Analysis of Algorithm","short_title":"Design and Analysis of Algorithm","author":"Arun Karthik","tracking":{"object_type":"document","object_id":47769724,"track":"flattened_recommender","doc_uuid":"gF3Tgqg6v/vYOl8q0MlRXqG96uA="},"url":"","top_badge":null},"258040":{"type":"document","id":258040,"thumb_url":"","retina_thumb_url":"","title":"Book","short_title":"Book","author":"anon-619902","tracking":{"object_type":"document","object_id":258040,"track":"flattened_recommender","doc_uuid":"H5tpYt3AfhKbqV3mtB5qdpvHLA8="},"url":"","top_badge":null},"39809348":{"type":"document","id":39809348,"thumb_url":"","retina_thumb_url":"","title":"Technical","short_title":"Technical","author":"Swadeesh","tracking":{"object_type":"document","object_id":39809348,"track":"flattened_recommender","doc_uuid":"6Dy0BUbbMI+DwTBN81CBjoI3KZg="},"url":"","top_badge":null},"59344566":{"type":"document","id":59344566,"thumb_url":"","retina_thumb_url":"","title":"Intro Notes","short_title":"Intro Notes","author":"knshiva","tracking":{"object_type":"document","object_id":59344566,"track":"flattened_recommender","doc_uuid":"k60cShRKM01STZ2yLlqjAgCaEis="},"url":"","top_badge":null},"17158103":{"type":"document","id":17158103,"thumb_url":"","retina_thumb_url":"","title":"GA Book","short_title":"GA Book","author":"nagaraju","tracking":{"object_type":"document","object_id":17158103,"track":"flattened_recommender","doc_uuid":"wotRDsHJ7fVdkpygTJ2BGbqKfIU="},"url":"","top_badge":null},"23293984":{"type":"document","id":23293984,"thumb_url":"","retina_thumb_url":"","title":"Problems on Algorithms (2002)","short_title":"Problems on Algorithms (2002)","author":"lcnblzr3877","tracking":{"object_type":"document","object_id":23293984,"track":"flattened_recommender","doc_uuid":"xnti7X/OICcu59u1vY58l7N1InI="},"url":"","top_badge":null},"55797032":{"type":"document","id":55797032,"thumb_url":"","retina_thumb_url":"","title":"\fGraph Theory & Algorithms - M. Ashraf Iqbal","short_title":"\fGraph Theory & Algorithms - M. Ashraf Iqbal","author":"Abdul Rehman Naeem","tracking":{"object_type":"document","object_id":55797032,"track":"flattened_recommender","doc_uuid":"F2ZVh+JvFXYIw5AEyn4FrsgHO5U="},"url":"","top_badge":null},"39971884":{"type":"document","id":39971884,"thumb_url":"","retina_thumb_url":"","title":"ln1ntroduction","short_title":"ln1ntroduction","author":"Alper Cezibarak","tracking":{"object_type":"document","object_id":39971884,"track":"flattened_recommender","doc_uuid":"PWNDuYk9NEFxFBW6WMfFBD5Q4H0="},"url":"","top_badge":null},"45826591":{"type":"document","id":45826591,"thumb_url":"","retina_thumb_url":"","title":"daa","short_title":"daa","author":"Rajesh Manoharan","tracking":{"object_type":"document","object_id":45826591,"track":"flattened_recommender","doc_uuid":"StbBdUU2dgbtbcTk0YCBKPlPouA="},"url":"","top_badge":null},"37162291":{"type":"document","id":37162291,"thumb_url":"","retina_thumb_url":"","title":"DAA Tutorials","short_title":"DAA Tutorials","author":"divyadarshan18","tracking":{"object_type":"document","object_id":37162291,"track":"flattened_recommender","doc_uuid":"nejxpUqApdPZ8X5cDQF3BBpePI4="},"url":"","top_badge":null},"124632256":{"type":"document","id":124632256,"thumb_url":"","retina_thumb_url":"","title":"Fundamentals of Algorithmics - Brassard, Bratley","short_title":"Fundamentals of Algorithmics - Brassard, Bratley","author":"dickcheese9742","tracking":{"object_type":"document","object_id":124632256,"track":"flattened_recommender","doc_uuid":"aZrU6sG7M/CElwQEwuX4HQn2DBc="},"url":"","top_badge":null},"48952795":{"type":"document","id":48952795,"thumb_url":"","retina_thumb_url":"","title":"StaticBS","short_title":"StaticBS","author":"Project_Omega","tracking":{"object_type":"document","object_id":48952795,"track":"flattened_recommender","doc_uuid":"y4BS8UBF6f40uMC4+E/DvnCTWKw="},"url":"","top_badge":null},"48641567":{"type":"document","id":48641567,"thumb_url":"","retina_thumb_url":"","title":"Materials Science and Technology","short_title":"Materials Science and Technology","author":"Juanito Jimenez","tracking":{"object_type":"document","object_id":48641567,"track":"flattened_recommender","doc_uuid":"pgxVCDxHZwj6xYirornQDH6qb4Q="},"url":"","top_badge":null},"48828415":{"type":"document","id":48828415,"thumb_url":"","retina_thumb_url":"","title":"theoryofwavetran","short_title":"theoryofwavetran","author":"Lilian Li","tracking":{"object_type":"document","object_id":48828415,"track":"flattened_recommender","doc_uuid":"Km3FU3Zbg44xitZDdC87fA4KPHQ="},"url":"","top_badge":null},"49047827":{"type":"document","id":49047827,"thumb_url":"","retina_thumb_url":"","title":"Weather_climate_and_Water","short_title":"Weather_climate_and_Water","author":"estebanmatsui4042","tracking":{"object_type":"document","object_id":49047827,"track":"flattened_recommender","doc_uuid":"m2sDRArAHrpC9U2M3P6KXZC8cKM="},"url":"","top_badge":null},"50080038":{"type":"document","id":50080038,"thumb_url":"","retina_thumb_url":"","title":"Econometrics-Creel (2005)","short_title":"Econometrics-Creel (2005)","author":"Atul Tripathi","tracking":{"object_type":"document","object_id":50080038,"track":"flattened_recommender","doc_uuid":"gc/STd/JVe95U/dx1SMBEeWmie4="},"url":"","top_badge":null},"30079211":{"type":"document","id":30079211,"thumb_url":"","retina_thumb_url":"","title":"Complexity of Algorithms","short_title":"Complexity of Algorithms","author":"naty-em777","tracking":{"object_type":"document","object_id":30079211,"track":"flattened_recommender","doc_uuid":"v/7kQMvqH5url0qjpKMOjoS6ZIk="},"url":"","top_badge":null},"49243611":{"type":"document","id":49243611,"thumb_url":"","retina_thumb_url":"","title":"3211766650","short_title":"3211766650","author":"Ramo Dženita","tracking":{"object_type":"document","object_id":49243611,"track":"flattened_recommender","doc_uuid":"MosbBby4mz6BWt2FwK7u4tapB4Y="},"url":"","top_badge":null},"61363785":{"type":"document","id":61363785,"thumb_url":"","retina_thumb_url":"","title":"CS2251-QB","short_title":"CS2251-QB","author":"Kishore Arun Kumar","tracking":{"object_type":"document","object_id":61363785,"track":"flattened_recommender","doc_uuid":"vyGrhEAyVLoC5f3Ny6QmW7HXfEU="},"url":"","top_badge":null},"12410508":{"type":"document","id":12410508,"thumb_url":"","retina_thumb_url":"","title":"Theory of Computation(toc) - Eitan Gurari","short_title":"Theory of Computation(toc) - Eitan Gurari","author":"hemant","tracking":{"object_type":"document","object_id":12410508,"track":"flattened_recommender","doc_uuid":"ZHqzinSEHumPFn7jRf3tIC3H+c8="},"url":"","top_badge":null},"49171291":{"type":"document","id":49171291,"thumb_url":"","retina_thumb_url":"","title":"Applied_Numerical_Computing","short_title":"Applied_Numerical_Computing","author":"Adi Sentosa","tracking":{"object_type":"document","object_id":49171291,"track":"flattened_recommender","doc_uuid":"t6xkRnp7lOsNItSwxeSW9qyS0fg="},"url":"","top_badge":null},"53421562":{"type":"document","id":53421562,"thumb_url":"","retina_thumb_url":"","title":"DAA2M","short_title":"DAA2M","author":"Mohan Mohanbagavathi","tracking":{"object_type":"document","object_id":53421562,"track":"flattened_recommender","doc_uuid":"bhFYImA18/BkwHZKM6FbAmwF49A="},"url":"","top_badge":null},"50920787":{"type":"document","id":50920787,"thumb_url":"","retina_thumb_url":"","title":"Against the Theory of ‘Dynamic Equivalence","short_title":"Against the Theory of ‘Dynamic Equivalence","author":"Spirit of William Tyndale","tracking":{"object_type":"document","object_id":50920787,"track":"flattened_recommender","doc_uuid":"fqhhAMI7LVp1L6cUUnsivzDrtw4="},"url":"","top_badge":null},"51588433":{"type":"document","id":51588433,"thumb_url":"","retina_thumb_url":"","title":"selva","short_title":"selva","author":"SIVAMCHITRA255958","tracking":{"object_type":"document","object_id":51588433,"track":"flattened_recommender","doc_uuid":"0GVSvKdMPfSoJeYMua8endNZmb0="},"url":"","top_badge":null},"50917268":{"type":"document","id":50917268,"thumb_url":"","retina_thumb_url":"","title":"A Working Method Approach for Introductory Physical Chemistry Calculations","short_title":"A Working Method Approach for Introductory Physical Chemistry Calculations","author":"Ridwan Baharum","tracking":{"object_type":"document","object_id":50917268,"track":"flattened_recommender","doc_uuid":"xMCi2QorjjPHG+S1HttmQ3ENIgU="},"url":"","top_badge":null},"51163936":{"type":"document","id":51163936,"thumb_url":"","retina_thumb_url":"","title":"daa 1mark questions and answers","short_title":"daa 1mark questions and answers","author":"koteswararaopittala","tracking":{"object_type":"document","object_id":51163936,"track":"flattened_recommender","doc_uuid":"Ho4tDxjMHYmpyK1eX4uFyDAtoLg="},"url":"","top_badge":null},"253264900":{"type":"book","id":253264900,"thumb_url":"","retina_thumb_url":"","title":"Sapiens: A Brief History of Humankind","short_title":"Sapiens","author":"Yuval Noah Harari","tracking":{"object_type":"document","object_id":253264900,"track":"flattened_recommender","doc_uuid":"Pv0jx8+XPudTqkdvskSNnpHwkWs="},"url":"","top_badge":null},"244157917":{"type":"book","id":244157917,"thumb_url":"","retina_thumb_url":"","title":"Yes Please","short_title":"Yes Please","author":"Amy Poehler","tracking":{"object_type":"document","object_id":244157917,"track":"flattened_recommender","doc_uuid":"ZDBqAxpVL3t3ZNHK07Bf4fEYD58="},"url":"","top_badge":null},"212863738":{"type":"book","id":212863738,"thumb_url":"","retina_thumb_url":"","title":"The Unwinding: An Inner History of the New America","short_title":"The Unwinding","author":"George Packer","tracking":{"object_type":"document","object_id":212863738,"track":"flattened_recommender","doc_uuid":"KYbqIwaMZ5+isJhO/ylg/2exWlU="},"url":"","top_badge":null},"239488191":{"type":"book","id":239488191,"thumb_url":"","retina_thumb_url":"","title":"The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution","short_title":"The Innovators","author":"Walter Isaacson","tracking":{"object_type":"document","object_id":239488191,"track":"flattened_recommender","doc_uuid":"DYqtuzZy7lGIxxICbv0vDb98oHg="},"url":"","top_badge":null},"263504218":{"type":"book","id":263504218,"thumb_url":"","retina_thumb_url":"","title":"Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future","short_title":"Elon Musk","author":"Ashlee Vance","tracking":{"object_type":"document","object_id":263504218,"track":"flattened_recommender","doc_uuid":"m7JdHuVt9I4Q6ZNoeb3s1AvGP6g="},"url":"","top_badge":null},"282766939":{"type":"book","id":282766939,"thumb_url":"","retina_thumb_url":"","title":"Dispatches from Pluto: Lost and Found in the Mississippi Delta","short_title":"Dispatches from Pluto","author":"Richard Grant","tracking":{"object_type":"document","object_id":282766939,"track":"flattened_recommender","doc_uuid":"U0QrYPKqrWbsXvFOR+D4BH5lTQo="},"url":"","top_badge":null},"224419023":{"type":"book","id":224419023,"thumb_url":"","retina_thumb_url":"","title":"John Adams","short_title":"John Adams","author":"David McCullough","tracking":{"object_type":"document","object_id":224419023,"track":"flattened_recommender","doc_uuid":"Y4aN81BTc9vnlj2L3rUkyBg+gE8="},"url":"","top_badge":null},"163646054":{"type":"book","id":163646054,"thumb_url":"","retina_thumb_url":"","title":"Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America","short_title":"Devil in the Grove","author":"Gilbert King","tracking":{"object_type":"document","object_id":163646054,"track":"flattened_recommender","doc_uuid":"SryaDV1L3ZOkmV3+ahOOItzKzCE="},"url":"","top_badge":null},"224258670":{"type":"book","id":224258670,"thumb_url":"","retina_thumb_url":"","title":"The Prize: The Epic Quest for Oil, Money & Power","short_title":"The Prize","author":"Daniel Yergin","tracking":{"object_type":"document","object_id":224258670,"track":"flattened_recommender","doc_uuid":"yGSsZLTnDujr35zYu93wA0PFtIk="},"url":"","top_badge":null},"238704340":{"type":"book","id":238704340,"thumb_url":"","retina_thumb_url":"","title":"This Changes Everything: Capitalism vs. The Climate","short_title":"This Changes Everything","author":"Naomi Klein","tracking":{"object_type":"document","object_id":238704340,"track":"flattened_recommender","doc_uuid":"1OyGRWOeZ3S9/urxMZU0StP2wDI="},"url":"","top_badge":null},"224355300":{"type":"book","id":224355300,"thumb_url":"","retina_thumb_url":"","title":"Grand Pursuit: The Story of Economic Genius","short_title":"Grand Pursuit","author":"Sylvia Nasar","tracking":{"object_type":"document","object_id":224355300,"track":"flattened_recommender","doc_uuid":"YIf3200JMBckyAyKhmwTvDdcyBo="},"url":"","top_badge":null},"225916486":{"type":"book","id":225916486,"thumb_url":"","retina_thumb_url":"","title":"A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story","short_title":"A Heartbreaking Work Of Staggering Genius","author":"Dave Eggers","tracking":{"object_type":"document","object_id":225916486,"track":"flattened_recommender","doc_uuid":"x5o4nFDApmbBB7/4PeNy4jy4jMw="},"url":"","top_badge":null},"224369806":{"type":"book","id":224369806,"thumb_url":"","retina_thumb_url":"","title":"The Emperor of All Maladies: A Biography of Cancer","short_title":"The Emperor of All Maladies","author":"Siddhartha Mukherjee","tracking":{"object_type":"document","object_id":224369806,"track":"flattened_recommender","doc_uuid":"gnoIz1HVRRkizQ9BlYFY6BMwZJY="},"url":"","top_badge":null},"224410295":{"type":"book","id":224410295,"thumb_url":"","retina_thumb_url":"","title":"Team of Rivals: The Political Genius of Abraham Lincoln","short_title":"Team of Rivals","author":"Doris Kearns Goodwin","tracking":{"object_type":"document","object_id":224410295,"track":"flattened_recommender","doc_uuid":"GFR6sJXmlMcoad9+3pIHq340AjM="},"url":"","top_badge":null},"293461549":{"type":"book","id":293461549,"thumb_url":"","retina_thumb_url":"","title":"The New Confessions of an Economic Hit Man","short_title":"The New Confessions of an Economic Hit Man","author":"John Perkins","tracking":{"object_type":"document","object_id":293461549,"track":"flattened_recommender","doc_uuid":"Am8iWbCpDhK/qX8Ijy/b6yyeMo8="},"url":"","top_badge":null},"239588474":{"type":"book","id":239588474,"thumb_url":"","retina_thumb_url":"","title":"Rise of ISIS: A Threat We Can't Ignore","short_title":"Rise of ISIS","author":"Jay Sekulow","tracking":{"object_type":"document","object_id":239588474,"track":"flattened_recommender","doc_uuid":"xLcDDJLOxvslcoGt4njeDH0vwLM="},"url":"","top_badge":null},"211302755":{"type":"book","id":211302755,"thumb_url":"","retina_thumb_url":"","title":"The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers","short_title":"The Hard Thing About Hard Things","author":"Ben Horowitz","tracking":{"object_type":"document","object_id":211302755,"track":"flattened_recommender","doc_uuid":"kDMwpvoRYRLQbez5embifq75/6s="},"url":"","top_badge":null},"202691564":{"type":"book","id":202691564,"thumb_url":"","retina_thumb_url":"","title":"Smart People Should Build Things: How to Restore Our Culture of Achievement, Build a Path for Entrepreneurs, and Create New Jobs in America","short_title":"Smart People Should Build Things","author":"Andrew Yang","tracking":{"object_type":"document","object_id":202691564,"track":"flattened_recommender","doc_uuid":"hL9wEwTuW8qDa58kFbqr31Zm54g="},"url":"","top_badge":null},"182553141":{"type":"book","id":182553141,"thumb_url":"","retina_thumb_url":"","title":"The World Is Flat 3.0: A Brief History of the Twenty-first Century","short_title":"The World Is Flat 3.0","author":"Thomas L. Friedman","tracking":{"object_type":"document","object_id":182553141,"track":"flattened_recommender","doc_uuid":"HEmhJkG8JXg3dY7Q3xfTp8nWAPU="},"url":"","top_badge":null},"234028503":{"type":"book","id":234028503,"thumb_url":"","retina_thumb_url":"","title":"Bad Feminist: Essays","short_title":"Bad Feminist","author":"Roxane Gay","tracking":{"object_type":"document","object_id":234028503,"track":"flattened_recommender","doc_uuid":"Qda4O1q6HjcY3h0TwLgI2obBznQ="},"url":"","top_badge":null},"224349281":{"type":"book","id":224349281,"thumb_url":"","retina_thumb_url":"","title":"How To Win Friends and Influence People","short_title":"How To Win Friends and Influence People","author":"Dale Carnegie","tracking":{"object_type":"document","object_id":224349281,"track":"flattened_recommender","doc_uuid":"GweCV1QaLjp9yMIiHJbAiRz3RGs="},"url":"","top_badge":null},"224326250":{"type":"book","id":224326250,"thumb_url":"","retina_thumb_url":"","title":"Steve Jobs","short_title":"Steve Jobs","author":"Walter Isaacson","tracking":{"object_type":"document","object_id":224326250,"track":"flattened_recommender","doc_uuid":"hG4AgxPDen+6cNrR3BE4qOe1HB0="},"url":"","top_badge":null},"224426877":{"type":"book","id":224426877,"thumb_url":"","retina_thumb_url":"","title":"Angela's Ashes: A Memoir","short_title":"Angela's Ashes","author":"Frank McCourt","tracking":{"object_type":"document","object_id":224426877,"track":"flattened_recommender","doc_uuid":"xg7f/dlUrWJ0EeXM9RTEniT1IFw="},"url":"","top_badge":null},"182565115":{"type":"book","id":182565115,"thumb_url":"","retina_thumb_url":"","title":"The Silver Linings Playbook: A Novel","short_title":"The Silver Linings Playbook","author":"Matthew Quick","tracking":{"object_type":"document","object_id":182565115,"track":"flattened_recommender","doc_uuid":"yRQKodmvtZXYToiRPJOP7q7Mecg="},"url":"","top_badge":null},"254039238":{"type":"book","id":254039238,"thumb_url":"","retina_thumb_url":"","title":"Leaving Berlin: A Novel","short_title":"Leaving Berlin","author":"Joseph Kanon","tracking":{"object_type":"document","object_id":254039238,"track":"flattened_recommender","doc_uuid":"oCY81RPtmlh155AoWz5zjJAJwmc="},"url":"","top_badge":null},"249309502":{"type":"book","id":249309502,"thumb_url":"","retina_thumb_url":"","title":"Extremely Loud and Incredibly Close: A Novel","short_title":"Extremely Loud and Incredibly Close","author":"Jonathan Safran Foer","tracking":{"object_type":"document","object_id":249309502,"track":"flattened_recommender","doc_uuid":"QcpYondZKLeMB0rA61ywkOI6tT8="},"url":"","top_badge":null},"337536061":{"type":"book","id":337536061,"thumb_url":"","retina_thumb_url":"","title":"The Sympathizer: A Novel (Pulitzer Prize for Fiction)","short_title":"The Sympathizer","author":"Viet Thanh Nguyen","tracking":{"object_type":"document","object_id":337536061,"track":"flattened_recommender","doc_uuid":"v68xmyhUh73zjJrjjkGmVbLWu4Q="},"url":"","top_badge":null},"224306619":{"type":"book","id":224306619,"thumb_url":"","retina_thumb_url":"","title":"The Light Between Oceans: A Novel","short_title":"The Light Between Oceans","author":"M.L. Stedman","tracking":{"object_type":"document","object_id":224306619,"track":"flattened_recommender","doc_uuid":"n2iqnqCHy7WqQKpYe+AWccJPx1M="},"url":"","top_badge":null},"273440534":{"type":"book","id":273440534,"thumb_url":"","retina_thumb_url":"","title":"The Incarnations: A Novel","short_title":"The Incarnations","author":"Susan Barker","tracking":{"object_type":"document","object_id":273440534,"track":"flattened_recommender","doc_uuid":"ySaIRersBk48IocCdfe+SuPe2gY="},"url":"","top_badge":null},"273582508":{"type":"book","id":273582508,"thumb_url":"","retina_thumb_url":"","title":"You Too Can Have a Body Like Mine: A Novel","short_title":"You Too Can Have a Body Like Mine","author":"Alexandra Kleeman","tracking":{"object_type":"document","object_id":273582508,"track":"flattened_recommender","doc_uuid":"9D/DyGkFH5c4lFNI7u3HC4TGSIw="},"url":"","top_badge":null},"249308781":{"type":"book","id":249308781,"thumb_url":"","retina_thumb_url":"","title":"Life of Pi","short_title":"Life of Pi","author":"Yann Martel","tracking":{"object_type":"document","object_id":249308781,"track":"flattened_recommender","doc_uuid":"7cz20HyUfbSBY5N/FTRkOsWZ1JU="},"url":"","top_badge":null},"182546874":{"type":"book","id":182546874,"thumb_url":"","retina_thumb_url":"","title":"The Love Affairs of Nathaniel P.: A Novel","short_title":"The Love Affairs of Nathaniel P.","author":"Adelle Waldman","tracking":{"object_type":"document","object_id":182546874,"track":"flattened_recommender","doc_uuid":"7MbT/5ra2U55AsF8NZtyQ6ehM20="},"url":"","top_badge":null},"231689346":{"type":"book","id":231689346,"thumb_url":"","retina_thumb_url":"","title":"The Blazing World: A Novel","short_title":"The Blazing World","author":"Siri Hustvedt","tracking":{"object_type":"document","object_id":231689346,"track":"flattened_recommender","doc_uuid":"3frFpouxAXFDYSDKV5zUOpAumhk="},"url":"","top_badge":null},"224248577":{"type":"book","id":224248577,"thumb_url":"","retina_thumb_url":"","title":"The Rosie Project: A Novel","short_title":"The Rosie Project","author":"Graeme Simsion","tracking":{"object_type":"document","object_id":224248577,"track":"flattened_recommender","doc_uuid":"BkQZdRAXVD7qH0ozlBRR6DNVEv4="},"url":"","top_badge":null},"250006950":{"type":"book","id":250006950,"thumb_url":"","retina_thumb_url":"","title":"The First Bad Man: A Novel","short_title":"The First Bad Man","author":"Miranda July","tracking":{"object_type":"document","object_id":250006950,"track":"flattened_recommender","doc_uuid":"OSUEtNirQYRwBXV1EWJ8PRVQZbE="},"url":"","top_badge":null},"235583696":{"type":"book","id":235583696,"thumb_url":"","retina_thumb_url":"","title":"We Are Not Ourselves: A Novel","short_title":"We Are Not Ourselves","author":"Matthew Thomas","tracking":{"object_type":"document","object_id":235583696,"track":"flattened_recommender","doc_uuid":"di5IS+it+uxNp/S1mHPUDiIKClQ="},"url":"","top_badge":null},"224285255":{"type":"book","id":224285255,"thumb_url":"","retina_thumb_url":"","title":"The Flamethrowers: A Novel","short_title":"The Flamethrowers","author":"Rachel Kushner","tracking":{"object_type":"document","object_id":224285255,"track":"flattened_recommender","doc_uuid":"prDRCP4qc3kWWD6ULyvShWKIOw8="},"url":"","top_badge":null},"224252178":{"type":"book","id":224252178,"thumb_url":"","retina_thumb_url":"","title":"Brooklyn: A Novel","short_title":"Brooklyn","author":"Colm Tóibín","tracking":{"object_type":"document","object_id":224252178,"track":"flattened_recommender","doc_uuid":"wV1nWUedkrHYGdOMavvQmg/oUKA="},"url":"","top_badge":null},"235411767":{"type":"book","id":235411767,"thumb_url":"","retina_thumb_url":"","title":"A Man Called Ove: A Novel","short_title":"A Man Called Ove","author":"Fredrik Backman","tracking":{"object_type":"document","object_id":235411767,"track":"flattened_recommender","doc_uuid":"zZBqHUpP7mdu5XMFY6TyAJg3R18="},"url":"","top_badge":null},"224270540":{"type":"book","id":224270540,"thumb_url":"","retina_thumb_url":"","title":"The Master","short_title":"The Master","author":"Colm Tóibín","tracking":{"object_type":"document","object_id":224270540,"track":"flattened_recommender","doc_uuid":"0ntLDhcj9xY/osr1omdIRVPWrQQ="},"url":"","top_badge":null},"163580087":{"type":"book","id":163580087,"thumb_url":"","retina_thumb_url":"","title":"Bel Canto","short_title":"Bel Canto","author":"Ann Patchett","tracking":{"object_type":"document","object_id":163580087,"track":"flattened_recommender","doc_uuid":"33sigbkKUi4o7qUeiC/SFsT9kyo="},"url":"","top_badge":null},"249308236":{"type":"book","id":249308236,"thumb_url":"","retina_thumb_url":"","title":"Interpreter of Maladies","short_title":"Interpreter of Maladies","author":"Jhumpa Lahiri","tracking":{"object_type":"document","object_id":249308236,"track":"flattened_recommender","doc_uuid":"2WQ4F9ZdH5Z+tACX+T8NWpT38Xo="},"url":"","top_badge":null},"224248520":{"type":"book","id":224248520,"thumb_url":"","retina_thumb_url":"","title":"The Kitchen House: A Novel","short_title":"The Kitchen House","author":"Kathleen Grissom","tracking":{"object_type":"document","object_id":224248520,"track":"flattened_recommender","doc_uuid":"wDUiw64SPtv4gV966YloF/f7MgI="},"url":"","top_badge":null},"163564256":{"type":"book","id":163564256,"thumb_url":"","retina_thumb_url":"","title":"Beautiful Ruins: A Novel","short_title":"Beautiful Ruins","author":"Jess Walter","tracking":{"object_type":"document","object_id":163564256,"track":"flattened_recommender","doc_uuid":"gq3Ya60xSpBwnKafhoIc3R659GM="},"url":"","top_badge":null},"163579056":{"type":"book","id":163579056,"thumb_url":"","retina_thumb_url":"","title":"The Art of Racing in the Rain: A Novel","short_title":"The Art of Racing in the Rain","author":"Garth Stein","tracking":{"object_type":"document","object_id":163579056,"track":"flattened_recommender","doc_uuid":"jrSvpYURGoFwR/HIadYgyZMtoX0="},"url":"","top_badge":null},"182560283":{"type":"book","id":182560283,"thumb_url":"","retina_thumb_url":"","title":"Wolf Hall: A Novel","short_title":"Wolf Hall","author":"Hilary Mantel","tracking":{"object_type":"document","object_id":182560283,"track":"flattened_recommender","doc_uuid":"m87IcUTueZWL6y2EzUYgmQHkisM="},"url":"","top_badge":null},"246897514":{"type":"book","id":246897514,"thumb_url":"","retina_thumb_url":"","title":"The Wallcreeper","short_title":"The Wallcreeper","author":"Nell Zink","tracking":{"object_type":"document","object_id":246897514,"track":"flattened_recommender","doc_uuid":"YwAY34TUsKhAr8gt3KbcbYldzzY="},"url":"","top_badge":null},"163603389":{"type":"book","id":163603389,"thumb_url":"","retina_thumb_url":"","title":"A Prayer for Owen Meany: A Novel","short_title":"A Prayer for Owen Meany","author":"John Irving","tracking":{"object_type":"document","object_id":163603389,"track":"flattened_recommender","doc_uuid":"NBL0B6a76Jdsio3UB9mI7HVCKJU="},"url":"","top_badge":null},"163647832":{"type":"book","id":163647832,"thumb_url":"","retina_thumb_url":"","title":"The Cider House Rules","short_title":"The Cider House Rules","author":"John Irving","tracking":{"object_type":"document","object_id":163647832,"track":"flattened_recommender","doc_uuid":"Asu234BbJasHiZgYD4hz+e22Mpk="},"url":"","top_badge":null},"216442529":{"type":"book","id":216442529,"thumb_url":"","retina_thumb_url":"","title":"Lovers at the Chameleon Club, Paris 1932: A Novel","short_title":"Lovers at the Chameleon Club, Paris 1932","author":"Francine Prose","tracking":{"object_type":"document","object_id":216442529,"track":"flattened_recommender","doc_uuid":"n64VjQMeW3JqTVsrHutdqPTuaig="},"url":"","top_badge":null},"182522032":{"type":"book","id":182522032,"thumb_url":"","retina_thumb_url":"","title":"The Bonfire of the Vanities: A Novel","short_title":"The Bonfire of the Vanities","author":"Tom Wolfe","tracking":{"object_type":"document","object_id":182522032,"track":"flattened_recommender","doc_uuid":"bKL6+b/f4NJ+gKkQej/i/icxS2E="},"url":"","top_badge":null},"224266633":{"type":"book","id":224266633,"thumb_url":"","retina_thumb_url":"","title":"The Perks of Being a Wallflower","short_title":"The Perks of Being a Wallflower","author":"Stephen Chbosky","tracking":{"object_type":"document","object_id":224266633,"track":"flattened_recommender","doc_uuid":"trQqievfxgLzlFuv+G2VCEEBL7g="},"url":"","top_badge":null},"224420008":{"type":"book","id":224420008,"thumb_url":"","retina_thumb_url":"","title":"The Constant Gardener: A Novel","short_title":"The Constant Gardener","author":"John le Carré","tracking":{"object_type":"document","object_id":224420008,"track":"flattened_recommender","doc_uuid":"65M2Q7Rw427A2QDjEZTP+kJ6DjQ="},"url":"","top_badge":null}}},"seo_roadblock_props_path":"/doc-page/seo-roadblock-props/328690982","signup_context":null,"toolbar":{"search_path":"/search-4gen?allowed_pages=1%2C2%2C3%2C4%2C5%2C6%2C7%2C81%2C82%2C83%2C84%2C85%2C86%2C87%2C274%2C275%2C276%2C277%2C278%2C279%2C280%2C330%2C331%2C332%2C333%2C334%2C335%2C336%2C339%2C340%2C341%2C342%2C343%2C344%2C345&auth_token=Wy1%2F4Sh37IEH6JewiriR%2BNWc1CY%3D&authenticity_token=0j3PKwgwlEQAyQSKkrJpgcPlZSpjFiYj3TtROpBFUlJXCq1NOGqYyA8aweNVfof1rLpMKCIQBXcH8qNY7vAgVg%3D%3D&expires=1540239082&wordDocumentId=328690982&wordUploadId=334515607"},"renewal_nag_props":null}-->