Cipolla's algorithm returns a square root, and each trial succeeds with probability (p - 1)/(2p) full
This proof shows: correctness.
Claim. Let p be an odd prime and n an integer. If p∣n, the algorithm returns 0. If n is a quadratic nonresidue modulo p, it returns NONE. Otherwise, each trial of lines L3 and L4 succeeds with probability exactly (p−1)/(2p), so the loop ends with probability 1, and line L11 returns x with x2≡n(modp).
Lines L1 and L2. After line L1, 0≤n<p, and 0 is a square root of 0. For n=0, Euler's criterion says that n(p−1)/2≡1 if n is a quadratic residue and ≡−1 otherwise. So line L2 returns NONE exactly for nonresidues. From now on n≡0 is a residue, say n≡r2 with r≡0.
The trials succeed often. Write χ(t)=t(p−1)/2modp, read as −1, 0 or 1; by Euler's criterion it is the Legendre symbol. For each a∈Fp, the number of b∈Fp with b2=a2−n is 1+χ(a2−n). So the number S of pairs (a,b) with a2−b2=n is ∑a(1+χ(a2−n))=p+∑aχ(a2−n). On the other hand, (a,b)↦(u,v)=(a−b,a+b) is a bijection of Fp2, because 2 is invertible, and a2−b2=uv. The equation uv=n with n=0 has exactly p−1 solutions, one for each u=0. So S=p−1 and ∑aχ(a2−n)=−1.
Let N0, N+ and N− count the a with χ(a2−n) equal to 0, 1 and −1. Then N0=2, for a=±r, which are distinct since p is odd. Also N0+N++N−=p and N+−N−=−1. Hence N++N−=p−2 and N−=(p−1)/2. Line L3 chooses a uniformly in {0,…,p−1}, and line L4 accepts exactly when χ(d)=−1 with d=a2−n. So each trial succeeds with probability (p−1)/(2p)≥1/3, independently of the other trials, and the loop ends with probability 1.
The field. Now d is a nonresidue, so X2−d has no root in Fp and is irreducible. The ring F=Fp[X]/(X2−d) is a field with p2 elements. Write ω for the class of X, so ω2=d, and every element is x+yω with x,y∈Fp, stored as the pair (x,y). The product is
(x+yω)(b+cω)=(xb+ycd)+(xc+yb)ω, which is line L8, and the square of b+cω is (b2+c2d)+2bcω, which is line L9.
The power. Lines L5 to L10 are binary exponentiation in F: with e0=(p+1)/2, the loop keeps the invariant (x+yω)(b+cω)e=(a+ω)e0, since line L8 moves one factor b+cω into x+yω when e is odd, and lines L9 and L10 square the base and halve e. When e=0, x+yω=(a+ω)(p+1)/2=:z.
z2=n. In a field of characteristic p, (u+v)p=up+vp, because the binomial coefficients (ip) with 0<i<p are multiples of p. By Fermat's little theorem, ap=a for a∈Fp. Also ωp=ω⋅(ω2)(p−1)/2=ωd(p−1)/2=−ω, since d is a nonresidue. So (a+ω)p=a−ω and
z2=(a+ω)p+1=(a+ω)p(a+ω)=(a−ω)(a+ω)=a2−d=n. z lies in Fp. In the field F, the polynomial T2−n has at most two roots. The elements r and −r of Fp are two different roots, so z∈{r,−r}⊆Fp. Hence y=0, and line L11 returns x=z with x2≡n(modp). ■