If I understand the code of G4MagHelicalStepper.cc right, I would replace the line CosT = 1.0 - 2.0 * SinT * SinT; by CosT = 1.0 - 4.0 * SinT * SintT;
Please, note this is still referring to Geant4 1.0 !
Problem now under investigation.
Report from V.Grichine analysis of the problem: G4MagHelicalStepper has been checked and did not find error. The formula below: cos(2*theta) = cos^2(theta) - sin^2(theta) = 1 - 2*sin^2(theta) is correctly represented.