Share the orbit and canonical form code with ModIsom

[fingolfin]

AutPGrp and ModIsom both compute stabilisers and canonical forms of subspaces under a hybrid group (a soluble normal subgroup given by a pcgs, and a permutation representation of the quotient). Bettina Eick suggested extracting that into a package both use. This issue records what is known and the order in which to do it.

State (September 2026)

The p-group layer is one algorithm in two copies. #30 ported ModIsom's gap/cfstab/pgroup.gi (Schwingel's canonical forms of vectors and subspaces under a unitriangular pcgs) to gap/cfstab.gi, and the four speed-ups on ModIsom's PR gap-packages/modisom#34 were applied to both. After renaming, VectorCanonicalForm / PGVectorCanonicalForm and SubspaceCanonicalForm / PGSubspaceCanonicalForm differ only in

• the identity: one is an argument in AutPGrp, pcgs[1]^0 in ModIsom;
• AutPGrp dropped the fallback for a base that is not square;
• Assert (AutPGrp) against the global CHECK_CNF (ModIsom).

Both take a pcgs of pairs [ elm, mat ] and use nothing else of their package, so this layer can be extracted as it is.

The hybrid layer did not converge. Each package got the other's idea written again in its own data model:

	ModIsom BlockCanonicalForm	AutPGrp PGHybridOrbitStabilizer
perm rep of A/S	glPerms	glOper
ag part	a p-group, no relative orders	soluble, agOrder
elements	pairs [ elm, mat ]	PGAutomorphism with !.mat, plus induce
result	minimum of the whole gl-orbit and a transversal element	the stabiliser only
has	canonical forms under the p-part; dictionary (new)	Schreier-vector transversal, OrbitLimit and work budget, set stabiliser (permstab.gi), niceing, canonical forms under the kernel on the Frattini quotient (#30)
series splitting	BlockCanonicalFormBySeries	PGOrbitStabilizerBySeries

The "result" row is the mathematical difference: ModIsom needs a canonical representative, so it must enumerate the gl-orbit; the set stabiliser and the early exit only make sense for a stabiliser. A shared routine needs two modes, or a canonical-image mode (as the images package provides for permutation groups).

Other duplicates: ModIsom ChainStabilizer = AutPGrp StabilizingMatrixGroup, BasisBySubspaces = ChainByCollection (gap/autiso/chains.gi, gap/matrix.gi). Sophus keeps its own copies of nicestab.gi and initauts.gi.

Consumers and what they need. Three primitives occur:

1. stabiliser of one subspace: AutPGrp, ANUPQ and Polycyclic through PGOrbitStabilizer, Sophus through PGOrbitStabilizer with its own element type (IsNilpotentLieAutomorphism, methods for PGMult, PGInverse, PGPower), Polycyclic's SchurCovers through PGHybridOrbitStabilizer with subgroups as points and the automorphisms in place of matrices (Accept six arguments in PGHybridOrbitStabilizer again #36 is what happens when that is forgotten);
2. canonical representative with transversal element: ModIsom; Sophus' AreIsomorphicNilpotentLieAlgebras enumerates a whole orbit with GAP's OrbitStabilizer for the same purpose;
3. all orbits on the allowable subgroups, points as integer labels (O'Brien): ANUPQ, Sophus gap/allowable.gi.

Where the code can live. ModIsom needs Polycyclic, which needs autpgrp >= 1.6; ANUPQ, Polycyclic and Sophus need AutPGrp directly. So everything can call AutPGrp today without a new dependency, and AutPGrp can depend on nothing. A separate package would have to be dependency-free and in GAP's default load list, as AutPGrp is.

Plan

1. Share the p-group layer: ModIsom calls PGVectorCanonicalForm and PGSubspaceCanonicalForm (or the file moves to a new package and both call it). Until then, keep the copies in sync with a differential test: the same random pcgs and points into both implementations, results compared element by element, for the fields GF(2), GF(4), GF(3^6), GF(257), GF(2^9).
2. Write the hybrid-group interface down once, and test it: product, inverse, power, matrix action, a faithful perm rep of A/S on which the ag part acts trivially, a pcgs with relative orders, and the start of the p-kernel (PGKernelTail reads PGAutomorphism internals for that; the caller should pass it). Sophus shows the record is already nearly element-agnostic. Add CI tests running autpgrp, modisom, polycyclic, sophus test suites #37 covers the tests on the consumers' side.
3. One orbit routine with two modes, stabiliser and canonical representative, and one series splitting. This is the design step; the first two are bookkeeping.

Not in scope: the third primitive (label-encoded orbits), and the flags of AutPGrp (#34).