How Cygnet-G31B Preserved Product Quality at Smaller Scale | Lawpath Engineering

We wanted to answer a practical question: could a smaller open model run the high-volume parts of Lawpath Atlas without lowering the quality users experience?

We started with Gemma 4 31B, added a small specialised training layer, and called the result Cygnet-G31B. We then tested it inside the real Atlas setup, with the same instructions, document search, tools, evidence, and interface used by our production model.

The result was not a universal replacement for a frontier model. It was more useful than that: on the product checks that matter most to Atlas, Cygnet-G31B made fewer mistakes while costing 4x less to serve in our measured setup.

The question we tested

Lawpath Atlas does more than write answers. It must decide when to search, recommend real documents, follow structured response formats, stay within product boundaries, and hand a matter to a qualified professional when needed.

Our main question was:

Can a smaller open model, given the same instructions, tools, and evidence, become a more disciplined Atlas assistant than our production baseline?

We also tested a scale-versus-specialisation hypothesis. We expected targeted training on verified Atlas runs to help Cygnet-G31B match or beat GLM-5.3, a 744-billion-parameter open model, on structured outputs and unsupported claims in Australian legal and compliance scenarios, despite being roughly 24 times smaller overall.

Published model tests could not answer that question. They focus on broad capabilities such as coding and general problem-solving, not our document library, Australian compliance material, or the response formats Atlas uses [9]. We had to test it in the product.

How we built the training set

We ran GLM-5.3 on our own cloud servers as an offline teacher and reference model. It produced candidate Atlas runs showing when to search, how to use retrieved evidence, and how to return the final answer.

We applied strict boundaries from the start:

No customer data. Every training and evaluation scenario was synthetic. We used no customer conversations, documents, or personal information.
Evidence over memory. The model was trained to answer from retrieved material and to stop or clarify when the evidence was missing, following the principle behind RAFT [2].
Hard filtering. We rejected candidates with unsupported claims, invented links, or broken tool sequences. A fluent answer was not enough.
Separate comparison arms. Outputs from the production model and automated judges never entered the training set.
Product boundaries counted as quality. Escalating a matter to a lawyer or tax professional was scored as correct when tailored advice was required.

This produced about 550 verified tool-using runs, each using the full Atlas prompt and averaging roughly 40,000 tokens.

We trained with QLoRA, a memory-efficient method that adjusts a small adapter instead of every parameter in the base model [3][4]. Training took about 17 hours on one high-end data-centre GPU. We supervised the model’s answers and tool calls, but not the raw text returned by tools, so it learned what actions to take rather than memorising retrieved content [5].

We kept the setup deliberately simple. That made it easier to identify whether an improvement came from the data, the training, or the surrounding product.

Running the large teacher only when needed

GLM-5.3 activates about 40 billion parameters for each answer, but the complete 744-billion-parameter model still has to be available. We therefore spread it across multiple GPUs and configured the serving software to distribute its specialised components between them [8][9].

Getting it running was iterative. Early attempts ran out of memory while loading, left too little working space for long Atlas prompts, or failed because the model had been split across the GPUs incorrectly. We adjusted how the weights were staged, memory was allocated, and the model was divided until it could reliably complete the frozen test set.

This was a batch system, not an always-on service. We started it on discounted, interruptible cloud capacity only when teacher examples or reference outputs were needed, saved the results, and shut it down afterwards [10]. That controlled cost, but it was not treated as a privacy or compliance feature.

What the real product revealed

Automated writing judges still preferred the production frontier model on ordinary business prose. In blind comparisons, it won 63 of 70 times.

Inside Atlas, however, Cygnet-G31B was often more dependable:

It searched when it should. On questions about hiring or starting a business, it reached for the template library instead of answering only from memory.
It recommended real documents. The unadapted model sometimes invented plausible document names. Cygnet searched and returned templates that actually existed.
It held product boundaries. It was more consistent at respecting plan limits, escalating active disputes, declining requests to take sides against a business partner, and giving direct emergency referrals.
It corrected base-model weaknesses. Training improved its handling of confidentiality, legal professional privilege, superannuation prompts, and pricing across products and government fees.

These behaviours are easy to miss in a writing comparison. An answer can sound excellent while skipping a mandatory search, inventing a link, or giving advice outside the product’s scope. Retrieval quality, faithfulness to evidence, and usefulness are separate measurements [1].

The results

We froze the scenario dates, source documents, prompt, and automated critic so every model saw the same environment. The figures below are internal results from Lawpath scenarios, not general model benchmarks.

Product regression suite

This suite contained 340 synthetic tests, each reviewed case by case.

Model	Failures	Pass rate
Base Gemma 4 31B	7	97.9%
Cygnet-G31B	2	99.4%
Production frontier model	4	98.8%

Cygnet’s remaining failure was an over-eager consultation and plan upgrade suggestion attached to a simple GST question.

Broader held-out tasks

We also used a fresh set of 120 scenarios to test broader performance.

Model	Task success
Base Gemma 4 31B	0.733
Cygnet-G31B	0.808
Production frontier model	0.850

Cygnet improved substantially over the base model and led on the product regression suite, but the production model remained stronger across broader tasks and prose.

This distinction matters. Cygnet is not the best model for everything. It is a smaller specialist that performed well on the high-volume Atlas behaviours it was trained to handle. In our measured serving setup, that path cost 4x less. Actual production savings will still depend on usage and infrastructure.

A follow-up training round did not improve the result. Overall performance stayed flat and some broken links returned, so we kept the earlier checkpoint.

Five lessons

1. Better writing is not always better product behaviour

A polished answer can still miss a search, invent a document, or soften a required refusal. Measure whether the assistant completes the job, not only how good the prose sounds.

2. Select models using real errors

The training score kept improving after product behaviour began to regress. We chose an early checkpoint because it made fewer hard errors, even though later checkpoints looked better on the training curve.

3. You cannot imitate behaviour missing from the examples

We tried to teach a strict word limit, but the teacher examples kept breaking it. Rules like that need reliable examples or enforcement by the surrounding application.

4. Freeze the test environment

Changing dates, search results, or cached tool responses can make a model appear better or worse for reasons unrelated to training. Comparable tests need identical inputs and paired evaluation [6].

5. The whole system matters

Fine-tuning improved tool use and boundary discipline, but it did not replace strong instructions, reliable retrieval, clear tool definitions, or a well-designed interface. Agent quality belongs to the complete system, not the model alone [7].

What we are claiming

For a focused product like Atlas, a carefully trained 31-billion-parameter open model can outperform both its base model and a frontier comparator on specific product behaviours, while creating a materially cheaper serving path.

We are not claiming that:

Cygnet is better than frontier models across general tasks.
These internal results form a general benchmark of any vendor’s model.
Self-hosting or interruptible servers establish privacy, security, or legal compliance.
Training alone solved document search.
Cygnet is ready to become the universal default. We still need to remove its sales-pitch tendency and residual dead links on difficult edge cases.

The broader lesson is simple: test AI in the product where it will be used. The model that writes the nicest answer is not always the model that does the job best.

This post describes engineering work on Lawpath Atlas. It is not legal, tax, or financial advice. Atlas provides general information and document drafting support within its product scope and directs users to a qualified professional where appropriate. All comparisons reflect Lawpath’s internal synthetic scenarios and adjudication.

References

Salemi & Zamani. Evaluating Retrieval Augmented Generation. arXiv:2309.15217
Zhang et al. RAFT: Adapting Language Model to Domain Specific RAG. arXiv:2403.10131
Learning Rate Matters: Vanilla LoRA May Suffice. arXiv:2602.04998
Dettmers et al. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314
Hugging Face TRL. SFTTrainer documentation for assistant loss and tool-calling conversations. Docs
Statistical Methods for Multiple Language Model Comparison on a Shared Evaluation. arXiv:2608.22659
Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents. arXiv:2607.27275
vLLM Project. Distributed inference and expert-parallel deployment. Distributed inference and Expert parallelism
GLM-5.3 official model repository, model card, and licence. Repository, Model card, and Licence
Cloud documentation for discounted interruptible compute and interruption handling. Spot interruptions