Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

In the rapidly evolving landscape of Large Language Models (LLMs), the ability to generate reliable, structured data is often the deciding factor in whether a model can be integrated into a production-grade software system. While benchmarks frequently prioritize broad reasoning or creative writing, the practical utility of an LLM often hinges on its capacity to return parseable, schema-compliant output.

A new, accessible methodology has emerged that demonstrates how even the smallest models can be significantly improved for these tasks. By utilizing Group Relative Policy Optimization (GRPO) via the TRL library, developers can fine-tune a compact LFM2.5-350M model to achieve substantial gains in structured-output compliance. Remarkably, this entire process requires only 500 training samples and 100 steps—a workflow lightweight enough to execute on free-tier GPU resources like Google Colab or Kaggle.

The Importance of Schema Compliance

Structured output is the backbone of modern AI-driven automation. Whether it is extracting data into JSON for a database or formatting responses for a specific API, the model must adhere strictly to a defined schema. Most existing benchmarks fold these capabilities into general-purpose metrics, obscuring a model's actual reliability in real-world integration.

The IFStruct benchmark serves as a critical tool for measuring this specific competency. It evaluates whether a model can reliably return valid, parseable output in the requested format and shape. This guide explores how task-specific fine-tuning can bridge the performance gap between compact, efficient models and their much larger counterparts.

"Whether a model reliably returns valid, parseable output in the requested format and shape — schema compliance — is often what decides whether it can be wired into a downstream system at all."

Setting the Stage: Prerequisites and Baseline

To replicate these results, the workflow is divided into two distinct environments:

• Fine-tuning: Executed on a GPU-enabled cloud environment (Colab or Kaggle).
• Evaluation: Performed locally on hardware such as a MacBook Pro (M5 Max, 36GB unified memory) using llama.cpp to host an OpenAI-compatible server.

Before applying the GRPO optimization, it is essential to establish a baseline. Using the LiquidAI/LFM2.5-350M base model, we conducted an evaluation against the IFStruct benchmark. The model was served via llama-server with a context size of 32,768 and offloaded layers to the GPU.

Baseline Performance Metrics

The initial evaluation yielded an overall pass rate of 22.6% (452/2000 samples). The breakdown of these results highlights the model's struggle with specific formatting requirements:

• JSON Compliance: 18.0% (180/1000)
• YAML Compliance: 27.2% (272/1000)
• Wrapper Key Structure: 28.5%
• Bare List Structure: 16.6%

Common failure modes included missing required fields, incorrect item counts, and type mismatches. These results align closely with the 21.1% score reported in the original IFStruct documentation, providing a reliable foundation for measuring the impact of our fine-tuning.

The GRPO Fine-Tuning Pipeline

The core of this improvement lies in the application of GRPO (Group Relative Policy Optimization). By leveraging the TRL (Transformer Reinforcement Learning) library, we can train the model to prioritize structural integrity through a reward-based system.

Training Data and Augmentation

We utilized the nvidia/Nemotron-RL-instruction_following-structured_outputs dataset. To ensure the model performed well on the IFStruct benchmark, we applied two specific augmentations: 1. Code Block Formatting: 40% of prompts were appended with instructions to return output inside a fenced code block, training the model to avoid raw JSON output. 2. Top-Level Array Tasks: 20% of prompts were converted into array-based tasks, forcing the model to learn bare-list output and item-count compliance.

Model Configuration

We attached a LoRA (Low-Rank Adaptation) adapter to the LFM2.5-350M architecture. Given the model's unique hybrid attention/convolution design, we targeted specific modules including q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, and w3. This approach trained approximately 6 million parameters, representing just 1.66% of the total model size.

Reward Functions

The model was trained using three distinct reward functions, each normalized on a [0, 1] scale:

• json_format_reward: Evaluates parseability and adherence to requested formatting (fenced vs. raw).
• field_count_reward: Measures the accuracy of top-level field counts, with linear decay for deviations.
• schema_validation_reward: Checks for JSON Schema violations and required-key coverage.

These were combined using a weighted sum (reward_weights=[1.0, 0.5, 2.0]) to prioritize schema validation and format adherence.

Results: Measuring the Impact

Following the 100-step training run, the LoRA adapter was merged back into the base model. The resulting checkpoint was converted into a BF16 GGUF format for local serving and re-evaluated against the IFStruct benchmark.

Performance Comparison

The results demonstrate a clear and significant improvement in structured output reliability:

| Metric | Base Model | GRPO-Tuned | Delta | | :--- | :--- | :--- | :--- | | Overall | 22.6% | 29.7% | +7.1% | | JSON | 18.0% | 31.9% | +13.9% | | YAML | 27.2% | 27.5% | +0.3% | | Wrapper Key | 28.5% | 29.7% | +1.2% | | Bare List | 16.6% | 29.7% | +13.1% |

The data confirms that the training successfully targeted the desired behaviors. JSON compliance saw a massive jump of nearly 14 percentage points, while bare-list output performance improved by over 13 points.

Key Takeaways for Developers

This experiment proves that you do not need massive compute clusters or billions of parameters to achieve high-quality structured outputs. By focusing on a specific, task-oriented reward signal, developers can significantly enhance the reliability of small models.

• Efficiency: A 100-step GRPO run is highly cost-effective and can be completed on entry-level hardware.
• Targeted Gains: Fine-tuning effectively closes the gap in specific areas like JSON formatting and list structure.
• Accessibility: By using open-source tools like TRL, llama.cpp, and the IFStruct benchmark, developers can iterate rapidly on their own custom schemas.

While the fine-tuned LFM2.5-350M model remains slightly below the performance of larger models like Qwen3.5-2B, the delta is closing. This methodology provides a blueprint for any team looking to deploy small, efficient, and highly reliable LLMs into production environments where structured data is non-negotiable.

For those interested in exploring the implementation, the full pipeline is available via the accompanying notebook, and the benchmark datasets can be accessed through the Liquid4All/ifstruct repository. This approach serves as a compelling reminder that in the world of AI, precision and targeted training often outweigh sheer parameter count.