Training a coding model to paint watercolours with TRL and OpenEnv

On August 23, a captivating video showcasing watercolours generated by a language model took the internet by storm, racking up over 1.5 million views in record time. The project, spearheaded by Surya Narreddi, utilizes p5.brush—a specialized library designed to bring naturalistic drawing tools to the p5.js framework—to instruct a model to "paint" via JavaScript code. While Narreddi’s initial blog post provided a glimpse into the project’s early stages, the community has been eagerly awaiting a full technical breakdown.

In the spirit of open-source collaboration, a new effort has emerged to reproduce this artistic pipeline using TRL (Transformer Reinforcement Learning) and OpenEnv. This initiative provides a fully transparent, end-to-end recipe, complete with training scripts, RL environments, and a curated reference dataset, all hosted on the Hugging Face Hub.

The Engineering Behind the Art

The goal of this reproduction was to validate the original concept while providing a reproducible framework for others to explore. The entire pipeline is built to run seamlessly on Hugging Face infrastructure. By leveraging Jobs for training, Spaces for the RL environment and the pairwise judge, and Inference Providers for model scoring, the project demonstrates how modern AI tooling can be orchestrated to create something entirely new.

The training process is triggered by a single command, utilizing the GRPO (Group Relative Policy Optimization) trainer. The configuration is precise, targeting a Qwen/Qwen3.5-35B-A3B model with LoRA adapters to ensure efficient learning.

"The paintings look loose, imperfect, and handmade—a stark contrast to the statistically average, overly polished images produced by current diffusion models. This project feels like a return to the early, exploratory days of generative AI."

Why This Project Resonates

At a time when generative AI is often criticized for producing sterile, "perfect" outputs, this project’s focus on the "imperfect" is refreshing. It draws a clear line back to the experimental era of DeepDream and Edmond de Belamy, where the focus was not just on the final image, but on probing the capabilities of neural networks.

The project’s brilliance lies in its medium: the model outputs roughly 150 lines of JavaScript. Because the output is code, the creative process—the decision-making behind every brushstroke—is transparent, editable, and reproducible. By restricting the model to a specific subset of the p5.brush library, the project forces the AI to operate within a constrained artistic space, resulting in a distinct, watercolor-like aesthetic.

RL Over Taste: Can We Quantify Beauty?

Most Reinforcement Learning (RL) applications for language models rely on verifiable rewards, such as solving math problems or passing unit tests. This project, however, ventures into the subjective realm of RLHF (Reinforcement Learning from Human Feedback), where the reward is aesthetic preference.

The reward function is a composite of four distinct metrics:

• Gate (5%): Ensures the code compiles and adheres to the library constraints.
• Length (5%): Encourages more complex, detailed code snippets.
• Pairwise Judge (60%): A Qwen3-VL-30B-A3B-Instruct model that compares the candidate painting against four reference images from a curated pool.
• HPSv3 (30%): An open 7B preference model that scores the aesthetic quality of the render.

The "taste" of the model is effectively defined by the 178-image reference pool. By adjusting the weights of these rewards, the trainer can steer the model toward different artistic outcomes.

The Challenges of Infrastructure

Building this system was as much an exercise in infrastructure management as it was in machine learning. The project required a stable, persistent websocket connection between the environment and the trainer. Early iterations were plagued by "silent failures," where rendering timeouts or scorer errors were incorrectly logged as zero-reward outputs, effectively training the model on noise.

Key technical hurdles included:

• LoRA Configuration: The initial assumption that standard dense model target modules would work failed due to the architecture of the Qwen 35B model. Switching to an all-linear approach was the breakthrough that allowed the adapter to successfully train.
• Environment Stability: A bug in OpenEnv regarding cached socket connections caused several runs to fail. Once patched, the pipeline achieved the stability required for long-duration training.
• Compute Costs: A single training run requires significant resources, with rendering accounting for 70–80% of the time. Because the environment lacks a GPU, the software-based rendering of WebGL canvases is a significant bottleneck.

What the Model Learned

The results are striking. In the early stages of training, the model primarily learns to avoid "bad" outputs—blank canvases or incoherent scribbles. As training progresses, the model moves beyond mere reliability and begins to capture the specific aesthetic nuances defined by the reference pool.

Interestingly, the model learned to ignore the system prompt’s instruction to use a specific number of shapes, instead prioritizing the reward function’s preference for "better" paintings. This confirms that in an RL-driven system, the reward function is the true architect of the model's behavior.

Future Directions

While this reproduction successfully mirrors the original vision, it opens the door to several exciting avenues for future research: 1. Multi-step Feedback Loops: Allowing the model to "see" its own work during the generation process, rather than training in a single-turn, blind fashion. 2. Smaller Models: Evidence suggests that smaller models (e.g., 4B parameters) are capable of writing valid sketches, which could reduce the cost of these experiments by an order of magnitude. 3. Dynamic Reward Scaling: Gradually shifting the reference pool from "easy" to "hard" as the model improves to maintain a consistent learning signal.

Conclusion

This project serves as a powerful demonstration of how AI can shift the bottleneck from "making" to "noticing." By curating a pool of images and training a model to align with that specific taste, the creator becomes a curator of aesthetic intent.

For those interested in exploring the results, the full gallery of paintings, the training curves, and the complete source code are available in the project’s repository. Whether you find the results beautiful or merely interesting, the project stands as a testament to the power of open-source collaboration and the ongoing evolution of generative art.