Welcome Inkling by Thinking Machines

Hugging Face Blog49d ago

Thinking Machines Lab has officially expanded its flagship model family with the introduction of Inkling-Small, a more compact, highly efficient iteration of its massive, multimodal powerhouse. Following the successful launch of the original 1-trillion-parameter Inkling model, this update brings significant improvements in deployment flexibility, performance optimization, and accessibility for developers looking to integrate advanced multimodal reasoning into their own applications.

The team has also released comprehensive deployment configurations for both the Inkling-Small and the Inkling-Small-NVFP4 variants, ensuring that users can leverage these models across a variety of hardware setups. To make adoption even smoother, Thinking Machines has enabled one-click deployment via Inference Endpoints, capable of reaching up to 160 tokens per second (TPS).

"Inkling is a large (1T params!) open model designed to natively accept image, text, and audio inputs. It represents a new wave of multimodal reasoning, supporting a 1M context window and agentic capabilities."

The Inkling Ecosystem: A Multimodal Giant

At its core, Inkling is a groundbreaking open-source model that bridges the gap between text, audio, and visual data. Trained on a massive corpus of 45 trillion tokens—spanning text, images, audio, and video—the model is engineered for deep reasoning across diverse modalities.

The release includes:

Inkling (Full BF16): The flagship 1T parameter model.
Inkling-NVFP4: A well-calibrated, high-efficiency variant.
Inkling-Small: A 276B total parameter model (12B active) that maintains the same architecture as its larger sibling, now with support for MXFP8 and NVFP4 weights.

Architectural Innovations

Inkling distinguishes itself through a sophisticated, decoder-only Mixture-of-Experts (MoE) architecture. By utilizing 256 experts, the model achieves high performance while keeping only 41B parameters active during any single inference pass, significantly boosting speed.

Key architectural highlights include:

Relative Attention: Departing from standard RoPE methods, Inkling learns positional information directly within the attention logits, utilizing a fourth projection to generate per-token, per-head relative features.
Hybrid Attention: The model alternates between global attention and sliding window attention (a 5:1 ratio), optimizing computational efficiency while maintaining rich feature representation.
Short Convolution (SConv): A 1D convolution layer processes the current token alongside previous hidden states, offloading local representation tasks from the primary attention and MoE modules.
Multimodal Integration: The model features a hierarchical MLP patchifier for vision and a discretized mel-spectrogram approach for audio, allowing for seamless ingestion of media without the need for separate, complex encoders.

Deployment and Inference

Thinking Machines has prioritized "day-zero" support for major frameworks, ensuring that Inkling can be integrated into existing production pipelines with minimal friction.

#### Transformers and Inference Engines Users can leverage the latest transformers library (v5.14.0+) to run the model using the any-to-any pipeline. For high-performance production environments, the model is fully compatible with SGLang and vLLM, allowing for tensor-parallel sharding across multiple GPUs.

SGLang: Ideal for high-throughput serving; supports sharding across 8 GPUs with an OpenAI-compatible API.
vLLM: The preferred choice for production-grade deployment, supporting custom hardware recipes and advanced tool-calling features.
llama.cpp: For local or resource-constrained environments, the model supports GGUF quantization, with Unsloth providing 1-bit precision versions that reduce VRAM requirements by up to 95%.

Agentic Capabilities and Tool Use

Inkling is not just a passive model; it is designed for agentic workflows. By utilizing tools like Pi—a minimal coding agent harness—developers can task the model with complex math, document processing, and audio analysis. The model’s ability to reason across modalities makes it particularly adept at tasks requiring the synthesis of visual or auditory information into actionable code or text.

Furthermore, the inclusion of Multi-Token Prediction (MTP) layers allows the model to act as its own "drafter" during speculative decoding. This significantly accelerates generation speeds without sacrificing the quality of the output, providing a seamless experience for real-time applications.

Performance Benchmarks

The "vibe evaluations" conducted by Thinking Machines demonstrate the model's robust reasoning capabilities across various domains, including physics, law, and medical diagnostics.

In audio reasoning, the model excels at transcribing and characterizing speech before evaluating logical fallacies or performing object counting, consistently resisting "decoy" information in multilingual tests.

Post-Training and Customization

For organizations looking to tailor Inkling to specific domains, Thinking Machines offers Tinker, a managed tool for post-training. The team has successfully utilized the ECHO algorithm to train the model to predict environment states without the need for external verifiers, effectively teaching the policy an implicit world model.

For those interested in knowledge distillation, Inkling serves as an ideal "teacher" model. Using the GOLD algorithm, developers can distill Inkling’s advanced document and multimodal understanding into smaller, on-device models, balancing high-level reasoning with edge-computing efficiency.

Deployment Requirements at a Glance

To assist in infrastructure planning, Thinking Machines has provided a breakdown of VRAM requirements for various configurations:

Inkling (BF16): 2 TB VRAM (Requires 8x NVIDIA B300/GB200 or 16x H200).
Inkling (NVFP4): 600 GB VRAM (Requires 4x B300 or 8x H200).
Inkling-Small (BF16): 600 GB VRAM (Requires 4x B300 or 8x H200).
Inkling-Small (NVFP4): 180 GB VRAM (Requires 1x B300 or 2x H200).

Conclusion

The release of Inkling and Inkling-Small marks a significant milestone in the open-weights AI landscape. By combining a 1-trillion-parameter architecture with native multimodal support, speculative decoding, and robust agentic tooling, Thinking Machines is providing the building blocks for the next generation of intelligent applications. Whether you are deploying on a massive GPU cluster or experimenting with quantized local models, Inkling offers the performance and flexibility required to push the boundaries of what multimodal AI can achieve.

Read Original Source