Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

For the modern AI engineering team, the infrastructure landscape has become increasingly fragmented. It is now standard practice for models and massive datasets to reside in a specific cloud bucket within a single region, while the high-performance GPUs required for training, fine-tuning, or inference are scattered across entirely different cloud providers. This geographical and vendor-based separation creates a "cross-cloud transfer tax"—a hidden, recurring cost incurred simply to move data to where the compute lives.

Today, SkyPilot and Hugging Face are announcing a joint solution to bridge this divide. By integrating Hugging Face Storage directly into the SkyPilot ecosystem, teams can now keep their models and datasets on the Hugging Face Hub while executing compute jobs on any cluster, regardless of the cloud vendor.

Eliminating the Egress Tax

The core of this integration is the ability to mount a Hugging Face Bucket or any Hub repository directly into a SkyPilot job using a simple hf:// URL. By utilizing existing Hugging Face credentials (HF_TOKEN), developers can launch tasks across more than 20 different clouds, Kubernetes environments, Slurm clusters, or on-premise hardware without worrying about data locality.

"Hugging Face Storage charges no egress or CDN fees, so wherever SkyPilot lands the job, it reads your models and datasets straight from the same bucket, with no per-cloud copies and no egress bill to pull them in."

This architecture fundamentally changes how teams manage their AI lifecycle. Because Hugging Face does not charge for data egress, the financial penalty for pulling large models or datasets onto GPUs in third-party clouds is effectively eliminated. This allows organizations to stop pinning their workloads to the specific cloud provider where their data happens to reside, granting them the freedom to utilize whichever GPUs are available and cost-effective at any given moment.

How It Works: The hf:// Scheme

The integration introduces store: hf as a first-class storage backend within SkyPilot. This allows for a seamless workflow where a single hf:// identifier covers the entire lifecycle: reading base models, streaming datasets, writing training checkpoints, and publishing final artifacts back to the Hub.

Key Implementation Features:

• Unified Access: Use hf:// to mount buckets (read-write) or repositories (read-only) as local paths within your container.
• Lazy Loading: The system utilizes hf-mount FUSE, meaning data is fetched at the filesystem layer. When your code triggers a read() operation, only the necessary bytes are pulled from the Xet-backed storage.
• On-Disk Caching: Repeat reads are cached locally, ensuring that subsequent epochs or training runs do not require redundant network transfers.
• Immediate Execution: Because reads are lazy, processes can begin consuming data before a full download is complete, preventing GPU idle time and reducing overall billing.

For developers, the configuration is straightforward. By defining file_mounts in a SkyPilot YAML file, you can specify whether a resource should be mounted or copied, providing flexibility for different workload requirements.

Xet-Backed Deduplication: Efficiency at Scale

A significant advantage of this new integration is the underlying use of Xet technology. Hugging Face Buckets leverage content-defined chunking, which splits files into roughly 64 KB segments. Because the system identifies unique chunks, it only stores and transfers the data that has actually changed.

This provides massive benefits for AI workflows: 1. Incremental Checkpointing: When training adapters or fine-tuning models, only the modified layers are uploaded, rather than the entire multi-gigabyte checkpoint. 2. Shared Base Models: If you are managing multiple fine-tunes or quantizations of a single base model, the shared chunks are stored only once, drastically reducing storage footprint. 3. Efficient Appends: For logs or large Parquet datasets, appending new rows does not require re-uploading the entire file. Only the new data is transmitted. 4. Rapid Re-uploads: If a file already exists in the bucket, the system recognizes the chunk hashes, allowing for near-instantaneous synchronization.

Benchmarking the Performance

To validate the impact of this integration, the teams conducted a fine-tuning test using the Qwen/Qwen3.5-4B model on the HuggingFaceH4/Multilingual-Thinking dataset. The experiment involved running the same SkyPilot YAML configuration across AWS, GCP, and Lambda, simply by toggling the --infra flag.

The results demonstrated that the model loaded efficiently across all providers. Because the lazy-read mechanism only pulls the data required by the from_pretrained call, training was able to commence in approximately 30 seconds.

| Cloud Provider | GPU Type | Checkpoint Write Speed | | :--- | :--- | :--- | | AWS (us-east-2) | L40S | ~168 MB/s | | GCP (us-central1) | L4 | ~123 MB/s | | Lambda (us-west-3) | H100 | ~112 MB/s |

These benchmarks highlight that the storage backend is capable of keeping up with high-performance GPU requirements, even when the compute is geographically distant from the storage origin.

Breaking the Vendor Lock-in

Historically, object storage has been the primary bottleneck for multi-cloud AI strategies. Because object stores are typically regional and vendor-specific, teams were forced to either maintain expensive, redundant copies of their data across every cloud provider or pay significant egress fees to move data between them.

With the new Hugging Face Storage integration, the storage layer no longer dictates the compute strategy. Teams can now maintain a single "source of truth" on the Hugging Face Hub. Whether you are scaling up on a hyperscaler, utilizing a specialized GPU cloud, or running on-premise, your data is reachable without the "egress tax."

Getting Started

The integration is available now. To begin, ensure you have the latest version of SkyPilot installed:

pip install "skypilot[huggingface]"
hf auth login

Once authenticated, you can add an hf:// mount to any SkyPilot task. The system is designed to be user-friendly, requiring only that your base image includes glibc 2.34+ and /dev/fuse.

This collaboration represents a significant step forward in the democratization of AI infrastructure. By combining the compute orchestration power of SkyPilot with the storage efficiency and accessibility of Hugging Face, developers can focus on building and training models rather than managing the complexities of cross-cloud data movement.

Resources

• SkyPilot Documentation: Official Storage Guide
• Hugging Face Storage: Buckets Guide
• Open Source Repositories: Explore the hf-mount and huggingface_hub client code on GitHub.
• Community Support: Join the SkyPilot Slack community to discuss implementation and share your results.