How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code

Three months ago, the team behind Papers with Code initiated a comprehensive revival of the platform. The core mission remains clear: to democratize access to artificial intelligence research, making it digestible and actionable for the community. By enabling users to easily locate research artifacts, track state-of-the-art (SOTA) benchmarks across diverse AI domains, and collaborate on shared findings, the platform aims to catalyze the next generation of breakthroughs—perhaps even the successor to the Transformer architecture.

To achieve this, the platform requires a search engine that transcends traditional text matching. Whether accessed via the website or the pwc search CLI command—which empowers AI agents to utilize the platform as a skill—the search engine must be both sophisticated and resilient.

The Challenge of Searching Scientific Literature

Searching for research is fundamentally different from standard web search. A robust academic search engine must handle:

• Precision: Finding exact paper titles or specific arXiv identifiers.
• Semantic Understanding: Interpreting complex queries like “small language models for code generation,” even when those exact terms are absent from the paper text.
• Navigational Intent: Recognizing that a query like “the original BERT paper” is a request for a specific, foundational document.
• Robustness: Handling typos, incomplete titles, and maintaining performance even when backend services are cold or temporarily unavailable.

To address these requirements, the team implemented a hybrid search architecture. Drawing on extensive experience developing Retrieval-Augmented Generation (RAG) systems, the team concluded that hybrid search—combining keyword-based lexical search with vector-based semantic search—consistently outperforms either method in isolation.

The Hybrid Architecture: Lexical Meets Semantic

The system leverages a PostgreSQL database as its foundation, utilizing its full-text search capabilities to provide a high-speed lexical baseline. To introduce semantic recall, the team integrated pgvector, with the Reciprocal Rank Fusion (RRF) algorithm serving as the glue to combine these two distinct retrieval streams.

The infrastructure relies on three core Hugging Face services to manage this complexity: 1. Hugging Face Jobs: Provides burstable GPU compute for embedding the massive paper corpus. 2. Hugging Face Storage Buckets: Acts as the durable, reliable handoff point between the database, experimental pipelines, and production jobs. 3. Hugging Face Inference Endpoints: Delivers low-latency, scalable embedding services for live user queries and incremental updates.

Currently, the system manages embeddings for over 110,000 papers sourced from arXiv and Daily Papers. By splitting the architecture into an offline corpus build and an online search service, the team ensures that the most resource-intensive tasks are handled asynchronously, while the online path remains lean and responsive.

"If that endpoint is cold, busy, or unhealthy, search immediately falls back to full-text retrieval. This separation makes the system both powerful and fast."

Establishing a Strict Embedding Contract

Embedding pipelines are notoriously fragile. Minor changes in model revisions, mismatched prompts, or inconsistent truncation can lead to silent failures. To mitigate this, the team treats the embedding format as a versioned API.

Every paper is processed using a standardized format: normalized title + "\n\n" + normalized abstract. For every vector generated, the system records:

• The model repository and exact revision.
• The output dimension.
• The input-format version.
• The content hash of the source material.

The production pipeline utilizes Qwen/Qwen3-Embedding-0.6B, pinned to a specific revision, producing 256-dimensional L2-normalized vectors. This choice was informed by the MTEB leaderboard, the industry standard for evaluating embedding models. Furthermore, by utilizing Matryoshka Representation Learning (MRL), the team can trade off embedding size against speed and storage costs, opting for 256 dimensions to ensure rapid search performance.

Scaling with Hugging Face Jobs

Full-corpus embedding is a classic batch workload. It demands high throughput and GPU acceleration, but only for short, intensive bursts. Hugging Face Jobs is perfectly suited for this, allowing the team to define hardware flavors and dependencies inline.

The process begins by exporting a repeatable-read snapshot from PostgreSQL. The exporter streams data to avoid memory bottlenecks, creating JSONL shards that are synced to a private Storage Bucket. Using hf-mount, these buckets are mounted directly into an l4x1 Job (NVIDIA L4 GPU, 24GB VRAM).

The worker node performs several critical steps:

• Verifies input manifests and shard checksums.
• Loads the pinned model revision.
• Sorts texts by length to minimize padding overhead.
• Executes batch encoding, with automatic memory management to prevent out-of-memory errors.
• Writes float16 Parquet shards atomically.

This design allows for seamless retries; if a job is interrupted, it can resume exactly where it left off by checking completed shards, rather than restarting the entire process.

Buckets: The Connective Tissue

Storage Buckets serve as the boundary between the production database, ephemeral compute jobs, and the final search index. By organizing artifacts under immutable run prefixes, the team ensures:

• Reproducibility: Every database generation can be traced back to a specific snapshot and model version.
• Safe Retries: Jobs can resume from the last successful shard.
• Controlled Rollout: New generations are validated for coverage and index integrity before being marked as active, allowing for simple, instant rollbacks if issues arise.

Putting Semantic Search on the Request Path

While batch jobs handle the corpus, user queries require real-time embedding. The team deploys the pinned model as an authenticated Inference Endpoint backed by Text Embeddings Inference (TEI).

When a user submits a query, the API performs a cosine-distance search over the active pgvector generation. The HNSW (Hierarchical Navigable Small World) index ensures that this lookup remains lightning-fast. In pilot testing, the 256-dimensional Qwen index achieved 0.9955 Recall@20 compared to exact search, with a p50 latency of just 1.31 ms.

To handle the reality of "scale-to-zero" infrastructure, the query client implements a strict, defensive design:

• A one-second production timeout.
• A non-blocking concurrency limit.
• A circuit breaker to prevent cascading failures.
• A fallback mechanism that defaults to lexical search if the semantic endpoint is unavailable.

Hybrid Retrieval: The Best of Both Worlds

The final search result is a fusion of two worlds. The lexical branch retrieves up to 50 candidates via PostgreSQL, while the semantic branch retrieves 50 candidates via pgvector. These are combined using Reciprocal Rank Fusion (RRF).

"Dense retrieval improves recall for conceptual queries. Full-text retrieval remains excellent for exact terminology, identifiers, and rare names."

This hybrid approach ensures that while the system is smart enough to understand conceptual queries, it never loses the ability to perform precise, deterministic lookups for specific paper titles or identifiers.

Continuous Updates and Future-Proofing

The platform is not static. New papers arrive hourly, and existing records are updated. Rather than launching a full-scale GPU job for minor changes, the team uses an incremental process. This process selects changed or missing papers and sends them to the same TEI endpoint in small batches. This ensures the index remains current without turning the online endpoint into an unbounded batch processor.

Furthermore, these document embeddings power "Related Papers" recommendations, which become essentially "free" at request time, as they require only a single nearest-neighbor query over the existing vector index.

Key Lessons Learned

The team’s journey to production offers several takeaways for developers building AI-powered search: 1. Separate Concerns: Optimize throughput for corpus embedding (Jobs) and latency for query embedding (Inference Endpoints). 2. Storage as a Contract: Use buckets to create a reviewable, checksummed boundary between compute and production. 3. Pin Everything: Beyond just the model name, pin revisions, dimensions, prompts, and normalization methods. 4. Design for Cold Starts: If using scale-to-zero, ensure your application has a graceful, high-quality fallback. 5. Vectors as a Feature: Use Matryoshka embeddings to balance memory, index size, and recall. 6. Boring Activations: Treat index updates as configuration changes, not emergency recomputations.

By combining these architectural patterns, Papers with Code has created a search engine that is as reliable as it is intelligent, ensuring that the next big breakthrough in AI research is only a query away.