TL;DR

Embedding-based vector search surfaces code that looks like the query, but it does not reliably surface code that depends on the query. In our benchmark on a 173-function TypeScript repo (zx), similarity search (even with modern embeddings) recovers only ~59% of true transitive dependents at top-10, while call-graph-aware retrieval recovers ~99%. The gap holds up across a second, smaller repo too. It's a structural mismatch, not a shortcoming of any particular embedding model. CodeFundi's call-graph mapping closes that gap by marrying structural analysis with vector retrieval.

Two Different Graphs: Similarity vs. Dependency

When you ask a retrieval-augmented generation (RAG) tool to "find code related to AuthMiddleware", the engine can build one of two very different graphs:

Diagram description: Imagine a set of nodes, each representing a function. In the similarity graph, clusters form around surface-level similarity: hashPassword, verifyToken, and logRequest might sit together because they all contain the word "token". In the dependency graph, clusters form around call chains: AuthMiddleware → verifyToken → UserService.getUser; logRequest would be isolated unless it is actually invoked by AuthMiddleware.

Both graphs are valid, but they answer different questions. Vector search traverses the similarity graph; a call-graph analysis traverses the dependency graph. When an AI-driven change modifies AuthMiddleware, the danger lies in the downstream callers that will now execute altered code, a set that lives only in the dependency graph.

Why Most RAG Tools Stop at Similarity

Most off-the-shelf code-search services (e.g., OpenAI embeddings, Cohere, or commercial semantic code search) expose an API that takes a query string, returns the top-K nearest vectors, and leaves structural information to the client. Building and maintaining an up-to-date call graph across a monorepo is non-trivial: it requires language-specific parsers, incremental indexing, and handling of dynamic imports. Consequently, many products publish only the similarity graph as a "quick win."

A Worked Example: Auth Middleware

Below is a realistic excerpt from a Node.js Express application:

    

      
1
      
// src/middleware/auth.js
    
  
    

      
2
      
export function authMiddleware(req, res, next) {
    
  
    

      
3
      
  const token = req.headers['authorization']?.split(' ')[1];
    
  
    

      
4
      
  if (!token) return res.status(401).json({ error: 'Missing token' });
    
  
    

      
5
      
 
    
  
    

      
6
      
  try {
    
  
    

      
7
      
    const payload = verifyJwt(token);          // ← verification function
    
  
    

      
8
      
    req.user = payload.sub;
    
  
    

      
9
      
    next();
    
  
    

      
10
      
  } catch (e) {
    
  
    

      
11
      
    res.status(403).json({ error: 'Invalid token' });
    
  
    

      
12
      
  }
    
  
    

      
13
      
}
    
  

Direct Callers (Dependency Graph)

If authMiddleware is altered (e.g., changed error handling or token parsing), all four callers are potential regression points.

What a Vector-Similarity Search Returns

Running a similarity query against openai / text - embedding - 3 - small (the same model used in our benchmark below) yields a top-5 list ordered by cosine similarity that looks like this:

1. verifyJwt: same file, shares token-related identifiers.
2. hashPassword: similar cryptographic vocabulary.
3. logRequest: generic logging helper.
4. generateRefreshToken: same "token" word.
5. sanitizeInput: generic utility.

Missing from the list are the true dependents (orders.js, profile.js, metrics.js, test files). The similarity graph surfaces semantic cousins but not the call-graph descendants that will actually break when the middleware changes.

The Benchmark

Methodology

• Repos tested: zx (173 named functions, 120 caller edges, 23 files) and firecrawl-mcp-server (64 named functions, 50 caller edges, 4 files). Both are TypeScript/JavaScript codebases; we haven't yet tested other languages, and we're not claiming these results generalize beyond JS/TS.
• Ground truth: a call graph built via TypeScript reference resolution, treated as authoritative. Two truth sets are evaluated separately:
  • direct: the query function's immediate (1-hop) callers.
  • transitive: the full reverse-reachability set (every function that can reach the query function through any chain of calls).
• Query sampling: query functions are stratified by fan-in bin (low: 1-2 callers, mid: 3-5, high: 6+) and sampled with a fixed seed (42) so runs are reproducible. Target-per-bin differed by repo size: 50/bin for zx (61 functions actually sampled, since zx has few high-fan-in functions), 15/bin for firecrawl-mcp-server (18 sampled). Because firecrawl-mcp-server is a much smaller repo (64 functions vs. 173), its sample is smaller and its confidence intervals are correspondingly wider; treat the firecrawl numbers as directional, not definitive.
• Methods compared:
  • Similarity (TF-IDF, lexical): a classic bag-of-words baseline. This is not a neural embedding method; it's included as a lexical floor.
  • Similarity (OpenRouter: openai / text - embedding - 3 - small): a modern neural embedding model, queried via nearest-neighbour search.
  • Call-graph ordering: ranking by graph distance in the dependency graph.
• Metric: Recall@K, the percentage of the true dependent set present in the top-K results, reported as mean ± 95% confidence interval across the sampled queries, for K = 5, 10, 20.
• Run details: Node v20.15.0, zx at commit 00a2c484e219c2e84bfc3a199febf7fbce2cfbf4, firecrawl-mcp-server at commit 3eb1115b1f2883ff2fb74e61b5c4acf5a9ac0fb0. The zx numbers below are from our largest and most recent run (2026-07-14); two earlier, smaller zx runs (26 and 51 sampled queries) produced consistent results and are superseded by this one.

Results: zx (173 functions, 61 sampled queries)

Results: firecrawl-mcp-server (64 functions, 18 sampled queries)

Notably, similarity search does relatively better on firecrawl-mcp-server than on zx. That's likely because firecrawl-mcp-server is smaller and shallower (50 call edges across 64 functions, in 4 files); with less depth to the call graph, there's more overlap between "looks similar" and "is actually called by." The gap between similarity and call-graph methods widens as the codebase gets larger and its call chains get deeper, which is the more common, and more consequential, case for real production repos.

Limitations

• Two repos, both JS/TS. We haven't benchmarked Python, Go, Rust, or other ecosystems yet, and we're not claiming these numbers hold there.
• Sample sizes are modest and unequal (61 queries for zx, 18 for firecrawl-mcp-server); confidence intervals reflect this, and the firecrawl-mcp-server numbers in particular should be read as directional rather than precise.
• Call-graph ordering's near-100% "direct" recall is partly a function of the evaluation design, not a discovery. The call-graph method is ranked by graph distance, and the "direct" truth set is also defined by graph distance, so a strong showing there is structurally expected rather than a surprising result. The more informative comparison is transitive recall, where call-graph ordering still requires correctly traversing multi-hop chains and yet reaches ~99% (zx) and ~99-100% (firecrawl-mcp-server), versus ~59-63% for the best similarity method at K=10.
• TF-IDF is a lexical baseline, not a modern neural method; it's included to show that the gap isn't just "old embeddings vs. new embeddings."

Why Better Embeddings Won't Solve the Gap

Improving the quality of embeddings (using a larger training corpus, fine-tuning on code, or switching to a newer model) does reduce semantic noise. Our benchmark shows exactly that: switching from TF-IDF to text - embedding - 3 - small improves transitive recall at K=10 from ~52% to ~59% on zx. That's a real, measurable gain, and it still leaves roughly 4 in 10 true dependents unfound.

The remaining gap is topological, not a matter of embedding quality:

1. Embedding space is a metric space, optimized for proximity of textual or lexical features.
2. Call-graph space is a directed graph, encoding execution flow and data-dependency information that is orthogonal to lexical similarity.

Even a very good embedding that captures every syntactic nuance would still place a function authMiddleware far from its callers if those callers don't share lexical tokens. The distance in embedding space does not correlate with reachability in the call graph. That's why, in our results, similarity search's transitive recall plateaus well below call-graph ordering even at K=20: more results returned doesn't fix a graph-shaped problem with a metric-space tool. The only way to push the metric substantially higher is to incorporate structural edges, i.e., to query the call graph in addition to the vector index.

What Closes the Gap

CodeFundi augments vector retrieval with a live, incremental call-graph map. By indexing each function both as an embedding and as a node in a directed dependency graph, CodeFundi can answer hybrid queries such as "show me the top-10 similar snippets plus all downstream callers of the matched node." This dual-graph approach yields the high transitive coverage demonstrated in the benchmark above, without discarding the semantic relevance that embeddings provide.

FAQ

Does RAG work for code?
Yes. Retrieval-augmented generation improves code completion and documentation by surfacing semantically similar snippets. However, RAG alone does not guarantee that all affected code paths are considered.

What's the difference between semantic code search and call-graph analysis?
Semantic search ranks snippets by embedding similarity, ignoring how they are wired together. Call-graph analysis traverses explicit import and call relationships, revealing runtime dependencies regardless of lexical similarity.

Can embeddings represent code dependencies?
In principle, a model could learn to embed structural cues (e.g., function names often called together). In practice, our benchmark suggests the signal for indirect (transitive) dependencies is still fairly weak relative to an explicit graph. Modern embeddings narrow the gap versus a lexical baseline like TF-IDF, but don't close it.

Run the same test on your own repository. Our live demo indexes your code, builds the call graph, and lets you compare a pure-vector search with a call-graph-augmented retrieval. -> Start the demo