Python Library for Semantic Similarity Calculation LLM Eval

Question 9. When implementing a Python script to evaluate LLM outputs, which library would be most useful for calculating semantic similarity between generated and referenced texts? NumPy, Sentence Transformers, Matplotlib, or pandas? The answer: Sentence Transformers.

The image contains a multiple-choice question regarding the most useful Python library for calculating semantic similarity between generated and reference texts, with the answer highlighted as "sentence-transformers." It includes an explanation of why that library is most suitable.

Why Sentence Transformers for LLM evaluation?

Sentence Transformers (the sentence-transformers library) is built to produce dense sentence and text embeddings that capture semantic meaning, not just lexical overlap.
Embeddings let you compare generated and reference texts by meaning using similarity metrics (commonly cosine similarity), which is essential for evaluating LLM outputs where many valid phrasings exist.
Pretrained models are available in sizes that trade off speed and accuracy, enabling both low-latency inference and higher-fidelity comparisons.

Minimal example: compute semantic similarity with Sentence Transformers

from sentence_transformers import SentenceTransformer, util

# Lightweight, fast model suitable for many evaluation tasks
model = SentenceTransformer('all-MiniLM-L6-v2')

generated = "A quick brown fox jumps over the lazy dog."
reference = "A fast brown fox leaps over a sleeping dog."

# Encode texts to embeddings (use convert_to_tensor=True for GPU/fast cosine)
emb_gen = model.encode(generated, convert_to_tensor=True)
emb_ref = model.encode(reference, convert_to_tensor=True)

# Cosine similarity between embeddings (value typically between -1 and 1)
similarity = util.cos_sim(emb_gen, emb_ref).item()
print(f"Cosine similarity: {similarity:.4f}")

Interpreting the result:

Values closer to 1 indicate higher semantic similarity.
Values near 0 indicate little semantic relation.
Values near -1 are rare for sentence embeddings but indicate opposite directions in vector space.

Comparison: which library to choose

Library	Use case for LLM evaluation	Example / Notes
Sentence Transformers	Best for semantic similarity, embedding-based comparisons, and clustering	`from sentence_transformers import SentenceTransformer`
NumPy	Fundamental numeric operations and array math used downstream (not for embeddings themselves)	Use for manipulating vectors once obtained
Matplotlib	Visualization of results (e.g., similarity distributions, ROC curves)	Not for computing semantic similarity
pandas	Organizing and analyzing tabular evaluation results (scores, metadata)	Great for storing per-sample metrics

Why not the others?

NumPy: Core numerical library but does not provide pretrained text embeddings or semantic models.
Matplotlib: Useful for plotting evaluation outputs, not for producing embeddings or similarity metrics.
pandas: Ideal for organizing results and aggregating metrics, but not designed to compute semantic similarity directly.

Tip: Select a model based on your accuracy/latency needs — all-MiniLM-L6-v2 is efficient for bulk evaluations, while larger models (or fine-tuned ones) can yield higher semantic fidelity. For large-scale comparisons, use batch encoding and enable GPU acceleration where available.

Links and references

Watch Video

Effective Chunking Strategy for RAG Systems

Software Design Principle for Evolving LLM Applications