> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Python Library for Semantic Similarity Calculation LLM Eval
> Recommending Sentence Transformers for computing semantic similarity between LLM outputs and references, with example code, interpretation, and comparisons to NumPy, pandas, and Matplotlib.
Question 9.
When implementing a Python script to evaluate LLM outputs, which library would be most useful for calculating semantic similarity between generated and referenced texts? NumPy, Sentence Transformers, Matplotlib, or pandas?
The answer: [Sentence Transformers](https://www.sbert.net/).
Why Sentence Transformers for LLM evaluation?
* Sentence Transformers (the `sentence-transformers` library) is built to produce dense sentence and text embeddings that capture semantic meaning, not just lexical overlap.
* Embeddings let you compare generated and reference texts by meaning using similarity metrics (commonly cosine similarity), which is essential for evaluating LLM outputs where many valid phrasings exist.
* Pretrained models are available in sizes that trade off speed and accuracy, enabling both low-latency inference and higher-fidelity comparisons.
Minimal example: compute semantic similarity with Sentence Transformers
```python theme={null}
from sentence_transformers import SentenceTransformer, util
# Lightweight, fast model suitable for many evaluation tasks
model = SentenceTransformer('all-MiniLM-L6-v2')
generated = "A quick brown fox jumps over the lazy dog."
reference = "A fast brown fox leaps over a sleeping dog."
# Encode texts to embeddings (use convert_to_tensor=True for GPU/fast cosine)
emb_gen = model.encode(generated, convert_to_tensor=True)
emb_ref = model.encode(reference, convert_to_tensor=True)
# Cosine similarity between embeddings (value typically between -1 and 1)
similarity = util.cos_sim(emb_gen, emb_ref).item()
print(f"Cosine similarity: {similarity:.4f}")
```
Interpreting the result:
* Values closer to 1 indicate higher semantic similarity.
* Values near 0 indicate little semantic relation.
* Values near -1 are rare for sentence embeddings but indicate opposite directions in vector space.
Comparison: which library to choose
| Library | Use case for LLM evaluation | Example / Notes |
| --------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| Sentence Transformers | Best for semantic similarity, embedding-based comparisons, and clustering | `from sentence_transformers import SentenceTransformer` |
| NumPy | Fundamental numeric operations and array math used downstream (not for embeddings themselves) | Use for manipulating vectors once obtained |
| Matplotlib | Visualization of results (e.g., similarity distributions, ROC curves) | Not for computing semantic similarity |
| pandas | Organizing and analyzing tabular evaluation results (scores, metadata) | Great for storing per-sample metrics |
Why not the others?
* NumPy: Core numerical library but does not provide pretrained text embeddings or semantic models.
* Matplotlib: Useful for plotting evaluation outputs, not for producing embeddings or similarity metrics.
* pandas: Ideal for organizing results and aggregating metrics, but not designed to compute semantic similarity directly.
Tip: Select a model based on your accuracy/latency needs — `all-MiniLM-L6-v2` is efficient for bulk evaluations, while larger models (or fine-tuned ones) can yield higher semantic fidelity. For large-scale comparisons, use batch encoding and enable GPU acceleration where available.
Links and references
* [Sentence Transformers (SBERT)](https://www.sbert.net/)
* [Hugging Face Models](https://huggingface.co/models)
* [NumPy Documentation](https://numpy.org/doc/)
* [pandas Documentation](https://pandas.pydata.org/docs/)
* [Matplotlib Documentation](https://matplotlib.org/stable/contents.html)