> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Python Package for Text Vectorization

> Guide comparing Python libraries for creating dense text embeddings, recommending spaCy with vector models and transformer alternatives, plus usage examples and similarity search tips.

Question 2.

Which Python package is most appropriate for creating dense vector representations of text for a semantic search application? Would it be NumPy, spaCy, matplotlib, or scikit-learn?

Answer: [spaCy](https://spacy.io)

spaCy is a production-ready natural language processing library that includes easy access to pre-trained dense word and document vectors when you use models that ship with vectors (for example, `en_core_web_lg`). While:

* [NumPy](https://numpy.org) is the core numerical library used to manipulate vectors (arrays)
* [matplotlib](https://matplotlib.org) is for visualization
* [scikit-learn](https://scikit-learn.org) offers ML utilities and some vectorizers (e.g., TF-IDF)

spaCy directly provides token and document `.vector` embeddings that are convenient for semantic search workflows.

<Callout icon="lightbulb" color="#1CB2FE">
  Use a spaCy model that includes vectors (for example, `en_core_web_lg`) when you need ready-made dense embeddings. For higher-quality sentence or paragraph embeddings, consider transformer-based libraries such as [`sentence-transformers`](https://www.sbert.net/) (SBERT) or Hugging Face models, and combine those embeddings with spaCy or your retrieval pipeline as needed.
</Callout>

## Quick comparison

| Library                                | Typical use for embeddings                          | Notes / Recommendation                                                 |
| -------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------------- |
| `spaCy`                                | Dense word & document vectors via model `.vector`   | Best for fast, production NLP with built-in vectors (`en_core_web_lg`) |
| `sentence-transformers` / Hugging Face | State-of-the-art sentence embeddings                | Often yields better semantic similarity for sentences/paragraphs       |
| `scikit-learn`                         | Feature extraction (TF-IDF, hashing)                | Good for sparse representations; not dense semantic embeddings         |
| `NumPy`                                | Numerical operations on vectors                     | Essential for handling arrays returned by embedding libraries          |
| `matplotlib`                           | Visualization of embeddings (e.g., PCA/t-SNE plots) | Not used to create embeddings                                          |

## Install and load a spaCy model with vectors

Shell:

```bash theme={null}
# Install spaCy and a model that includes vectors
pip install spacy
python -m spacy download en_core_web_lg
```

Python example:

```python theme={null}
import spacy

# Load a spaCy model that includes pre-trained vectors
nlp = spacy.load("en_core_web_lg")

doc = nlp("This is a sample sentence for semantic search.")
vector = doc.vector  # dense NumPy array representing the whole document

print(type(vector))   # <class 'numpy.ndarray'>
print(vector.shape)   # commonly (300,) for many spaCy vectors
```

Notes:

* spaCy’s `.vector` on a `Doc` or `Token` returns a dense `numpy.ndarray` that you can use directly for similarity computations (cosine similarity), indexing into vector stores, or as input to downstream models.
* For improved semantic-search accuracy at sentence/paragraph level, consider transformer-based encoders such as [`sentence-transformers`](https://www.sbert.net/) or Hugging Face embedding models; these typically yield higher-quality embeddings than classic spaCy vectors for sentence semantics.
* Use `sklearn.metrics.pairwise.cosine_similarity` or `scipy.spatial.distance.cosine` (or faiss/annoy for large-scale search) to compute nearest neighbors efficiently.

## References and further reading

* spaCy models: [https://spacy.io/models/en\_core\_web\_lg](https://spacy.io/models/en_core_web_lg)
* sentence-transformers: [https://www.sbert.net/](https://www.sbert.net/)
* Hugging Face: [https://huggingface.co/](https://huggingface.co/)

<CardGroup>
  <Card title="Watch Video" icon="video" cta="Learn more" href="https://learn.kodekloud.com/user/courses/nvidia-generative-ai-llms-associate-certification/module/875d98e8-3b09-4f35-b877-2758b84443ca/lesson/2ab2521e-4861-45d9-a718-e201f5d20011" />
</CardGroup>