Skip to main content
Question 3. When conducting zero-shot testing of an LLM on a new task, which of the following represents the most rigorous experimental design?
  • A. Testing on a dataset that was partially included in the model’s training data
  • B. Testing on a completely unseen dataset with well-defined evaluation metrics
  • C. Testing with prompts that include examples of expected outputs
  • D. Testing on tasks that are similar to those in the training data
Answer: B — Testing on a completely unseen dataset with well-defined evaluation metrics. Why B is correct:
  • True zero-shot evaluation requires strictly held-out data that the model has never seen during training to avoid data leakage and inflated performance estimates.
  • Clear, objective evaluation metrics (accuracy, F1, exact match, BLEU/ROUGE for generation, etc.) are essential for reproducibility and meaningful comparison across models.
  • This combination isolates the model’s ability to generalize rather than its memorization or familiarity with task-specific examples.
The image displays a question about conducting zero-shot testing on a new task with LLMs, identifying testing on an unseen dataset with defined metrics as the most rigorous method. It includes an explanation that emphasizes evaluating on unseen data to maintain the zero-shot nature.
Why the other options are suboptimal:
  • Option C: Including example outputs in the prompt converts the setup to few-shot evaluation, which is a different experimental condition and no longer zero-shot.
  • Option A: Using datasets that overlap with training data introduces data leakage and can artificially boost performance.
  • Option D: Testing on tasks similar to training tasks risks overestimating generalization because the model may rely on pattern reuse rather than genuine transfer.
Comparison table Practical checklist for rigorous zero-shot evaluation
  • Ensure dataset provenance: document sources, timestamps, and any possible overlap with model training corpora.
  • Hold out a strict unseen test set: never use it during prompt development, hyperparameter tuning, or any model selection.
  • Define metrics before running experiments: choose task-appropriate metrics (e.g., accuracy/F1 for classification, EM/F1 for QA, BLEU/ROUGE for generation).
  • Standardize preprocessing and tokenization: report these steps so others can reproduce results.
  • Control prompts: keep prompts consistent across models and document exact wording and temperature/top-p settings.
  • Run statistical tests and report variance: include confidence intervals or bootstrap estimates to show result stability.
  • Compare to baselines: rule-based or simple heuristics help contextualize model performance.
  • Publish artifacts when possible: test set (or a reproducible sampling procedure), prompts, and evaluation scripts.
Common evaluation metrics by task
  • Classification: accuracy, precision, recall, F1
  • Question answering (extractive): exact match (EM), F1
  • Generation/summarization: BLEU, ROUGE, METEOR (and human evaluation)
  • Ranking/retrieval: MRR, MAP, NDCG
Resources and further reading
For rigorous zero-shot testing: use strictly held-out datasets, define evaluation metrics up front, and document any pre-processing or prompt engineering to ensure reproducibility.

Watch Video