CLARITY

Detecting evasive answers in political interviews

Research code for the SemEval 2026 CLARITY shared task, comparing transformer encoders, prompt-engineered LLMs, and parameter-efficient fine-tuning on the QEvasion dataset.

Project report · Experiment notebooks · Recorded results · SemEval task page

Chapter 1 — Overview

1.1 Research Objective

CLARITY is an experimental NLP pipeline for determining whether a politician answers a journalist's question directly and, when the answer is evasive, identifying the rhetorical strategy being used.

The repository investigates one central question:

Does fine-grained supervision about how an answer is evasive improve the coarser prediction of whether it is clear?

Three model families are evaluated under the same data split and with Macro-F1 as the primary metric:

• Transformer Encoders: a configurable registry for BERT, DistilBERT, RoBERTa, XLNet, and Longformer; the recorded report comparisons focus on DistilBERT and RoBERTa.

• Prompted LLMs: Llama 3.1 8B Instruct with Zero-Shot, Chain-of-Thought, taxonomy-aware, RAG, tone-aware, and Few-Shot prompts.

• Fine-Tuned LLMs: Llama 3.2 3B Instruct adapted with 4-bit QLoRA and DoRA for direct clarity and fine-grained evasion classification.

1.2 Main Finding

Fine-grained evasion labels provide a useful intermediate representation. The strongest prompted configuration first predicts one of nine evasion strategies and then maps that prediction to a clarity class, reaching $0.608$ Macro-F1 on the held-out test set. For encoder models, however, this mapping is not universally beneficial: errors made during nine-class classification can propagate into the final three-class prediction.

The complete analysis, including ablations and error discussion, is available in the project report.

Chapter 2 — Task Definition

2.1 Prediction Tasks

The project evaluates three related settings over question-answer pairs:

ID	Task	Output Space	Purpose
A	Direct Clarity	3 classes	Predict the overall clarity of the answer directly.
B	Direct Evasion	9 classes	Identify the fine-grained rhetorical strategy.
C	Evasion $\rightarrow$ Clarity	9 then 3 classes	Predict Task B and deterministically map it to Task A.

2.2 Label Taxonomy

Clarity Class	Fine-Grained Evasion Labels
Clear Reply	`Explicit`
Ambivalent	`Implicit`, `General`, `Partial/half-answer`, `Dodging`, `Deflection`
Clear Non-Reply	`Declining to answer`, `Claims ignorance`, `Clarification`

Task C applies this mapping through the EVASION_TO_CLARITY constant. For the official test split, fine-grained labels are reconstructed from three annotators using majority voting and reliability-weighted tie-breaking.

Chapter 3 — System Design

3.1 Experimental Pipeline

flowchart LR
    D["QEvasion<br/>question + answer"] --> S["Reproducible split<br/>seed = 42"]
    S --> E["Encoder track<br/>BERT family"]
    S --> P["Prompting track<br/>Llama 3.1 8B"]
    S --> F["PEFT track<br/>Llama 3.2 3B"]

    E --> R["Smart Resampling<br/>+ rhetorical tone"]
    P --> Z["Zero/CoT/Few-Shot<br/>+ taxonomy/RAG/tone"]
    F --> Q["4-bit QLoRA<br/>+ DoRA"]

    R --> A["Task A<br/>3-class clarity"]
    Z --> A
    Z --> B["Task B<br/>9-class evasion"]
    Q --> A
    Q --> B
    B --> M["Deterministic mapping"]
    M --> C["Task C<br/>mapped clarity"]

Figure 3.1: Shared data flow and the three experimental model tracks.

3.2 Encoder Track

Input: the journalist's question and the politician's answer are jointly tokenized for sequence classification.

Model Registry: per-model sequence length, batch size, gradient accumulation, learning rate, and tokenizer behavior are centralized in config/model_configs.py.

Imbalance Handling: the enhanced pipeline combines:

• Semantic Downsampling: KMeans selects representative real examples from majority classes.

• Paraphrase Upsampling: T5 generates minority-class candidates, filtered by semantic similarity.

• Tone Injection: facebook/bart-large-mnli assigns an Assertive, Guarded, or Dismissive rhetorical feature before tokenization.

Losses: Cross-Entropy, weighted Cross-Entropy, Focal Loss, and label-smoothed Cross-Entropy are implemented in src/training/losses.py.

3.3 Prompting Track

The prompted experiments use meta-llama/Meta-Llama-3.1-8B-Instruct with 4-bit quantization. The study progresses from direct Zero-Shot inference to structured reasoning:

Zero-Shot
Chain-of-Thought
Chain-of-Thought + Taxonomy Table
Chain-of-Thought + Taxonomy Table + RAG
Chain-of-Thought + Taxonomy Table + Tone
One-, Three-, and Nine-Shot Prompting
Dynamic Few-Shot Retrieval
Nine-Shot + Tone

Dynamic Few-Shot retrieval uses sentence-transformers/all-MiniLM-L6-v2 embeddings and a FAISS index. The experiments show that semantic similarity by topic does not necessarily imply similarity in rhetorical strategy.

3.4 Fine-Tuning Track

The fine-tuned experiments use meta-llama/Llama-3.2-3B-Instruct with 4-bit QLoRA + DoRA. Separate adapters are trained for:

• Direct Clarity: three-class Task A.

• Direct Evasion: nine-class Task B, followed by Task C mapping.

The final notebook also evaluates Prompt Chaining: multi-part questions are decomposed into atomic sub-questions, classified independently, and aggregated into a single result.

Chapter 4 — Results

4.1 Evaluation Protocol

All reported experiments use the same held-out test set of $N=308$ question-answer pairs. Macro-F1 is the principal score because both the three-class and nine-class targets are imbalanced.

4.2 Representative Results

The table below highlights the best or most informative configurations from each model family. It is a compact snapshot, not a leaderboard across incomparable task formulations.

Task	Model Family	Configuration	Accuracy	Macro-F1
A — Direct Clarity	Encoder	RoBERTa baseline	$0.640$	$0.558$
A — Direct Clarity	Prompted LLM	CoT + Table + Tone	$0.667$	$0.384$
A — Direct Clarity	Fine-Tuned LLM	Direct	$0.698$	$0.474$
B — Direct Evasion	Encoder	Enhanced RoBERTa	$0.308$	$0.308$
B — Direct Evasion	Prompted LLM	Nine-Shot	$0.386$	$0.288$
B — Direct Evasion	Fine-Tuned LLM	Direct	$0.263$	$0.220$
C — Mapped Clarity	Encoder	Enhanced RoBERTa	$0.640$	$0.568$
C — Mapped Clarity	Prompted LLM	Nine-Shot	$0.685$	$0.608$
C — Mapped Clarity	Fine-Tuned LLM	Direct	$0.721$	$0.571$

Interpretation:

• Best direct Task A Macro-F1: the RoBERTa baseline remains the strongest direct clarity classifier among the tested systems.

• Best direct Task B Macro-F1: Smart Resampling and tone injection make Enhanced RoBERTa the strongest fine-grained classifier.

• Best Task C Macro-F1: static Nine-Shot prompting benefits most from the evasion-to-clarity mapping.

• Best Task C accuracy: the fine-tuned 3B model achieves $0.721$, showing the value of specialized fine-grained supervision even with a smaller LLM.

4.3 Few-Shot Progression

Figure 4.1: Macro-F1 across static, dynamic, and tone-aware Few-Shot strategies.

Machine-readable metrics, predictions, and confusion matrices are committed under results/. The paper contains the full comparison tables and error analysis.

Chapter 5 — Quick Start

5.1 Prerequisites

Required:

• Python $3.10+$

• Git

• JupyterLab or Jupyter Notebook

Recommended for LLM notebooks:

• An NVIDIA GPU with CUDA support

• A Hugging Face account with access to the selected Meta Llama model

• A Hugging Face token exposed as HF_TOKEN

The encoder experiments can be adapted to CPU execution, but training and LLM inference are intended for a GPU environment. The notebooks include dedicated Google Colab setup paths.

5.2 Local Installation

git clone https://github.com/itsPinguiz/CLARITY.git
cd CLARITY

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Listing 5.1: Clone the repository and install its core dependencies.

Optional notebook-specific dependencies:

# Notebooks 04–07
python -m pip install pyyaml peft trl sentence-transformers faiss-cpu

Listing 5.2: Install the retrieval and parameter-efficient fine-tuning extras.

On Windows PowerShell, activate the environment with:

.venv\Scripts\Activate.ps1

Listing 5.3: Activate the virtual environment on Windows.

5.3 Hugging Face Authentication

The QEvasion dataset is downloaded automatically. Llama-based experiments may also require model access and authentication:

export HF_TOKEN="your_token_here"
hf auth login

Listing 5.4: Authenticate with Hugging Face for gated model downloads.

Never commit the token. Local .env loading is supported through python-dotenv; Colab users can use the notebook environment's secret store.

5.4 Launching the Experiments

jupyter lab

Listing 5.5: Start the local notebook environment.

Open a notebook from notebooks/ and run it from top to bottom. The numbered notebooks document the intended progression, but experiments 02–06 are largely independent after dataset preparation. Notebook 07 expects the Task B adapter produced or loaded by notebook 06.

5.5 Loading the Dataset Programmatically

from src.data.dataset_loader import load_and_split_dataset

dataset = load_and_split_dataset(seed=42)

print(dataset)
print(dataset["train"][0])

Listing 5.6: Load the reproducible QEvasion split from Python.

Output: a Hugging Face DatasetDict containing train, validation, and test. The original QEvasion training split is partitioned reproducibly into training and validation data, while the original test examples remain the final evaluation set.

Chapter 6 — Experiment Guide

6.1 Notebook Index

Notebook	Experiment	Primary Output
`01_data_analysis.ipynb`	Label distributions, answer lengths, tone analysis, and class imbalance	Exploratory plots and statistics
`02_train_encoders.ipynb`	Multi-model encoder benchmark with augmentation and custom losses	`results/encoder/`
`03_cot_llm.ipynb`	Zero-Shot and CoT prompting ablations	`results/cot/`
`04_few_shots_llm.ipynb`	Static, dynamic, and tone-aware Few-Shot prompting	`results/few_shots/`
`05_clarity_tuned_llm.ipynb`	QLoRA + DoRA for direct three-class clarity	`results/task1/`
`06_evasion_tuned_llm.ipynb`	QLoRA + DoRA for nine-class evasion and mapped clarity	`results/task2/valutazione/`
`07_prompt_chain_llm.ipynb`	Multi-question decomposition and prediction aggregation	`results/task2/chaining/`

6.2 Suggested Execution Paths

Encoder Benchmark:

01_data_analysis.ipynb → 02_train_encoders.ipynb

Listing 6.1: Suggested encoder experiment sequence.

Prompt Engineering:

01_data_analysis.ipynb → 03_cot_llm.ipynb → 04_few_shots_llm.ipynb

Listing 6.2: Suggested prompting experiment sequence.

Parameter-Efficient Fine-Tuning:

05_clarity_tuned_llm.ipynb
06_evasion_tuned_llm.ipynb → 07_prompt_chain_llm.ipynb

Listing 6.3: Suggested parameter-efficient fine-tuning sequence.

Chapter 7 — Repository Architecture

7.1 Directory Structure

CLARITY/
├── config/
│   ├── config.yml              # Dataset, model, training, and output settings
│   └── model_configs.py        # Encoder model registry
├── notebooks/
│   ├── 01_data_analysis.ipynb
│   ├── 02_train_encoders.ipynb
│   ├── 03_cot_llm.ipynb
│   ├── 04_few_shots_llm.ipynb
│   ├── 05_clarity_tuned_llm.ipynb
│   ├── 06_evasion_tuned_llm.ipynb
│   └── 07_prompt_chain_llm.ipynb
├── report/
│   └── paper__LLM_project_2026__CLARITY.pdf
├── results/
│   ├── cot/
│   ├── encoder/
│   ├── few_shots/
│   ├── task1/
│   └── task2/
├── src/
│   ├── data/                   # Loading, labels, augmentation, resampling
│   ├── training/               # Trainers, losses, and metrics
│   └── utils/                  # Environment and result persistence helpers
├── LICENSE
├── README.md
└── requirements.txt

Listing 7.1: Repository directory structure.

7.2 Reusable Modules

Module	Responsibility
`dataset_loader.py`	Reproducible splitting, majority voting, and reliability-aware tie resolution
`augmentation.py`	Length and tone features plus a common augmentation registry
`resampling.py`	Semantic downsampling, paraphrase upsampling, and Smart Resampling
`label_utils.py`	Label maps, class weights, and integer label assignment
`trainers.py`	Hugging Face `Trainer` construction with custom losses
`metrics.py`	Accuracy, Macro-F1, Weighted-F1, and mapped clarity evaluation
`results_utils.py`	JSON/CSV result persistence and run comparison

Chapter 8 — Configuration and Reproducibility

8.1 Central Configuration

config/config.yml controls:

• Dataset: source paths, seed, and augmentation settings.

• Model: the default Llama model identifier.

• Training: batch size, gradient accumulation, epochs, and maximum length.

• Outputs: result directories for Task A and Task B.

Encoder-specific settings are kept separately in config/model_configs.py, allowing new encoders to be registered without changing the notebook training loop.

8.2 Reproducibility Guarantees

Fixed Split: load_and_split_dataset(seed=42) always reproduces the same training, validation, and test partition.

Shared Evaluation Set: every reported system is evaluated on the same $308$ test examples.

Structured Outputs: aggregate metrics are stored as JSON, row-level predictions as CSV, and diagnostic plots as PNG.

Known Variability: exact LLM outputs can still vary across GPU architectures, library versions, quantization kernels, and decoding behavior. For the closest reproduction, retain the configured seed and the dependency versions used by the notebooks.

Chapter 9 — Report, Citation, and License

9.1 Project Report

The full nine-page report documents the methodology, equations, experimental protocol, complete results, and error analysis:

CLARITY — SemEval 2026 Challenge: LLMSE 2026 Course Project

9.2 Citation

If this repository supports your work, cite the project as:

@techreport{zizzi2026clarity,
  title       = {CLARITY: SemEval 2026 Challenge},
  author      = {Zizzi, Stefano and Perrotti, Alessio and Vaccari, Riccardo
                 and Candela, Davide and Lodesani, Luca},
  institution = {Politecnico di Torino},
  year        = {2026},
  type        = {LLMSE Course Project}
}

Listing 9.1: Suggested BibTeX entry for this project.

For the shared task and dataset, also follow the citation guidance on the official CLARITY task page and the QEvasion dataset card.

9.3 Authors

Author	Student ID
Stefano Zizzi	`s346595`
Alessio Perrotti	`s346737`
Riccardo Vaccari	`s348856`
Davide Candela	`s347245`
Luca Lodesani	`s346978`

9.4 License

This project is released under the MIT License.

Name		Name	Last commit message	Last commit date
config		config
notebooks		notebooks
report		report
results		results
src		src
.DS_Store		.DS_Store
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation