ML · 2023
LLM Fallacy Detection
Benchmarked large language models for logical-fallacy detection and co-developed a detector with reproducible evaluation pipelines.
Overview
Logical fallacies are easy for humans to miss and inconsistent for language models to catch, which makes fallacy detection a useful stress test for LLM reasoning. This project benchmarked several LLMs on the task, using Python (transformers, scikit-learn) to build baselines and report precision/recall/F1 across models. We curated datasets and prompts, implemented ETL and feature-engineering steps to keep the evaluation pipeline reproducible, and co-developed a fallacy detector, running ablation studies and error analysis to understand where and why each model's detection broke down.
Highlights
- Benchmarked LLMs for fallacy detection and built baselines in Python (transformers, scikit-learn) with precision/recall/F1 reporting.
- Curated datasets and prompts; implemented ETL, feature engineering, and reproducible evaluation pipelines.
- Co-developed a fallacy detector and ran ablation studies and error analysis to refine model design.
Tech & topics
- NLP
- LLMs
- Python
- scikit-learn
- Machine Learning