LLM is Not All You Need: An Evaluation of ML vs. Foundation Models for Medical Classification

Quick Overview

The study concludes that while newer, larger Language Models (LLMs) like those from Meta perform significantly better than classical ML models on complex medical classification tasks, they are not a universal solution and can fail spectacularly on simple, linear tasks, suggesting that architectural diversity and understanding fundamental data properties remain crucial for reliable medical AI.

Key Points: The research evaluates LLMs against classical ML models for text and image-based medical classification. The fine-tuned LLM (Laura) achieved 99.8% accuracy on the diabetic data set, significantly outperforming the classical model's 88.12% accuracy. The classical ML model (LightGBM) achieved 97.8% accuracy on the diabetic data set, but failed completely (0% accuracy) on the skin cancer image classification task. The LLM also failed the skin cancer classification task (22.4% accuracy) on the zero-shot test, but performed better (66.61%) after fine-tuning. The authors suggest the failure of classical models on complex tasks like image classification is due to their inability to handle language/visual nuance, whereas LLMs excel at this but struggle with simple tabular data if not specifically trained for it. The optimal approach appears to be a hybrid system that leverages the strengths of both LLMs for complex reasoning and classical models for structured data tasks.

Context: This video analyzes a research paper comparing the performance of Large Language Models (LLMs) against established Machine Learning (ML) models, specifically LightGBM and a custom CNN, across different medical classification tasks involving both structured tabular data (like patient vitals) and unstructured data (text and images). The paper titled, "LLM is Not All You Need: An Evaluation of ML vs. Foundation Models for Medical Classification," investigates whether the massive scale of modern LLMs inherently makes them superior across all medical domains, or if specialized models still hold an advantage in specific scenarios.