# LLM is Not All You Need: An Evaluation of ML vs. Foundation Models for Medical Classification

Source: https://www.youtube.com/watch?v=n3p1nyp1hYI
Recap page: https://rapidrecap.app/video/n3p1nyp1hYI
Generated: 2026-01-29T18:04:49.347+00:00

---
## Quick Overview

The study concludes that while newer, larger Language Models (LLMs) like those from Meta perform significantly better than classical ML models on complex medical classification tasks, they are not a universal solution and can fail spectacularly on simple, linear tasks, suggesting that architectural diversity and understanding fundamental data properties remain crucial for reliable medical AI.

**Key Points:**
- The research evaluates LLMs against classical ML models for text and image-based medical classification.
- The fine-tuned LLM (Laura) achieved 99.8% accuracy on the diabetic data set, significantly outperforming the classical model's 88.12% accuracy.
- The classical ML model (LightGBM) achieved 97.8% accuracy on the diabetic data set, but failed completely (0% accuracy) on the skin cancer image classification task.
- The LLM also failed the skin cancer classification task (22.4% accuracy) on the zero-shot test, but performed better (66.61%) after fine-tuning.
- The authors suggest the failure of classical models on complex tasks like image classification is due to their inability to handle language/visual nuance, whereas LLMs excel at this but struggle with simple tabular data if not specifically trained for it.
- The optimal approach appears to be a hybrid system that leverages the strengths of both LLMs for complex reasoning and classical models for structured data tasks.

![Screenshot at 03:08: The comparison is set up with classical models on one side and Foundation Models \(LLMs\) on the other, highlighting the core conflict of the evaluation.](https://ss.rapidrecap.app/screens/n3p1nyp1hYI/00-03-08.jpg)

**Context:** This video analyzes a research paper comparing the performance of Large Language Models (LLMs) against established Machine Learning (ML) models, specifically LightGBM and a custom CNN, across different medical classification tasks involving both structured tabular data (like patient vitals) and unstructured data (text and images). The paper titled, "LLM is Not All You Need: An Evaluation of ML vs. Foundation Models for Medical Classification," investigates whether the massive scale of modern LLMs inherently makes them superior across all medical domains, or if specialized models still hold an advantage in specific scenarios.

## Detailed Analysis

The video discusses research comparing LLMs against classical ML models for medical classification tasks involving text, images, and structured tabular data. The main takeaway is that LLMs are not universally superior; while they excel at tasks requiring nuanced understanding like language-based classification, they can fail disastrously on simple structured data tasks where classical models excel. Specifically, on the diabetes data set (structured, tabular), the classical LightGBM model achieved 88.12% accuracy, while the fine-tuned LLM (Laura) reached 99.8%. However, on the skin cancer image classification task, the classical model failed completely (0% accuracy), while the zero-shot LLM scored 22.4%, improving to 48.75% after fine-tuning. The authors note that the classical model's failure on image tasks stems from its inability to handle visual reasoning, while the LLM's failure on tabular data suggests its core strength is language processing, not basic arithmetic or tabular structure recognition. The ideal future likely involves a hybrid system where LLMs handle complex reasoning and language, while classical models handle structured data efficiently, avoiding the high cost and potential for hallucination associated with over-relying on large LLMs for every task.

### Experimental Setup and Data

- The evaluation compared LLMs against classical ML models across four medical data sets: text, image, tabular patient data (diabetes), and image data (skin cancer lesions).

### Diabetes Classification Results

- The fine-tuned LLM achieved 99.8% accuracy, vastly outperforming the classical LightGBM model at 88.12%, demonstrating LLM superiority in this specific context.

### Skin Cancer Image Classification Results

- The classical CNN failed entirely (0% accuracy) on zero-shot image tasks, while the LLM scored 22.4% zero-shot, improving to 48.75% after fine-tuning.

### The Core Weakness of LLMs

- LLMs struggle with simple tasks like predicting a number from a spreadsheet, performing worse than random chance (e.g., 1.5% accuracy on a simple regression task), showing they lack fundamental mathematical reasoning if not explicitly trained for it.

### The Core Weakness of Classical Models

- Classical models like LightGBM or CNNs fail on tasks requiring nuanced understanding, such as interpreting visual patterns in medical images or understanding complex language prompts.

### Conclusion and Future Direction

- The optimal path is a hybrid architecture that uses LLMs for complex reasoning and language tasks, while using specialized models for structured data and visual tasks, leveraging the strengths of both.

![Screenshot at 00:00: Title card for the Really Easy AI podcast, featuring an illustration of two podcasters and the call to action 'Become a Member Today!'](https://ss.rapidrecap.app/screens/n3p1nyp1hYI/00-00-00.jpg)
![Screenshot at 03:15: A slide comparing the performance of the LLM \(higher score\) against the classical model \(lower score\) on the diabetes data set.](https://ss.rapidrecap.app/screens/n3p1nyp1hYI/00-03-15.jpg)
![Screenshot at 06:44: A visual representation comparing the performance of the LLM \(high score\) versus the classic model \(low score\) on the text classification task.](https://ss.rapidrecap.app/screens/n3p1nyp1hYI/00-06-44.jpg)
![Screenshot at 09:32: A comparison showing the performance gap between the LLM and the classical model on the diabetes task after increasing epochs for the LLM.](https://ss.rapidrecap.app/screens/n3p1nyp1hYI/00-09-32.jpg)
![Screenshot at 11:57: A visual summarizing the comparison, implicitly contrasting the reliability of LLMs versus classical models.](https://ss.rapidrecap.app/screens/n3p1nyp1hYI/00-11-57.jpg)