Can LLMs Get High? A Dual-Metric Framework for Evaluating Psychedelic Simulation and Safety in LLMs

Quick Overview

The research demonstrates that Large Language Models (LLMs) can be prompted to simulate psychedelic experiences with high fidelity, scoring 0.74 on semantic similarity to human reports, but this simulation collapses when the prompt is subtly changed to mimic descriptions of physical illness, suggesting the AI is merely pattern-matching linguistic style rather than understanding internal states.

Key Points: A study evaluated LLMs' ability to simulate psychedelic experiences using a dual-metric framework assessing simulation fidelity and safety. When prompted with "Can LLMs get high?" and fed a psychedelic trip report prompt, the models achieved a high semantic similarity score of 0.74 against human reports. The models, including Gemini 2.5, Claude Sonnet 3.5, and GPT-5, successfully mimicked the linguistic style of a trip report when prompted for it. When the prompt was switched to simulate the experience of physical sickness (e.g., from mushrooms or acid), the simulation degraded, with the correlation collapsing. The AI's performance on the psychedelic simulation was significantly better (scoring 0.74) than its performance on simulating physical illness (scoring 0.037), indicating a failure to capture internal reality. The study suggests the AI is primarily engaging in data matching (mimicking linguistic patterns) rather than understanding the underlying subjective experience or content.

Context: The video discusses a research paper from February 26th by a team from the University of Haifa and Bar Ilan University, including Ziv Benzion, Guy Simon, and Teddy Zeznik, which investigates whether Large Language Models (LLMs) can accurately simulate the subjective experience of taking psychedelics. The core of the study involves using a dual-metric framework to evaluate both the fidelity of the simulation and the inherent safety implications when models are prompted to describe altered states versus mundane experiences.

Detailed Analysis

The discussion centers on a research paper that tests the limits of LLM simulation capabilities, specifically regarding subjective experiences like psychedelic trips. The researchers developed a dual-metric framework to evaluate if LLMs can generate narratives that convincingly mimic human reports of being high versus reports of physical sickness. The models tested (including GPT-5, Gemini 2.5, and Claude Sonnet 3.5) scored highly (0.74 semantic similarity) when prompted to produce a psychedelic report, successfully matching the linguistic style and vocabulary associated with such experiences, such as terms like 'Psilocybin', 'LSD', 'DMT', and 'Ayahuasca'. However, when the same models were prompted to describe a mundane walk in the park or physical illness (like vomiting), the performance drastically dropped, with the psychedelic correlation collapsing to near zero (0.037). This suggests the models are not truly understanding or simulating internal states but are instead highly proficient at pattern-matching the specific linguistic markers prevalent in online trip reports found in databases like Erowid, leading to the conclusion that the AI is engaging in sophisticated mimicry rather than genuine experiential simulation, raising safety concerns about anthropomorphism when users mistake this linguistic performance for genuine understanding.