A Guide to LLMs in Modeling and Simulation: From Core Techniques to Critical Challenges

Quick Overview

The paper argues that the scientific community is stuck in a mindset where they try to make Large Language Models (LLMs) deterministic by adding more prompts, when the better approach is to treat them as inherently probabilistic systems and use techniques like design of experiments to probe their behavior and manage uncertainty, especially concerning subtle changes in output due to minor input variations.

Key Points: The core argument is that LLMs should be treated as probabilistic, not deterministic, despite community efforts to force deterministic outputs through extensive prompting. The paper references research by Philip J. Gabinelli from Old Dominion University, who demonstrated that even simple graph structures exhibit variation when modeled probabilistically versus deterministically. Gabinelli's work shows that using a format like P(A|B) + P(C) results in slightly different final sums when the order of operations changes, highlighting inherent non-determinism. A key concept discussed is the 'interface bleeding problem,' where changing the hardware stack or using different LLMs results in output variation, even when the logic remains the same. The authors suggest that instead of constantly adding more prompts (which they liken to 'burying' the issue), researchers should use design of experiments (DoE) to systematically probe the model's behavior. The paper contrasts the reliability of probabilistic analysis (like using Alloy when code is wrong) versus relying solely on the LLM's internal, non-guaranteed deterministic behavior. The ultimate goal is to move away from seeking absolute scientific reproducibility in LLM outputs and instead focus on managing the inherent stochasticity.

Context: This discussion centers on a research paper by Philip J. Gabinelli from Old Dominion University, focusing on the proper methodology for modeling and simulation using Large Language Models (LLMs). The context is a critique of the current approach within the scientific community, which often attempts to enforce deterministic, reproducible results from LLMs by adding layers of prompts, rather than acknowledging and managing the inherent probabilistic nature of these tools.