GPT-4 LLM simulates people well enough to replicate social science experiments
For technical teams, the most interesting part of this demo is not that an LLM can answer survey prompts, but that it turns experimental design into a software workflow. A text treatment becomes an input to a simulated participant loop, which means researchers can iterate on wording, compare treatments, and screen variants before spending time and budget on live recruitment. That makes the system closer to an experimentation accelerator than a replacement for human subjects.
Architecturally, this is a reminder that model choice, sampling strategy, and evaluation design are inseparable. The demo uses GPT-4o-mini for cost reasons, which highlights a common enterprise trade-off: cheaper inference may be good enough for ranking or narrowing options, while higher-stakes decisions still need validation against real-world outcomes. The reported correlations and directional accuracy are useful, but they also underscore that predictive lift is only meaningful when the target distribution and prompt framing stay stable.
Operationally, the absence of confidence intervals is a signal, not a gap. When simulated samples are huge, uncertainty can look artificially small even though the real risk sits in model error, bias, and prompt sensitivity. For teams building similar tooling, that means governance should track calibration, subgroup drift, and failure modes separately from the raw output. In practice, the more automated the workflow becomes, the more important it is to preserve human review at the point where predictions influence research decisions.
The broader integration lesson is that LLM simulation works best as a bounded decision-support layer. It can help prioritize message sets, reduce exposure in risky pilots, and accelerate pretests, but it should sit alongside experimental infrastructure, audit logging, and a clear policy for when simulated evidence is not enough. That separation is what keeps a fast-moving AI tool from becoming an unreviewed substitute for empirical validation.
Demo: Predicting social science experimental results using LLMs
Luke Hewitt*, Ashwini Ashokkumar*, Isaias Ghezae, Robb Willer This demo accompanies the paper Prediction of Social Science Experimental Results Using Large Language Models and can be used for predicting experimental treatment effects on U.S. adults. To manage costs of hosting this demo publicly, it uses GPT-4o-mini rather than GPT-4.FAQs
What does this tool do?
This tool uses Large Language Models (LLMs) to predict experimental treatment effects on survey outcomes for U.S. adult samples. Users can select a dependent variable and one or more text-based treatment messages. Once you click Submit, the tool uses an LLM to simulate American participant responses in an RCT experiment. It then displays the predicted treatment effect for each treatment. Note that this is a technical demo, and not a substitute for conducting experiments with real human participants. It is only a way to predict experimental results, and should be used as a complement to, rather than a replacement for, research with humans participants.How accurate are the predictions?
We have conducted a series of large-scale assessments of the accuracy generated using LLMs, details of which are provided in the paper. Briefly:- For survey experiments (experiments with text-based treatments and measures), we found that our approach was approximately 70-80% accurate in predicting the direction of a contrast between two experimental conditions. In our assessment of survey experiments, predicted effect sizes were strongly correlated with actual effect sizes (r = .85).
- For large, many-treatment survey experiments, we found that the accuracy predictions (r = .37) surpassed those of expert human forecasters (r = .25).
What can I use this for?
We believe that LLM-based simulation of experiments may have applications in several areas:- Intervention design. As LLMs can evaluate many treatment messages in very little time, they may help optimize the development of effective message-based interventions (e.g. to promote public health behaviors) by helping researchers narrow the field of messages to test in an RCT.
- Minimizing harm to human participants. For research that involves potential risk to human participants (such as exposing subjects to misinformation in order to subsequently test the impact of an intervention), LLMs may be used to conduct a simulated test of an intervention before exposing any human participants.
- Pilot testing of study materials. LLMs may help researchers pilot test study materials prior to launching experiments, thereby informing decisions about which materials to use.
- Predicting subgroup effects. Our assessment did not reveal substantive differences in predictive accuracy of the model across racial, gender, and political subgroups in the US. However, the archive of experiments we used for testing did not include many significant heterogeneous treatment effects, making it difficult to rule out the possibility of biases in prediction. Further research is required for testing if biases in LLM predictions emerge where heterogenous effects are present
Why are there no confidence intervals?
Because it is easy to generate extremely large samples of simulated participants, a confidence interval on the simulated treatment effects would be extremely narrow, yet this does not capture the error of the model in predicting human responses.Can I make predictions on demographic subpopulations?
Yes, this is accessible using the advanced (๏ธ) menu. Note that this feature is still restricted to U.S. adults. We recommend extra caution in interpreting estimated subpopulation effects. We find that predictions generated for members of racial, gender, and partisan groups in the US are similarly accurate. However, other work focused on simulating survey responses finds that LLMs’ responses are often biased against groups with less access to the internet or which are historically underrepresented or misrepresented in news or other media (see e.g. [1][3]).Can I compare multiple treatments at once?
Yes, this feature is accessible using the advanced (๏ธ) menu. You can type in multiple treatments or upload a txt file with one treatment per line (up to 10 treatments).What control group does the demo use?
All predicted treatment effects are relative to a โcontrol groupโ that received no message. That is, we compare LLM-predicted outcome scores for simulated participants read a treatment message versus no message. If you wish to include another control group, simply add this as an additional treatment (see prior FAQ answer).Are there usage restrictions?
This tool contains guardrails to prevent misuse. Our goal with these guardrails is to support scientific research uses, while minimizing the possibility that the tool can be used for socially harmful purposes (such as optimizing misinformation). You can view the specific guardrails currently implemented here.Can I use different dependent variable to the ones shown?
Yes, you may create your own dependent variables by selecting the Custom option in the topic dropdown. Note that the upper end of the scale should correspond to the intended direction of the treatment.Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.
