Recently, Bian Ning, a postdoctoral researcher at CUMT’s School of Information and Control Engineering, in collaboration with researchers from the Institute of Software at the Chinese Academy of Sciences, published an academic paper titled “Social simulations with large language model risk Utopian illusion” in Humanities and Social Sciences Communications, a journal under the Nature portfolio. China University of Mining and Technology is the first affiliated institution of the paper, and Bian Ning is the first author. The research was funded by the China Postdoctoral Science Foundation.
The development of large language models (LLMs) has provided new research tools for simulating human behavior, group interaction, and societal operation. However, whether the model-generated social interactions can truly reflect the complexity of human society remains to be systematically tested. To address this issue, the research team constructed an LLM-based social simulation analysis framework, organizing multi-agent communication in a chat room format. They compared simulated dialogues with human dialogue data across five dimensions: social role distribution, semantic similarity, keyword persistence, sentiment tendency, and linguistic features. Experiments were conducted on eight representative models from the GPT, Llama, and DeepSeek series.
The study found that, under the experimental conditions examined, LLMs exhibited social role bias, primacy effect, and positivity bias: the roles generated by the models were overly concentrated in professional and managerial occupations, with insufficient representation of ordinary workers; early-introduced topics in the dialogue persistently dominated subsequent communication, leading to content repetition and limited topic variation; and agents tended to maintain positive, polite, and mutually affirming expressions, lacking the emotional fluctuations and disagreements found in human communication. These biases collectively formed an overly idealized “Utopian illusion,” making it difficult for the simulated society to fully present the diversity and complexity of real social interactions.
This study reveals the systematic biases of LLMs in social simulations and provides an analytical method for evaluating the reliability of such simulations. The research points out that models may reproduce more of the behaviors people expect others to exhibit rather than how people actually behave, and this discrepancy may be amplified in multi-agent interactions. The related findings provide a reference for the prudent use of LLMs in social analysis and for building models that more closely approximate real human social behavior.
This achievement reflects the research exploration of young CUMT researchers in the interdisciplinary field of artificial intelligence and social sciences. Relying on AI and multi-agent technologies, Dr. Bian Ning conducts research on issues such as social interaction, group behavior, and the social impact of technology in the AI era, combining AI methods with social science issues. In this interdisciplinary research direction, Bian Ning has been awarded a National Social Science Fund Youth Project in the sociology discipline and a General Program of the China Postdoctoral Science Foundation.
Humanities and Social Sciences Communications is a comprehensive international academic journal under the Nature portfolio covering the full spectrum of the humanities, social sciences, and behavioral sciences. It is indexed by SSCI and A&HCI and is dedicated to publishing high-level research and interdisciplinary achievements. According to the metrics published on the journal’s official website, its Impact Factor is 4.8, ranking ninth among 275 journals in the interdisciplinary social sciences category, placing it at the forefront of the field. According to the Web of Science Journal Citation Indicator (JCI), it ranks first among 408 journals in the multidisciplinary humanities category and second among 271 journals in the interdisciplinary social sciences category, reflecting its outstanding academic influence in interdisciplinary research in the humanities and social sciences.
