国产生成式人工智能解决物理问题能力研究——以“智谱AI”、“讯飞星火认知大模型”、“天工”、“360智脑”、“文心一言”为例

Research on the Ability of Domestic Generative Artificial Intelligence to Solve Physical Problems—Taking “Zhipu AI”, “SparkDesk”, “Tiangong”, “360 Wisdom Brain”, “ERNIE Bot” as Examples

摘要: ChatGPT是一款功能强大的预训练语言模型，自2022年发布以来，引发了人们的广泛关注。为紧跟人工智能的发展潮流，我国也相继出品了自己的生成式人工智能模型。为了检测国产模型解决实际物理问题的能力，本文选取了“智谱AI”、“讯飞星火”、“天工”、“360智脑”和“文心一言”等五大模型，以经典力学问题为例，分别测试了其概念理解、推理计算和实验设计能力。研究发现，上述五个模型在概念理解方面的解题能力最强，推理计算次之，实验设计最差，实际解题过程存在“计算失误”、“前后回答不一致”、“情境分析能力欠缺”等问题。横向比较：“天工”在概念理解方面的表现占优，而“文心一言”在推理计算方面的表现最好。总的来说，我国国产模型实现替代人类解题似乎还有很长的一段路要走。

Abstract: ChatGPT is a powerful pre-trained language model that has attracted widespread attention since its release in 2022. Following the development trend of artificial intelligence, China has also successively produced its own generative artificial intelligence models. In order to test the ability of domestic models to solve practical physical problems, this paper selects five domestic models, namely, “Zhipu AI”, “SparkDesk”, “Tiangong”, “360 Wisdom Brain” and “ERNIE Bot”, to test their conceptual understanding, reasoning and calculation, and experimental design capabilities, taking classical mechanical problems as examples. Research has found that the above five models have the strongest problem-solving ability in concept understanding, followed by reasoning and calculation, and the worst experimental design. In the actual problem-solving process, there are problems such as “calculation errors”, “inconsistent answers before and after”, and “lack of situational analysis ability”. Horizontal comparison: “Tiangong” performs better in concept understanding, while “ERNIE Bot” performs best in reasoning and calculation. Overall, it seems that there is still a long way to go for domestically produced models in China to replace manual problem-solving.