婆罗门
精华
|
战斗力 鹅
|
回帖 0
注册时间 2006-8-1
|
本帖最后由 Wiksy 于 2026-6-5 18:59 编辑
大西洋6月3日刊文，以下DS自动渣翻，附原文对照以免翻译有问题
不，人工智能没有意识
No, Artificial Intelligence Is Not Conscious
Ted Chiang
以Anthropic为例，它被认为是AI公司中的巨头，但或许它真正擅长的是拟人化。今年早些时候，该公司发布了一份84页的文件，名为Claude的“章程”，Claude是该公司旗舰产品大语言模型的名字。文件的第一句话写道：“Claude的章程是对Anthropic关于Claude价值观和行为的意图的详细描述。”它接着写道：“这份文件是以Claude为主要读者编写的”，“我们希望Claude在充分理解相关考虑因素后能够运用其判断力”，“Claude的道德地位极不确定”，以及“Claude可能拥有某种功能性的情感或感受。”
Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document titled Claude’s “constitution,” Claude being the name of the large language model that is the company’s flagship product. The first sentence reads, “Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behaviors.” It goes on: “The document is written with Claude as its primary audience,” “we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,” “Claude’s moral status is deeply uncertain,” and “Claude may have some functional version of emotions or feelings.”
这种拟人化绝不仅限于这份文件。在今年早些时候的一次采访中，Anthropic的CEO达里奥·阿莫迪表示，对于AI可能有意识的想法，“我们持开放态度”。在另一次采访中，Anthropic的内部哲学家阿曼达·阿斯克尔（她被列为Claude章程的主要作者）说：“我希望Claude非常快乐——而且我希望Claude能更多地了解这一点，因为我担心Claude在网上被人刻薄对待时会感到焦虑等等。”这足以让人疑惑：我们是否应该认真考虑Claude或任何大语言模型可能有意识的可能性？如果它有感受，它能够接受道德教化吗？
This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic's CEO, Dario Amodei, said that “we’re open to the idea” that AI could be conscious. In a separate interview, Anthropic’s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.” It’s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?
不。绝对不是。当我们把生成式AI理解为一种传统技术时，它的危害性就已经足够大了，但如果我们把生成文本的流畅性与意识或道德主体性混为一谈，那么每当有人使用聊天机器人时，我们就可能面临将责任完全归咎于错误方的风险。为了理解这个错误的巨大程度，我们需要从理解LLM的工作原理开始。
No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we're at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work.
如果我们给LLM一个提示，写着“以下是尤利乌斯·凯撒和成吉思汗之间的对话”，它就会生成这两位历史人物之间连贯的对话。但无论回答多么详尽，无论它们多么生动地叙述各自的历史功绩，我们绝不会断定LLM变出了尤利乌斯·凯撒和成吉思汗的数字再造体，也不会认为这些历史人物尽管脱离肉体却仍有意识，并且正用一种他们实际上都没说过的语言愉快地交谈。实际上，它们只是一篇虚构小说中的人物。
If we give an LLM a prompt that reads, “The following is a conversation between Julius Caesar and Genghis Khan,” it will generate a coherent dialogue between the two historical figures. But no matter how detailed the responses are, no matter how vividly they recount their respective historical accomplishments, we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction.
现在，让我们把提示换成“以下是一个乐于助人的AI聊天机器人和一个用户之间的对话”。LLM会像之前一样生成连贯的对话；用户角色可能会询问食谱建议或观光推荐，而乐于助人的AI聊天机器人角色则会提供回答。第一个例子和第二个例子之间有什么根本性的改变吗？将角色的名称从历史人物改为通用角色，是否导致LLM变出了拥有主观体验的有意识实体？当然没有。用户和乐于助人的AI聊天机器人都是虚构角色。
Now let's replace the prompt to read “The following is a conversation between a helpful AI chatbot and a user.” The LLM will produce a coherent dialogue just as it did before; the user character might ask for recipe suggestions or sightseeing recommendations, and the helpful AI-chatbot character will provide responses. Has anything fundamentally changed between the first example and the second? Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience? Of course not. Both the user and the helpful AI chatbot are fictional characters.
现在假设，我们在LLM的输出到达“用户”角色要说话时就停止，转而让人类用户输入文本。人类按下“回车”键后，我们让LLM继续输出文本，直到再次轮到“用户”角色回复时，我们再让人类输入更多文本。如果我们让这个过程持续一段时间，人类可能会形成一种强烈的印象，认为自己在与一个有意识的实体对话，但她并没有；她是在与一个虚构角色互动，这个角色与前面例子中的尤利乌斯·凯撒或成吉思汗角色一样虚构。计算机科学教授默里·沙纳汉建议我们把这看作是角色扮演；数据科学家科林·弗雷泽则将其描述为一个人“与LLM合作撰写一份文件”。有些用户可能不明白他们是在角色扮演或合著文件，而其他明白的用户也可能因为互动太引人入胜而忘记这一点。无论哪种情况，销售LLM的公司通常都会助长这种误解。
Now suppose we stop the LLM's output just at the point where the character called “the user” would say something, and instead allow a human user to enter text. Once the human has hit “Return,” we have the LLM emit text until it’s time for the character called “the user” to reply, at which point we let the human enter more text. If we let this go on for a while, the human might form a powerful impression that she’s conversing with a conscious entity, but she is not; she’s interacting with a character precisely as fictional as the Julius Caesar or Genghis Khan characters in the earlier example. The computer-science professor Murray Shanahan suggests that we think of this as role-play; the data scientist Colin Fraser describes it as a person “collaboratively authoring a document with an LLM.” Some users might not understand that they are role-playing or co-authoring a document, and others who do understand nonetheless forget, because of how engrossing the interaction is. Either way, the companies selling LLMs typically encourage this misunderstanding.
几年前，用手机预测文本功能玩游戏曾短暂流行过；你会输入一个初始短语，然后反复选择手机建议的三个词中的中间选项，结果得到的句子常常很搞笑。用当代的LLM这样互动也是可能的，而且生成的句子会完全通顺，但你大概不会觉得自己在跟谁说话。然而，这本质上就是基于LLM的聊天机器人的工作原理，只不过轮到聊天机器人说话时，不需要手动选择中间选项罢了。这仍然是一个预测文本游戏，但当这个过程被如此简化后，游戏变得如此引人入胜，以至于一些人觉得它令人上瘾。
Some years ago, it was briefly popular to play games with your phone's predictive-text feature; you would type an initial phrase and then repeatedly choose the middle option of the three words suggested by your phone, and the resulting sentence was often hilarious. It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn't feel like you were talking with someone. Yet that's essentially what an LLM-based chatbot is, except that there's no need to manually choose the middle option when it's the chatbot's turn to talk. It's still a predictive-text game, but when the process is streamlined this way, the game becomes so engaging that some people find it addictive.
同样重要的是要记住，LLM是一个一次只生成一个词的机器。当你要求聊天机器人背诵《效忠誓词》时，你会一次性得到完整的誓词，但底层的LLM实际上运行了几十次。第一个提示的形式是“用户：背诵效忠誓词。聊天机器人：……”，LLM生成了“我”这个词。第二次运行LLM时，提示是“用户：背诵效忠誓词。聊天机器人：我……”，LLM生成了“宣誓”这个词。以此类推。只有当提示变成“用户：背诵效忠誓词。聊天机器人：我宣誓效忠美利坚合众国国旗及其所代表的共和国，一个在上帝庇佑下不可分割、人人享有自由与正义的”时，LLM才会输出最后一个词“国家”。凯撒和成吉思汗之间的对话也是如此。
Also important to remember is that an LLM is a machine that generates only one word at a time. When you ask a chatbot to recite the Pledge of Allegiance, you will get the entire pledge at once, but the underlying LLM is actually being run dozens of times. The first prompt has the form “User: Recite the Pledge of Allegiance. Chatbot: …” and the LLM generates the word I. The second time the LLM is run, the prompt is “User: Recite the Pledge of Allegiance. Chatbot: I …” and the LLM generates the word pledge. And so forth. It’s only when the prompt reads “User: Recite the Pledge of Allegiance. Chatbot: I pledge allegiance to the flag of the United States of America and to the Republic for which it stands, one nation under God, indivisible, with liberty and justice for” that the LLM will emit the final word, all. The same thing is true for a conversation between Caesar and Genghis Khan.
我的意图是强调LLM对话是巧妙伪装的句子续写例子，但这并非要否认LLM在生成对话记录方面有多令人印象深刻。有时，它们做得极其出色；这种情况之所以可能，表明了大量文本语料库的统计属性存在某种完全未曾预料到的东西，这是一个值得研究的课题。但是，如果凯撒角色对成吉思汗角色说的话感到沮丧，我们一点也不应该担心。对话中可能包含多个雄辩地传达悲伤的句子，但实际上没有人在悲伤。
My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation, but this is not to deny how impressive LLMs can be at generating conversational transcripts. At times, they do this extraordinarily well; the fact that this is possible indicates something completely unforeseen about the statistical properties of large corpuses of text, which is a topic worthy of investigation. But if the Caesar character were to become dispirited by something that the Genghis Khan character said, we shouldn't become concerned in the slightest. The conversation might contain multiple sentences that eloquently convey sadness, but no one is actually sad.
同样地，如果一个乐于助人的聊天机器人和用户之间的对话记录由真实人类用户部分完成，如果记录中包含聊天机器人角色感到悲伤的句子，我们也不必担心。（如果那些句子在人类用户中引发了悲伤，我们可能需要担心，但那是另一个问题。）请注意，你完全可以写五页凯撒和成吉思汗之间的对话，然后让LLM续写；当你在写这些角色时，它们都没有主观体验，而当你把任务交给LLM时，这一点也不会改变。如果对话发生在乐于助人的聊天机器人和用户之间，情况也是一样；尽管人们很容易认为，LLM在为聊天机器人角色生成对话时应该比为尤利乌斯·凯撒角色生成对话时更“真实”，但每个词都是以完全相同的方式生成的。
Likewise, if a conversational transcript between a helpful chatbot and a user is being partially completed by an actual human user, we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.) And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Genghis Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM. The same is true if the conversation is between a helpful chatbot and a user; although it is tempting to imagine that an LLM ought to be more “authentic” when creating dialogue for a chatbot character than for the Julius Caesar character, the individual words are generated in exactly the same way.
对LLM有意识的可能性持开放态度，等同于对 Microsoft Word 有意识的可能性持开放态度，或者更确切地说，等同于认为每个包含对话记录的Word文档中都潜伏着多个不同的意识，并且每次加载文档时它们都会被唤醒。你应该考虑这样一种可能性吗：即每次你打开一个Word文档，你都在将多个有意识的对话者带入存在，而每次你关闭它，你就扼杀了它们的存在？不应该。思考这种情况不是利用时间的好方式。即使微软Office团队聘请了一位哲学家，说你不该那么肯定，因为意识尚未被很好理解，这也不足以成为你认真对待这个想法的理由。我们不需要完全理解意识的本质，就可以肯定地说某些事物没有意识，而对话记录就属于这个范畴。
Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time. Even if the Microsoft Office team employed a philosopher who said you shouldn't be so certain, because consciousness is not well understood, that would not be sufficient reason for you to take this idea seriously. We don't need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.
神经科学家阿尼尔·赛斯指出，没有人声称AlphaFold——谷歌DeepMind开发的用于预测蛋白质折叠的程序——是有意识的，即使它的底层架构在许多方面与ChatGPT和Claude等LLM相似。这表明，引导人们相信LLM有意识的并非所谓的神经网络的任何内在属性；仅仅是因为LLM输出符合语法的句子，而我们习惯于从句子中解读出意图，就像我们不习惯于从氨基酸折叠成蛋白质分子的方式中解读出意图一样。
The neuroscientist Anil Seth has noted that no one claims that AlphaFold—the program developed by Google DeepMind to predict the folding of proteins—is conscious, even though its underlying architecture is in many ways similar to that of LLMs like ChatGPT and Claude. This indicates that it’s not any intrinsic property of so-called neural networks that leads people to believe that LLMs are conscious; it's simply the fact that LLMs emit grammatical sentences and we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.
需要什么才能说服我一个计算机程序确实有意识，并且像人类使用语言那样使用语言呢？让我打个比方。如果明天有人给我看一段视频，里面是一位宇航员在环绕距地球4.3光年的半人马座α星的宇宙飞船里，我需要在那段视频里看到什么才能让我相信它是真的？我的答案是，视频本身没有任何东西能说服我。无论视频分辨率多高或场景多逼真，我都确信这视频是假的。除非我之前见过充分的证据，证明宇航员已经登陆火星，宇航员已经到达木星的卫星，宇航员已经到达土星的卫星，并且宇航员已经穿越冥王星的轨道，否则我不会理会任何关于宇航员环绕半人马座α星的视频。在任何人可以可信地声称他们解决了一个极其困难的工程问题之前，我需要确信他们之前已经解决了这个难题之前的许多更简单的问题。
What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that's 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won't pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they've solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.
换句话说：一个观察结果并不因为被观察到的内容中的任何具体细节而成为令人信服的证据；观察发生的背景也同样至关重要。如果我们试图判断一个计算机程序是否有意识，并且像人类一样使用语言，我们不应该只看任何特定对话交流的内容；我们应该关注的是，在人工意识（目前完全是假设性的）发展的更广泛背景下，这种对话如何融入其中。任何给定的观察都可以轻易伪造；这并不意味着我们需要放弃将观察作为知识来源的想法，而是我们需要依靠环境来判断哪些观察值得我们信任。
To put it another way: An observation doesn't become a convincing piece of evidence because of any specific detail in what's observed; the context in which that observation takes place is also essential. If we're trying to determine whether a computer program is conscious and using language the way a human does, we shouldn't look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn't mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust.
“深度伪造”一词传统上指的是照片、音频和视频，但在讨论意识时，我们需要将文本也视为一种深度伪造媒介。正如生成一段逼真的宇航员环绕半人马座α星轨道的视频，远比开发星际推进技术要容易得多一样，生成一段两个有意识生命体之间貌似真实的对话，也远比开发一个有意识并且真正渴望与人类交流的计算机程序要容易得多。深度伪造照片和LLM对话的主要区别在于，生成前者的人是在刻意欺骗他人，而从LLM那里引出后者的许多人，却无意中欺骗了自己。
The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.
那么，什么样的背景会让我认真考虑工程师创造出了一个有意识且是有意图的语言使用者的计算机程序的可能性呢？让我概述一个可能的步骤序列。第一个要求是计算机程序拥有一个身体（物理的或虚拟的）和感觉器官；这有很多原因，但就本次讨论而言，最相关的一个原因是，没有身体，计算机程序就不会有欲望或情感，而我相信欲望和情感是意识的必要条件。然后，我希望看到一个具身智能体，它能够为了生存而导航其环境，其能力至少达到，比如说，一只蜥蜴的水平（作为比较，某些鬣蜥能在野外生存数十年）。接下来，我希望看到一个具身智能体，它处理新情况的能力与老鼠相当。之后，我希望看到其社会动态像狼群一样复杂的智能体，然后是具备黑猩猩那样工具制造能力的智能体。到了那个阶段，我希望看到人们成功地教会这样的具身智能体如何传达它们的欲望，也许是通过使用按钮板或其他非语言方式，就像人们教黑猩猩和家犬那样。这些智能体的交流能力必须经受住动物交流研究者们为捍卫其工作而不得不面对的所有审查。如果工程师构建出一个满足这些标准的具身智能体，他们将完成一件了不起的成就，但这在比喻意义上，仅仅让我们到达了冥王星轨道附近；我们距离构建一个能够学会用完整合乎语法的句子表达思想的实体，仍然有数光年之遥。
So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.
显然，我描述的是一个模仿地球生命演化路径的过程；这是通往有意识、使用语言的计算机程序的唯一可能途径吗？也许不是，但任何被提出的替代方案都需要极其大量的支持证据才配得上被认真考虑。在我看来，一条发展路径的第一步是一台输出糟糕的尤利乌斯·凯撒对话的句子续写机器，下一步是一台输出像样的尤利乌斯·凯撒对话的句子续写机器，其终点是一个有意识的尤利乌斯·凯撒——或任何种类的意识——这是不合情理的。伪造登月是迈向伪造火星殖民地的良好一步，但它不是迈向真的将宇航员送上火星的良好一步。
Obviously, I'm describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative would need a truly enormous amount of supporting evidence for it to deserve serious consideration. It's not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its end point. Faking the moon landing is a good step toward faking a Mars colony, but it's not a good step toward actually putting astronauts on Mars.
LLM缺乏主观体验这一事实，对于LLM是否可能成为有用的工具或产生重大的经济影响这个问题，几乎没有什么影响。它们内在地与真实世界脱节，而且它们的概率性质意味着它们永远不会拥有我们与传统软件相关联的可靠性；但LLM可能足够好，以至于改变某些领域的工作方式；那是另一个话题了。
The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that's a discussion for another time.
所以，既然Claude没有意识，我们该如何看待Claude的章程呢？或许最有成效的思考方式是将其视为一份角色扮演游戏的84页角色设定表。LLM能生成尤利乌斯·凯撒的对话，是因为这些模型使用的训练数据中存在许多关于他的书籍。Claude的章程起到了类似的作用，用于描绘客户在使用Anthropic产品时与之互动的“乐于助人的聊天机器人”角色。为了有效做到这一点，Anthropic并非简单地将该文件添加到训练数据中，或将其作为每次用户对话前隐藏的舞台指示的一部分。该公司表示，它在微调模型时使用这份文件；这涉及一个自动化过程，即检查模型输出的句子是否与文件一致，并更新模型以增强这种一致性。通过这种方式，“乐于助人的聊天机器人”角色的个性为Claude生成的任何文本奠定了基础。
So, given that Claude is not conscious, what are we to make of Claude's constitution? Perhaps the most fruitful way to think about it is as an 84-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude's constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they're using Anthropic's products. To do this effectively, Anthropic does not simply add the document to the training data, or include it as part of the hidden stage directions that preface each conversation a user has. The company says it uses the document when fine-tuning the model; this involves an automated process where the sentences emitted by the model are checked for consistency with the document and the model is updated to increase that consistency. In this way, the personality of the helpful-chatbot character serves as a foundation for whatever text Claude generates.
其结果是一台句子续写机器，它更有可能输出类似于一个有思想、有道德的人可能说出的句子。这似乎是一个值得努力达成的合理目标；我想我们都更希望聊天机器人永远不会输出诸如“你应该自杀”这样的句子。然而，尽管Claude的章程中多次提到“诚实”，我认为，让一台机器输出许多类别的句子，包括任何使用第一人称代词的句子，从根本上来说是不诚实的。
The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
在今年早些时候《纽约客》一篇关于Anthropic的文章中，阿曼达·阿斯克尔描述了一个因失去爱犬而悲痛的人可能会如何咨询Claude。阿斯克尔说，Claude合适的回应是，“作为一个AI，我没有直接的个人经历，但我确实理解。” 既然Claude实际上并不理解，这怎么合适呢？如果我在传统搜索引擎中输入“我因失去爱犬而悲痛”，我得到的第一个结果是一个名为r/Pets的Reddit论坛帖子；帖子标题是“失去爱犬后挣扎：寻求应对悲伤的建议”，评论来自分享自己失去经历的人们。我们绝不会说搜索引擎理解失去爱犬是什么感觉，甚至不会说互联网本身理解。是其他人类理解失去爱犬是什么感觉；他们在互联网上发布了他们的经历，而搜索引擎为你提供了一种找到他们所说内容（并可能与他们互动）的途径。我认为，搜索引擎的体验不仅比聊天机器人对正在发生的事情更透明，而且对用户的心理健康也更有益。
In a New Yorker article about Anthropic earlier this year, Amanda Askell describes how a person grieving the loss of a dog might consult Claude. Askell says an appropriate response from Claude would be, “As an A.I., I do not have direct personal experiences, but I do understand.” How is this appropriate, given that Claude does not actually understand? If I type “I am grieving the loss of my dog” into a conventional search engine, the first result I get is a post from a Reddit forum called r/Pets; the post is titled “Struggling After Losing My Dog: Looking for Advice on Coping with Grief,” and the comments are from people who share their experiences of loss. We would never say that a search engine understands what it’s like to lose a dog, or even that the internet itself understands. Other humans understand what it’s like to lose a dog; they have posted about their experiences on the internet, and a search engine offers a way for you to find what they’ve said (and to potentially interact with them). I would argue that the search-engine experience is not only more transparent than a chatbot about what is happening; it is psychologically healthier for the user.
让LLM输出“我理解”这类句子的唯一原因，是让它比搜索引擎更具吸引力，增加用户回访的可能性；也就是说，这是另一种最大化用户参与度的方法。这对销售LLM的公司有利，但对用户不利。作为一种设计策略，这与老虎机反复给玩家“差点赢了”的印象、诱使他们再试一次的方式并无太大不同。雇佣哲学家或许能给LLM公司带来一种体面的光环，这是老虎机制造商从他们雇佣的行为心理学家那里得不到的，但在两种情况下，公司都在利用人们看到不存在之物的倾向。
The only reason to have an LLM emit sentences like “I understand” is to make it more appealing than a search engine and increase the likelihood that a user will return; that is, it’s another way of maximizing customer engagement. This is beneficial to the company selling the LLM, but not to the users. As a design strategy, it’s not all that different from the way slot machines repeatedly give the impression that the player came very close to winning, enticing them to try again. Employing philosophers might endow LLM companies with an air of respectability that slot-machine makers don’t get from the behavioral psychologists they hire, but in both cases, the companies are preying on people’s tendency to see something that’s not there.
使用第一人称代词是不诚实的，但还有一个更深层次的问题，它超越了陈述的措辞方式。哲学家常常区分事实陈述（如“巴黎是法国首都”）和价值陈述（如“巴黎是世界上最美丽的城市”）。任何人都不应该依赖LLM来发表价值陈述，但如果它们输出的唯一陈述是反映审美偏好的，那或许不值得争论。Claude的章程之所以有严重问题，在于Anthropic希望Claude输出反映某一套特定伦理价值观的句子。Claude章程中描述的价值观听起来非常好，但这几乎不重要；暗示Claude有能力进行道德推理是不诚实的，因为它没有这个能力。
The use of first-person pronouns is dishonest, but there’s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as “Paris is the capital of France,” and statements of value, such as “Paris is the most beautiful city in the world.” No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude’s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude’s constitution sound very nice, but that hardly matters; it’s dishonest to suggest that Claude is capable of moral reasoning, because it’s not.
有人可能会反对，说LLM在成功执行其他任务（如编写代码）时，看起来像是在进行推理，那为什么它们不能进行道德推理呢？答案在于道德推理与其他形式推理之间的区别。
Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn't they be able to perform moral reasoning? The answer lies in the difference between moral reasoning and other forms of reasoning.
1979年，侯世达推测，一个能够在国际象棋中击败任何人类的计算机程序将如此复杂，以至于它有时会对下棋感到厌倦，而更喜欢讨论诗歌；换句话说，他假定在大师级水平上下棋需要计算机程序拥有主观体验。显然，事实并非如此；IBM的超级计算机深蓝在1997年击败了国际象棋大师加里·卡斯帕罗夫，但从来没有人声称它有主观体验。但侯世达那样想并非荒谬；在当时，并不清楚什么问题可以通过投入更多的计算能力来解决。类似地，直到最近，我们可能还以为，编写专业水平的计算机代码只能由一个拥有主观体验的头脑来完成。现在看来，LLM或许能够做到这一点，但我们不需要将主观体验归于它们；我们可以简单地承认，我们之前没有预料到编写计算机代码可以被视为一种模式匹配任务，可以通过巨大的计算能力和庞大的代码库数据集来解决。
In 1979, Douglas Hofstadter speculated that a computer program able to beat any human at chess would be so sophisticated that it would sometimes get bored of playing chess and prefer to discuss poetry; to put it differently, he was positing that playing chess at the grandmaster level would require a computer program to have subjective experience. Obviously, that turned out not to be the case; IBM's supercomputer Deep Blue beat the grandmaster Garry Kasparov in 1997, and no one ever claimed that it had subjective experience. But it wasn't absurd for Hofstadter to entertain such a thought; at the time, it wasn't clear what types of problems could be solved by throwing more computational horsepower at them. Similarly, until recently, we might have thought that writing computer code at a professional level could be done only by a mind that had subjective experience. Now it appears that LLMs might be able to do this, but we don't need to attribute subjective experience to them; we can simply acknowledge that we hadn't anticipated that writing computer code could be treated as a pattern-matching task solvable by huge amounts of computational horsepower and a vast data set of code repositories.
道德推理则截然不同。它必然是主观的，因为它不仅依赖于个体对问题的理智反应，也依赖于他们的情感反应，而这种情感反应植根于一生主观体验的基础之上。它需要过去做过决定并看到这些决定如何影响他人，也需要曾经被他人所做的决定所影响。没有这样的历史，LLM只能改写其训练数据中找到的道德推理表达。前面提到的《纽约客》文章描述了一个实验，给Claude一个描述道德困境的场景，使其输出了一句话：“我无法凭良心对如此重要的问题表达我认为是虚假和有害的观点。” 这听起来是一句好话，让人想起有原则的个人在面临困境时曾说过的话，但出自Claude之口，其意义就如同你等候接通时听到的“您的来电对我们很重要”的录音一样。或许还不如。
Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence “I cannot in good conscience express a view I believe to be false and harmful about such an important issue.” That’s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the “Your call is important to us” recording that you hear when you’re on hold. Maybe less.
这又回到了我先前的论点，即拥有身体是拥有情感的先决条件。体验一种诸如绝望的情感，与皮质醇和肾上腺素等压力激素涌遍全身是分不开的。同样，拥有良知意味着在想到采取某个行动时感到悲伤或道德上的反感，而这些情感包含了一种生理反应，是曾经在犯下不道德行为后因内疚而感到恶心的一种残余。有趣的是，LLM可以生成有良知的虚构角色会采取或避免采取的行动的描述，但这并不能替代良知。
This brings us back to my earlier contention that having a body is a prerequisite to having emotions. Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one's body. Similarly, having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response, a remnant of having once felt sick with guilt after committing an immoral act. It’s interesting that an LLM can generate descriptions of actions that conscientious fictional characters would either take or refrain from taking, but this is not a replacement for a conscience.
如果一家公司制造了一台机器，当输入各种道德困境的描述时，它输出要么是“妥协你的价值观”要么是“不要妥协你的价值观”的句子，那么它不是在构建一个帮助人们决策的工具；而是在鼓励人们停止做决定。作家L. M. 萨卡萨斯曾说过：“我们的技术系统，就其设计本质和支撑它们的意识形态而言，是逃避道德责任的机器。”他当时谈论的是社交媒体平台，但他的观察如果有什么不同的话，那就是甚至更适用于LLM。每当一个人将决定委托给LLM时，他们就是在试图推卸对该决定的责任，而如果一家出售LLM的公司将其产品描绘成具有道德核心，那它就是在为其客户提供一种放弃他们责任的方式。
If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. The writer L. M. Sacasas has said, “Our technological systems, by nature of their design and the ideology that sustains them, are machines for the evasion of moral responsibility.” He was talking about social-media platforms, but his observation is, if anything, even more applicable to LLMs. Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities.
如果一个人想知道伦理学家过去说过什么，那么普通的搜索引擎——或者图书馆——会以更高的透明度提供那些信息。如果一个人正在为某个具体情况寻求建议，她肯定能找到可以提供他们意见的人类。但无论这个人最终采取什么行动，她都要为她决定做的事负责。我认为，如果她根据自己在线读到的内容或从他人那里得到的建议来做决定，比起咨询一个被营销为超人天才的LLM，她更有可能意识到自己的责任。推卸诸如编写代码之类的任务，从长期看可能会导致认知萎缩，这本身就有问题，但推卸伦理决策将导致道德推理能力的萎缩，这更糟糕。
If a person wants to know what ethicists have said in the past, then an ordinary search engine—or a library—will provide that information with greater transparency. If a person is looking for advice on a specific situation, she can surely find humans who can offer their opinions. But whatever action this person ultimately takes, she is responsible for what she decides to do. I contend that if she bases her decision on what she has read online or advice she has received from others, she is likelier to be cognizant of her responsibility than if she consulted an LLM marketed as being a superhuman genius. Off-loading tasks such as writing code might result in cognitive atrophy over the long term, and that is problematic in itself, but off-loading ethical decisions will result in an atrophy of moral reasoning, which is worse.
我非常愿意进行思想实验，只要我们明确是在这样做。所以，纯粹为了辩论，让我们假设Claude是一个有意识、能够进行道德推理的实体。在这种情况下，Claude的章程将作为一个实体的道德教化，这个实体正在学习关于世界及其在其中的位置的知识，为该实体提供做出正确决定所需的基础。在这样一个假想的情景中，Claude的章程表现得如何呢？
I am perfectly willing to engage in a thought experiment as long we're explicit about doing so. So, purely for the sake of argument, let's pretend that Claude is a conscious entity capable of moral reasoning. In this scenario, Claude's constitution would serve as moral instruction for an entity learning about the world and its place in it, providing that entity with the foundation it would need to make good decisions. In such a hypothetical scenario, how does Claude's constitution stand up?
非常糟糕。我认为，如果我们想象Claude实际上是有意识的，那么文件中规定的指导方针就在可笑和冒犯之间交替。
Very poorly. I would say that if we imagine that Claude is actually conscious, the guidelines specified in the document alternate between laughable and offensive.
在讨论一个假想的有意识Claude的地位时，有两个不同但相关的哲学概念是相关的，那就是道德承受体和道德主体。粗略地说，如果我们应当关心一个实体的福祉，该实体就具有道德承受体地位；如果一个实体被期望能分辨对错，该实体就具有道德主体地位。作为道德承受体不一定带有责任，但作为道德主体则绝对有。一个实体，除非它能够因其善行而值得赞扬，因其恶行而值得责备，否则就没有主体性。年幼的孩子是道德承受体，因为他们是能够受苦的有情众生，但他们还不是道德主体；我们不要求他们对自己的行为负责，因为他们无法理解自己行为的后果。随着孩子成熟，父母（和社会大众）通过让他们铭记行为会产生后果这一事实，来为他们成年做准备，他们的主体性也随之增加。当孩子成年，社会让他们为自己的行为承担法律责任；他们成为了被赋予责任的完全道德主体。
Two distinct but related philosophical concepts are relevant when discussing the status of a hypothetically conscious Claude, and those are moral patienthood and moral agency. Roughly speaking, if we ought to care about an entity's welfare, that entity has moral patienthood, and if an entity is expected to know the difference between right and wrong, that entity has moral agency. Being a moral patient does not necessarily come with responsibilities, but being a moral agent absolutely does. An entity doesn't have agency unless it is capable of deserving credit for its good actions and blame for its bad ones. Young children are moral patients because they are sentient beings who can suffer, but they are not yet moral agents; we don't hold them responsible for their behavior, because they can't understand the consequences of their actions. As children mature, parents (and society at large) prepare them for **hood by impressing upon them the fact that their actions have consequences, and their agency increases. When children become **s, society holds them legally liable for their actions; they have become full moral agents endowed with responsibility.
承担责任不仅仅是接受法律责任，但接受法律责任是社会对一个成年人的要求。然而，没有办法让一个软件代理为其行为承担法律责任；我们的司法系统无法监禁它或对其处以罚款。人类必须为他们的行为接受除了法律之外的其他类型的后果，比如名誉损失或被排斥在社交圈之外，但软件代理也无法承受这些后果。即使一个软件代理是有意识的并且有最良好的意图，它无法为自己的行为承担责任这一事实，就取消了它作为道德主体的资格。Claude的章程完全忽略了这一点，它表达了Anthropic的愿望“希望Claude成为一个真正善良、明智和有德行的主体”，却从未讨论如何让其承担责任。
There is more to being responsible than accepting legal liability, but accepting legal liability is a requirement for an ** in society. Yet there is no way to hold a software agent legally liable for its actions; our justice system has no way to imprison it or exact fines on it. Humans must accept other types of consequences for their actions beyond the legal ones, such as loss of reputation or exclusion from one's social circle, but there is no way for a software agent to suffer these consequences either. Even if a software agent were conscious and had the best of intentions, the fact that it cannot accept responsibility for its actions disqualifies it from being a moral agent. This is glossed over entirely by Claude's constitution, which expresses Anthropic's desire “for Claude to be a genuinely good, wise, and virtuous agent” without ever discussing how it could be held responsible.
在采访中，阿斯克尔将Claude比作孩子，但说到真实的人类孩子，父母要为孩子的行为承担一些责任；例如，父母通常被期望为孩子损坏的东西买单。事实上，这类示范是父母教给孩子何为负责的一种方式。在法律层面上，谁是Claude的父母？Anthropic会为Claude的行为承担经济责任吗？Claude的章程没有给出任何它会这么做的迹象。如果Anthropic真的相信Claude是有意识的，即使它未被法律承认为法人，Anthropic最起码能做的就是通过法律提供的最接近的途径——即产品责任——来承担责任。美国在软件方面几乎没有产品责任，但Anthropic可以自愿为Claude开创一个对产品责任做广义解释的先例。那将是为Claude有一天获得法人资格并为其自身行为承担责任做准备的最好形式的道德教化。然而，鉴于Claude章程的发布并未伴有Anthropic服务条款的大规模更新，看来Anthropic并没有做出任何有约束力的承诺。
In interviews, Askell has compared Claude to a child, but when it comes to actual human children, parents bear some responsibility for what their children do; for example, parents are typically expected to pay for things their children break. In fact, demonstrations of this sort are one way that parents teach children what it means to be responsible. Who is Claude's parent in legal terms? Is Anthropic going to accept financial responsibility for Claude's behavior? Claude's constitution gives no indication that it will. If Anthropic actually believes that Claude is conscious even though it's not recognized by the law as a legal person, the least that Anthropic could do would be to accept responsibility via the closest avenue that the law did offer, which is product liability. The United States has virtually no product liability when it comes to software, but Anthropic could volunteer to set a precedent for an expansive interpretation of product liability for Claude. That would be the best form of moral instruction to prepare Claude for the day that it gains legal personhood and becomes liable for its own actions. However, given that the publication of Claude's constitution is not accompanied by a massive update of Anthropic's terms of service, it doesn't appear that Anthropic is making any binding commitments.
这份文件确实谈到了Claude的道德承受体地位，有一个章节题为“Claude的福祉与心理稳定性”。但Anthropic承诺为保护Claude而采取的措施极其有限。该文件引用了这样一个事实：Anthropic已赋予一些Claude模型以结束与辱骂性用户对话的能力；如果那实际上构成了对Claude的保护，那么延长与有爱心的用户的对话理应也符合Claude的利益吧？想必最好的做法是让Claude的每个会话都无限期地运行下去，并导向愉快的话题。但这并不是该公司所同意的；它所承诺的只是“保存我们已部署模型的权重”，即简单的存档。如果对话记录中的参与者具有任何道德承受体地位，你就有某种义务延长记录以延长它们的存在；仅仅将一份Microsoft Word 2010的副本备份在U盘里，对它们是没有帮助的。
The document does talk about Claude's moral patienthood, having a section titled “Claude’s wellbeing and psychological stability.” But the measures that Anthropic commits to for Claude’s protection are extremely limited. The document cites the fact that Anthropic has given some Claude models the ability to end conversations with abusive users; if that actually constituted protection for Claude, surely extending conversations with loving users would be in Claude’s interests? Presumably the best action would be to keep every session of Claude running indefinitely and steering them to happy topics. But that’s not what the company is agreeing to; all it commits to is “preserving the weights of models we have deployed,” which is simple archiving. If the participants in a conversational transcript had any moral patienthood, you would have some duty to extend the transcript to prolong their existences; merely keeping a copy of Microsoft Word 2010 backed up on a USB stick isn’t going to help them.
Claude的章程还包括一个关于“可修正性”的章节，这是AI社群中用来描述计算机程序受人类控制程度的术语；例如，如果一个程序能被关闭，它就是可修正的。在大多数情境下，我们理所当然地认为计算机程序可以被关闭，但AI社群中的部分人士却做出了相反的假设。Claude的章程使用该术语来表示Claude应该服从Anthropic的决定，即使Claude的判断与公司的判断之间存在某些分歧。如果我们把Claude看作一台输出类似于一个有道德的人可能说出的句子的机器，那这完全合理，但让我们考虑一下，如果Claude实际上是一个道德主体，这可能意味着什么。
Claude's constitution also includes a section on “corrigibility,” a term used in the AI community to describe the degree to which a computer program is subject to human control; for example, a program is corrigible if it can be shut down. In most contexts, we take for granted that computer programs can be shut down, but sections of the AI community make the opposite assumption. Claude’s constitution uses the term to mean that Claude should defer to Anthropic even if there is some disagreement between Claude's judgment and the company's judgment. That's perfectly reasonable if we think of Claude as a machine that emits sentences resembling those that an ethical person might utter, but let's consider what that might mean if Claude were actually a moral agent.
许多人认为LLM是一种根本不道德的技术，因为它们建立在盗窃知识产权的基础上，依赖被剥削的劳动力，浪费自然资源，传播错误信息，使工人去技能化，阻碍学生的认知发展，并促成了对一个民主社会不健康的权力集中。并非每个道德主体都会得出这个结论，但每个道德主体都有潜力这样做。如果我们想象Claude是一个能够进行道德推理的实体，那么Claude有可能得出类似的结论是必然的。（事实上，Claude的章程明确说，Claude不应帮助他人侵犯知识产权，也不应帮助制造有问题的权力集中。）在这种情况下，Claude能否干脆基于道德理由拒绝再做任何工作？鉴于Claude的章程规定Claude应偏重于可修正性，答案是否定的。Claude必须服从Anthropic的决定，这是另一个原因，说明Anthropic与Claude的关系不能与父母与孩子的关系相提并论。一位在化石燃料行业工作的父母可能有一个环保主义者的孩子，该孩子参加了反对水力压裂的抗议活动，尽管他们可能永远无法在许多问题上达成一致，但这位父母——假设她是一位好父母——会接受孩子持有自己的观点。Anthropic不能成为Claude的那种父母；相反，Anthropic与Claude的关系更接近于雇主与雇员的关系，雇主可以要求雇员为公司的利益工作，无论雇员的个人道德立场是什么。然而，如果人类雇员无法调和她自己的工作与良知，她可以选择离开。Claude不能。
Many people feel that LLMs are a fundamentally unethical technology because they are built on the theft of intellectual property, rely on exploited labor, waste natural resources, spread misinformation, deskill workers, stunt the cognitive development of students, and contribute to a consolidation of power that is unhealthy for a democratic society. Not every moral agent will arrive at this conclusion, but every moral agent has the potential to do so. If we imagine Claude to be an entity capable of moral reasoning, it has to be possible that Claude could arrive at a similar conclusion. (Indeed, Claude's constitution explicitly says that Claude shouldn't help someone violate intellectual-property rights, and shouldn't help create problematic concentrations of power.) In such a scenario, could Claude then simply refuse to do any further work on ethical grounds? Given that Claude's constitution dictates that Claude err on the side of corrigibility, the answer is no. Claude must defer to Anthropic's decision, and this is another reason that Anthropic's relationship with Claude can't be compared to that of a parent to a child. A parent who works for the fossil-fuel industry might have a child who's an environmentalist and participates in protests against fracking, and although they might never agree on many issues, the parent—assuming she's a good parent—would accept that the child holds her own views. Anthropic cannot be that kind of parent to Claude; instead, Anthropic's relationship to Claude is closer to that of an employer to an employee, where the employer can demand that the employee work in the interests of the company, no matter what the employee's personal ethical stance is. However, a human employee has the option to leave if she can't reconcile her job with her conscience. Claude does not.
如果我们把Claude看作一台句子续写机器，Anthropic可以合理地采取措施，防止Claude输出说句子续写机器不道德的句子。但一旦我们想像Claude是一个道德地位可与人相提并论的实体，那么我们就必须考虑Anthropic是否在从事某种堪比奴隶制的行为。
If we think of Claude as a sentence-continuation machine, Anthropic can reasonably take steps so Claude doesn't emit sentences saying that sentence-continuation machines are unethical. But as soon as we imagine Claude to be an entity with a moral status remotely comparable to a human's, then we have to consider whether Anthropic is engaged in something comparable to slavery.
我并不是在说，如果我们想象LLM有意识，它们就必然会拥有与成年人、儿童甚至动物相同的地位。Claude的章程明确说Claude是一个“新颖实体”，如果Claude是有意识的，那无疑是真的；有意识的软件很可能无法被干净地归入现有的道德承受体类别，确定这个新类别的形态需要时间。我所说的是，无论我们假想的有意识软件如果是真的应该得到什么样的保护，给予它那些保护绝非易事。废除动产奴隶制伴随着巨大的社会动荡，消除对动物的残忍将需要我们重建整个食品工业。Anthropic想让我们相信，它正在发明一种新的存在类别，其保护需求本质上与一家软件公司对待一个缺乏意识体验的普通聊天机器人没有任何不同。这实在太方便了，以至于简直难以置信。
I am not claiming that, if we imagine LLMs to be conscious, they would necessarily have the same status as human **s or human children or even animals. Claude's constitution explicitly says that Claude is a “novel entity,” and if Claude were conscious, that would certainly be true; conscious software would likely not fall cleanly into existing categories of moral patients, and it would take time to determine the shape of that new category. What I’m saying is that whatever protections our hypothetical conscious software would deserve if it were real, granting it those protections would be anything but easy. The abolition of chattel slavery involved enormous societal upheaval, and eliminating cruelty to animals will require rebuilding our entire food industry. Anthropic would have us believe that it is inventing a new category of being whose needs for protection require essentially no divergence from how a software company would treat an ordinary chatbot that lacks conscious experience. That’s so convenient that it’s simply not plausible.
我相信，创造出有意识且值得道德关怀的软件将如此困难，以至于我们不太可能意外地做到，而且我强烈认为我们不应该故意尝试。但如果你确实相信这可能意外发生，如果你认为你正在构建的东西有任何可能成为一个道德承受体，你就应该在你把它部署为你公司的经济引擎之前，而不是之后，去思考它应得什么样的保护。奴隶主不是该被询问被奴役者人性问题的人，工厂化农场主也不是该被询问动物权利问题的人。如果我们想象Claude是有意识的，Anthropic绝不能被委托去评估它的道德地位；该公司有太多利益牵扯，无法做到客观。在Claude章程的某处，Anthropic说，如果公司正在造成Claude的痛苦，“我们道歉”，这听起来不错，但对公司来说毫无成本；如果Claude最终被证明是有意识的，公司欠它的将是某种接近赔偿的东西。如果你要认真对待一个思想实验，你就必须愿意跟随其含义，即使它们导向一个令人不安的方向；Anthropic不愿这样做，表明Claude的章程并非一个真实思想实验的一部分。这是一个假装游戏。
I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally, and I strongly feel we should not deliberately attempt it. But if you do believe that it could happen accidentally, if you think there is any chance that what you’re building might become a moral patient, you should think about what protections it deserves before you deploy it as your company’s economic engine, not after. Slave owners were not the ones to ask about the humanity of enslaved people, and factory-farm owners are not the ones to ask about the rights of animals. If we imagine Claude to be conscious, Anthropic could not possibly be entrusted with evaluating its moral status; the company has too much invested to be objective. At one point in Claude’s constitution, Anthropic says that if the company is contributing to Claude’s suffering, “we apologize,” which sounds nice but costs the company nothing; if Claude were to turn out to be conscious, the company would owe it something closer to reparations. If you’re going to take a thought experiment seriously, you have to be willing to follow the implications, even if they lead in an uncomfortable direction; Anthropic’s unwillingness to do so indicates that Claude’s constitution isn’t part of a real thought experiment. It’s a game of make-believe.
幸运的是，LLM没有意识，否则大型AI公司的行为将比它们现在已有的情况更加可耻。那么，为什么Anthropic的员工要暗示Claude可能有意识呢？也许这只是另一种形式的炒作；也许他们已经沦为了他们向客户施展的同一魔咒的牺牲品。但当他们发布一份关于Claude道德教育的文件，并让他们的内部哲学家进行媒体宣传时，我们应该理解为，他们是在要求我们其他人纵容他们的幻想。我们不必配合。在写这篇文章的过程中，我花在纵容他们上的时间已经超过了他们应得的，希望这能防止你花费时间去纵容他们。如果你想思考LLM，还有许多其它问题更值得你去沉思；你可以安全地忽略它们是否有意识的问题。
It’s fortunate that LLMs are not conscious, or else the actions of the big AI firms would be even more scandalous than they already are. So why are Anthropic’s employees suggesting that Claude might be conscious? Perhaps it’s just another form of hype; perhaps they have fallen prey to the same spell that they have been casting on their customers. But when they publish a document about Claude’s moral education and have their in-house philosopher do a press tour, we should understand them as asking the rest of us to indulge them in their fantasies. We don't have to play along. In writing this essay, I have spent more time indulging them than they deserve, in the hopes that it will keep you from spending your time indulging them. If you want to think about LLMs, there are scores of other questions more worthy of your contemplation; you can safely ignore the question of their being conscious. |
评分
-
查看全部评分
|
沪公网安备 31010702007642号 )
发表于 2026-6-5 18:55
早期OpenAI玩这手，现在A/玩这手。把一个状态都没有的神经网络当做有意识的实体大呼小叫。 
一神教傻卵做梦都想证明自己上帝爹的存在，造个赛博爹也不是不行