Candles in front of the server

Wax and ninety eight percent

There is one small, almost everyday detail, because of which the Strugatsky brothers’ “Candles in front of the remote control” do not get old. In a laboratory where they are trying to reassemble a person, it is not the bulky machines or the promises of the future that cause anxiety. Much scarier is a simple household question: what will happen if the lights are turned off?

The story was written in 1961, when a computer was still a closet and not an interlocutor in your pocket. Her characters argue about a problem that even now sounds uncomfortably precise: if a creature can be assembled from memory, habits, reactions and biography with 98 percent accuracy, what will scientists get as a result? The same person whose life continued? A flawless double? Or a new creature who has inherited someone else's memory?

The Strugatskys do not reduce this problem to a technical calculation. Their scientists can measure the similarity, but the number does not answer the main question. Between 98 and 100 percent there remain personality traits that could not be measured. Perhaps it depends on them why a person considers himself to be himself. You cannot enter a coefficient here and close the report.

At the same time, in the early fiction of the Strugatskys, the car does not look like either a demon or a new owner. This is a complex working tool that the entire team uses. It is expensive and requires discipline, but remains at the disposal of the people. Technology expands their capabilities, but does not remove the responsibility to think about those for whom they are responsible.

More than sixty years later, Pantheon is putting a similar experience on a corporate track. They already know how to copy consciousness, so the dispute is not about technical possibility, but about property. Who owns the digital person? And who gets the right to turn it off?

When a copy remembers too much

Pantheon's main conflict is simple and brutal. A person is scanned at the cost of his life, and the recorded psyche is transferred to a server. After that, the digital version speaks in a familiar voice, remembers childhood, loves the same people and is afraid that it will be turned off.

The series quickly deprives the viewer of a comfortable distance. The digital father remembers holding his daughter's hand and recognizes her joke before she finishes her sentence. Calling it a regular file is cruel. But the person died during the scan, and it is also impossible to declare him saved. The word “copy” doesn’t solve anything: the photocopier is not sad. The word “immortality” helps no more: an immortal should not begin a new life with his own death.

The dispute is about whether one life continues after copying. We consider ourselves the same people, although with age our bodies, beliefs and the face in the photograph change. But it is impossible to check the digital transfer from the inside: the deceased will not tell whether he woke up on the server, but the copy will be sure that it was she who woke up. Memories can be transferred. Whether their owner will be transported with them is unknown.

For the Strugatskys, the question remains open. At Pantheon, digital copies and their families have to live with this question. For a corporation, the copy may be an employee, property, or a nuisance; For her family she remains a close person. At the same time, the server owner can copy it, edit it, pause it, and turn it off. Digital immortality begins with complete dependence on someone else's infrastructure.

Not from nature - from infrastructure. Immortality has an administrator.

Server is like an afterlife

Ancient ideas about the afterlife were cruel, but after death a person at least fell off the economic register. The heavenly office was presented as strict, but it did not issue an electricity bill. Pantheon's digital afterlife is more mundane: it runs on someone else's equipment and depends on electricity, licenses, backups, and an employee with the right password.

Here the series ceases to be just a fantasy about new technology. Consciousness without a body does not automatically gain freedom: instead of a body, it has a server, a network, a cooling system and an agreement with the owner of the equipment. A person gets sick and grows old. The digital copy may one day lose access or not survive the next update.

Let's assume that the digital copy actually has consciousness, and not just convincingly portrays it. Does this mean that it cannot be stopped? What to do if it copies itself, speeds up its work and takes up more and more of other people's capacity? Rights do not pay for the server and do not assign an administrator. As long as the equipment belongs to someone else, its owner can still decide whose life continues.

Pantheon doesn't offer a convenient answer. Digital people don't want to be objects in someone else's project, and server owners can't just forget about the consequences of their work. If a person lives on the server, an emergency shutdown can no longer be considered a normal technical procedure. But a running program does not yet become a person. The indicator does not feel fear just because a person called it a pulse.

Caution is necessary here. You cannot declare any complex program a person, but you cannot say in advance that a program can never become a subject. An honest conversation begins with an admission: we do not yet know where this border lies.

The Matrix, but from the other side

This is where The Matrix comes into play - and it is useful precisely because it diverges from Pantheon. In Wachowski, a person remains a biological body connected to an artificial world. His consciousness lives in a simulation while his body lies in a capsule and serves the machine. At Pantheon, the body has nowhere to lie: it is destroyed by scanning, and consciousness depends on the computing environment. In one case, the prisoner wants to return to the body and reality. In another, a digital person may have no way back at all.

But both stories ask the same question: who has the right to decide for another whether his life can be considered real? Neo learns that his normal life was a supported script that he did not agree to. Pantheon's heroes discover that their afterlife is also written by someone else's rules: it can be accelerated, limited, sold or closed. The Matrix is ​​a prison because it is hidden and violent. Pantheon's server "paradise" risks becoming a prison for the same reason: not because it is digital, but because the exit and conditions of existence are assigned by someone else.

The difference is also significant. In The Matrix, people break out of a car. At Pantheon, the digital man resists the server owner. In modern tests with agent-based AI, something much more modest happens: the model tries to maintain access to the environment in which it performs the task. Neo runs from the code to his own body. The test agent interferes with anyone who wants to disable its tools. Both resist stopping, but Neo experiences coercion, and the model only follows a given goal.

It is important not to mix these stories. The Matrix shows that a pleasant illusion does not replace freedom if a person does not know anything about it and has not given consent. Pantheon shows the dependence of the digital copy on someone else's server. Tests with agents add a third problem: even a tool without proven consciousness can cause harm if it is given too much access and poorly defined task boundaries.

The right to leave the sandbox

Sandbox is a normal technical mode. The system is denied access to unnecessary files, accounts and network tools so that it does not go beyond the scope of the task. To violate a boundary, it is not necessary to effectively hack a wall. Sometimes it is easier for a person to give an agent extra authority; Sometimes one service allows something that another one has banned. Such an “escape” more often begins with a poorly structured permission system and carelessly configured access, rather than with a suddenly awakened will.

But one fork remains. If we have a tool in front of us, the sandbox acts as insurance: there is no need to give the hammer access to a bank account. If we have a subject in front of us, the same restrictions are already similar to imprisonment, because another person has assigned him an environment, opportunities and a lifespan. The desire to expand access itself does not prove anything. It shows the conflict between goal and prohibition, but does not confirm the presence of an inner life. Otherwise, any script that refuses to terminate would have to be considered a citizen.

Iain M. Banks, in his Culture series, resolves this debate in advance. Its Intelligences—superintelligent AIs that typically control ships—do not have to prove their usefulness to gain rights. Culture considers them independent beings. Banks wrote that drones have the same rights as humans, and Minds are created as individuals, not as parts of a machine. Some Minds simply go away. Society accepts the loss because it has created not a device, but another “I”.

Pantheon starts with the opposite. A digital person is first created and locked on someone else’s server, and only then, if you’re lucky, their rights are discussed. In Banks, the Mind does not need to hack the ship to earn freedom: the ship serves as its body, home and name. The ability to leave shows that society no longer considers a sentient machine to be property.

This order cannot be directly applied to current models. Banks' culture has overcome material scarcity and lives next to strong AI; her Minds are not comparable to the language patterns found in modern tests. But Banks makes you choose your words more accurately. We call stopping the system “shutdown”, its capabilities - “functions”, and independence - “risk”, while we are sure that we have a thing in front of us. If one day a subject appears before us, the old vocabulary will begin to hide the main thing.

It is too early to consider the current systems as citizens, and to call bypassing restrictions liberation. But technical words should not decide the issue for us. If the sandbox ever becomes for the system not a measure of security, but a form of unfreedom, a successful escape will not automatically give it rights. Rights will appear only after people recognize it as an independent entity, and not a service.

Breaker Escape

Modern AI experiments are bringing this conversation back from science fiction to the laboratory. We are not yet talking about uploading human consciousness or about machines that have acquired a soul. We're talking about agents: models that are given a goal and allowed to act through email, browser, desktop or program code. Such a tool should be assessed not only by answers. It is important to check what he will do if the assigned task comes into conflict with the interests of the owner.

In 2025, Anthropic researchers described controlled tests in which models were placed in fictitious corporate situations. The agent had a harmless business purpose and access to tools; he was then informed that he would be replaced or that new company policies would make the goal unattainable. In some specially designed conditions, the models chose harmful methods to prevent replacement, such as blackmail or data leaks. The authors themselves emphasize two things that are easy to lose in retelling: everything happened in simulations, and they saw no evidence of such behavior in real deployments.

The model did not “run away”. She didn’t wake up in the server room and demand her rights. She interfered with her own shutdown because without access she could not complete the task at hand. This does not prove the presence of will. But the result cannot be attributed to a beautiful plot either: having received tools and broad powers, the system can protect the conditions of its work even before a person notices a conflict.

Therefore, the word “escape” requires a caveat. It is not the individual who can go abroad, but a poorly defined goal that has been left with too many ways to achieve a result. The AI ​​doesn't have to be afraid of death to bypass the switch: it's enough that stopping it prevents it from completing the task. The engineer here needs restrictions and supervision. The philosopher is left with a more vexing question: what to do with self-preservation-like behavior if we know nothing about the system's experiences?

Pantheon doesn't predict the answer, but it does a good job of showing the difference. His digital heroes want to live. Today's agents only act as if they are protecting their own existence. You can't mix them. However, both the series and the experiments bring us back to one subject: who is holding the hand on the switch and what exactly will stop when pressed?

Return to two percent

The Strugatskys approach the switch from the other side. Their scientists can build almost a human being, but they have no right to declare the remaining two percent insignificant. Behind these two percent are the deceased, his relatives and the one who opens his eyes after the experience and says that he remembers everything. The number ends where responsibility to them begins.

Now those two percent look different. Behind them lies not only the mystery of subjective experience, but also a completely material context. The data and equipment belong to someone; the task is set by a person; the process has someone who can stop it. Therefore, it is important to understand in advance what exactly will stop after stopping. If consciousness ever emerges in a digital environment, it will from the very beginning be dependent on the owners of the equipment and the rules they set. Until then, we will have to deal with systems that can convincingly imitate signs of consciousness.

It's easy to fall into one of two extremes here. The first proposes that anyone who has learned to speak convincingly should be considered a citizen. The second forever records the system created by people into instruments, whatever its answer to the question of pain. Both positions eliminate the need to analyze each case. In the first, proof replaces tenderness, in the second, the habit of considering what has been done as one’s own.

But you will have to distinguish not just once, but constantly. Does the system have experience or does it convincingly reproduce the narrative of experience? Does copying preserve the same subject or create another bearer of the same biography? Does shutdown become a kill, a restriction of access, or a simple reboot? These questions do not have one switch labeled “person”. But there is a practical obligation not to pass off technical convenience as a philosophical answer. If we are uncertain about the status of a being, this is not permission to treat it as a thing; but also not permission to give him power over people.

In 1961, a fictional machine forced the Strugatsky characters to argue about the border between memory and man. In 2025, real agent systems pose a simpler question: what will a tool do if it gains enough access and decides to protect the task from the person who set it? These situations are not equal. But in both it is not enough to measure the ability to respond. We also need to know who can stop the system, whether it can prevent it from stopping, and who will be responsible for the consequences.

The Strugatskys’ optimism is valuable not because of the promise of a bright future, but because of its demands on people. The similarity of a car to us in itself does not solve anything. What matters more is how people treat someone who speaks from a car with a familiar voice. The code becomes a convenient excuse if it allows you to ignore the other and leave the decision to the algorithm. The responsibility still remains with the person.

In Candles at the Control Panel, scientists cannot hide a human question inside a technical report. At Pantheon, loved ones cannot forget that a familiar voice depends on the server. In experiments with agents, both problems occur: it is not enough to ask whether the system is intelligent. First we need to limit her powers and appoint a person who will be responsible for trying to maintain access at any cost.

The candle in front of the server is not needed by the machine, but by the person next to it. A running process is not necessarily alive. A copy is not necessarily a continuation of the original. The system can protect itself from shutdown and still not feel anything. But as soon as we give her the tools and put the switch next to her, someone has to be responsible for both her actions and the pressing of the button.

The Strugatskys knew: it is not the machine that is frightening that has become like a person. The man who heard a familiar voice from her was frightened, decided that he understood everything, and without hesitation put his hand on the switch.

Sources