This post comes courtesy my brilliant student Michael Tarn in my TO433: Artificial Intelligence for Business class. One of the class assignment is to talk about a mass media depiction of AI from the point of view of trust in AI. This is a popular assignment as you might imagine.
Most of the times I get predictable submissions around I, Robot; 2001: A Space Odyssey and recently Ex Machina and a couple episodes of Black Mirror. Michael though decided to leverage his expertise in gaming and wrote about his experience playing the award winning game: NieR Automata. Michael wrote about his reflections on trust in AI in the context of playing as AI agents in the game. It comes down to the basic question – what does it mean to be human? Is humanity defined by who we are or how we behave? Can AI be more human than humans?
Below is lighted edited version of Michael’s thoughts.
NieR Automata
Disclaimer
We will be discussing NieR Automata, a game released in 2017. Please do keep in mind that this discussion will include massive spoilers, as in, the entire premise and plot twist revealed near the end. So proceed at your own risk.
Context
The story takes place in a post-apocalyptic world where humanity has been driven to near extinction as a result of a machine army invasion from another world. After the primary war, humans fled to take refuge on the moon as they were unable to protect and live on earth after being nearly wiped out during the invasion. In this game, you essentially play as 3 main characters: 2B, 9S, and A2.
These characters are AIs. Specifically, androids created by humanity. They were created during the war to serve, protect, and fight for the humans. However, post-war, they remained on earth to fight the machine invaders while the last remains of humanity were on the moon.
What defines human?
This is an essential driving factor throughout the entire story of NieR and the primarily focus of this write up – “what defines human?”
2B is a combat android whose main traits are calmness and composure. 9S is a scanner android whose main defining trait (often mistaken for malfunctioning) is displaying more emotion than other units. A2 is an attack android who has a very solitary and lonesome demeanor. What’s interesting is that this is how they were “made” to be by the humans, and often that becomes a really harsh reality.
Throughout a player’s time playing through the game, the conversations that often occur between these characters and other androids they meet are essentially indistinguishable between that of an android vs human, and it is like that for a reason. Their conversations down to inflection and tone mimic that of a human almost entirely. However, what’s harrowing at times is the stark contrast between the duo of 2B and 9S.
Due to their drastically different personalities, 2B and 9S share dialogue that really brings up the question on who is really human? 9S is a curious character that has immense development throughout the game but starts off always asking questions on why things are the way they are and what 2B thinks of certain things. 2B, being a rather stoic and collected (a rather more generalized idea of what a robot should be like), always answers with logic and reason, or dismisses 9S’s questions with comments about the unnecessary “thoughts” of 9S. Here’s where this gets interesting. To us, 9S seems like the most “human” character due to the curiosity, emotion, and wonder he displays. However, as a player, you often forget that he simply is not human. Every time, 2B responds that they are doing what they are doing – wiping out the machines – for humanity, and that everything they do is for humanity, and that is what they were made for. This begins to bother 9S as the story progresses as he starts to wonder more on the “why” they are doing what they do.
So, now we know a little bit about the personality/humanity of these characters. But what about the actual humans? Well, here are the big spoilers. As I mentioned, I’d highly recommend you don’t look at the next section if you have not had the chance to experience this game in person. I believe this game is honestly one of the most “must-play” experiences available.
Spoilers follow after the break
**SPOILERS START**
In the second half of the story we learn that humanity is actually dead. The last of humanity died away on the moon long ago. As a matter of fact, they left orders behind, knowing of their incoming demise, to ensure the androids continued to wipe out the machines on Earth. However, they left a major insurance policy, to ensure that any form of corruption to human history would not be done – they timed a virus to infect every android they created. This, in turned, caused the androids to go berserk and kill themselves, the machines, and each other.
Throughout this story, we are treated to the interactions of 9S and 2B. As their bond grows, we see a trust and relationship between the two that truly parallels that of two humans who have been through so much together. Even A2, who displays anger and resentment after the humans tried to destroy her (for being an older model), shows very real human emotions in her solitude and the way she lives. A massive part of the first half of the story was to make us feel like these characters were human, and to empathize with them. However, in this second part of the story, we begin to realize a very harrowing reality – they just aren’t.
As we begin to see these bonds torn apart by the virus, we also see another very eye-opening scene. In A2’s playthrough, she befriends a group of machines that have displayed peace and no hostility towards the androids (which is a big part of her development from a lone-wolf soldier), and as we keep playing, we keep realizing that these machines are not as brutal as they may seem.
During the events of the virus, the home of these peaceful machines catch fire and destroys their village. We then get a glimpse of these machines crying for help, some even committing suicide out of fear when attacked. These were the machines. The ones that we killed all throughout the first half of the story because the humans told us to. Here we were, friends with some that we learned could be peaceful, and watching them die, and now out of all times feeling dreadful witnessing the scene.
**SPOILER END**
So, the big takeaway from all this was that, in the end, we saw the androids and machines more so than the humans. The humans labeled these robots as monsters, but in the end, the actions of the humans seemed more like the monsters than those of the robots. So, what does this mean for what is means to be human?
Aftermath & Analysis
As you play the game, it gets so difficult to decide and determine what really defined being human. Therefore, it becomes hard to decide who to trust and who not to trust. So, you start to question yourself and your motives. NieR made me really think what defined being “human”, and if I could trust even that. We really have to question how we can trust AI first by how we view it. If we were to find ourselves in a scenario where we were in this world, would we trust the AI? Or the humans? At first glance, we think, “of course, we trust humans because we are humans too”. But, in the story, humanity betrayed us while the androids ended up seeming like the most “human” characters in the game universe.
Therefore, it’s really difficult to define how human-like characteristics can result in trust, and what it actually means to trust an AI. This game spawns more questions than answers, and although we are far from the future and the level of technology displayed in the game, these are questions we should be wondering ourselves now.
Creator of the game: Yoko Toro sums up the essential thought behind the game:
“I don’t think that humankind is worthy of trust when we can’t let go of war; draw borders between neighboring countries; seek to become richer than others; find joy in defeating others at sports; and choose someone of the opposite gender based on their appearance,”
Other Resources
Below are some articles I found that relate relatively well to the whole humanity aspect of the game.
Polygon’s article about NieR and “being human”
Nier: Automata, an action RPG about weaponized fetish dolls, is one of the saddest games ever made. It is a tragedy riddled with dead parents, dead gods and the death of self…. There is no god, the robots’ parents are dead, and those who survive will have to figure out how … 2B.
USGamer.net’s article about the hidden humanity
Nier: Automata, a game about a bunch of non-human things struggling with their distant grips with humanity, doesn’t play with clothes in precisely the same way as humans do, but they signal different messages all the same. Clothes, instead of signaling a personal mood as they do in today’s society, signal a broader wish: to show signs of being human at all. Clothes are a small part in the game’s broader reaches to understand humanity, as they’re one of the few ties the world’s remaining survivors even have to past societies.