Motion capture, or motion capture, brought a level of realism never seen before to games. 20 years ago, for example, it was necessary to improvise to try to give a more natural look to simple actions, such as hugging or synchronizing the lines with the characters’ lips. The need for a technology that allows bringing the real to the digital has always existed in games, but due to technical restrictions, we are only now starting to see more expressive results.
But such technical restrictions have not stopped, for example, game makers of the past from finding a way to capture more realistic actions. The classic running, jumping and climbing moves of the hero of Prince of Persia (1989), for example, were captured by the game’s creator, Jordan Mechner, who filmed his younger brother, David, running and jumping. Mechner’s idea was seen as a breakthrough in game animation in the early 1990s.
But what is motion capture?
Briefly, the technique uses cameras that track the actions of actors and objects that are using motion sensors, that is, those little balls (of varying sizes) that are glued to everything that needs to be captured. All data obtained by these cameras are sent to computers equipped with a program that creates a virtual skeleton, in which the actors’ movements are reproduced in real time.
From there, the director and the other responsible for the production, can have a preliminary view of the performance of the actor/actress in the game and make the necessary adjustments, if necessary, before the character goes to post-production and is animated about the virtual skeleton. This step can also be done in real time.
Of course, other processes, techniques and even improvisations take place during this work, which can last from weeks to years. So, nothing better than bringing experts on the subject to go deeper into the topic and talk about working with motion capture for games, from production to delivery of the final product.
For this Special, the Techblog talked to professionals who were part of games, such as: Resident Evil Village, Assassin’s Creed Valhalla, Ghost of Tsushima, The Dark Pictures Anthology – House of Ashes and Days Gone.
why use mocap in a game
Generally speaking, whether or not to use motion capture varies by project. The technique can be applied throughout the game or in some parts. To Masato Miyazaki, Director of Presentation in Resident Evil Village, the decision to use mocap (short form to refer to technique) in Capcom’s game stemmed from the design of the characters’ movements and the need to create specific animations.
“We use mocap quite often, but there are definitely situations where it doesn’t make sense to apply it. For example, for movements that are impossible to do with the human body or if we want to amplify or emphasize a movement, we work with the artists to manually create a specific type of animation.” (Masato Miyazaki – Resident Evil Village)
Emmanuel Roth, animator in Days Gone, says that there are different aspects and stages of motion capture. During the production of the game at Bend Studios, Roth claims that the mocap was also used a lot for research purposes, generating data that helped in the development of the game.
“We work with mocap at the beginning of production to get an early idea and try different moves quickly, because it’s an almost instantaneous process,” says Roth, who points out how motion capture can be useful in the prototyping and format testing phase. to the game.
François Harvey, Mocap Recording Manager at Alice Studio (an arm of Ubisoft Montréal), agrees with Emmanuel Roth about the agility of motion capture. Harvey, who worked for Assassin’s Creed Valhalla, says that “the mocap process is very fast compared to the ‘key frame animation’ [animação feita à mão, onde cada gesto precisa ser criado pelo animador],” explains Harvey.
While in some productions the mocap is used in specific parts of the development, in other situations motion capture is one of the main characters of the game. In more cinematic-looking titles, like until dawn, Detroit: Become Human, Death Stranding, The Last of Us – Part II and the franchise The Dark Pictures Anthology, the constant use of this technology is evident.
“We capture the moves even for our monsters!”
The statement is from Dan McDonald, Executive Producer at Supermassive Games (responsible for The Dark Pictures Anthology – House of Ashes), in an interview with the Techblog.
“A lot of the moves you see in creatures in House of Ashes was captured using a stuntman and mocap,” explains McDonald. “It gave us a great base of movement for the monsters and helped the human characters [durante as gravações] to have a clearer reference.”
The decision to use motion capture in a game must be well planned, not least because the technology is not cheap. From the necessary equipment, creating and/or purchasing prostheses to hiring professionals and, sometimes, renting recording studios, it is a lot of investment to be made. Time really is money in this case.
For the mocap process of Resident Evil Village, Masato Miyazaki says that both Capcom’s own studios, in Japan, and another one based in Los Angeles (California – USA) were used. “The goal of using a studio abroad was to capture content for cutscenes. RE Village uses total mocap, where voice and movement are recorded simultaneously”, explains the director. “Unlike recording at Capcom itself, hiring an outside motion capture studio requires a lot of planning.”
François Harvey (Assassin’s Creed Valhalla) also highlights that finding the best actors and stunt doubles is crucial during pre-production. “If the recording is to capture parkour moves, we’ll look to experts in the sport so we can have the best animation for those specific actions in the game.”
“Choosing the cast is half the battle in pre-production,” Harvey points out to Techblog. “The other important parts are establishing whether there is a need to build fixtures, structures, or anything that has a significant impact on performance and integration with the 3D layout.”
And talking about choosing the best actors and analyzing the need to build accessories, which in mocap can also be called “props” (or prosthetics), Billy Harper, animation director at Sucker Punch (responsible for Ghost of Tsushima) says that selecting actors and stunt people who feel comfortable shooting in tight outfits and full of props (motion sensors, cameras and drums) is also a challenge.
“Usually, we spend extra time in our casting process making sure actors are able to feel comfortable in the outfits they wear on the mocap. Not least because there’s nothing we can really do, at least for now,” explains Harper.
Foam, plastic and whatever you have at hand
The creation of mocap accessories and backdrops is impressive in itself. Anyone who has seen behind-the-scenes videos of these recordings was certainly curious about the whole process and even found it funny that heap of pieces of wood, aluminum and foam – all properly decorated with balls of various sizes.
Like François Harvey (Assassin’s Creed Valhalla) added, part of the production requires analyzing the need to create prostheses, buying them or even reusing something that has already been used in other projects.
Dan McDonald (House of Ashes) comments that during production, the team creates basic accessories that reflect the real object to be used in the game. “Usually this is a prop to help actors understand what is needed in the scene. For example, to give them a sense of distance if they are holding a weapon, or to allow them to react when hit by something.”
“It’s all a matter of seeing what we need”, complements Emmanuel Roth (Days Gone). “If we want to open a port, for example, we won’t bring a real port for the recording because it would harm the capture, since the camera would not be able to see through it”, he explains to Techblog. “Then we would use some pieces of wood, a tripod and something to make a doorknob.”
François Harvey also emphasizes that the prostheses used in the mocap must be the same size and shape as the game’s version. “Here in the studio, we work with over 500 props. These accessories physically exist and each one of them also has a digital version to allow the team to carry out the post-processing”, he details. “If our prostheses do not meet the needs of a recording, we seek to buy them at a specialized store. Ultimately, we hire professionals to create them.”
But what if something goes wrong…
Even with everything organized in advance, sometimes things don’t go as well as planned. This is normal in any type of work and professionals who deal with motion capture often need to solve recording problems without being able to stop production. As stated before, this is an expensive process and there are deadlines to be met.
But then, in 2020, the pandemic came and many studios, as well as several other companies, were forced to reinvent themselves to continue operating even with some restrictions. Those who had projects in progress, when the social isolation began, had to find a way.
“As you can imagine, COVID-19 had a big impact on production,” comments Masato Miyazaki (Resident Evil Village) to the Techblog. “The endgame cutscenes were shot after the pandemic started. When the pandemic came, we faced the unforeseen situation of not having filmed the game’s climax.”
Miyazaki adds that remote work was a first for the team and one of the biggest challenges of this new work environment was dealing with the language barrier between the US and Japanese studios. “Communicating technical issues and detailed acting instructions made it up much harder. To solve this problem, we built a dedicated recording environment, where we could exchange videos and recorded data in real time.”
The pandemic really took everyone by surprise, but even games that were filmed before were also subject to unforeseen events. Emmanuel Roth (Days Gone) says that at times the studio needed to reuse mocaps: “we took a recorded recording to the studio and the designer said ‘no, I don’t want it that way. I want it that way. So yes. You had to hunt down what had already been made, change some things and reuse the mocap”. Roth points out that some parts of the game may still need to be rewritten, which also impacts filming.
And that unforgettable moment?
Epic battles with hundreds of soldiers on all sides, climbing and jumping off cliffs, facing gigantic monsters or even escaping a zombie tsunami are as exciting during gameplay as shooting all these scenes. As in film productions, capturing motions with many elements involved is a huge challenge, but rewarding just the same (when everything works out).
Billy Harper (Ghost of Tsushima) recalls how complex it was, in the beginning, to work with horses – something the Sucker Punch studio had never done before. “We started with a little test. We did a specific scene, inside the premiere trailer, where we had a large number of samurai on horseback advancing through a fog. We just shot this scene and took a lot of notes and figured out what the problems were.”
From these observations, Harper explains that it was possible to ramp up production and decide what needed to be recorded for the full game. Including, the opening scene of Ghost of Tsushima, with the samurai advancing along the beach and facing the Mongol army for the first time, had to be rewritten a few times.
“Everything we filmed with the actors [na cena de abertura] it was actually mounted on horses built in the studio with them,” Harper tells Techblog. “That way, we had more control over the actors’ gaze on the action that was taking place. Then, using animation, we mixed real horses with those captured with prostheses.”
François Harvey (Assassin’s Creed Valhalla) says that the most challenging part of filming, of the whole game, was the introduction. “What you see is not just ten minutes of footage shot by a stabilized camera. There are also several small gameplays from when Eivor [ainda criança] crosses through a vinking hall to seal loyalty between two clans.”
Harvey explains that this scene took a few days of planning and preparation. This entire introduction is made up of little vignettes of people drinking, partying and interacting with each other. All of this was properly planned, spatially allocated and captured in mocap in order to be in balance with Eivor’s movements.
“We rehearsed for a whole day with the stunt team. We split the action into five sections that needed to blend seamlessly into the camera,” says Harvey. “We also had to capture the actors’ facial and audio performance to complete each character’s performance on screen. There were a lot of elements worked together to make the scene work. This was certainly a complex process to coordinate, but it contributed to an impressive introduction to the game.”
Masato Miyazaki (RE Village) points out that there were no significant technical difficulties during the game’s mocap, but dealing with Lady Dimitrescu’s incredible height was a real challenge. Miyazaki explains that the height difference between the actress and the character created a large margin of error.
“For example. Imagine a scene like in Lord of the Rings, where a little hobbit and a human are together. If done by normal means, the performance of both actors would not align correctly”, explains the director. “To solve this problem, we planned a scenario where the actors could imagine the correct height. We also had to adjust the character’s rigs to minimize, as much as possible, the error caused by the height difference.”
What is it like to act for motion capture
Actors working with motion capture for games need to deal with difficulties that maybe movie actors, for example, do not face so often, such as: having to imagine real scenarios, sometimes different stories for the same character, monsters and developing emotions even when surrounded by monochromatic walls and improvised objects decorated by polka dots.
Daisuke Tsuji is a Japanese-American actor and he played Jin Sakai, the protagonist of Ghost of Tsushima. He claims he knew from the beginning of the casting process that he was running for one of the main characters in the game. “I just didn’t know how important he was,” says Tsuji in an interview with Techblog.
The actor also comments that despite having voiced some games, this was his first experience acting with motion capture. “After I was cast, I was privileged to audition for some supporting roles as well. It was part of my job to do some scenes with these actors and exchange experiences.”
Of course talking about Ghost of Tsushima it also arouses the curiosity to know how the sword fight scenes were recorded. Daisuke said he didn’t participate in the gameplay mocap involving combat and that his work was more focused on cutscenes.
Billy Harper (animation director at Sucker Punch) adds that the studio used stunt doubles for these fight scenes. “We were lucky to work with a stuntman who knew samurai and Mongolian fighting techniques,” says Harper. “We also work with two Japanese teachers who have a school in Tokyo that specializes in the type of samurai combat at the time the game is set.”
About acting wearing tight outfits, full of polka dots and with a camera attached to a helmet, Daisuke says that all these body adornments are not the best of experiences: “it’s never comfortable. It wasn’t comfortable the first day and I don’t think it was comfortable the last day.”
At least for the horses it looks like it was a little more comfortable, Daisuke:
Fun fact: Our horses were captured at Sony. The volume was enlarged just for this shoot. For safety, the layer of plywood and pretty much the ENTIRE current world’s supply of rubber matts were ordered for the floor. It arrived just days before shooting. #TheLastofUsPartII #horse pic.twitter.com/bC7M6rMkiT
— Jeremy Yates (@Jeremy_Yates) July 14, 2020
“The costume is very tight, there are a lot of things hanging. So, if there’s some action involved, like someone going behind their back or something that might move some of the marbles out of place, it’s likely they won’t be able to use the scene. And then there are batteries. So, depending on the movement, if I’m going to fall on my back, for example, they need to move the batteries forward. You have little balls stuck all over the place and it could hurt if you land on them”. (Daisuke Tsuji – Ghost of Tsushima)
Even with all the difficulties during recording, it is common for the actor to learn something from the character he played. At the end of the game, the question that remains is: does the Jin Sakai of real life consider himself more of a samurai or a ghost?
“Wow… Well, if we go for the game, I would say the ghost. Not that I’m violent or use poison, but I think the story is a lot about tradition versus moving on and adapting. And I respect tradition, but at the same time it is necessary to change to evolve, sometimes. So yes, I would say I’m more of the ghost,” declares Tsuji.
The future of motion capture
It is undeniable that all the work involved in motion capture has substantially scaled the technical quality and realism of game animations. But like all technology, it needs to be constantly improving so it doesn’t stagnate and become obsolete.
In that regard, Masato Miyazaki (RE Village) comments that since the introduction of motion capture in the world, its accuracy and efficiency have dramatically improved. However, he says that from the perspective of the data obtained from the captures, nothing very significant has changed over the years.
“In that sense, I think we can say that we’ve reached the peak of what motion capture is capable of,” says Miyazaki. “On the other hand, there are new technologies that don’t have cameras, which can even be taken outdoors, and new systems that can extract motion from videos.”
“I believe that, in the future, mocap will not be something special, but a tool for daily use that can be combined with other hardware, software and ideas to create valuable content”, concludes the Director of Presentation.
Dan McDonald (House of Ashes) agrees with Miyazaki’s view and adds: “there are constantly new methods and processes being developed and used. (…) We are looking for more specialization, getting better trained people for this job and this is helping us to reach higher levels of quality.”
François Harvey (AC Valhalla) emphasizes that motion capture is not just a tool. “This is a complete way to bring the characters to life. (…) Not only does it provide a more natural movement, but, more importantly, it gives us more vivid characters to inhabit the worlds we create.”
Whether motion capture will remain one of the stars of ultra-realistic game productions remains to be seen in the future. What is already notorious is the industry’s movement to adapt or create new techniques, develop more effective methods of analyzing recorded data and, above all, study how to create more fluidity and versatility during filming. If a new technology emerges, in this regard, we gamers are eager to see and test it.
Extra: did you know?
- The first time the Sucker Punch studio used motion capture was in inFamous 2.
- During recordings of Days Gone, the team needed to create a motorcycle made of wood for the filming. They even tried to use a real one, but it was too heavy.
- Daisuke Tsuji took a real (but blunt) katana to his audition for the role of Jin Sakai.
- the hordes, in Days Gone, were made in part using mocap. The actors were recorded bumping and tripping over each other to add more realism to the zombie sea.
- the directors of House of Ashes they used to move like the monsters in the game and make strange noises to help the actors react correctly.
- Sucker Punch and Naughty Dog exchanged references when doing horse motion capture for both games, Ghost of Tsushima and The Last of Us.