Early analysis of the 3rd test, Part III

Discovering affordances on your own might strengthen the game experience
In line with the results of the first user test, there were some claims made during the third user test regarding the joy of discovery. Two of the persons taking part in the third test didn't mind that it was hard to discover which objects that could be smashed to pieces, with one claiming that the discovery of this affordance is rewarding enough to compensate for the lack of affordance amplification. Another participant had some complaints about the fact that the game didn't make him aware of the possibility to break apart objects, but suggested forcing the player to discover the interaction by "locking him into a room" with a breakable object, rather than using a UI affordance amplifier. Note: the remaining two participants didn't find the affordance hard to discover.

Regarding the breakable boxes in Dead Space, another test participant claimed the boxes to be too attention grabbing. He wanted a more subtle affordance amplification, as he - as well as the other participant - found the rewarding feeling of discovery strengthening for the game experience.

Once again, this suggests that sense of freedom is important for engrossing oneself into the game experience, and that dumbing down the gameplay by being to explicit might take players out of the game experience. Personally, I think that this might be especially relevant considering optional affordances.

Early analysis of the 3rd test, Part II

Continuing on last weeks post, here's two additional conclusions from the test.

Attitude towards hand-holding depends on player type
As in the second user test, two groupings among the participants can be noticed. This time, the participants could be divided into those who prefer the Resident Evil 5 approach of amplifying affordances with explicit button prompts throughout the whole game, and those who thought that the button prompt approach might be ok in the introductory part of the game but not justified later on. Perhaps not very surprisingly, the person that spent most time on games in the former group (two persons) spent about 2h/week playing games while the latter group (three persons) consisted of players that could be described as hardcore gamers.

Accordingly, hand-holding seems to be preferred by players with a causual perspective, while experienced players might accept a tutorial approach of button prompts in the beginning of a game but might be taken out of the game experience if these button prompts are persistently used.

Design mistake: Introducing interactive objects while player interaction is blocked
One of the first things the test participants encountered in the Dead Space session was a box with a twinkling green light. This box immediately caught the attention of every participant in the test - the moment they saw it, they tried to interact by pressing buttons or pushing it around, without any response. These boxes are in fact destroyable and are encountered throughout the whole game, however during the first minutes of the game the player's ability to interact with the environment is limited and this interaction is blocked.

While the appearance of these breakable boxes - containing items like ammo and health packs - easily catches the player's attention and hence could be seen as a design success (if this is the aim), introducing them when the player is not able to interact with them is a risky design choice. When the test participants were finally able to use the full range of actions (shooting or foot stomping to break boxes), their interest for the twinkling boxes had been lost; two out of five persons did not break any box at all throughout the session, and the only participant discovering the affordance instantly when permitted by the game was familiar with Dead Space.

While these results might not be as homogenous as the results from the Resident Evil 5 session, it seems quite unnecessary to introduce interactive environmental elements during moments when player interaction is blocked. The undesired effect of this might be that the player draws the conclusion that objects are not interactive despite the fact that they're clearly communicating the affordance, and hence from that point make no further attempt of interaction.

Early analysis of the 3rd test, Part I

(test description)

Non-apparent affordances are hard to discover without proper amplification

In the introductory Resident Evil 5 chapter that was included in the test, there was several possible interactions with objects - affordances - that was difficult for the participants to discover. In an action scene that was a part of the session, the player is able to block the doors of a shelter by pushing shelves in front of them. This affordance (that a shelf affords pushing) is not amplified at all. None of the test participants discovered this interaction.

Earlier in the same session, the player is introduced to breakable objects that contain usable or valuable items. This affordance (object is breakable) is amplified in three ways: 1) a "Investigate" button prompt when the player stands near the object (a wood box) and looks in its direction, followed by a textual message ("This box looks conveniently placed"); 2) a tutorial message popping up about one seccond after the box is "investigated" (press LB + RT to swing knife, hence break the box); 3) the player characters partner Sheva demonstrating the affordance by breaking objects randomly. The first two amplifiers is used only the first time a breakable item (a wood box) is encountered. The third amplifier occurs at random moments during gameplay, if the player happens to look in his partner's direction he see's the partner breaking an object.

Far from all test participants dicovered that certain objects were breakable. Only one test participant breaked the first box encountered (affordance amplified by "Investigate" message + tutorial message), and one other participant discovered the affordance later on during the game session when encountering a group of barrels (no amplification at all).

Apart from the "smash object to pieces" affordance, participants had problems discovering that a window afforded jumping through it when trapped inside a building. The affordance "jump through window" was only amplified when standing close to it, by a "Jump through window" button prompt. The problem was that the players rarely positioned themselves close enough to the window for the affordance amplifier to occur. This resulted in a state of confusion, when the players did not know how to escape the building. Only one of the participants discovered the affordance instantly.

It should be noted that if enough time passes by, the player's partner Sheva positions herself next to window and communicates by dialogue that the window is a way out of the building (an affordance amplifier as such). However this occurs after a significant period of time and the participants tended to find the way out by trial-and-error before it happened.

This somewhat falls in line with our hypothesis. The "break object" affordance was not discovered by the majority of the test participants, resulting in poorer interactivity with the game geometry. The "Push shelf" affordance was not discovered at all, also resulting in the players missing out on possible interaction with the environment. The "jump through window" affordance was only discovered instantly by one test participants, for the majority who didn't this resulted in a state of confusion until the affordance was found by trial-and-error. Note that decreased immersion/engrossment isn't necessarily a direct result of the player missing out on optional affordances such as breaking boxes or pushing shelves, but might be an indirect result as affordances are hopefully put into the game to flavor and enrich the game experience. For affordances required to make progress through the game (jump through window) it should be important not to frustrate the player and force him/her to rely on trial-and-error, which might take the player out of the gameplay experience.

3rd user test: Description

So we're officially done with user testing, after some struggling with the third and final user test.

The third user test consisted of two gameplay sessions, in which the persons taking part in the test experienced the first ~20 minutes of Resident Evil 5 and Dead Space respectively. During these sessions, observations were made of what the participants interacted with in the environment. Both sessions were followed up with questions regarding the way the game eases player interaction with objects and the environment, if this is positive or negative for the game experience, and if the diegesis of the UI solutions affects the game experience. As in the first test, the participants were asked to rank the two games according to 1) usability (how much the game eased interacting with the environment), and 2) immersion/engrossment (to what extent this strengthened the game experience).

The aim of the test was to prove our hypothesis regarding spatial interaction; the amount of affordances deduced from an 3D game space is aways greater than the amount of affordances actually allowed by the rule framework (in other words, code) - only certain doors can be opened in Mirror's Edge yet some of our participants in the first user test tried to open doors that could not be opened - hence UI designers must amplify affordances allowed and supress non-existing but deducable affordances. This to enable as rich spatial interaction as possible, and avoid having the player try to do things not allowed by the rule framework which might block the players precense within the game world.

As in previous tests, we can notice a number of interesting trends in the results, which will be posted shortly.

Early analysis of the second user test

In this test, the “feeling of being damaged” part of the player-avatar sensory link was tested, by letting the test participants experience the way FPS games communicate damage to the players. The participants once again experienced four gameplay sessions, but this time four different games; Battlefield: Bad Company, Call of Duty 4: Modern Warfare, Halo 3, and Killzone 2. After each session, questions were asked regarding how well the game communicates the feeling of being damaged, and how this affected the game experience. For the games containing an overlay health bar or digits indicating level of health (Halo 3, Battlefield: Bad Company) the test participants were asked about how the diegesis of the UI element affected the attitude to this information. As in the first test, the participants was asked to rank the four games solutions according to 1) how well the game communicates the feeling of being damaged, and 2) to what extent this strengthened the game experience.

The results from this test is slightly more scattered than from the first one, but nevertheless a number of trends can be identified here as well:

Communicating the feeling of being damaged in an accurate way enhances the game experience.
The test participants were asked to rank the games included in the test according to 1) how well the game communicates the feeling of being damaged, and 2) to what extent this strengthened the game experience. While players ranked the games quite differently (more on this below), they ranked the games in the same order for both question 1 and 2. This indicates that communicating the feeling of being damaged in a accurate way strengthens the game experience.

The player attitude towards different strategies for communicating the feeling of being damaged is affected by the player's earlier experience with games.
Some of the test participants opinions seemed to be heavily influenced by the games they have played before. Two out of the five test participants had played earlier games in the Halo series extensively, but were not very familiar with the other games tested. Both ranked Halo 3's way of communicating the feeling of being damaged high, while other test participants placed Halo 3 in the last place when asked to rank the games.

Non-diegetic and meta-perceptual UI elements (image filtering and full-screen image overlay) does not break immersion/engrossment, despite being outside the diegetic part of the UI design space.
As in the first test, no participant questioned about the non-diegetic nature of the health digits in Bad Company or the health bar in Halo 3 (which could be argued to be diegetic if one is familiar with the Halo fiction) mentioned this as a immersion/engrossment breaker. No participants was taken out of the game experience because of the overlay nature of perceptual emulation either. Once again, genre conventions was mentioned as a reason for accepting this kind of information. Instead, players mentioned loss of control or unclear communication of the higher end of the health spectrum to be breaking the game experience.

Meta-perception can strengthen the feeling of being damaged, hence enhancing the game experience.
A majority of the test participants (the three persons that didn't favor Halo 3's way of communicating the feeling of being damaged) seemed to think that meta-perception (image filtering and full-screen image overlay) affected their game experience in a positive way. Some however had issues with meta-perception that was used to the extent that it causes a loss of control over the situation (for example image blur), while others seemed to find it irritating that it only made them notice the low end of the health range (critical health level) and that it was hard to notice it when it was used to indicate being hit by fire. Generally, the participants seemed to think that the Killzone 2 approach was a good way to go, with the overlay blood stains symbolizing both being hit by fire and eventually critical health level when the perifery of the screen is covered with blood, without causing loss of control with blur or fading out the screen.

First user test, observations

Apart from feedback in the form of answers from the questions asked to the participants in our first user test, a few interesting observations could be done during the Mirror's Edge play sessions (it should be pointed out that these observations was not a part of the planned test).

1. In situations when sense of direction was lost, players tried to interact with doors not colored red by Runner Vision, resulting in a slightly confusing situation.

2. With Runner Vision turned off, participants interacted with the environments using a poorer range of actions than with Runner Vision turned on. None of the five participants made use of the springboard technique to launch themselves into higher jumps, allthough the opportunity was given on at least two places. The persons taking part in the test also had trouble discovering the "swing on crossbar" affordance, despite having experienced it during the games tutorial just 5-10 minutes earlier.

3. Some players experienced a slight inconsistency with Runner Vision vs other geometry colored red. In one case the red communication tower in the introductory level became the center of attention of one participant, when he was supposed to run for the helicopter to finish the level; the helicopter was judged to be hostile as the player could not see that its splints were colored red, and he instead searched for a way to reach the clearly red tower. Another participant initially thought that he was supposed to interact with(wallrun) red walls. The same player thought a orange/red colored flag was confusing.

These three observations justifies, if not proves, our spatial interaction hypothesis: with an increased level of player interaction within the game geometry, it becomes increasingly important to separate affordances allowed by the geometry/rules from those that are deducable from the geometry but not actually existing within the game world.. which can be done using signifiers and spatial UI elements.

Early analysis of the first user test

We have finished our first user test regarding how UI designers - without breaking player immersion/engrossment - can make up for the loss of sense of direction that occurs when connecting the player to an in-game avatar via visuals, audio and haptics. As Erik has described earlier, the test participants played through three Mirror's Edge sessions with different types of navigational aid (1. Runner Vision, 2. Time trial checkpoint beacons, 3. No aid at all) and one Bioshock session experiencing the navigational aid of Bioshock (an overlay compasse arrow giving very detailed guidement). After each session, the participants answered questions regarding ease of navigation, immersion and diegesis, as well as ranking the four different UI solutions according to 1) usability (to what extent the navigational aid made it easier to navigate) and 2) immersion/engrossment (to what extent the navigational aid strengthened the game experience).

Looking at the results, a number of interesting trends can be noticed.

Spatial and non-diegetic UI elements does not necessarily break the immersion
When questioned about the fact that the navigational aid was something not visible to the player's in-game avatar, none of the participants (five persons participated) stated that this was an issue with Runner Vision and the Bioshock arrow, and only one person felt that the ME checkpoint beacons was out of place. The art style of Mirror's Edge and general UI/HUD conventions within the FPS genre was mentioned as factors preventing the UI elements from breaking the experience. Issues connected to the navigational guidance was instead concerning inconsistency, or the information being either to ambiguous or to unambiguous.

Spatial representations can strengthen the game experience
When questioned about if Runner Vision strengthened the game experience, three out of five participants answered yes and only one person was taken out of the experience, irritated when he misstook other red details in the game geometry for runner vision. When answering the same question for the Bioshock arrow, no player thought that these approaches had a strengthening effect. The same could be said for the ME checkpoint beacons, suggesting that spatial UI navigational elements does not automatically increase immersion. It does however indicate that if done right (for example in the style of Runner Vision), spatial representations can strengthen the game experience.

Being dumbed down spoils the game experience
When told to rank the four different approaches to navigational guidance according to 1) usability and 2) immersion/engrossment, every single test participant selected not having a navigational aid at all as the approach that strengthened their game experience the most, stating that it gave a larger sense of freedom and gave a more rewarding experience when finding the right way through the environments on their own. The Bioshock arrow and the ME checkpoint beacons, both generally ranked high when considering the extent in which the navigational aid made navigation easier, was the approaches least preffered from a immersion/engrossment point of view. As previously mentioned, the participants had issues with the navigational guidement being either too ambiguous or too unambiguous, which suggests that players do not want to be told explicitly where to go and what to do, but they do need to be hinted.

This suggests that considering the informational content of a UI element might be just as important as the way it is presented. Sense of freedom might be just as important as sense of direction. There is a fine line between providing too ambiguous and too unambiguous information, which should be considered by UI designers.

Sense of direction connected to spatial UI representations
The hypothesis we set out to answer with this user study was the following: if player navigation within the game geometry is important for a game, it is also important to maintain a high sense of direction for the players, which can be achieved with spatial information and/or signifiers. We didn't explicitly ask the test participants any question about their "sense of direction" within the game per se, but we did ask them how easy it was for them to navigate through the environments using the different approaches to navigational aid. While the navigational UI elements eased player navigation, the players seemed to prefer exploring the game environments without navigational aid, stating that it gave a larger sense of freedom and gave a more rewarding experience when finding the right way through the environments on their own. Hence, we might have to rethink this hypothesis, alternatively the hypothesis should be tested using a different test strategy.

Of course, we have to take into account that the players only experienced a small part of the games tested, and for a quite small period of time. The results might be different if the test participants had experienced several hours of gameplay rather than the first ~20-30 minutes of the game. However, one might also argue that these initial 20-30 minutes is the time a game has to convince the player to explore the game further and keep on playing, fortifying the test results.

Hypotheses testing (Week Summary 20/4-24/4)

This week we have been creating and conducting game tests in order to validate some of our hypothesis concerning fps UI solutions.

Our first test concerns (short-term) navigation. Our hypotesis is that a spatial navigational aid like Runner's Vision in ME are perferable to solutions like the one in BioShock (and perhaps that the beacons in ME's time trial mode are the most usefull - but might be considered to intrusive outside the scope of a highly competetive game mode.)

In this test players get to play the first level in Mirror's edge in order to experience navigating with the help of Runner's Vision. Next they get to try a segment of the next level, with RV turned off. After that they get to try playing a time-trial level, and navigate using the (much less subtle) beacons.

Finally they get to play the first 15-20 minutes of BioShock in order to experience navigating both with and without the games "Golden Compass".

After each short sessions the player will be interviewed about their experience with each navigational aid. How well they conveyed where to go, their level of intrusiveness, their level of helpfullness/usefullness and so on.

The second test is about "Meta Perception" - This time the players get to play brief, action packed episodes of Call of Duty 4, Battlefield BC, Killzone 2 and Halo 3 in short succession. The goal is to familiarize players with four quite different solutions of how to indicate that the player is taking damage (and the direction from where it originates). Players will then get to rank the games and elaborate around how well the solutions conveyed a sens of damage/injury/danger.

Our hypothesis in this case is; the more visceral the experience of being hurt the more immersive the game experience.

Prototyping Approaches

Since we start to feel content with our gathered knowledge, it's time to start thinking about how we can prototype and evaluate our ideas and hypotheses regarding engrossing/immersive UI solutions.

Since neither of us have massive coding skills, one reasonable option is to look for games where UI elements can be disabled (for example runner vision in ME, the guiding arrow in Bioshock, hardcore mode in CoD, etc), and do user tests with the UI elements of interest turned on and off, and observe any differences in player engrossment. The obvious advantage of this is that we don't have to spend any effort prototyping our own UI solutions, however this will also limit us to only test others ideas and also not be completely free to choose the ideas that we are interested in testing (since enabling/disabling UI elements is a rare option in games).

Another option that we have carried with us since the start of the project is to prototype ideas by superimposing information on top of pre-recorded gameplay footage. Here, at least in my opinion, the advantages does not make up for the disadvantages; the prototypes will not be interactive which will most certainly affect users opinions, and we will be limited to overlay information (the only reasonable approach to prototyping spatial info this way is to pause the movie and put the element into the game space).

I found a potentially interesting tool to develop our own prototypes; FPS Creator. As the name implies, this is an application for developing FPS games without writing any code at all (in extreme cases at least). The HUD is fully customizable, overlay filter effects is used to indicate health and hopefully this is fully customizable as well it. If the application allows some fairly straight-forward scripting I think this might be really suitable for us when testing ideas.