An interactive demo that explores how mixed reality could benefit hybrid (in-person + remote) learning in design studio environments. The persistent, holographic map of the design studio uses colored blinking lights to give users passive awareness of what other designers in the studio space are working on while minimizing distractions. Intuitive gaze and gesture interactions effectively leverage the affordances of mixed reality to lower the friction in engaging with other studio members. While it was challenging to work around technologies that aren’t fully established yet and lack readily available low-fidelity prototyping tools, I found it incredibly rewarding to apply design methods and research to such an open problem—considering not just what already exists, but simply what was possible.
Process > Exploring the Problem Space
Since the main users of my mixed reality product would be designers in a studio, I started by interviewing my fellow design students about their typical workflow, methods, and pain points regarding remote learning. Seeing how their process had changed because of the pandemic helped me identify two key problems I wanted to address—difficulty in giving meaningful feedback and keeping up with other design peers’ progress. In addition, patterns that were easily missed, like the correlation of device use with productivity, became more obvious from conducting interviews.
One facet of this larger issue that exacerbated many other factors is the the lack of passive awareness between individuals in the same studio. Without a physical shared space, remote students who aren’t able to see when others were working, busy, or willing to talk cannot readily reach out and give or receive feedback, often forcing them to resort to look at static documentation or send email threads with long intervals between responses. Creating a system that allows students to quickly understand another's progress and availability in real-time facilitates more social bonding, design collaboration, and impromptu inspiration.
I was inspired by interfaces like Spotify or Discord that use minimal UI to give the user information about what their friends are doing. Being able to see state changes live is a crucial part of the experience since it conveyed to users the opportunity to start a conversation. Not only would it provide situational awareness, showcasing the transitions between working and available would also give cues to when a user was open and wanted feedback.
Process > Analyzing Workflows and Solution Spaces
In order to explore how best to utilize the physical space that mixed reality affords, I first determined how I intuitively partition the space on my desk and how each device fits into my workflow. I documented my own remote work setup and noted that almost all of the programs I used to design fit into two categories. Technical programs, like Adobe Illustrator or Cinema4D, were computationally taxing and stay on my laptop or larger desktop computer while documentation programs, like Notion or Goodnotes, were more easily accessible when used in conjunction with another display or tablet on the desk.
I found my attention and time are almost exclusively directed at the region right in front of me where my laptop or desktop would be. The secondary region just outside this primary focus zone is used by my tablet and mousepad. While my focus isn't directed at this area as much, the items in this area are essential to how I work. However, there was an opportunity to use the vertical space above my laptop or the open peripheral space along the far left and right of the desk.
Process > Storyboarding Spatial Interactions
Sketching out storyboarding interactions was my fastest method of generating a wide range of ideas quickly. Determining the key features early on and making specific, problem-oriented goals for each interaction made it easier to find what worked and what needed improvement. I learned that establishing clear thresholds, experimenting with what objects were shared in the digital space, and determining how much information needed to be conveyed at each level of the experience guided both the design of the visual hierarchy as well as the ways users engaged with the map.
Before I could create a prototype, I had to decide how I wanted users to interact with the experience. Rather than creating two-dimensional planes that exist in the three-dimensional space, I knew the most intuitive, powerful interactions moved interfaces directly into the space. Existing solutions like Gravity Sketch's color picker allow the user to modify parameters across all three axes.
I decided on a draggable, expandable studio map of fellow design students to best leverage the physical, three-dimensional affordances of mixed reality. I focused on establishing passive awareness through abstracting the studio across synchronous but spatially distant peers, using gaze and gesture interactions as low-friction methods of moving through the experience.
Process > Information Hierarchy
It was important to consider what information could be displayed without disrupting the user's workflow because the map remained open in the user's peripheral vision. More abstract options like colors and symbols could convey the same types of information as text without needing the user to spend time and attention reading through each person's status. My initial attempt at using a facial expression-driven approach ran into the obstacle of needing the user to constantly update their own emotional state, something that dramatically increased the friction in using the system. Taking a page from cognitive science, I noticed that representing one's context gave equally accurate cues towards their situational awareness. Instead, I chose to show one's goals, expectations, and costs through the interface and leverage our ability to decipher social cues.
Left: Snapchat UI uses customizable Bitmojis displayed next to one's username to give insight into emotional states through facial cues. Right: From "How We Understand Emotions" Talk by Rebecca Saxe. A sample story that lacks any facial expressions still gives accurate emotional information because of its included context.
Since these are the most important in establishing awareness, the studio map uses vertical height to show time spent working as a measurement of cost and a distinctive color to indicate their working program in order to set expectations about their work. Adjusting whether one's icon is blinking, glowing, or dimmed also creates immediate goals for whether the user wants feedback, is willing to give feedback, or doesn't want to be disturbed. More information about specific programs, software or tools, and detailed goals are available upon glancing at another user's sphere but remain hidden when the user is not in immediate focus.
Process > Settling on a Design Scope
Process > Technical Implementation
With the details of the interaction in place, I started prototyping the interaction in mixed reality using Unity’s ARFoundation and Apple’s ARKit. While storyboarding was helpful in communicating the idea of an interaction, there was no substitute for being able to feel out the details in real-time. I found that prototyping directly in mixed reality allowed me to gain incredibly valuable feedback around scale, vision, and ease of use. My first few mockups were proof-of-concept demonstrations of AR tracking and gaze/gesture interactions. A package called Manomotion helped streamline the process for identifying different hand poses and gestures such as dragging and clicking while other behaviors were scripted in with C#.
The main scene consisted of a target plane to indicate where the map would appear after acquiring a surface to place AR content onto, a translucent map, selection icons closer to the user, and spheres for each student in the map. The experience itself was set up in three stages: finding the ground plane for AR tracking, observing the map with gaze interaction, and calling another student with hand tracking and pose recognition. While the app was in one state, other elements were hidden until necessary. Small touches like faked cast shadows to make the map more believable and grounded helped sell the illusion.
After smoothing out the AR tracking, I added in features to make the interactive experience more immersive for the user. Unity’s Universal Render Pipeline allowed me to add in lighting effects, giving each sphere its corresponding program’s color and glow. The render pipeline also enabled scripting in blinking lighting, making it possible to subtly grab the user’s attention when someone wanted feedback. For some extra flair, I used Bellus3D to scan in the heads of our actual studio mates, giving the interactions a degree of authenticity and engagement that a simple figurine lacked.
Some final user experience improvements like increasing contrast, scaling up the size of the popups, and tweaking the tracking accuracy made the demo smoother to use. I then created a short video to help pitch mixed reality ideas to a client detailing the individual interactions and how a user might benefit from using mixed reality in their hybrid studio learning.
