Ultimate Interface Lab's 2026 Papers

Redirected Pinch: Efficient and Comfortable Bare-Hand Interaction for 2D Windows in VR

(Abstract) Virtual Reality (VR) offers portable and flexible workspaces. However, enabling efficient and comfortable interactions without external input devices remains challenging. We propose leveraging redirected input to enable comfortable and touch-like interaction for quick and intuitive control. Our design study revealed that while touch interaction performs well with direct input, its performance degrades significantly under input redirection. In contrast, using pinch improves redirected input by providing self-haptic feedback and reducing input dimensionality, thereby compensating for spatial discrepancies. Based on these findings, we introduce Redirected Pinch, a bare-hand interaction technique that combines input redirection with pinch confirmation. It creates a virtual plane at waist height, remapping hand movements on the plane to a vertical window, with pinch gestures used for confirmation. A user study demonstrated that Redirected Pinch achieves a strong balance of accuracy, efficiency, comfort, and sense of agency across fundamental interactions.

(Introduction) Virtual Reality (VR) is emerging as a promising platform for productivity tasks, offering portable, extendable, and distraction-free workspaces that adapt to user needs and contexts [29, 39, 59, 66, 70]. Commercial headsets like Meta Quest 3 and Apple Vision Pro already support desktop-like workflows while enabling large, reconfigurable virtual screens accessible anywhere. Despite the threedimensional nature of VR environments, many productivity applications continue to rely on two-dimensional (2D) interfaces, such as media editing, document review, and web browsing, as 2D layouts remain familiar, efficient, and cognitively lightweight [12, 13, 24, 28, 36, 75]. However, supporting efficient and comfortable interaction with such 2D windows in VR remains a critical challenge.

Common VR interaction methods, such as using handheld controllers, often lack both precision and comfort during prolonged use [71]. External devices like mice [31], keyboards [30, 40, 42], and tablets [12, 13, 28] can provide more accurate control but at the cost of portability, while also requiring physical surfaces for support. With handtracking integrated into commercial headsets, bare-hand interaction offers a natural, lightweight, and always-available alternative [43, 45], capable of enabling more dexterous input than device-constrained methods [48, 58]. However, bare-hand interaction lacks tangibility and stability, making it inefficient for precise input [34, 49], and it induces fatigue when arms are kept elevated, known as the “Gorilla Arm” effect [9, 32].

To reduce fatigue and extend reach, researchers have explored input remapping, which manipulates the spatial relationship between real and virtual hands. The Go-Go technique [64] introduced nonlinear reach extension, and subsequent work has amplified small arm movements for large-window interaction [52] or redirected near-body hand motions onto far-away windows for more comfortable input [16]. While effective for ergonomics, input remapping can introduce visual–motor mismatch, disrupting proprioception and making mid-air touch less reliable [41, 54]. Complementary work improves input redirection using physical surfaces or custom haptic devices [27, 54], but such approaches reduce mobility and are not always available for everyday VR use.

Self-haptic gestures offer an appealing lightweight alternative [25, 33, 61, 86]. Pinch provides tactile confirmation through thumbfinger contact and supports robust gesture recognition [18, 62, 79], leading to wide adoption in commercial systems [6, 55]. However, the performance benefitsofpincharenotconsistent across contexts. Pinch can be slower, more error-prone, and physically demanding in direct mid-air selection [14, 21, 57], yet has shown benefits in indirect interactions such as text entry [26, 41]. We hypothesize that pinch becomes more beneficial in redirected mid-air interaction than in direct interaction for two reasons: (1) its immediate tactile feedback without external devices, compensating for the uncertainty introduced by spatial remapping, and (2) its reduced input dimensionality, as 3D positioning is transformed into 2D motion with a binary pinch confirmation gesture, mitigating accuracy issues caused by depth perception challenges in redirected spaces.

In this paper, we present Redirected Pinch, a novel bare-hand interaction technique that combines pinch with input remapping to enable efficient and comfortable interaction with 2D windows in VR (Figure 1). Redirected Pinch creates a tilted virtual plane at waist height, decoupling the visual workspace from the control space for more ergonomic interaction [4, 16]. Hand movements relative to this plane are remapped in both position and orientation to the application window, while pinches provide self-haptic confirmation. This design reduces fatigue from elevated postures and compensates for spatial inaccuracies caused by remapping. We developed Redirected Pinch through a design study that explored different input mappings (i.e., direct vs. redirected) and confirmation gestures (i.e., touch vs. pinch). Our findings revealed that while touch worked effectively for direct input, its performance degraded significantly under redirected input. In contrast, pinch enhanced redirected input across accuracy, efficiency, and sense of agency, supporting our hypothesis about the value of pinch in remapped interactions.

Finally, weevaluatedtheperformanceofRedirectedPinchagainst commonly used VR interaction techniques, including direct pinch, gaze pinch, and handray pointer, in both simple selection tasks and more complex docking tasks. The results showed that Redirected Pinch provided the best overall balance of comfort, efficiency, and sense of agency in both tasks. In addition, Redirected Pinch was consistently preferred because it required less effort and offered easier control, particularly during prolonged, complex interactions involving continuous and multi-touch input.

(Conclusion) In this work, we investigated mid-air bare-hand interaction techniques for discrete, continuous, and multi-touch inputs on 2D windows in VR. Through design studies comparing input mapping and confirmation gestures, we propose Redirected Pinch, a novel interaction technique that allows users to perform pinch gestures with comfortable postures while enabling efficient interactions by hand redirection and input remapping. We compared Redirected Pinch with three commonly used methods: Direct Pinch, Gaze Pinch, and Handray Pointer. The comparative evaluation showed that Redirected Pinch provides a strong balance of speed, accuracy, agency, and comfort across both selection and docking tasks. While we tested Redirected Pinch in a controlled study, it can be flexible and adapted to more realistic applications. Future work could explore similar techniques in real-world productivity tasks involving windows of varying sizes, distances, orientations, and multi-window setups.

"It Feels Like I am Invited to Communicate": Mediating Ad-Hoc Bystander-VR User Interruptions Through Proactive Proxies

(Abstract) As VRexpands into public spaces, new challenges emerge around spontaneous interactions between bystanders and unfamiliar VR users. While current VR systems often prioritize user awareness of their physical surroundings, they overlook the social dynamics affecting nearby bystanders. We conducted a deception-based study (N=80) examining how interface availability influences bystanders’ comfort, confidence, and hesitation when interrupting VR users. We compared traditional static interruption interfaces (e.g., button on screen) with a proactive proxy that actively approached bystanders upon detecting interruption intent. Static interfaces, due to insufficient cueing, frequently caused bystander discomfort, leading to hesitant physical interruptions or complete communication avoidance. In contrast, the proactive proxy implicitly conveyed social permission, significantly enhancing bystanders’ comfort and confidence. Our findings provide empirical insights into how bystanders assess availability and initiate interruptions with unfamiliar VR users in shared spaces, offering design implications for VR systems that support bystander agency and comfort during these interactions.

(Introduction) Human interruptions in shared spaces are inevitable- whether it is asking a colleague for their input on a shared project [80] or asking a fellow passenger to move [81]. In these moments, gaining someone’s attention is essential. However, interruptions become more challenging when individuals are deeply immersed in activities that render their primary communication channels unavailable (e.g., listening to music occupies their hearing, reading occupies their vision, etc.). In such cases, secondary communication channels like peripheral vision and ambient awareness serve as subtle cues for initiating interaction.

When individuals are immersed in virtual reality (VR), this dynamic becomes especially complex because their audiovisual immersion effectively limits the secondary use of these primary communication channels [19, 28, 40, 53]. Several alternative modalities have thus been proposed to manage interruptions in VR [50]. Modern VR headsets, such as the Meta Quest and Apple Vision Pro, incorporate open-ear audio technology that enables users to hear verbal interactions. However, this technology can reduce immersion and may not align with all users’ preferences [50]. Additionally, such technology can be problematic in shared spaces, as sound leakage can disturb nearby individuals and make VR users selfconscious about audio spillover [50, 63]. Users already seek full auditory isolation when completing focused work in public environments by using noise-canceling headphones [46, 55], and as portable VR headsets become productivity tools, extending this need for isolation to the visual domain is a natural progression. Both the Apple Vision Pro and Meta Quest now include dedicated travel modes for use on airplanes and trains [4, 41], explicitly supporting fully immersive use in shared spaces. Moreover, airlines have also begun providing VR headsets to passengers as part of in-flight entertainment services [16], while academic and public libraries increasingly offer VR headset lending programs and dedicated VR spaces [2, 34]. These developments signal that encounters between unfamiliar individuals (i.e., one immersed in VR and the other needing to interrupt them) will become increasingly common. Critically, because bystander awareness features within headsets remain user-controlled, bystanders have no guaranteed means of initiating contact when VR users opt for full immersion. Touch, while potentially the most direct way to interrupt a VR user, is often socially inappropriate in public settings, where interactions among strangers are governed by implicit boundaries [50, 53]. As VRheadsets transition beyond private environments into shared spaces where bystanders may need to interrupt VR users, developing socially appropriate and effective interruption mechanisms becomes increasingly critical [1, 14, 62, 72, 81].

One approach to this challenge has been to enhance VR users’ awareness of their surroundings. Previous work has explored embedding awareness systems within VR headsets to help users avoid physical accidents and maintain spatial boundaries [22, 40, 42, 49, 51, 53, 58, 73, 84]. Such systems, however, cannot adequately replicate the dynamics of face-to-face interruptions and the subtle social cues that initiators (i.e., bystanders) carefully weigh when determining whether an interruption is appropriate [44]. By shifting control from initiators to VR users, awareness systems invert the natural negotiation process that governs interpersonal interruptions. Beyond immersion trade-offs and the risk of information overload in public spaces, embedded awareness systems may also prematurely redirect a VR user’s attention to bystanders, triggering unwanted face engagements [20]. Although awareness systems can dynamically adjust how they deliver information by withholding alerts until a verbal interaction is attempted [52], such techniques overlook how bystanders assume others’ unavailability and make decisions not to engage [76, 80]. In crowded public spaces, VR users may even disable these systems to avoid constant notifications about nearby bystanders, especially when they do not anticipate needing to interact with others.

Although limited, prior work has explored bystander-initiated interruption interfaces, such as the HTC’s Knock Knock feature [71] and a physical doorbell peripheral [81], which offer explicit mechanisms for bystanders to attract a VR users’ attention. However, these interfaces typically assume that bystanders are aware of their existence, limiting their effectiveness in public spaces where interruptions are often spontaneous and involve unacquainted individuals. While interruptions between acquaintances may be more frequent, stranger-to-stranger interactions represent the most challenging case due to the absence of established social rapport and the heightened psychological cost of initiating contact [12]. Solutions effective in such demanding contexts are also likely to generalize to less demanding scenarios. While prior research underscored the importance of observing spontaneous interactions between unfamiliar individuals [50], little is known about how such encounters unfold when unacquainted VR users share physical spaces. Yet, insights into these dynamics are essential for designing VR headsets that integrate more seamlessly into public environments. In this work, we investigate how bystanders naturally navigate spontaneous interactions with unfamiliar VR users in shared environments.

As studying such unplanned encounters at scale is challenging duetotheir situational constraints, we conducted a deception-based study that recreated these constraints in a controlled setting. Unlike prior work that relied on solicited interruptions [17, 23, 50], acquainted participants [50, 53], or anecdotal reports [53], our study placed 80 unsuspecting bystanders in scenarios requiring urgent interaction with an unfamiliar VR user in a shared space. Our study had two interruption interface conditions. The baseline condition, experienced by 40 participants, used a typical desktop VR setup with a static interruption interface (e.g., doorbell peripheral). For the second condition, we hypothesized that explicitly visible, bystander-initiated interfaces could improve interruption experiences. Inspired by public display research [27, 30], we created a robot-based interruption interface, i.e., a proactive proxy that served as a design probe to examine whether explicit interface discoverability impacted bystander experiences. This deliberately extreme intervention was intended to isolate the effect of proactive presentation on bystander comfort and behavior. This condition was experienced by an additional 40 participants.

During our study, we found that, similar to public kiosk research [30], the participants who encountered the baseline condition suffered from the first-click problem [30], where they failed to notice or use the static interface. In contrast, the participants who encountered the proactive proxy reported increased comfort and engagement. While participants preferred mediated interaction over direct interruption, the static interface was largely ignored. These findings underscore the importance of interface visibility and design in shared VR spaces, offering actionable insights for future VR headset development that better supports both users and bystanders.

This research contributes: (1) Empirical evidence that bystanders feel significant discomfort during spontaneous interactions with unfamiliar VR users in shared spaces, with key insights revealing how physical boundaries, perceived safety, and social dynamics influence their interruption behaviors. (2) Identification of key limitations in existing static interruption interfaces, showing how privacy concerns and low situational awareness hinder their discoverability and usability during spontaneous interactions. (3) Design implications for bystander-aware VR interfaces that support comfortable and effective spontaneous interactions, emphasizing the need to explicitly integrate bystander perspectives into VR system design.

(Conclusion) As VRsystems increasingly extend beyond private spaces, understanding and supporting interactions between VR users and bystanders becomes essential. Our research provides empirical evidence that bystanders often feel significant discomfort when interrupting unfamiliar VR users and that traditional static interruption interfaces frequently go unnoticed during spontaneous encounters. Our proactive proxy interface, on the other hand, demonstrated that explicitly discoverable interfaces can significantly improve bystander comfort and interaction quality by signaling social permission to initiate contact. These findings suggest that as VR systems increasingly move into shared and public environments, their design must also consider the needs of nearby bystanders. In particular, incorporating interfaces that clearly communicate availability and guide bystander attention can help trigger interruptions in a comfortable and socially appropriate way.

We advocate for integrating bystander-centric design into VR systems, demonstrating that giving bystanders agency in managing interruptions enhances comfort and engagement. These interfaces can complement existing awareness systems by offering explicit communication channels without compromising immersion. We outline key design considerations for such systems and suggest that future research explore how to effectively combine these approaches across diverse contexts. As VR adoption expands into offices, transit systems, and public venues, this integration will be vital for creating socially attuned VR experiences that respect the needs of all participants.

PrevizWhiz: Combining Rough 3D Scenes and 2D Video to Guide Generative Video Previsualization

(Abstract) In pre-production, filmmakers and 3D animation experts must rapidly prototype ideas to explore a film’s possibilities before fullscale production, yet conventional approaches involve trade-offs in efficiency and expressiveness. Hand-drawn storyboards often lack spatial precision needed for complex cinematography, while 3D previsualization demands expertise and high-quality rigged assets. To address this gap, we present PrevizWhiz, a system that leverages rough 3D scenes in combination with generative image and video models to create stylized video previews. The workflow integrates frame-level image restyling with adjustable resemblance, time-based editing through motion paths or external video inputs, and refinement into high-fidelity video clips. A study with filmmakers demonstrates that our system lowers technical barriers for film-makers, accelerates creative iteration, and effectively bridges the communication gap, while also surfacing challenges of continuity, authorship, and ethical consideration in AI-assisted filmmaking.

(Introduction) Previsualization (previz) is a central practice in filmmaking, enabling directors and creative teams to explore the visual and narrative structure of a scene before production [54]. By creating early visualizations, filmmakers can test ideas for camera angles, blocking, pacing, and emotional beats without the expense of full-scale sets, actors, or detailed assets. Beyond its role as a creative sketching tool, previz also functions as a collaborative artifact, helping directors, cinematographers, production designer, and other stakeholder align around a shared vision [2, 4, 18].

Despite its importance, existing approaches force filmmakers to make trade-offs between speed, fidelity, and control. Storyboards and moodboards are quick and expressive, allowing for early exploration and communication of creative intent [16, 44]. However, these are static, offering limited spatial and temporal representation: they cannot adequately represent motion or timing, making it difficult to visualize complex shots or sequences. 3D previz tools on the other hand allow filmmakers to compose scenes, experiment with camera blocking, and ensure continuity across shots [12, 34, 54]. However, these tools require high-fidelity 3D assets, rigging, and animation expertise [40]. Many existing 3D previz tools also fail to convey fine-grained nuances like emotional beats and microactions.

Recent advances in generative AI can accelerate previsualization by producing images or videos directly from textual prompts, allowing filmmakers to quickly generate outputs with a compelling visual style [17]. Yet, they pose new challenges. Text-to-image and text-tovideo models often struggle with temporal consistency, making coherent motion across frames challenging [36]. They also lack spatial grounding: precise placement of objects and camera, blocking, and continuity are difficult to control. As a result, current approaches using generative AI risk producing highly polished looking results that are disconnected from the filmmaker’s intended structure. Filmmakers need a lightweight and flexible approach that combines the spatial grounding of 3D tools with the expressive richness of generative video tools.

We present PrevizWhiz, a system that allows filmmakers to rapidly explore and visualize their shots by combining rough 3D scene blocking for timing and spatial structure, 2D video references for detailed character motion, and generative stylization guided by images and text. Filmmakers begin by arranging rough 3D proxies to establish prop positions, character movement, as well as camera paths (Figure 1a). They can restyle frames from their 3D scenes to experiment with different aesthetic styles, ranging from strict adherence to loose reinterpretation of their compositions (Figure 1a). Finally, PrevizWhiz allows filmmakers to specify three levels of motion fidelity: (1) coarse motion from 3D blocking, (2) stylized animations that combine motion from 3D blocking with the restyled frame, and (3) control-video animation that augments the stylized animation with 2D reference videos for detailed character motion (Figure 1b). These frames of scene composition, and time-based elements can guide the video generation of final outputs (Figure 1c), shaping style, lighting, composition, and movement in ways that balance the structural consistency of 3D blocking with the expressiveness of 2D generative tools to create previsualization for film.

Our contributions are: (1) PrevizWhiz, a system that combines rough 3D blocking, frame stylization, and granular animation control to enable lightweight yet expressive previsualization, and (2) findings from a user study with filmmakers and 3D artists showing how the system enables rapid ideation during previsualisations and probes their thoughts on generative tools for pre-production.

(Conclusion) We presented PrevizWhiz, a system that combines rough 3D scene blocking, detailed character motion, and video stylization through generative AI to support flexible, rapid previsualization. Through a user study with filmmakers, we found that the system enabled lightweight scene setup, iterative refinement, and expressive authoring across modalities. Our findings suggest that AI-assisted previz can augment creative practice, lowering barriers for independent creators to communicate their creative intent. At the same time, issues of latency, consistency, and fear of displacement highlight the need for careful future design.

DataSpeck: An AI-Driven Human-in-the-Loop System for Automating Transformations in Data Conversion Workflows

(Abstract) In data-driven systems, integrating disparate data sources becomes challenging when incoming data does not conform to the system’s specifications. Despite advances in automated schema matching systems, data integration tasks involving complex semantic interrelationships still require users to manually identify and define transformations between datasets, which can be cognitively demanding and time-consuming. We present DataSpeck, an end-to-end system that automates the conversion of disparate data sources to fit any pre-existing data specification. DataSpeck employs an AI-driven human-in-the-loop design, using LLMs to analyze semantic relationships and generate step-by-step transformation pipelines autonomously, while only requesting user attention to resolve semantic ambiguities. In our technical evaluation, DataSpeck successfully automated ~86% of varied data transformations while generating interpretable strategies with confidence scores and targeted clarification requests. In a user study (N=12), participants completed data conversion tasks ~53% faster with significantly reduced cognitive load using DataSpeck compared to Microsoft Excel with Copilot.

(Introduction) Data-driven systems are often built around specific data formats and structures optimized for their intended use cases. However, the growing variety of data sources poses a major challenge, as incoming data frequently deviates from expected formats and must be adapted before it can be utilized [58]. For example, a healthcare analytics platform designed to process standardized patient records may struggle to integrate external research datasets with differing schemas [11]. Integrating new data sources within established data architectures requires manually designing elaborate transformation pipelines which map incoming data to match the specification format [113]. This process consumes time and effort and must be repeated every time a new data source is introduced - creating significant bottlenecks in data integration workflows.

Reconciling different data formats has been a longstanding research goal spanning decades in data management and information systems, with schema matching and mapping techniques being the primary approaches to enable interoperability between heterogeneous data formats [85, 98]. While these systems excel at finding match candidates between source and target schemas, they often leave the semantic relationships and necessary transformations between matched attributes for users to define manually [6, 10, 13, 98, 126]. In data workflows, this transformation step remains a significant bottleneck, requiring detailed preparation before data can be utilized [75]. To address this, various interaction techniques seek to simplify the process by reducing manual coding requirements. Programming-by-example [8, 20, 44, 49, 62, 106] and Programming-by-demonstration [47, 64] systems have proven particularly effective in eliminating the need to manually write transformation code and allow users to automate data formatting through demonstrated examples. Natural language interfaces [56] have further enabled users to describe transformation requirements in plain language, though they often require precise phrasing to avoid ambiguity [66, 109, 123]. Despite these advancements, users must still understand data structures, interpret relationships, and devise appropriate transformations—a process that remains cognitively demanding for complex or unfamiliar datasets. While human insight remains crucial for exploratory analysis [105], scenarios with predetermined data structures could benefit from more nuanced automation approaches that reduce unnecessary overhead when adapting diverse sources to existing specifications.

In this paper, we present DataSpeck, an AI-driven system designed towards converting a dataset into a prescribed specification. Unlike previous approaches, DataSpeck does not require users to provide examples or describe the transformation process explicitly. Instead, it leverages LLMs to analyze the semantic relationships between the new data source and the pre-existing data specification, and uses this understanding to automatically generate both transformation strategies and the corresponding data conversion scripts. However, data conversions can involve ambiguities that require additional context. To address such scenarios, DataSpeck incorporates a human-in-the-loop design, classifying the need for human input based on system confidence, and prompting users to provide additional context when necessary.

To evaluate the effectiveness of our human-in-the-loop system, we first conducted a technical evaluation to understand the boundaries of our system’s automation capabilities. We tested DataSpeck against 43 isolated transformation scenarios and 5 complex realworld data conversion scenarios. Our system was able to successfully automate ~86% of the data transformation operations and yielded appropriate system confidence scores and clarification requests. Our technical evaluation also highlighted transformation scenarios where the automation capabilities struggled. Using these insights, we designed a user study with 12 participants who had prior data integration experience to measure the system’s impact on user performance and effort. As a baseline, we compared DataSpeck against Microsoft Excel with Copilot, which represented the counterpart human-driven, AI-in-the-loop approach where users infer the semantic relationships on their own, and then use natural language interactions to design the transformations. Participants achieved significantly higher performance and efficiency in the given data conversion tasks, completing tasks 53% faster on average using DataSpeck, and reported significantly lower levels of mental demand, effort, and frustration on the NASA-TLX scale. They found DataSpeck’s ability to automatically infer transformations from source-specification pairs highly usable and valuable for their professional settings, noting that such an interaction could transform the tedious task of manually analyzing datasets and writing migration scripts into simply reviewing system-generated strategies and answering clarifications, thereby significantly reducing manual effort and cognitive load.

We summarize our contributions as follows: (1) The design of DataSpeck, an end-to-end system that automates the transformation of disparate data sources into preexisting data specifications through an AI-driven, human-inthe-loop approach. (2) Technical findings demonstrating that DataSpeck successfully automates a comprehensive set of data transformation operations across diverse scenarios, while generating appropriate confidence scores and targeted clarification requests when human input is needed. (3) User performance results showing that DataSpeck’s AI-driven, human-in-the-loop approach enables users to complete data conversion tasks more efficiently while significantly reducing cognitive load across all NASA-TLX dimensions compared to human-driven AI-in-the-loop approaches.

(Conclusion) Data conversions often require significant manual effort to align diverse data sources to specific structural requirements. Traditional methods do not directly solve for this problem and using existing tools requires extensive manual efforts; in contrast, DataSpeck employs an end-to end AI-driven human-in-the-loop approach that understands the data, strategizes the conversion pipeline, and executes the transformation steps in a highly transparent manner, employing a tiered confidence-based human intervention mechanism when it may be needed. A technical evaluation shows that DataSpeck was able to handle a large number of transformation tasks without human intervention, automate a large part of the pipeline for realworld end-to-end conversions, and successfully surface the need for human intervention for the rest. Our user study demonstrates that DataSpeck’s AI-driven human-in-the-loop approach is significantly more efficient in terms of time and effort compared to a familiar human-driven AI-assisted method. While fully human-driven methods offer complete control, they are often time-consuming and cognitively demanding. In contrast, a human-in-the-loop approach, where the AI automates the majority of transformations and requests user input for clarification or uncertainties, demonstrated potential to significantly enhance efficiency and reduce cognitive overheads for such data tasks.