Ultimate Interface Lab's 2025 Papers

DialogLab: Authoring, Simulating, and Testing Dynamic Human-AI Group Conversations

(Abstract) Designing compelling multi-party conversations involving both humans and AI agents presents significant challenges, particularly in balancing scripted structure with emergent, human-like interactions. We introduce DialogLab, a prototyping toolkit for authoring, simulating, and testing hybrid human-AI dialogues. DialogLab provides a unified interface to configure conversational scenes, define agent personas, manage group structures, specify turn-taking rules, and orchestrate transitions between scripted narratives and improvisation. Crucially, DialogLab allows designers to introduce controlled deviations from the script—through configurable agents that emulate human unpredictability—to systematically probe how conversations adapt and recover. DialogLab facilitates rapid iteration and evaluation of complex, dynamic multi-party human-AI dialogues. An evaluation with both end users and domain experts demonstrates that DialogLab supports efficient iteration and structured verification, with applications in training, rehearsal, and research on social dynamics. Our findings show the value of integrating real-time, human-in-the-loop improvisation with structured scripting to support more realistic and adaptable multi-party conversation design.

(Introduction) The simulation and creation of conversations with embodied agents have long been integral to virtual entertainment environments, especially for non-player characters (NPCs). Beyond gaming, these systems are increasingly deployed in diverse contexts such as education [60], healthcare [77], professional training [47], and retail [53]. Initially constrained by rule-based or scripted mechanisms and limitations in natural language understanding (NLU), the design of embodied agents have more recently leveraged Large Language Models (LLMs) to deliver more adaptive, context-aware dialogues.

Commercial tools like Character.AI1 and Replika2 illustrate this trend, enabling users to create custom AI personalities for one-onone conversations. However, as human-AI interaction continues to evolve, user needs extend beyond single-agent dialogues. In many real-world scenarios, users participate in multi-party conversations, where multiple agents and humans interact under a blend of prescripted and emergent conditions. These settings introduce new complexities—users must navigate shifting roles, manage turn-taking protocols, and negotiate control of the conversation as they move between reactive and proactive stances. For instance, a human trainee may lead with structured questions in a medical simulation [75] but revert to a reactive role when questioned by an AI panelist. In multi-agent contexts, these transitions require sophisticated system support to maintain coherence across overlapping speaking turns, interruptions, and dynamic topic shifts.

Through a formative study, we sought to identify key challenges faced by conversation designers and developers: (1) Simulating how scripted agents balance rigidity and flexibility. (2) Testing interactions with human participants who deviate from expected behaviors. These challenges reveal two core design tensions: First, the authoring and simulating of multi-party hybrid human-AI dialogue systems remain complex, often demanding significant programming or prompt engineering [103], hindering rapid iteration by designers. Even with scripted guidance, balancing narrative structure reflecting diverse group dynamics with unpredictable human input requires extensive testing. Second, designers must navigate a trade-off between structured scripts, which provide consistency and safety, and improvised human-led dialogue, which fosters realism, user agency, and spontaneity [65, 104]. Current approaches often represent two extremes: fully scripted interactions, which ensure control but lack adaptability and spontaneity [55, 108]; and fully generative and automated systems, which allow open-ended dialogue but are difficult to steer toward specific goals [59, 106]. While structure is critical in domains like healthcare or education, rigid scripting can make users feel passive and limit adaptability. Generative systems, by contrast, allow open-ended interaction but lack control for task-oriented or training scenarios. These challenges become even more pronounced in multi-party and group settings due to the increased complexity of managing roles, turntaking, and recovery from interruptions [38, 39, 93].

To address these challenges, we present DialogLab, a unified authoring tool for creating multi-party conversations with hybrid scripted/LLM-driven embodied agents. Our system enables designers to: (1) configure basic persona, scene, and conversation setups that help define diverse group dynamics (Fig. 1a-b); (2) embed LLMdriven agents during prototyping to simulate human unpredictability within pre-defined narratives (Fig. 1c); (3) incorporates features designed to facilitate verification and reflection of these complex group and conversational dynamics.

To evaluate DialogLab, we conducted a user study with five regular users and nine domain experts who used DialogLab to author and test multi-party conversation scenarios. Participants appreciated the diverse conversation flexibilities and agency during the authoring, and reported clear verification of group behaviors, and strong support for real-time improvisation and testing. These findings demonstrate DialogLab ’s ability to support structured authoring, dynamic simulation, and reflective verification of complex hybrid human-AI conversations.

In summary, our contributions are:
A flexible framework and authoring paradigm for multi-party human-AI conversations, enabling configuration of group dynamics, social protocols, and hybrid scripted/improvised interactions.
DialogLab, an open-source system3 implementing this framework, supporting the workflow of authoring, simulating, and interactively testing multi-party dialogues.
A human evaluation comparing human agents with reactive agents in group settings and a user study of DialogLab with regular users and domain experts, demonstrating its effectiveness in building and simulating human-AI group conversations. 

(Conclusion) We present DialogLab, a prototyping toolkit for creation and iterative design of multi-party conversations between humans and AI agents. DialogLab allows users to configure various conversation attributes, including agent roles, interaction patterns, and turn-taking dynamics, to simulate realistic and dynamic dialogues. The toolkit supports both scripted and adaptive conversational elements, offering a hybrid approach that combines the structure of predetermined dialogues with the flexibility of spontaneous interactions. DialogLab aims to streamline the development process by providing an intuitive interface for conversation setup and realtime testing and validation, addressing key challenges such as the complexity of configuration, uncertainty of AI responses, and lack of systematic refinement methods. Our evaluation with 14 participants demonstrates its effectiveness in enhancing user control, reducing development time, and facilitating realistic multi-party conversation simulations across various domains. We hope that DialogLab’s approach and contributions will help advance and inspire the emerging field of Human-AI group conversation dynamics.

Thing2Reality: Enabling Spontaneous Creation of 3D Objects from 2D Content using Generative AI in XR Meetings

(Abstract) During remote communication, participants often share both digital and physical content, such as product designs, digital assets, and environments, to enhance mutual understanding. Recent advances in augmented communication have facilitated users to swiftly create and share digital 2D copies of physical objects from video feeds into a shared space. However, conventional 2D representations of digital objects limits spatial referencing in immersive environments. To address this, we propose Thing2Reality, an Extended Reality (XR) meeting platform that facilitates spontaneous discussions of both digital and physical items during remote sessions. With Thing2Reality, users can quickly materialize ideas or objects in immersive environments and share them as conditioned multiview renderings or 3D Gaussians. Thing2Reality enables users to interact with remote objects or discuss concepts in a collaborative manner. Our user studies revealed that the ability to interact with and manipulate 3D representations of objects significantly enhances the efficiency of discussions, with the potential to augment discussion of 2D artifacts.

(Introduction) Shared artifacts, including digital resources (e.g., text, images, videos), and physical objects (e.g., prototypes, printouts), play a crucial role in facilitating effective communication of spatial concepts and the generation of design ideas, especially in creative fields such as product design, architecture, and marketing strategy development. They provide common spatial reference points that bridge gaps between collaborators, enhancing creative exploration and ideation [4]. Besides physical artifacts, designers frequently use online platforms like Google and Pinterest to source relevant digital artifacts that can support their design processes [24]. However, using shared artifacts in remote meetings often pose challenges, especially in scenarios that require quick and spontaneous sharing, such as brainstorming and early stage design sessions. First, artifacts shared via remote meetings are typically in 2D, whether they are captured via camera or retrieved from online repositories, limiting the understanding compared to interactions with physical objects or 3D models. Second, in physical meetings, participants can easily observe and interact with tangible artifacts, which facilitates creative exploration and idea generation processes [4]. However, in remote meetings, this level of interaction is often unavailable or limited.

Several methods have attempted to address these challenges, such as preparing 3D models in advance via CAD or 3D scanning [34], or employing specialized real-time 3D capture setups [62, 78]. While effective, these approaches have limitations: pre-made 3D assets do not support spontaneous sharing, and specialized setups are often impractical for general use. Recent advances in AI-driven text-to-3D and image-to-3D technologies [75] present an accessible and intuitive alternative, lowering barriers to 3D content creation and enabling broader participation in collaborative efforts.

In this paper, we aim to investigate how on-the-fly transformations between 2D content and 3D representations can support object-centric ideation and spatial sense-making in collaborative XR meetings, and how the design of such a system can enhance user experiences by integrating interactive image-to-3D workflows across various XR meeting context.

We designed and implemented Thing2Reality, an XR meeting platform that enables fluid interactions with 2D and 3D artifacts. Thing2Reality allows users to segment content from any source (video streams, shared digital screens) within the XR environment (Figure 1a), generate multi-view renderings (Figure 1b) with conditioned multi-view diffusion models, and transform them into shared 3D Objects with Gaussian splatting for interactive manipulation (Figure 1c). We conducted two user studies: a preliminary study (N=12) analyzing the usability of using digital and physical sources for 3D object creation, and an exploratory study (N=18) exploring how the co-existence and transformation between 2D and 3D formats shape collaboration patterns across tasks such as avatar decoration, spatial layout, and open-ended design brainstorming. Our findings suggest that 3D objects facilitate intuitive explanations, detailed visualization, and interactive collaboration, whereas 2D representations are more often used for final pitch deliverables, suggesting a context-dependent trade-off between the two formats based on task objectives.

In summary, we contribute: Thing2Reality, an XR meeting platform that provides onthe-fly 3D objects generation by enabling users to present and share spontaneous thoughts, and augment their shared digital and physical artifacts with remote collaborators. Findings from a usability study (𝑁 =12) examining the digital and physical inputs of Thing2Reality. Findings from an exploratory user study (𝑁 =18) evaluating the use of Thing2Reality (both 2D-to-3D and 3D-to-2D workflow) for discussing and presenting both 2D and 3D objects in XR meetings.

(Conclusion) We believe that XR communication has tremendous promise for co-presence and for bridging distances between humans, yet much focus today is on realistic rendering of avatars and remote participants. However, as XR systems mature and become increasingly realistic, it will also become increasingly important to support a similar level of spontaneity with objects and artifacts, as what people experience in real environments. In this paper, we presented Thing2Reality, an XR communication system that allows users to instantly materialize ideas or physical objects and share them as interactive conditioned multiview renderings or 3D Gaussians for realistic 3D rendering. Thing2Reality is one of many necessary building blocks towards increasingly realistic co-presence in XR, and we hope that our work will inspire continued work towards augmented communication in both physical and mirrored world [9].

CustomSight: Enhancing LLM-Powered Visual Assistance for Blind Individuals using Goal-Directed Dynamic Filters

(Abstract) LLM-powered assistive technologies (ATs) have enabled blind and visually impaired (BVI) users to query personalized, goal-oriented information about their visual environment. However, the accuracy of system responses depends heavily on well-framed, queryrelevant images, which can be difficult for BVI users to capture. We present CustomSight, an LLM-powered AT that helps BVI users effectively query visual information by providing task-aware, real-time guidance to frame the camera and automatically capture images when relevant content is in view. When a user issues a query, CustomSight generates a Dynamic Filter—a custom pipeline that encodes logic tied to the user’s intent, monitors the live feed, and triggers context-aware feedback and image capture. The captured image is sent to the LLM to fetch accurate visual information.

(Introduction) Blind and visually impaired (BVI) individuals frequently rely on smartphone-based ATs to independently access visual information in daily life. Several computer vision (CV) models—such as text recognition, object detection, and scene captioning—have been integrated into popular ATs (e.g., Seeing AI [9], Tap Tap See [5]) to help users explore their surroundings and identify objects [6, 7]. However, despite widespread adoption, these solutions typically provide context-agnostic responses, forcing BVI users to creatively “hack” around limitations or cope with significant cognitive load to find relevant information [7].

Recently, LLM-powered ATs (e.g., Be My AI [2]) have introduced natural language querying, allowing BVI users to ask targeted visual questions conversationally for diverse use cases [1, 8, 10]. However, these systems depend critically on capturing relevant, well-framed images to produce useful responses—an inherently difficult task without sight [3]. For instance, if a user wants to know a food item’s expiry date, the image must clearly show it. Given the asynchronous nature of LLM responses [4], capturing a usable image may take multiple attempts, reducing efficiency and limiting practicality.

To address this, we designed CustomSight, an assistive system that creates custom pipelines to automatically capture images relevant to a user’s goal and query the LLM, helping BVI individuals access goal-oriented visual information effectively. CustomSight retains a chat-based interface similar to existing LLM-powered apps like Be My AI. However, instead of simply facilitating question answering, CustomSight first checks whether the submitted image contains enough information to meaningfully answer the query. If not, it assesses whether available on-device sensing and feedback modules can help the user better aim the camera and trigger capture when relevant content is likely in frame. It then generates a Dynamic Filter—a tailored pipeline that processes the live feed using goal-specific logic, provides context-aware audio cues to assist framing, and automatically captures an image once the user’s goal criteria are met, passing it to the LLM for a task-specific response (Fig 2a). For example, if a user looks for expiry date, CustomSight may generate a Dynamic Filter that monitors text recognition output for relevant keywords like “expiry date”, “use by”, and “best before”, provides continuous sonification and speech feedback, and captures when relevant content is detected and steady (Fig 2b).

(Conclusion) We presented the design of CustomSight, an LLM-powered visual assistive tool that helps blind users efficiently capture task-relevant images through real-time, goal-directed guidance. Our preliminary study highlighted its potential to improve the efficiency of visual information access for BVI users. Future work includes co-design workshops to explore new sensing and feedback modules (e.g., depth, haptics), supporting more expressive framing guidance, and real-world deployments to assess everyday utility and impact.

SpaceShare: Leveraging Multimodal Context for Fluid Sharing of Spaces in Video Meetings

(Abstract) In video meetings, people not only talk but also share objects and their environments. However, conventional video calls offer limited support for spatial sharing, typically relying on a live video feed. We present SpaceShare, a system that integrates real-time 3D space reconstruction into video meetings and leverages multimodal context to fluidly support shared spatial understanding. The reconstructed 3D environment enables users to independently explore the space, while avatar visualization and shared views convey each participant’s location and viewpoints. SpaceShare also stores conversation content and contextual metadata, such as where the conversation took place. This allows users to retrieve spatial features using both physical attributes and conversational context.

(Introduction) Video meetings are now widely used across many areas. Beyond simple conversation, people increasingly share physical environments through video meetings. For example, they take virtual guided tours [6], view real estate remotely [4], and collaborate on spatial projects [1]. While modern mobile devices can capture scenes in high fidelity, effectively communicating spatial information remains challenging. The video feed is momentary and shows only what is currently in view. Referencing parts of the space is difficult and inefficient, especially when the location is out of view or far from the users. As the shared space becomes larger, problems become more pronounced.

To effectively support space sharing in video meetings, prior work has identified several key aspects: enabling view independence, fostering spatial awareness, and supporting referencing of past scenes. View independence allows users to explore remote environments freely, improving task efficiency and confidence [15], though solutions like 360° video remain limited by fixed viewpoints [1, 16, 17]. Real-time 3D reconstruction using Simultaneous localization and mapping (SLAM) methods [14] or neural techniques like NeRF and Gaussian Splatting [7, 12] may address this by allowing interactive, live exploration. For spatial awareness, systems have used avatars, gaze, and gestures to convey partner context [2, 10], but these become less effective in large spaces. Finally, referencing previously visited scenes remains challenging. Landmarks and minimaps help [18], but are often insufficient without integration of conversational context [9].

To address these challenges, we developed SpaceShare, a system that integrates real-time 3D space reconstruction with multimodal contextual features to enhance shared spatial understanding in video meetings. Using SLAM, SpaceShare reconstructs the environment on the fly, enabling both local and remote users to independently explore the shared space. The system visualizes each user’s position and viewpoint through virtual avatars and ray-based pointers to facilitate view communication. As users converse, SpaceShare captures not only the 3D environment but also multimodal contextual data, including user locations, and conversation content. It generates panoramic images by stitching keyframes, labels objects in the scene, and stores these representations alongside the conversation history. This enables rich, context-based retrieval of prior scenes. For instance, instead of navigating manually or issuing a generic search like “chair,” a user could search for “the chair that reminded us of our childhood memories,” leveraging the conversational context to find a specific moment or location.

Our key contribution is the introduction of a novel approach that incorporates multimodal contextual summaries to support spatial sharing and retrieval in remote meetings.

(Conclusion) While our preliminary study demonstrated the potential of SpaceShare for enhancing spatial recall and collaboration in remote meetings, the SUS scores indicate that the overall interface and interaction design still require improvement. Future work will include iterative design refinements to improve usability and a more comprehensive user study to gain deeper insights into the effects of the space summarization and retrieval features.

Evaluating the Physics and Repeatability of Human Brushers in Delivering Affective Touch

(Abstract) Research in affective touch often utilizes a pleasant stroking paradigm, with touch typically delivered at velocities between 0.3–30 cm/s and forces about 0.4 N. These values are derived from perceptual and physiological responses of touch receivers rather than natural behaviors of touchers. Herein, we observe untrained touchers in delivering pleasant touch as gentle strokes to a receiver’s forearm through high-resolution force and position measurement using an instrumented brush. Twenty participants delivered about eleven strokes in each of five trials, yielding data on stroke velocity, force, duration, and length. The cohort deployed forces (0.37 N ± 0.24, 2 SD) near 0.4 N, but stroking velocities (13.7 cm/s ± 8.8, 2 SD) slightly higher than typically considered “optimal” (1–10 cm/s). We observed no correlation between stroke force and velocity, which led us to consider these factors jointly in characterizing an individual’s brushing strategy. Moreover, while the cohort of participants exhibited a compact range of forces and velocities, individuals tended to occupy only subsets of this range with high repeatability across trials. Altogether, the findings suggest a need to further evaluate the velocity range between 10–30 cm/s and to jointly consider force, velocity, and consistency in characterizing an individual’s brushing strategy.

(Introduction) Affective touch typically refers to slow moving, low-force mechanical stimuli, exemplified by gestures such as a gentle caress or comforting stroke that carry emotional and social meaning, extending beyond the mere detection of physical properties of stimuli [1], [2], [3]. Affective touch often, but not always, overlaps with social touch, which occurs in interpersonal contexts and is influenced by relational factors including familiarity, intimacy, and cultural norms [4], [5]. Research into touch receivers’ perceptions of affective touch has sought to determine what makes a touch “pleasant” and has considered inherent traits between participants, including relationships [4], [6], [7], [8], the touch receiver’s sex [9], age [10], [11], culture [5], [12], skin site [13], [14], and touch delivery via person or robot [13], [15]. In an effort to understand what makes affective touch pleasant, researchers have hypothesized that alignment between the intent of touch delivery and its interpretation by a touch recipient plays a large role [2]. Thus, to understand how the intent of touch delivery is communicated, works have evaluated how cues at physical contact—such as force, velocity, temperature, contact area, duration, and frequency [16], [17], [18], [19]—influence social and emotional perception [20] and evoke neural responses [21], [22]. Notably, even slow, gentle stroking delivered by a stranger has been shown to elicit predominantly positive affect, suggesting a potential universal language of touch that operates independently of contextual or relational factors traditionally considered essential to affective touch [23].

Pleasant touch, the perception of received touch as positive affect, sits at the intersection of affective and social touch. Studies have shown that pleasant touch can enhance physiological regulation, which contributes to stress reduction, interpersonal bonding, and social wellbeing [24]. The delivery of pleasant touch has received particular attention, as it is thought to engage a specialized class of unmyelinated low-threshold mechanoreceptors in hairy skin, C-tactile (CT) afferents, that respond optimally to gentle, caress-like stroking [3], [19]. A common experimental paradigm to research factors contributing to pleasant touch involves stroking the skin with a soft brush [13], [14], [15], [19], [20], [21], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34]. As a substitute to skin-to-skin stroking, the use of a brush can remove variability in skin softness, hand size, and temperature. The brush also more readily affords instrumented sensing of force, position, and velocity, which are challenging to measure in skin-to-skin touch [15], [35]. Although stroking another person with a brush may not entirely reflect the way people touch one another in natural contexts, this controlled approach has been invaluable in studying pleasant touch.

One of the most influential findings from this line of research has been the identification of an inverted U-shaped relationship between stroking velocities and pleasantness ratings, which mirrors the firing profile of CT afferents, with peak responses in the range of ~1 – 10 cm/s and forces in the range of 0.2 – 0.4 N [19]. The parameters of force and velocity have become a de facto standard for delivering pleasant touch, with an “optimal range” of 1–10 cm/s at about 0.4 N.

Force and velocity have been emphasized because they are both salient and intuitive to receivers. Touch can be readily described as “fast” versus “slow,” or “hard” versus “soft.” For the touch deliverer, they also represent straightforward adjustments, i.e., to change velocity, one moves faster or slower; to change force, one presses harder or softer. This bidirectional clarity has contributed to their prominence in both perceptual and mechanistic studies of affective touch. Accordingly, the present study also focuses on these features. Yet critically, the six original experimental velocities (0.1, 0.3, 1, 3, 10, 30 cm/s) were selected arbitrarily to reflect “slow,” “average,” and “fast” stroking with a brush, rather than being empirically derived from naturalistic touch behavior [19]. Over time, these same six velocities have become the default in most stroking studies [18], [25], [27], [28], [31], with an emphasis on the “optimal range” of 1 – 10 cm/s. That said, velocities outside this range have not necessarily been deemed unpleasant, just less pleasant comparatively. Relying on these six velocities may foster a somewhat incomplete picture of how pleasant touch is naturally produced and experienced. Because these values originated from early experimental design choices rather than observations of real interactions, they may not capture the full range of motions people chose to deliver as pleasant.

In addition to the possibility of not fully capturing the preferred delivery of pleasant touch, these six velocities may mischaracterize our articulation of an “optimal range”, given notable limitations recently demonstrated at the level of individual participants. In particular, a review of five stroking studies found that the characteristic U-shaped relationship between velocity and pleasantness in the “optimal range” held for only 40% of individuals, revealing the masking effect of aggregate statistics [26]. This raises questions about which velocity ranges should guide experimental paradigms, given that some studies report positive pleasantness ratings even at 50 cm/s [36]. Advancing studies of touch delivery requires complementing perceptual research with observations of how untrained individuals naturally produce stroking gestures. Convergence between the kinematics of natural stroking patterns and pleasantness in an “optimal range” would provide evidence of receiver–deliverer attunement, a defining characteristic of affective touch.

Prior attempts to observe natural stroking behaviors have produced mixed results. For instance, some studies observing the velocity of untrained stroke delivery report velocities exceeding the CT-optimal range (e.g., 13.9 ± 4.0 cm/s, [37] and 3.9 – 27.7 cm/s [38]), whereas others more closely align [39]. Indeed, systematic observation of stroking-based delivery is necessary, not only to assess whether its dynamics resemble or diverge from skin-to-skin stroking, but also to evaluate whether the six commonly employed experimental velocities are naturally preferred by touch deliverers.

A related issue concerns the lack of standardization in brush-based stroking delivery paradigms. Existing studies differ in several facets related to brush type, whether strokes are delivered by humans [8], [15], [20], [21], [29], [32], [33], [34] or robots [13], [14], [19], [25], [27], [28], [32], how onset and offset of skin contact are specified, and how velocity is calculated. In many cases, human experimenters are asked to approximate stroking velocity with the aid of a metronome [33], [34], or a visual cue [29], and to practice force control on a scale [15]. While each method offers a degree of calibration, the precision and consistency of the resulting strokes have not been empirically validated. Without such validation, it remains unclear to what extent the delivery of experimental stroking is consistent and reproducible. Direct measurement of the physics of stroke delivery is essential to ground laboratory approaches in objective, naturalistic observation, while also offering a framework for experimental validation of stroking delivery.

The present study addresses this gap by quantifying the force and velocity of pleasant stroking as delivered by untrained individuals with a brush. Specifically, we examine these parameters at both group and individual levels, compare them against canonical ranges, and evaluate individual consistency across repeated strokes. By capturing how people naturally apply brush strokes within a controlled setup, this study aims to provide an observational foundation for naturalistically valid, yet reproducible, models of pleasant touch delivery.

(Conclusion) This study addresses a need to describe the characteristics of untrained human brushers in delivering affective touch with a soft brush. Through direct observation in the delivery of force velocity, brushstroke length, and duration, we examined characteristics of the aggregate population and individual brushers. The observed force range aligns with that used in prior stroking studies, while the velocity range is slightly higher than the “optimal range” of 1 – 10 cm/s. This result suggests that future stroking studies may need to evaluate velocities at a higher resolution between 10 – 30 cm/s. Within brushers, we observed a high level of consistency in their deployment of force and velocity, which suggests that individuals employ distinctive brushing strategies.

Observations of higher brushstroke velocities
The cohort of 20 participants deployed stroke velocity with a mean of 13.38 cm/s and range of 4.87 – 22.70 cm/s, which is somewhat higher than the typically considered “optimal range” of 1 – 10 cm/s [14], [19], [25], [27], [32]. Likewise, other recent studies focused upon skin-to-skin contact with the hand have also reported higher ranges of velocity [37], [38]. For example, one study observing gentle stroking differences based on a receiver’s relationship to the touch deliverer reported a velocity range almost identical to that identified herein, 3.9 – 27.7 cm/s [38]. Likewise, a mean velocity of 13.9 cm/s was observed between mothers and their 1 – 12 month old infants [37]. It is important to note that most of this prior work evaluated the velocity of stroking in skin-to-skin touch, with comparatively little observation of velocity in brush-delivered touch. Despite the differences in delivery method, the range of velocities observed resemble one another [37], [38].

That said, with the agreement in velocities exceeding the “optimal range” of 1–10 cm/s, a finer-grained investigation of pleasantness at higher velocities (10–30 cm/s) is warranted. Notably, in the present study, the highest mean velocity recorded by an individual brusher was 20.33 cm/s, which was still rated as pleasant. This finding aligns with the idea that the range of 1 – 10 cm/s represents a subset, rather than an exclusive, range of pleasantness. A study utilizing brushing stimuli similarly found that stroking was rated as pleasant even at 50 cm/s [36]. Furthermore, the apparent “U-shaped” relationship between pleasantness and velocity may, in part, reflect averaging across individuals, as suggested by a recent review of five stroking paradigm studies in which only 40% of participants exhibited the canonical U-shaped pattern [26]. Thus, a focus on brush velocities in the “optimal range” of 1–10 cm/s may represent the recipient’s perspective, whereas incorporating naturally observed patterns of deliverers may enhance the ecological realism of affective touch paradigms.

Notably, in the present study, the highest mean velocity recorded by an individual brusher was 20.33 cm/s, which was still rated as pleasant. This finding aligns with the idea that the range of 1 – 10 cm/s represents a subset, rather than an exclusive, range of pleasantness. A study utilizing brushing stimuli similarly found that stroking was rated as pleasant even at 50 cm/s [36]. Furthermore, the apparent “U-shaped” relationship between pleasantness and velocity may, in part, reflect averaging across individuals, as suggested by a recent review of five stroking paradigm studies in which only 40% of participants exhibited the canonical U-shaped pattern [26]. Thus, a focus on brush velocities in the “optimal range” of 1–10 cm/s may represent the recipient’s perspective, whereas incorporating naturally observed patterns of deliverers may enhance the ecological realism of affective touch paradigms.

Observation of brushstroke forces confirm current practice
We found that participants applied force magnitudes of 0.37 ± 0.24 N, consistent with typical ranges of 0.2 – 0.4 N [14], [18], [19], [28], [30]. We note that we calculate force magnitude as an aggregate of forces along X, Y, and Z axes. In contrast, most prior studies have derived force from the normal force calibration of a rotary tactile simulator [14], [19], [25], [27], [32], i.e., a brush attached to a single motor to produce a sweeping motion. Such normal force corresponds to Z-axis force of the instrumented brush used herein. Since our experiment accounts for an aggregate of forces along the X, Y, and Z axes, it is reasonable that our reported force range is slightly higher, as it also includes lateral forces (Fig. 1C).

Moreover, the alignment of our force measurements and typical ranges may be attributed, in part, to the properties of the brush bristles. Similar to von Frey monofilaments, brush bristles act as a force-limiting mechanism, preventing excessive force application beyond a threshold [43]. A study examining the perception of brushing from various brushes, found that different brush types (goat hair, horse hair, plastic bristles) deliver varying force values, even when stroked in a similar manner [15]. This suggests brush stiffness naturally constrains force application, contributing to experimental consistency, offering a simple way to modulate force delivery.

Human Brusher Repeatability and Robotic Delivery
Within the delivery strategies of individuals, brushers demonstrated repeatable stroke delivery across trials – as indicated by low coefficients of variation, non-significant results in Levene’s tests, and high ICC values for both force and velocity – without formal training. While the degree of precision varies between individuals, as observable in the relative variability in their cluster sizes, each participant’s variability was characteristic and internally consistent. For example, a highly precise brusher (Fig. 6: small cluster, Participant 4) and a less precise brusher (Fig. 6: large cluster, Participant 1) each could readily replicate their unique cluster sizes between their 5 trials.

Moreover, in attempt to regulate the velocity and force of human delivered stroking, prior studies have used metronomes [33], [34] or moving visual bars [29] and pre-training with force-calibrated scales [15]. Although we did not utilize or compare the precision of strokes with or without such aids, the results herein demonstrate that untrained participants are nevertheless consistent in their stroking delivery. Further, our study neither evaluated trained participants, nor their ability to deliver accurate force and/or velocity in accordance with a control condition. To address such questions, future studies might incorporate real-time measurement of stroking velocity and force to verify that commanded delivery parameters correspond with actual performance.

Beyond pre-study training and visual guidance to standardize touch delivery of human touch deliverers, rotary tactile simulators [13], [14], [19], [25], [27], [28], [32] and other robotic systems [31] have sought to enable repeatable touch delivery. As with human brushing, robotic systems are not immune to delivery errors, and their performance has rarely been directly compared with human-delivered stroking. While many rotary tactile stimulators record applied force and velocity, their sweeping actuation motion may introduce discrepancies between the commanded and actual dynamics and lack resemblance to human-delivered brushing. A recent study redesigned a robotic apparatus to incorporate a cord-driven mechanism to more closely replicate the natural trajectory of human motion [31]. This sort of systematic evaluation of robotic performance may reveal accuracy in reproducing target velocity and force profiles and approximating human stroking. In some cases, robotic touch may be overly consistent, or too “robotic.” The introduction of controlled variability may help robotic systems better emulate the natural dynamics of human touch in experimental contexts.

Gaussian Mixture Models for Touch Delivery Characterization
Within the delivery strategies of individuals, brushers demonstrated repeatable stroke delivery across trials – as indicated by low coefficients of variation, non-significant results in Levene’s tests, The lack of correlation between delivered force and velocity in pleasant brushing delivery was unexpected and, to our knowledge, has not been previously reported. Given their independence, we developed a two-dimensional clustering model to represent individualized brushing profiles within a force-velocity space (Figs. 5 and 6). Compared to traditional univariate methods, this approach offers a more nuanced framework for characterizing brushing strategies, capturing the complexity of individual behaviors across all five trials. Moreover, it provides interpretable visual outputs that facilitate direct comparisons between individuals.

To contextualize our approach, we drew inspiration from other multi-dimensional applications of GMMs in behavior classification, such as ambulatory monitoring, and individual path analysis for gesture recognition for telemanipulation [44], [45]. In ambulatory monitoring, GMMs have been used to classify behaviors such as falls using data from a single hip-worn accelerometer. For example, a 2006 study by Allen and colleagues could effectively classify postures and movements, e.g. sitting, standing, walking, standing-to-lying, etc. The study found that time-based features improved accuracy and reduced computational demand compared to frequency-based measures [44]. However, the resulting model relied on a 25-dimensional feature vector at each sample and a 32-mixture GMM, which limited direct interpretability of individual features. This “black-box” nature constrains transparency and makes it difficult to understand why specific classifications are made. While such models can detect behavioral differences, they often do not explain the underlying mechanisms driving those differences. In contrast, our focus on both populationlevel and individual clustering while limiting complexity allows us to extract meaningful insights about the features themselves, including force and velocity ranges for the participant pool, as well as individual-specific ranges and consistency. For future touch-focused GMM applications, care should be taken to ensure that model decisions are interpretable, and that the contributions of individual features are clearly understood and contextualized in relation to both the population and individual participants.

Importantly, GMMs are a form of soft clustering, making them particularly well-suited for observation discovery for behaviors where multiple solutions are valid. A similar challenge arises in robot-assisted surgery, where complex procedures can be executed in multiple ways. Haptic feedback is often integrated into robotic systems to guide the surgeon along a predefined trajectory or provide interactive assistance. To better understand when and how this feedback should be deployed, Pérez-del-Pulgar et al. applied GMMs to break down a multi-path task, inserting a peg into a hole, into stages, revealing the numerous valid strategies participants can use and identifying patterns that inform stepwise feedback [45]. The resulting GMM encoded surface contact patterns (e.g., up–down, left–right), enabling the system to detect consistent interactions across multiple trials and prioritize cues for haptic guidance. Observation-based experiments such as this and our study benefit from GMMs’ soft clustering, which allows patterns to emerge from complex, high-dimensional data without enforcing rigid classifications. In our touch delivery study, just as in robotic surgery, multiple brushing strategies may be equally effective, corresponding to individual clusters, while commonalities across participants reveal shared tendencies, reflected in our identified cluster range. By leveraging soft clustering, GMMs enable us to capture both individual variability and population-level agreement, revealing the structure of force and velocity patterns across participants. This approach allows meaningful patterns to be extracted from naturalistic behavior, providing insight into how untrained individuals deliver touch, all derived from direct observation rather than imposed experimental constraints.

ThingMoji: User-Captured Cut-Outs For In-Stream Visual Communication

(Abstract) Live streaming has become increasingly popular, driven by the desire for direct and real-time interactions between streamers and viewers. However, current text-based interactions and pre-defined emojis limit expressiveness, especially when referring to specific stream moments. We propose ThingMoji, a type of user-captured cut-outs to enhance user expression and foster more effective communication between streamers and their audience in the comment section. ThingMojis are unique digital icons created by users by capturing snapshots and annotating specific areas at any point during the stream. We developed StreamThing, a live-streaming platform integrated with ThingMojis, to explore their use during object-focused live streaming contexts. In a user study with three in-the-wild deployments reveals the expressive use of ThingMojis in diverse live-streaming scenarios with rich visual contents. Our findings show that ThingMojis enable viewers to reference specific objects, express emotions, and create shared visual narratives. Streamers found ThingMojis valuable for facilitating on-the-fly communication around visual content and fostering playful interactions. The study also uncovered challenges in ThingMoji comprehension, issues for long-term uses of ThingMojis, and potential concerns regarding misuse. Based on these insights, we discussed new opportunities for supporting object-focused communication during live streaming environments.

(Introduction) Live streaming has become an integral part of online communication and is used across various domains such as gaming, crafting, and e-commerce [51]. What sets live streaming apart from other digital media is its real-time, dynamic nature, where streamers and audiences engage in a real-time exchange, fostering a sense of community and participation [4, 12, 19, 24, 25]. Viewers can comment and ask questions during the stream, allowing streamers gauge viewers’ interests and dynamically adjust the content and direction. Consequently, viewer-streamer communication has become a key aspect of the streaming experience, moving beyond passive video consumption.

A critical challenge in live streaming emerges from the inadequacy of current communication tools — predominantly text chat and pre-defined emojis [4] — in supporting precise object-focused discourse. While viewers frequently need to reference specific objects, moments, and spatial relationships within streams [17, 50, 62, 64, 72, 81], existing tools often fall short in facilitating these nuanced interactions. These communication challenges are especially evident in streams with complex visual content. For instance, when a jewelry maker demonstrates intricate wirewrapping techniques or when a model builder showcases various assembly components, viewers struggle to precisely indicate which elements they’re discussing. Similarly, in e-commerce streams where multiple products are displayed simultaneously [72], viewers often resort to ambiguous temporal references (“the necklace you showed earlier”) or imprecise spatial descriptions (“the blue piece on the left”). Such descriptions not only consume time to compose but frequently lead to miscommunication and reduced engagement between streamers and viewers. Such descriptions lack the details and can be time-consuming to write and difficult for other viewers and the streamer to quickly understand, resulting in communication gaps and reduced engagement [73].

Previous research has attempted to address these referencing challenges through various visual annotation approaches [49, 73]. These systems [49, 73] have integrated pre-defined and free-form annotation tools for users to attach snapshots and sketches with chat-based comments, aiming to provide a more direct and visual form of communication. However, these snapshot-based methods with screenshot attachment can clutter the comment section and disrupt focus. These methods often result in coarse-grained area of interest identification and disjointed interactions, as they lack integration with textual communication. Moreover, while these approaches have shown promise in creative live streams like visual arts [49, 73], they struggle with object-focused communication tasks [41, 50, 56], e.g., hands-on activities like maker projects, crafting, and model building, e-commerce interactions involving shared physical objects, and specific gaming streams that emphasize tracking objects of interest rather than general areas of interest.

We believe that addressing these challenges and facilitating fluid visual communication of objects in live streams is critical to ensuring effective communication between viewers and the streamer. To achieve this goal, our work draws inspiration from how digital icons and emojis have successfully enhanced expressivity in online communication [16, 63]. While emojis excel at conveying emotions and simple concepts succinctly, current live streaming platforms limit viewers to pre-defined sets of emojis that cannot capture or reference the dynamic visual content being shared [49]. This suggests an opportunity: combining the communicative efficiency of emojis with the ability to reference specific objects and moments from the stream.

Our work investigates the use of user-captured cut-outs, named ThingMojis, designed for object-focused communication. Unlike typical pre-defined emojis, ThingMojis allow users to efficiently communicate context in text messages while preserving the natural flow of conversation. ThingMoji offers a unique way to enhance live-stream interaction by representing key objects of interest. It is object-based, capturing items identified by viewers; context-based, preserving the whole imagery from the stream and allowing users to create narratives enriched with contextual meanings; and time-based, turning selected live-stream moments into expressive cut-outs aligned with the video timeline.

With design principles derived from prior literature, we developed a live streaming system, StreamThing, as a prototype to investigate the use of ThingMojis during object-focused live streaming contexts. StreamThing enables viewers to efficiently create ThingMojis by brushing over objects they want to reference in the live video. These ThingMojis can then be seamlessly embedded into chat messages like regular emojis, carrying with them the rich context of when and where they were captured in the stream. The system also helps viewers and streamers keep track of these visual references through a timeline that shows when different ThingMojis were used throughout the stream. We conducted a deployment study with three streamers and 29 viewers using StreamThing. Our investigation focused on understanding how viewers create and use ThingMojis, howthese visual elements integrate into stream communication, and how their meaning and utility evolve over time. The findings demonstrated that viewers used ThingMojis as an expressive visual language channel for referencing and self-expression, while also revealing important challenges in comprehension and organization that inform future design of similar systems.

In summary, our contributions include:
The design and implementation of ThingMoji in a live stream platform that streamlines referencing and communicating spatial and temporal contexts in live video.
Results from the deployment study about the use of ThingMoji for in-stream communication during object-focused live-streaming contexts.

(Conclusion) Weintroduce ThingMoji, a user-captured cut-out designed to enhance streamer-viewer communication. ThingMojis serve as a powerful tool for spotlighting, condensing, and summarizing crucial contextual information related to object-oriented elements across various application scenarios, including creative crafting, e-commerce product presentations, and gaming. Our user study with streamers and viewers revealed promising use cases for ThingMojis across diverse live-streaming contexts, with design implications for the continued exploration of the ThingMoji concept. We anticipate that ThingMoji will pave the way for novel opportunities in object-focused sharing within the realm of live streaming, sparking further exploration of such design principles within the HCI community.

Exploring Older Adults Personality Preferences for LLM-powered Conversational Companions

(Abstract) Studies have shown promise in using conversational agents (CAs) to reduce loneliness among older adults. Recent advances in large language models (LLMs) have enhanced these systems by enabling more natural, human-like interactions. However, little is known about how CA personalities influence user experiences, despite personality being a key factor in human conversation. To explore this, we developed a smart speaker-based CA powered by an LLM and conducted a two-phase user study consisting of in-lab sessions and home deployments. We investigated how older adults perceive different CA personalities and how these personalities affect their interaction experiences. Our findings show that participants could distinguish between different personality characteristics and had varying preferences for different personalities during both shortterm and long-term interactions.

(Introduction) Studies have shown that older adults are generally more susceptible to loneliness than other age groups, due to loss of social connections, declining health and mobility that limit social activities, and reduced interactions with family and friends [13]. A study also showed that loneliness has been linked with poor health and wellness outcomes for older adults [4]. As a result, there is growing interest within the research community in developing interven-tions to address loneliness in this population. One promising area of research in Human-Computer Interaction focuses on conversational agents (CAs) to alleviate loneliness among older adults. Research has shown that such CAs can reduce feelings of loneliness by providing easily accessible social interaction, especially for older adults who are more likely to have limited opportunities for social engagement [8]. Multiple studies have shown some promise in the potential of companion CAs in reducing loneliness among older adults [2, 20]. As this is an emerging field, most prior studies have focused on exploring the use of various CAs by older adults, including voice assistants like Amazon Alexa [33] and chatbots such as ChatGPT[1]. Limited research has explored how personalizing CAs can enhance user experience, with most existing studies focusing on contextualizing agent responses based on user information like preferences or routines [2]. However, many important aspects of personalization remain underexplored. One such aspect is the CA’s personality, as psychological studies show that personality traits play a significant role in shaping communication patterns [25].

Building on these insights, this work aims to explore the effects of different personalities of Large language model (LLM)-powered CAs on the user experience and perceptions of older adults to answer the following research questions:
RQ1: How do older adults perceive different personality traits in LLM-powered voice conversational agents?
RQ2: What are the observed effects of different personality traits in LLM-powered voice conversational agents on older adults’ experience with using a conversational agent?

To answer these research questions in a naturalistic setting, we implemented a smart speaker device that includes a mini PC, speaker, and microphone sothat users can useit intheir homes, similar to the setup used in a previous study on the use of CAs [1]. We also developed a CA program that uses an LLM for processing the conversations. The CA system we developed supports three distinct personality profiles, each characterized by a dominant trait: Extroversion, Agreeableness, or Conscientiousness. To understand how CA’s personality traits affect older adults’ perceptions and experiences, we conducted a two-phase user study. In the first phase, older adults participated in an in-lab session, engaging in a 10-minute conversation with each personality variant of the agent. The second phase involved a deployment study, where the device was set up in participants’ homes for a 12-day period. Throughout the study, we examined participants’ perceptions and experiences with each personality type using a multi-method approach, including personality assessments, preference ratings, interviews, daily diary entries, and analysis of conversation transcripts between the older adults and the agents. We found that most participants preferred an Agreeable CAin shorter conversations while preferring an Extroverted CA for long-term use. We also found that Agreeableness can sometimes be perceived as sycophantic, while Extroversion may come across as insincerely positive. We discuss how these traits can be balanced to make them more suitable for conversational companions. We then offer insights into potential next steps that can help achieve our goal of designing the ideal conversational companions for older adults.

(Conclusion) We observed that personality perceptions and preferences varied significantly among participants, with interesting differences observed across the contexts of the Lab Study and the Deployment Study. For example, participants who favored a particular personality during the short conversations in the Lab Study sometimes expressed a preference for a different personality during the longerterm Deployment Study. Phase 1 and 2 had 5 and 3 participants respectively, and while this is not a large enough sample size to understand participant experiences in depth, the findings from this study provide insights into how different personalities can be balanced to better suit the role of conversational companions for older adults across different contexts.

The next natural step in our research is to conduct a more thorough study with more participants and a longer deployment. These future investigations could also explore how the personalities of older adult participants influence their interaction experiences with CAsexhibiting distinct personalities, as previous studies have found relationships between participant characteristics and their perceptions of CAs [6, 35, 36, 38]. The findings from in-depth investigations will inform the design of a conversational agent as a conversation companion for older adults. This will allow us to assess how effectively these interactions help older adults reduce loneliness. Another promising direction could be identifying and tailoring a personality for a conversational agent, and then assessing the effectiveness of the CA’s interactions with the older adult participants in mitigating loneliness. This could be achieved by conducting pre and post-study loneliness assessments for over an extended deployment period.

Additionally, drawing from prior research and our observations that older adults value responses contextualized by previous interactions, we can consider developing a dynamic system capable of adapting its personality based on retained context. This approach would reduce the possibility of older adults having to frequently adjust personality settings, minimizing cognitive load to ensure a smoother user experience. Such a system has the potential to enhance both the usability and the emotional connection between older adults and their conversational agents.

Our eventual goal is to design personalized CAs for older adults that can alleviate loneliness and encourage or facilitate socialization with other people. These CAs should be seen as complementary tools rather than substitutes for human companionship. In this paper we take the first step toward that goal, presenting preliminary work from a multi-phase qualitative study on evaluating different induced personalities in LLM-powered CAs for older adults. We present initial findings on how older adults perceived and interacted with the LLM-powered CAs, each featuring a distinct induced personality. Based on our insights, we give recommendations on the design of personalized companion CAs for older adults and outline possible next steps to achieve our goal.

Your Hands Can Tell: Detecting Redirected Hand Movements in Virtual Reality

(Abstract) In-air hand interactions are prevalent in Virtual Reality (VR), and prior studies have shown that manipulating the visual movement of the hand to be different from the actual hand movement, i.e., hand redirection, could create a more immersive and engaging VR experience. However, this manipulation risks degrading task performance and, if maliciously applied, poses a threat to user safety. Such manipulations may arise from VR applications developed with intentional or inadvertent perceptual manipulations that yield harmful outcomes. We advocate for a user’s prerogative to be informed of any such potential manipulations before application usage. To address this, our study introduces an Autoencoder-based anomaly detection technique that leverages users’ inherent hand movements to identify hand redirection, thereby preserving the integrity of application use. Our model is trained on regular (i.e., non-manipulated) hand movement patterns and employs a stochastic thresholding approach for anomaly detection. We validated our method through a technical evaluation involving 21 participants engaged in reaching tasks under manipulated and non-manipulated scenarios. The results demonstrated a high accuracy of hand redirection detection at 93.7%, with an F1-score of 93.9%.

(Introduction)  Over the past few years, Virtual Reality (VR) has gained a significant rise in popularity, driven by advancements in high-fidelity 3D visual displays and 6-degree-of-freedom (DOF) tracking technologies. These developments have expanded VR’s application across diverse fields, including entertainment, education, and work-related tools [4, 28, 47, 54, 56]. VR’s unique capability to immerse users in a completely virtual environment, distinct from the physical world, offers unprecedented opportunities for enhancing user experiences. One of the ways VR utilizes to enhance user experiences is perception manipulation, which leverages visual dominance to manipulate users’ perception by subtly altering the perceived reality [19]. Hand redirection is one of perception manipulation techniques to manipulate perception, such as diverging the visual representation of hand movements from their actual trajectory [5, 13, 31], creating illusions of weight [45], and altering perceived object size [2], have been extensively studied. Additionally, the use of dynamic movement gains to improve ergonomic interactions within VR environments has also been explored [55].

Despite their promise, these dynamic manipulation techniques may inadvertently impact the precision of skilled motor movements, which are developed through extensive repetition. Humans rely on both visual and kinesthetic feedback to refine and optimize their movements, learning to adjust based on the disparity between intended and actual movements [33, 38, 46, 50]. The acquisition of such motor skills and the development of muscle memory is crucial for performing tasks with increased speed, consistency, and stability, with minimal cognitive load [34]. However, the dynamic nature of perception manipulation, which alters the mappings between virtual and real hand movements, poses challenges to the development of such automaticity. Moreover, there are concerns that perception manipulation could be maliciously exploited to harm users, with potential adversaries including VR developers who might intentionally or unintentionally introduce harmful manipulations [51]. Instances of such exploitation have been documented, including manipulation of VR safety boundaries to control user movement without their awareness [10]. These studies show that perception manipulation can be harmful and pose significant risks when used maliciously. Such manipulations may degrade performance, compromise user safety, or exploit vulnerabilities. Detecting these manipulations is challenging, especially when developers do not disclose their code or inform users about the application’s manipulation state. To protect users from such manipulations, it is crucial to detect such manipulations through alternative means. In this work, we propose a semi-supervised anomaly detection approach that detects subtle changes in the user’s behavior during reaching tasks.

This work introduces a novel approach to detecting hand redirections, which is a type of perception manipulation in VR. We leverage natural hand movements within the theoretical framework of Woodworth’s two-component model of goal-directed movements, established in 1899 [57]. According to this model, goal-directed actions comprise an initial ballistic phase, directing the limb toward the target, followed by a corrective phase that adjusts for trajectory errors using visual feedback. Our observations indicate that the speed of users’ corrective movements varies in response to perception manipulation, a variation not present under normal conditions. This phenomenon occurs when perception manipulation is applied in a particular direction, causing users to adjust their hand movements in the opposite direction. One can easily notice this adjustment, which is evident in the hand movement trajectories shown in Figure 1 and Figure 5. This observation forms the basis of our hypothesis that the characteristics of hand movements during corrective phases, such as speed, acceleration, and jerk, can serve as indicators to differentiate between manipulated and normal movement patterns.

To test our hypothesis, we developed a semi-supervised machine learning model based on an Autoencoder (AE) [6, 15] for anomaly detection. The key aspect of using Autoencoder is that we do not need to train the model on manipulated data. Instead, the model is trained solely on the user’s normal movement data, identifying anomalies in movement patterns during testing. In other words, by leveraging on the normal movement data, the model distinguishes between manipulated and non-manipulated instances by learning to reconstruct normal movement patterns, with the premise that anomalies—manipulated movements—will exhibit higher reconstruction errors [6, 62]. This approach leverages the assumption that normal instances are more accurately reconstructed from learned representations, while anomalies deviate significantly, resulting in higher reconstruction errors. The determination of an anomaly is contingent upon a reconstruction error exceeding a predetermined threshold. To optimize this threshold, we conducted multiple evaluations of our model, selecting the optimal threshold based on the differentiation between normal and manipulated data.

The efficacy of our approach was validated through a user study involving 21 participants, where we collected hand movement data in VR under both normal (non-manipulated) and manipulated conditions. Our study design incorporated orientation manipulation at two levels (±10 ° ) to simulate manipulated movement conditions. Data collection was structured into three sessions: an initial session to establish baseline hand movements (S1), a second session to collect data under manipulated conditions (S2), and a final session to gather additional baseline data (S3). The model was trained in a semi-supervised manner using only S1 data to learn regular hand movement patterns, with S2 and S3 data serving to evaluate the model’s ability to accurately detect manipulated instances as anomalies and baseline instances as normal. Our findings, derived from a within-subject analysis employing a 70-30 train-test split, demonstrate that our model achieves an anomaly detection accuracy of 97.7% (F1-score: 98.1%). Further validation through leave-one-participant-out (LOPO) cross-validation confirmed the robustness of our model in detecting anomalies across new users, with an accuracy of 93.7% (F1-score: 93.9%).

The contributions of this paper are twofold:
Introduction of a novel deep learning approach for detecting the presence of hand redirection in VR from user’s hand movement behavior. To our knowledge, this is the first method to automatically detect the hand redirection in VR from the user’s behavior data.
The dataset of manipulated and non-manipulated hand movement data1 from VR tasks, addressing the absence of publicly available datasets for hand redirection research.

(Conclusion) This study introduced a semi-supervised AE model for detecting hand movement manipulations in VR environments, leveraging velocity and acceleration profiles from users’ hand movements. Through a comprehensive analysis, employing both within-subjects and LOPO approaches, the model demonstrated high accuracy (93.7%) in identifying anomalies, underpinned by the careful selection of an optimal anomaly threshold. Despite facing limitations such as the absence of SSs in certain instances and the study’s duration, the research highlights the model’s potential to enhance VR security by preemptively identifying and warning users of manipulative applications. Future investigations will aim to broaden the model’s applicability by incorporating varied manipulation types and extending the observational period. This work lays a foundational step towards developing robust, user-centric anomaly detection systems in VR, promising a safer interactive experience for users across diverse VR applications.

Diversifying Grain-Based Compliance Illusion by Varying Base Compliance

(Abstract) Grain-based compliance illusion mimics the mechanical vibrations that occur when a compliant object deforms with grain-like, short (∼15 ms) impulse-response vibrations. Previous work has demonstrated its robust effect on various types of devices. However, the impact of the device’s inherent compliance (i.e., base compliance) on perceived compliance remains unclear. This paper investigates the influence of base compliance on the perception of illusory compliance through three psychophysical experiments. The results show that (1) the compliance illusion remained effective with base compliance, (2) the description of compliance was affected by both illusory and base compliance, and (3) it is possible to render the compliance with the same magnitude but multiple different feelings.

(Introduction) It is crucial for people to feel the mechanical properties of an object, such as its softness and elasticity, as well as its behavior, to conduct dexterous manipulation in the real world, as it allows people to understand the material and the status of the object. For humancomputer interaction, haptic interfaces render these properties to allow users to manipulate 3D models effectively [39], precisely conduct teleportation [19], and increase the immersiveness of the virtual experience [32]. However, rendering such mechanical properties often comes at the cost of an expensive, large, and grounded device that uses mechanical or magnetic mechanisms to alter the physical properties of the haptic interface. These constraints limit their portability and usability in mobile scenarios.

To address this limitation, many illusion-based techniques have been developed to alter users’ perception rather than the physical properties of the interface. One widely used method is the visuohaptic illusion, which leverages the visual dominance of human perception. By manipulating the visual representation of an object, these techniques can make users perceive changes in stiffness [2, 48], shape [1], and weight [38]. Although effective, these methods depend on users’ visual attention and only work when the object is visually rendered and users are looking at the object. In contrast, tactile illusions create sensations of movement [6, 35] or deformation [11, 16, 21] without requiring structural changes in the object. These illusions rely on tactile actuators, such as vibrotactile mechanisms, to simulate different material properties through direct touch. This approach enables a more natural sensation, as users feel the object directly with their hands, akin to real-world interaction. Moreover, tactile illusions do not require users to look at the object, freeing their visual attention to focus on other objects or tasks.

Among tactile illusion techniques, grain-based compliance illusions have gained significant interest within the HCI community in recent years. These techniques are inspired by the psychophysical study conducted by Srinivasan and Lamotte [40], which demonstrated that participants could successfully discriminate between objects of varying stiffness using only tactile sensations. Building on this insight, researchers developed techniques utilizing grainlike short vibrations that mimic the tactile feedback caused by movement or deformation in response to force input. These studies revealed that such vibrations could create the illusion of compliance without actual physical movement or deformation [15, 16, 20, 21, 23]. Moreover, studies have shown that the properties of these illusions can be programmatically manipulated. For example, altering the virtual displacement threshold for triggering grain vibrations changes the perceived stiffness, while capping feedback for forces exceeding a predefined threshold restricts the sensation of further movement or deformation. This programmability has enabled grain-based compliance illusions to render tactile buttons with customizable force-displacement characteristics [22, 23] and virtual joints with varying movement constraints [16]. The effectiveness, adaptability, and simplicity of grain-based compliance techniques—requiring only a force sensor and a vibrotactile actuator—may explain their growing popularity in HCI research. These techniques have been demonstrated to enhance the precision [25] and realism [16, 41] of VR experiences, support high-fidelity tangible user interfaces [37], and facilitate rapid prototyping of tangible interfaces [44].

While grain-based illusion techniques have shown promising results in creating the illusion of compliance on a rigid surface enhancing the usability and user experience of interfaces, the setups used in these studies often involve an unclear parameter that may significantly influence the perception of compliance—the rigidity of the so-called rigid object. Materials that we consider rigid, such as plastic or steel, have some level of compliance. Given that grain-based illusion methods utilize force as input, devices used in previous studies may have a different level of deformation during the study. For example, some of the previous studies used acrylic plates under the finger [15, 25], 3D printed blocks [23] and handles [16], and an unknown rigid material [21] that vary in different sizes. In these studies, while the surface itself may not locally deform around the finger pad to alter the pressure distribution in the area of contact to make it feel as soft [27, 34, 42], the deformation of the base structure may have caused kinesthetic compliance (i.e., device being bent). This reveals two key issues: first, the diverse effects of base material compliance on perceived compliance properties have been largely overlooked; and second, this makes designers and engineers seeking to use grain-based illusion techniques to properly account for critical design properties and understand the resulting illusion.

In this paper, we investigate the effect of base compliance on the perception of grain-based compliance illusion, including the range of perceived compliance, Just Noticeable Difference (JND), and the perceived properties of compliance through a series of psychophysical experiments. We implemented a prototype device with a load cell and a vibrotactile actuator that plays short bursts of vibrations (i.e., grain vibrations) per a certain level of force changes to create a grain-based compliance illusion. From the top, the device consisted of a rigid finger-resting plate with a vibrotactile actuator, an interchangeable base compliance block, and a force sensor. By changing the base compliance block, we could investigate three conditions with different levels of base compliance while keeping the illusory compliance and the surface condition consistent.

Our results showed how base compliance affects perceived compliance. First, the grain-based compliance illusion remained effective with the base compliance. The softer the base compliance is, the softer the device feels with the compliance illusion. We found that the overall range of compliance illusion shifts a certain amount depending on the base compliance (Figure 1). Second, the existence and the level of base compliance affect the description of perceived compliance. While the illusory compliance dominated, the existence and the level of base compliance made perceived difference. The illusory compliance with more compliant base material was perceived lighter, softer, deeper, and more granular compared to the hard base material. Thirdly, we found a new design opportunities of compliance created by base × illusory compliance. Based on our finding in Experiments 1 and 3, by changing both base and illusory compliance, one can adjust both magnitude and feeling of compliance. In other words, one can create multiple feelings of compliance with the same magnitude of compliance. Finally, as a first investigation towards the unknown perceptual space created by base and illusory compliance, we provide design guidelines, baseline data in an interactive visualization, and areas to discover for future researchers.
The contributions of this paper are as follows:
The findings of three psychophysical experiments investigating the effect of base compliance on the perception of grain-based compliance illusion
The method of diversifying the range and the quality of perceived compliance by combining the base compliance and the illusory compliance
The findings on unknown perceptual spaces created by combining base and illusory compliance 

(Conclusion) With a prototype device with a rigid surface, interchangeable compliant blocks, and different levels of grain-based compliance illusion thresholds, we explored the effect of base compliance on the perception of grain-based compliance illusion, including the range of perceived compliance, JNDs, and the quality of compliance. The results showed that the grain-based compliance illusion worked robustly regardless of base compliance. Finally, we found that the existence of base material affects the quality of compliance using multidimensional scaling and adjective rating. We hope our findings foster future HCI research on rendering, augmenting, and quantifying grain-based compliance illusions.