Ultimate Interface Lab's 2024 Papers

MoiréTag: A Low-Cost Tag for High-Precision Tangible Interactions without Active Components

(Abstract) In this paper, we present MoiréTag—a novel tag-like device that magnifies displacement without active components for indirect sensing of subtle tangible interactions. The device consists of two overlapping layers of stripe patterns with distinct pattern frequencies. These layers create Moiré fringes that can move faster than the actual movement of a layer. Using a customized image processing pipeline, we show that MoiréTag can reliably detect sub-mm movement in real-time (mean error = 0.043 mm) under varying lighting conditions, camera angles, and camera distances. We also demonstrate five applications of MoiréTag to showcase its potential as a low-cost solution to capture and monitor small changes in movement and other physical properties, such as force and volume, by converting them into displacement.

(Introduction) With the advancements in computing technologies, computers are now more closely connected to the real world than ever. Computers sense and utilize interactions that happen in the real world, such as tapping and scratching of a surface [12, 13, 41], and interacting with physical objects [17], enabling tangible interactions in scenarios where traditional user interfaces are not available or not effective. Many of these are done through active sensors worn on the user’s body [17, 41] or attached to the object surface [12, 13]. Some use conductive material to cover the object surface to sense interaction with the surface [44]. While useful, these methods need wires and active components to be worn on the user’s body or active circuits to be instrumented on the object surface.

On the other hand, recent advancements in computer vision and machine learning algorithms enabled non-contact indirect sensing of real-world input using a camera. For example, MediaPipe [1] and DeepPose [34] allow human body posture and movement to be tracked with a regular 2D camera by using deep learning algorithms, and YOLACT allows sensing and tracking of objects in a video [3] without instrumenting the user or the object with active sensors. These vision-based methods have the limitation of precision and stability of the sensed movement, making it more suitable for sensing large-scale movements such as moving an arm or a hand. However, in the real world, people also utilize small movements, such as precisely aligning objects, and even tiny actions that may not be easily noticeable, such as applying a force on a physical object. Other studies have shown that using laser speckles can enable high-accuracy sensing of small movements, such as the movement of hand and object [45] and the movement caused by the force interaction [23]. However, methods that use laser speckle require a speckle projector and a speckle sensor [45] or a defocused camera [23].

In this paper, we present MoiréTag, a novel low-cost tag that enables camera-based sensing of precise movement from physical interactions. It utilizes the Moiré phenomenon, which is seen when two repetitive patterns with similar spacings are superimposed in the form of dark and light fringes. If one of the superimposed patterns moves, the Moiré fringe will also move, but at a different rate than the pattern movement. This allows the small displacement caused by the subtle interaction to be significantly magnified so that it can be easily captured by a regular camera. This Moiré phenomenon has been shown to be effective in various applications, such as pose [21] and camera position [37]. With the well-established theory framework of Moiré pattern as the basis, MoiréTag provides an easy way for people to use the Moiré phenomenon to create and use ad-hoc interfaces. MoiréTag consists of two overlapping paper layers that have stripe patterns with different grating periods that create Moiré fringes that serve as a displacement magnifier for a camera to precisely capture the displacement. We implemented an image processing pipeline that recognizes the MoiréTag from a camera image and detects the movement of the Moiré fringes. Our technical evaluation showed that MoiréTag and the image processing pipeline could enable the sensing of sub-mm displacement using a regular smartphone camera with an average error of 0.043 mm. The evaluation also showed that the displacement could be reliably detected in various lighting conditions and when captured from different camera angles and distances.

MoiréTags can be configured in various ways to be used for measuring different types of physical properties. For example, MoiréTags with a handle attached to the moving layer can be used as a tangible user interface for applications requiring multiple precise inputs, such as video editing. When coupled with an elastic material, MoiréTags can serve as a force sensor by magnifying the small displacement caused by the compression or extension of the elastic material. MoiréTags can also be attached to a strap wrapped around the human chest to measure chest volume changes during breathing. Our evaluation showed that the force and the breathing count can be accurately measured using MoiréTags.

In summary, the main contributions of this work include the development of MoiréTag, a novel tag device that enables low-cost and precise camera-based sensing of subtle changes in displacement for tangible interactions, and the design and implementation of force and volume change sensing mechanisms using MoiréTag.

(Conclusion) We presented a novel camera-based approach to detect small displacement changes in daily interactions by utilizing a low-cost passive tag that magnifies displacement using Moiré fringes. The validation experiments show consistent and accurate sensing of displacement under varying lighting conditions, camera angles, and camera distances. Additionally, we demonstrate the conversion of touch force and volume changes into displacement and the ability to sense them using repurposed MoiréTag prototypes and our image processing pipeline in practical applications. MoiréTag offers the following advantages over existing sensing methods:
Affordable and easy-to-make tag design. The prototype can be made with paper alone, using a printable template and machine-cut stripe patterns for the fixed layer. If the chosen grating period for the fixed paper layer is sufficiently large, it can also be printed and manually cut.
Robust sensing in various lighting and different camera placements. The customized image processing pipeline is able to detect displacement with sub-mm errors under different lighting conditions and from varying camera angles and distances.
Adjustable sensing range and sensitivity. The sensing range can be adjusted by changing the grating periods of the two overlapping paper layers of stripe patterns to meet the different sensing needs.
Versatile sensing capabilities. MoiréTag can be repurposed as a sensor for touch force or volume changes during respiration by adding an elastic block or a strap.

Like all camera-based sensing methods, MoiréTag requires uninterrupted visibility within the camera’s field of view, without any obstructions. Our validation results indicate that MoiréTag can function effectively with a camera angle of 60 degrees with a camera distance of 60 cm. A greater camera angle might cause the camera to fail in capturing the Moiré fringes. Using a lens with a wider field of view may enable detection with a larger camera angle. Moreover, an extended camera distance may render MoiréTag too small for detection or cause the Moiré fringes to become too blurry for tracking. However, these issues could potentially be mitigated by employing a MoiréTag with a larger size or a camera with a higher resolution.

In addition to the jittering of cropped Moiré fringes resulting from slight variations in contours detected by OpenCV across frames, another issue contributing to errors in displacement sensing is related to drastic camera movement and angle changes. The gap between two layers of stripe patterns may also affect the grating period of Moiré fringes, and may cause noticeable movement if the camera moves drastically. In our design of prototypes, we tried to avoid creating such a gap with the 3D-printed structures, and tapes for the paper prototype. If the camera angles change greatly, for example from left perpendicular to the tag to right perpendicular to the tag, the structure above the screen may affect the rectification of the fringes and thus lead to detected Moiré fringes movement. Lastly, it is important to note that if the camera moves too quickly, motion blur can potentially impact the detection and tracking of Moiré fringes, especially in low-light conditions. If Moiré fringes move rapidly, the image processing pipeline’s ability to detect displacement changes may be constrained by the frame rate. A future work will be needed to investigate the tracking performance in more dynamic use scenarios and to optimize the pipeline to improve the frame rate.

MoiréTag was designed to measure movement in only one direction. One potential future improvement that could be made to MoiréTag is the use of Moiré patterns with different shapes, such as circular ones, to enable displacement measurement in multiple directions. Another improvement could be creating Moiré fringes on multiple surfaces to allow for displacement detection from more camera angles. Overall, MoiréTag demonstrates the potential of using Moiré fringes to accurately detect small displacements and indirectly infer various physical properties during everyday interactions.

Experiencing Thing2Reality: Transforming 2D Content into Conditioned Multiviews and 3D Gaussian Objects for XR Communication

(Abstract) During remote communication, participants share both digital and physical content, such as product designs, digital assets, and environments, to enhance mutual understanding. Recent advances in augmented communication have facilitated users to swiftly create and share digital 2D copies of physical objects from video feeds into a shared space. However, the conventional 2D representation of digital objects restricts users’ ability to spatially reference items in a shared immersive environment. To address these challenges, we propose Thing2Reality, an Extended Reality (XR) communication platform designed to enhance spontaneous discussions regarding both digital and physical items during remote sessions. With Thing2Reality, users can quickly materialize ideas or physical objects in an immersive environment and share them as conditioned multiview renderings or 3D Gaussians. Our system enables users to interact with remote objects or discuss concepts in a collaborative manner.

(Introduction) Shared artifacts, such as physical objects, printouts, and digital images, play a crucial role in facilitating efective communication and idea generation [3]. They help bridge gaps between collaborators by providing a common spatial reference point and facilitating creative exploration [1]. In addition to physical artifacts, designers often use online platforms like Pinterest and Google to fnd relevant digital artifacts that can support their design processes [2]. However, using shared artifacts in remote meetings can pose several challenges, especially in scenarios that require quick and spontaneous sharing, such as brainstorming sessions. First, artifacts shared in remote meetings are often in 2D, whether they are captured using a camera or retrieved from an online repository [3]. These 2D representations may not provide the same level of understanding as interacting with a physical object or a 3D model. Second, in physical meetings, participants can easily rotate, manipulate, and interact with the artifacts, which can facilitate creative exploration and idea generation processes [1]. However, in remote meetings, this level of interaction with virtual artifacts generated on-the-fy is often unavailable or limited.

Several methods have been used to address these challenges. One is to prepare 3D models before the meeting by creating or retrieving CAD models, or by 3D-scanning an object [4]. Another is to use a special setup that can capture the physical world in real-time and reconstruct it in 3D [5]. While these methods efectively enable richer sharing of artifacts, they have their own limitations. For instance, using pre-made 3D assets does not efectively support the spontaneous sharing of objects, and using a special scanning setup may not be accessible for many people. On the other hand, recent advances in AI-driven text-to-3D and image-to-3D technologies [7] address the need for a more accessible and efcient way of creating and sharing 3D assets. These technologies can signifcantly lower the barriers to 3D content creation, enabling individuals without specialized skills to contribute to the co-creation process, thereby democratizing access to 3D modeling and enhancing collaboration.

To address the challenges of summoning spontaneous 3D representations into the existing information space, we seek to enable fuid communication in an XR environment comprised of paired 2D and 3D artifacts. In this demonstration, we present Thing2Reality, a distributed communication system that enables users to segment any content from any container (video streams, shared digital screens) in the XR environment (Figure 1a), explore the perspectives with multi-view renderings (Figure 1b), and transform them into shared 3D Gaussians (Figure 1c-d) for 3D manipulation.

(Conclusion) In this demonstration, we present Thing2Reality, an XR communication system that allows users to instantly materialize ideas or physical objects and share conditioned multiview renderings or 3D Gaussians for realistic 3D rendering. We believe that XR communication has tremendous promise for co-presence and for bridging distances between humans, and enabling the spontaneous creation and sharing of 3D objects and artifacts in XR will allow for a more fuid and efective exchange of ideas, beyond what is possible in real-world communication.

Enhancing VR Sketching with a Dynamic Shape Display

(Abstract) Sketching on virtual objects in Virtual Reality (VR) can be challenging due to the lack of a physical surface that constrains the movement and provides haptic feedback for contact and movement. While using a flat physical drawing surface has been proposed, it creates a significant discrepancy between the physical and virtual surfaces when sketching on non-planar virtual objects. We propose using a dynamic shape display that physically mimics the shape of a virtual surface, allowing users to sketch on a virtual surface as if they are sketching on a physical object’s surface. We demonstrate this using VRScroll, a shape-changing device that features seven independently controlled flaps to imitate the shape of a virtual surface automatically. Our user study showed that participants exhibited higher precision when tracing simple shapes with the dynamic shape display and produced clearer sketches. We also provided several design implications for dynamic shape displays aimed at enabling precise sketching in VR.

(Introduction) Leveraging spatial perception and 6-DOF manipulation, virtual reality environments provide designers with an immersive platform for 3D model interaction [3, 25]. Although typical mid-air interaction methods, such as using controllers and hands, facilitate fluid manipulations, they may not be able to support the natural and precise on-surface interactions essential for design tasks such as fine-line drawing [5]. This limitation stems from the lack of resistance and tactile feedback provided by physical contact, which is critical for accurately perceiving and manipulating virtual geometry [24].

To improve the performance of VR sketching, previous research has incorporated virtual surfaces with flat plates or tablets, which can rest the user’s hands, guide pen movement, and prevent strokes from penetrating or leaving the virtual surface [4, 5, 12, 13, 44, 48]. However, utilizing planar surfaces to represent non-planar shapes, including architectures, creatures, and characters, can be challenging because of the incongruities of their geometric features. While some systems enable interactions on non-planar surfaces by projecting the virtual surface on a touchscreen [13, 17] or a graphic tablet [4, 12, 29], the absence of geometry information, and the discrepancy between the tablet and the virtual shape might results in unintuitive [36], technically challenging, and learningdemanding [4, 6] design process in VR. 3D-printed physical proxies that exactly replicate a virtual object have also been exploited to improve drawing performance [48]. However, it may be impractical to fabricate such proxies for the diverse virtual objects users may interact with in VR.

To tackle this challenge of enabling precise sketching on virtual objects with various shapes, we propose using a dynamic shapechanging surface that can mimic the shape of virtual object surfaces. Many shape-changing devices, also known as shape displays, have been developed and shown to be effective for immersive in-hand interactions in VR [8, 20, 31, 39, 45, 49], however, their effectiveness in supporting precise interactions, such as sketching, has rarely been studied. To investigate how using a shape-changing device affects the precision, quality, and usability of sketching in VR, we developed VRScroll, a scroll-like dynamic shape display (Figure 1). VRScroll consists of seven 3D-printed flaps that are controlled by DCmotors to form a surface that mimics the surface of a virtual object in VR. We conducted a user study to understand the effect of having a physical surface using a dynamic shape display by comparing it with in-air sketching on planar and non-planar surfaces. The results showed that the participants could more precisely trace simple shapes and make smoother strokes. The free-form sketches participants made with VRScroll were rated to be clearer by Amazon Mechanical Turk workers. The participants also found sketching with VRScroll felt more realistic and natural.

In summary, the main contributions of this work include the design and implementation of VRScroll, a shape-changing device developedtosupportpreciseon-surfaceinteractions, and theresults from a user study that investigates the effect of a physical surface on the sketching performance on various surfaces.

(Conclusion) This paper proposed using a dynamic shape display to support precise sketching on virtual objects in VR. We developed VRScroll, a scroll-like shape-changing device designed to support on-surface pen interaction. Through a comparative user study, we investigated the effects of having a physical surface created using VRScroll on sketching performance. The study revealed that utilizing VRScroll could significantly improve the quality of sketches and provide a more realistic and intuitive experience. Based on our findings, we presented several design implications for equipping dynamic shape displays to facilitate precise sketching in VR. We hope our work will inspire future investigations into shape-changing interfaces, which may move beyond immersive shape manipulation and toward supporting more inventive and productive interactions in VR.

ViObject: Harness Passive Vibrations for Daily Object Recognition with Commodity Smartwatches

(Abstract) Knowing the object grabbed by a hand can offer essential contextual information for interaction between the human and the physical world. This paper presents a novel system, ViObject, for passive object recognition that uses accelerometer and gyroscope sensor data from commodity smartwatches to identify untagged everyday objects. The system relies on the vibrations caused by grabbing objects and does not require additional hardware or human effort. ViObject’s ability to recognize objects passively can have important implications for a wide range of applications, from smart home automation to healthcare and assistive technologies. In this paper, we present the design and implementation of ViObject, to address challenges such as motion interference, different object-touching positions, different grasp speeds/pressure, and model customization to new users and new objects. We evaluate the system’s performance using a dataset of 20 objects from 20 participants and show that ViObject achieves an average accuracy of 86.4%. We also customize models for new users and new objects, achieving an average accuracy of 90.1%. Overall, ViObject demonstrates a novel technology concept of passive object recognition using commodity smartwatches and opens up new avenues for research and innovation in this area.

(Introduction) Despite the massive advancements in technology today, there still remains a distinct divide between our physical world and the technological world. While most of us interact with electronic devices on a daily basis, we exist in twoseparate realms in a sort of symbiosis, where we benefit from the convenience of technology while technology builds and improves with our use over time. However, it is still easy to point at an object and identify which realm it belongs to, the technological world or our own. In this work, we explore a world heavily inspired by the idea of changing the world itself into an interface [34].

Daily object recognition is one of the technologies to help bridge the divide between our world and the technological world by using mundane objects as triggers for specific services. Attaching tags to objects has been widely proposed, where tags are used to retrieve information about the objects. QR codes, RFID tags [55], near-field communication (NFC) [22], and acoustic barcodes [29] have been utilized to recognize and automatically select the target service from the mobile devices. Vision-based solutions utilize computer vision and machine learning techniques to identify objects captured within the frame of the cameras [19, 35]. Capacitivo [58] and Tessutivo [23] recognized objects placed on customized fabrics, using capacitive sensing. Electromagnetic (EM) based sensing approaches require specialized EM sensors as well, and are applicable to only electrical appliances that emit EM signals [40, 53]. Other approaches also require customized devices to emit active signals (e.g., vibrator [46], millimeter wave[64], etc) to identify objects. Despite these advantages, customized devices reduce accessibility and thus their deployability is limited. Some acoustic-based approaches recognize objects through different sounds by knocking on different objects [24, 43, 51]. However, these approaches require users to have extra actions to indicate users’ intentions, which disrupts people’s daily activities and sets a boundary between the technology world and the physical world.

This paper introduces ViObject, a system designed for tangible interactions that harnesses passive vibrations—originating from the grasp of everyday objects—instead of active vibrations produced by vibrators, to enable object recognition. By capturing the unique passive vibrations that propagate through the hand when an object is grabbed, ViObject can identify untagged everyday objects. Leveraging the accelerometer and gyroscope sensors embedded in commodity smartwatches, ViObject can capture and process these grab-object-induced vibrations, enabling seamless and intuitive object recognition without requiring additional hardware or user effort.

One of the key advantages of ViObject is its focus on smooth integration with people’s daily lives. Unlike other techniques that require users to take extra actions, such as attaching sensors or taking pictures, ViObject reads data from users’ daily movements using a hand-worn smartwatch without disrupting their habits or daily routines. This facilitates borderless and fluid interactions between the technological world and our daily lives. Additionally, the popularity of commodity smartwatches and the development community surrounding them makes ViObject a practical and accessible solution for a wide range of applications.

The development of ViObject poses several challenges that need to be addressed. Firstly, the vibration signal generated by grabbing an object is relatively weak and can be overwhelmed by the hand’s movement signal, particularly with a limited Inertial Measurement Unit (IMU) sampling rate. Moreover, unlike active sensing approaches that use customized signals with specific frequencies as a signal source for sensing, the passive signals captured from grabbing objects are unmodulated and have varying frequencies, which makes detection and feature extraction more challenging. Additionally, the vibration signal induced when the user touches an object can vary depending on applied pressure and speed. Objects vary in shape and size, so the induced vibration signals also depend on object-touching positions. Lastly, new end users may need to identify new objects that are unique to their homes.

To overcome these challenges, we propose several techniques in ViObject. Firstly, we eliminate interference through signal processing and augmentation techniques. Secondly, we use interpolation to enhance samples and leverage attention-based residual networks to improve the system’s accuracy. Thirdly, we design an adversarial training regularization with center loss to mitigate the impact of orientation changes. Finally, we employ generalized few-shot learning with data synthesis for object customization, enabling the system to recognize new objects and users with minimal training data. These techniques enable ViObject to effectively recognize objects passively using smartwatches’ accelerometer and gyroscope sensor data, addressing new research and innovation opportunities.

The performance of ViObject was evaluated using data collected from 20 participants interacting with 20 different daily objects while wearing a smartwatch. The collected data was analyzed to assess the system’s basic performance, and the average recognition accuracy was found to be 86.4%. A subsequent experiment was conducted with 10 additional participants to test ViObject’s ability to recognize new objects customized by users. The results demonstrated excellent accuracy (90.1%) for new users and new objects, even when the objects were grabbed from different angles with varying pressure and speed, and different parts of the object were grabbed. The performance of ViObject was consistent across different smartwatches and over a week. An end-to-end standalone app was implemented on an Android smartwatch for object recognition in real-time, which included the Android TextToSpeech module to play the sound of the prediction. A user experience study was conducted to evaluate the app, and the results showed positive user feedback. Furthermore, ViObject achieved low latency and low power consumption (see section 5), making it a practical and efficient solution for everyday use.

This system has its inherent limitations. Numerous objects have yet to be tested in real-world scenarios, especially when objects bear close resemblance to each other. As the variety of objects increases, distinguishing between some may pose challenges. Nevertheless, ViObject strives to validate a new concept: recognizing objects from daily grasp actions. Moreover, most applications only need to employ a selective and manageable set of objects. For instance, recognizing a single pill bottle can facilitate medication reminders, while identifying a singular dumbbell can support fitness tracking. A curated set of daily objects could be instrumental in assessing Alzheimer’s disease progression. A few specified daily objects can also activate corresponding smart home services. In escape room scenarios, ViObject can detect when participants pick up a few particular objects, subsequently triggering clues or furnishing additional insights via the smartwatch to aid puzzle-solving. By identifying passive vibrations from daily object interactions, ViObject bridges the gap between our physical and digital realities, transforming the physical world around us into an interactive interface.
To summarize, our main contributions are:
To the best of our knowledge, we are the first to harness passive vibrations to recognize daily objects using the IMU sensor found in many common smartwatches. We also conducted a series of feasibility studies to understand the principle behind this new concept.
We havedesignedanoveltechnical process to eliminate interference (e.g., hand movement, grasp speed, and strength), extract fine-grained features from various object-touching positions using a transformer-based network, and customize the model for new users and new objects.
We have developed an end-to-end standalone system in commercial smartwatches, which can achieve realtime object recognition. Extensive experiments have been conducted to demonstrate ViObject’s accuracy, robustness, and user experience.

(Conclusion) In conclusion, we have introduced ViObject, a novel system for passive object recognition that utilizes accelerometer and gyroscope sensor data from commodity smartwatches to identify untagged everyday objects. Through our design and implementation process, we have successfully addressed challenges such as motion interference, grasp speed/pressure variations, object-touching position changes, and customization for new users and new objects. We have demonstrated the feasibility of passive object recognition using commodity smartwatches and shown promising results in terms of recognition performance and user feedback. ViObject has the potential to revolutionize the way we interact with our physical environment, from smart home automation to healthcare and assistive technologies. With the rapid development of smartwatch technology and its widespread adoption, we believe that ViObject will have a significant impact on the field of human-object interaction in the future.