UIST '20 Adjunct: Adjunct Publication of the 33rd Annual ACM Symposium on User Interface Software and Technology

Passive haptic feedback for virtual reality can provide immersive sensations but suffers from scalability issues. To tackle these, two independent concepts have been proposed previously: Dynamic Passive Haptic Feedback (DPHF), leveraging actuated props that change their physical state, and Haptic Retargeting, redirecting the user's hand during interaction. While past research on both techniques reported promising results, up to now, these concepts remained isolated. This paper advocates the combined use of DPHF and Haptic Retargeting. We introduce two thought experiments showcasing that the combination of both techniques in an example scenario is beneficial to solve two central challenges of prop-based VR haptics: haptic similarity and co-location of proxies and virtual objects.

Augmented reality (AR) is an emerging technology in mobile app design during recent years. However, usability challenges in these apps are prominent. There are currently no established guidelines for designing and evaluating interactions in AR as there are in traditional user interfaces. In this work, we aimed to examine the usability of current mobile AR applications and interpreting classic usability heuristics in the context of mobile AR. Particularly, we focused on AR home design apps because of their popularity and ability to incorporate important mobile AR interaction schemas. Our findings indicated that it is important for the designers to consider the unfamiliarity of AR technology to the vast users and to take technological limitations into consideration when designing mobile AR apps. Our work serves as a first step for establishing more general heuristics and guidelines for mobile AR.

While sighted users leverage both keyboard and mouse input devices to interact with desktops, non-visual users, i.e., users who are blind, cannot use a mouse as it only provides feedback through a visual cursor. As a result, these users rely on keyboard-only interaction, which is often cumbersome, inefficient, and error-prone. Prior work has shown that using a small, rotary input device benefits blind users significantly, as it simulates mouse-like operations. In this paper, we extend this prior work by proposing Wheeler, a multi-wheel based input device that provides simultaneous access to UI elements at three different hierarchies, to facilitate rapid navigation and mouse-like interaction. We designed Wheeler from scratch in multiple iterations and assembled it using 3D printed models and commercially available electronics. A preliminary user study with six blind-folded sighted users revealed its potential to become an essential input device for blind users, as well as a training and learning tool.

We present LightTouch, a passive gadget to enhance touch interactions on unmodified capacitive touchscreens. It simulates finger operations such as tapping, swiping, or multi-touch gestures using conductive materials and photoresistors embedded inside the objects. The touchscreen emits visible light and the photoresistor senses the level of this light, which changes its resistance value. By controlling the screen brightness, it connects or disconnects the path between the GND and the touchscreen, thus allowing the touch inputs to be controlled. In contrast to conventional physical extensions for touchscreens, our technique does not require continuous finger contact on the conductive part nor the use of batteries. Our technique opens up new possibilities for touch interaction such as for enhancing the trackability of tangibles beyond the simple automation of touch inputs.

Dynamic difficulty adjustment (DDA) is a technology that adapts a game's challenge to match the player's skill. It is a key element in game development that provides continuous motivation and immersion to the player. However, conventional DDA methods require tuning in-game parameters to generate the levels for various players. Recent DDA approaches based on deep learning can shorten the time-consuming tuning process, but require sufficient user demo data for adaptation. In this paper, we present a fast user adaptation method that can adjust the difficulty of the game for various players using only a small amount of demo data by applying a meta-learning algorithm. In the video game environment user test (n=9), our proposed DDA method outperformed a typical deep learning-based baseline method.

With the rise of the Maker Movement and STEM education, toolkit research for beginners in electronics has become popular. Most of them are focused on the efficient creation and debugging of electrical circuits. Therefore, users were implicitly required to understanding of the nature of electricity and some prerequisites knowledge. In this paper, we propose tangible interface to express electrical properties such as voltage and resistance. This system aims to bring a deep interest and understanding of the nature of electricity to the beginners explicitly. In this research, we discuss the interaction between the laws of electricity and another several physical laws, present an example of a prototype implementation, and describe the results of a preliminary user questionnaire.

We present a novel method to render color patterns using electrolysis applied onto open wet surfaces. By implementing electrodes within a wet object and electrifying them, electrolysis can occur and generate ions. Using color indicators reacting to such ions, we can create a color-forming display. By applying common techniques, such as a printed circuit board, arbitrary patterns can be displayed by computer control. By reversing the polarity of electrodes and varying the number of ions, it is possible to fade the existing pattern and display a contrasting color. The proposed method can be used to create a color-forming display using various objects more common in everyday life than those used in conventional substantial displays.

Museum visitors can often be distanced from directly engaging with historical artefacts due to their fragile nature. ARcheoBox is a novel interface that lets users physically pick up a digitized version of a real artefact and inspect it closely from all angles through a screen. The system combines augmented reality and tangible interactions to remove barriers between visitors and artefacts. We piloted our prototype with eight archaeological experts at a local exhibition centre. The results indicate that this is a promising way to remove barriers between humans and historical artefacts, and that it allows expert users to inspect, understand and interact with these artefacts in a way not previously possible.

Nowadays, customer virtual reality (VR) devices can offer high-quality presence experiences in a relatively low cost, which opens opportunities for popularizing situated learning. We investigate situated learning of soft skills, in particular, elevator pitches, through simulating an interactive agent with multimodal feedback in VR. A virtual agent acts as a coach and offers both verbal and nonverbal feedback to users. We developed a prototype system and conducted a quantitative experiment to investigate the potential of situated learning in VR, especially on how users perceive the virtual agent. We summarize design considerations on developing interactive agents with multimodal feedback in VR to facilitate situated learning of soft skills.

We present an early stage prototype of an acoustic labeling system using Velcro, a two-sided household adhesive product. We create labels by varying the shape of Velcro pieces to produce distinct sounds when the two sides are separated, and we use an automatic audio classification pipeline to detect and classify small sets of labels. We evaluate our classifier on four sets of three simple Velcro labels, present a demo highlighting potential use cases of these labels, and discuss future applications.

Recent years have witnessed the rapid progress of perception algorithms on top of LiDAR, a widely adopted sensor for autonomous driving systems. These LiDAR-based solutions are typically data hungry, requiring a large amount of data to be labeled for training and evaluation. However, annotating this kind of data is very challenging due to the sparsity and irregularity of point clouds and more complex interaction involved in this procedure. To tackle this problem, we propose FLAVA, a systematic approach to minimizing human interaction in the annotation process. Specifically, we divide the annotation pipeline into four parts: find, localize, adjust and verify. In addition, we carefully design the UI for different stages of the annotation procedure, thus keeping the annotators to focus on the aspects that are most important to each stage. Furthermore, our system also greatly reduces the amount of interaction by introducing a lightweight yet effective mechanism to propagate the annotation results. Experimental results show that our method can remarkably accelerate the procedure and improve the annotation quality.

Compression-based haptics instrumented into soft, wearable systems have the potential to enhance/augment gaming and entertainment experiences. Here, we present a computer-mediated Shape Memory Alloy (SMA) garment capable of delivering spatially and temporally dynamic compression and thermal stimuli for the purposes of enhancing a movie-watching affective experience based on 4 different emotions: sadness, love, gratitude, fear. Preliminary findings, still pending publication, were obtained from an MTurk survey (n=53) that seek to gather how different emotions can/should be represented physically through a haptic garment. A proposed study in a virtual reality (VR) environment is also introduced as a next step to evaluate how the presence/absence of haptics changes a user's video-watching experience, including immersion, enjoyment, presence, and felt emotional intensity, evaluated through objective (biometrics) and subjective (self-report questionnaire and qualitative interview) measures.

In this poster, we explore the potential of using a tangible user interface (TUI) to enable blind and visually impaired (BVI) developers to design web layouts without assistance. We conducted a semi-structured interview and a co-design session with a blind participant to elicit insights that guided the design of a TUI prototype named Sparsha. Our initial prototype contains 3D printed tactile beads that represent HTML elements, and a 3D printed base that represents the web page layout. BVI users can add an HTML element to the layout by placing tactile beads on top of the base. The base senses the type and location of the beads and renders the HTML element in the corresponding location on the client browser. The poster concludes with a discussion and enumerates future work.

High-fidelity localized feedback has the potential of providing new and unique levels of interaction with a given device. Achieving this in a cost-effective reproducible manner has been a challenge in modern technology. Past experiments have shown that by using the principles of constructive wave interference introduced by time offsets it is possible to achieve a position of increased vibration displacement at any given location. As new interface form factors increasingly incorporate curved surfaces, we now show that these same techniques can successfully be applied and mechanically coupled with a universal actuation plate.

We have explored a fluid-based interface(Hydrauio) on the pen body to extend interaction space of human-pen interaction. Users could perform finger gestures on the pen for input and also receive haptic feedback of different profiles from the fluid surface. The user studies showed that Hydrauio could achieve an accuracy of more than 92% for finger gesture recognition and users could distinguish different haptic profiles with an accuracy of more than 95%. Finally, we present application scenarios to demonstrate the potential of Hydrauio to extend interaction space of human-pen interaction.

Modern users typically open multiple software, websites, and documents for daily tasks. Retrieving previously used digital resources, such as web pages, documents, local files, and software, for knowledge workers is inevitable, but can be time-consuming. People may fail to remember where the resource is. In addition, sometimes, users need to retrieve digital resources across multiple applications to resume a computing task. However, the current methods - bookmarks, file systems, and recent history - for curating such resources are limited in their capacity and dispersed over multiple applications. To address the above problems, we test the idea of curating digital resources by developing Scrapbook, a software that allows users to curate digital resources with screenshots. Scrapbook allows users to take a screenshot of their computer screen and stores metadata of captured applications in the screenshot. Later, users can utilize multimodal (visual and textual) information to retrieve information that they want to recall or to restore the working context of a certain task. In this poster, we acknowledge previous relevant research, introduce ongoing efforts of developing Scrapbook, and share our plan to validate the idea via a user study.

The mediation of haptic feedback largely depends on physical properties of the surface of interaction and its internal structure. Most mobile devices today are assembled using a wide range of components with varied physical properties that can limit the propagation of tactile signals. For that reason, it is important to understand how common materials used in assembling mobile devices relay vibration signals and develop techniques of effectively mediating haptic feedback uniformly throughout the entire device. This research compares three off-the-shelf waveguide materials and one custom-designed 3D-printed ABS structure for creating feedback signals on flat interaction surfaces. Preliminary results indicate that by altering the internal structure of the interaction surface we can reduce attenuation and integration of the applied signal and improving the overall haptic experience.

Conventional pneumatic haptic displays need a complex and weight system because air is fed through a tube from an air compressor. To address this problem, we propose a haptic display that uses a liquid-to-gas phase change actuator and Peltier device. The actuator is composed of a thin plastic pouch containing a low boiling point liquid. The temperature of the liquid is controlled by a Peltier device in close contact with an actuator. This approach allows soft actuators to be inflated without the use of a compressor and tubes, allowing us to realize a small and light haptic display. We implemented the haptic display that has a pouch of 10 mm square and evaluated the characteristics of it. We confirmed that the maximum output force reached almost 1.5 N, which is a similar level to off-the-shelf wearable haptic display.

Moving in virtual reality without causing cybersickness is still an unsolved and difficult problem, especially if the virtual space is much larger than the real space and the virtual reality environment asks for quick movements. Many methods to reduce cybersickness are proposed but most of them also reduce immersion. In this paper, we explore the use of fNIRS as an additional modality to detect the level of cybersickness for movement events in VR. We try to mitigate the sickness an individual feels by narrowing the field of vision based on the sickness level detected via measuring increased deoxygenated hemoglobin (Hb) with fNIRS. Our overall goal is to reduce cybersickness in virtual reality applications using physiological signals and appropriate adjustments with little to no impact on immersion.

In this paper, we explore the use of a real-time speech visualization overlay to help native English speakers (NES) reflect on their speech speed and allow them to understand how English as a Foreign Language speakers (EFLS) perceive their speech. Our visual system generates a sequence of bubbles based on the speaking speed overlaid close to the user's mouth; the faster the speaking speed, the denser the bubble sequence, and vice versa. The results suggest that the presence of the speech visualization helps NES to understand their speech speed, and subsequently, it helps EFLS to feel comfortable speaking during the online group discussion.

ARCAR is a proof-of-concept headset-based mixed reality experience, for use by a driver in full control of a moving vehicle. The functional prototype has been implemented using a high end video pass-through mixed reality headset, and tested on roads (closed to external traffic) at conventional driving speeds. Our implementation embodies acceptable solutions to a number of current challenges that have been in the way of in-car XR applications, and enables a wealth of research. ARCAR is intended to serve as a research platform in the near future, enabling investigations on topics which include safety, design, perceived quality, and consumer applications.

Programming-by-example (PBE) can be a powerful tool for reducing manual work in repetitive data transformation tasks. However, having few examples often leaves ambiguity and may cause undesirable data transformation by the system. This ambiguity can be resolved by allowing the user to directly edit the synthesized programs, but this is difficult for non-programmers. Here, we present a novel approach: data-centric interaction for data transformation, where users resolve the ambiguity in data transformation by examining and modifying the output rather than the program. The key idea is focusing on the given set of data the user wants to transform instead of pursuing the synthesized program's completeness. Our system provides interactive visualization that allows users to efficiently examine and fix the transformed outputs, which is much simpler than understanding and modifying the program itself. Our experiment shows that the number of candidates is much smaller than the number of synthesized programs, which implies the effectiveness of the proposed method.

In this work, we present the VRQuestionnaireToolkit, which enables the research community to easily collect subjective measures within virtual reality (VR). We contribute a highly customizable and reusable open-source toolkit which can be integrated in existing VR projects rapidly. The toolkit comes with a pre-installed set of standard questionnaires such as NASA TLX, SSQ and SUS Presence questionnaire. Our system aims to lower the entry barrier to use questionnaires in VR and to significantly reduce development time and cost needed to run pre-, in between- and post-study questionnaires.

Brain-Computer Interfaces (BCIs) are progressively adopted by the consumer market, making them available for a variety of use-cases. However, off-the-shelf BCIs are limited in their adjustments towards individual head shapes, evaluation of scalp-electrode contact, and extension through additional sensors. This work presents EasyEG, a BCI headset that is adaptable to individual head shapes and offers adjustable electrode-scalp contact to improve measuring quality. EasyEG consists of 3D-printed and low-cost components that can be extended by additional sensing hardware, hence expanding the application domain of current BCIs. We conclude with use-cases that demonstrate the potentials of our EasyEG headset.

Accurate 3D registration of real and virtual objects is a crucial step in AR, especially when manipulating those objects in space. Previous work simplifies mid-air 3D manipulations by removing one or more degrees of freedom by constraining motion using automatic algorithms. However, when designing objects, limiting the user's actions can affect their creativity. To solve this problem, we present a new system called Context-based 3D Grids that allows users to do precise mid-air 3D manipulations without constraining their actions. Our system creates 3D grids for each object in the scene that change depending on the object pose. Users can display additional reference frames inside the virtual environment using natural hand gestures that are commonly used when designing an object. Our goal is to help users visualize more clearly the spatial relation and the differences in pose and size of the objects.

Paper can be a powerful and flexible user interface that lets programmers read through large amounts of code. Using off-the-shelf equipment, how can we generate a paper-based UI that supports code review, annotation, and teaching? To address this question, we ran formative studies and developed Papercode, a system that formats source code for printing on standard paper. Users can interact with that code on paper, make freehand annotations, then transfer annotations back to the computer by taking photos with a normal phone camera. Papercode optimizes source code for on-paper readability with tunable heuristics such as code-aware line wraps and page breaks, quick references to function and global definitions, moving comments and short function calls into margins, and topologically sorting functions in dependency order.

We present PViMat, a portable photovoltaic mat that can recognise six unique useful dynamic hover gestures up to 30 cm above the surface over a large area with more than 97% accuracy. We utilised an off-the-shelf portable and rollable outdoor solar tape and employed it to harvest indoor natural and artificial light energy to trickle charge a LiPo battery for self-powering. We demonstrate a low-power operation technique of continuous charging with DC photocurrent and simultaneous event-driven gesture recognition with AC photocurrent. The PViMat prototype harvests 30 mW in a general livingroom light level of 300 lux and consumes 0.8 mW per gesture. Lastly, we propose applications of PViMat with its large-area, flexible and rollable form-factors and the gesture set.

Swipe-based methods for text entry by gaze allow users to swipe through the letters of a word by gaze, analogous to how they can swipe with a finger on a touchscreen keyboard. Two challenges for these methods are: (1) gaze paths do not possess clear start and end positions, and (2) it is difficult to design text editing features. We introduce Swipe&Switch, a text-entry interface that uses swiping and switching to improve gaze-based interaction. The interface contains three context regions, and detects the start/end of a gesture and emits text editing commands (e.g., word insertion, deletion) when a user switches focus between these regions. A user study showed that Swipe&Switch provides a better user experience and higher text entry rate over a baseline, EyeSwipe.

People with limited digital literacy struggle to keep up with our increasing dependence on websites for everyday tasks like paying bills or booking flight tickets online. They often get in-person assistance from their friends and family but such help may not always be possible. Remote assistance from peers such as phone calls, or written instructions via email or text messages can be useful. However, remote methods of assistance may lead to communication issues between the helper and the help-seeker, due to a lack of shared visual context; a helper cannot see the help-seeker's screen. Moreover, help-seekers are often unacquainted with the terminology associated with web navigation. In order to bridge the gap between in-person support and remote help, we develop Remo, a web browser extension, which will allow helpers to create interactive tutorials by demonstration. These tutorials will be embedded within a web page and will make use of visual cues to direct users to specific parts of the page. Remo aims to provide opportunities for people with limited digital literacy to complete online tasks by following the step-by-step and task specific tutorials generated by their peers. Using Remo, we anticipate that the target population will be able to get personalized assistance, similar to in-person support, and eventually learn how to complete broader online tasks independently.

Qualitative coding, the process of assigning labels to text as part of qualitative analysis, is time-consuming and repetitive, especially for large datasets. While available QDAS sometimes allows the semi-automated extension of annotations to unseen data, recent user studies revealed critical issues. In particular, the integration of automated code suggestions into the coding process is not transparent and interactive. In this work, we present Cody, a system for semi-automated qualitative coding that suggests codes based on human-defined coding rules and supervised machine learning (ML). Suggestions and rules can be revised iteratively by users in a lean interface that provides explanations for code suggestions. In a preliminary evaluation, 42% of all documents could be coded automatically based on code rules. Cody is the first coding system to allow users to define query-style code rules in combination with supervised ML. Thereby, users can extend manual annotations to unseen data to improve coding speed and quality.

Digital fabrication allows users to produce physical objects from digital models. However, conventional fabrication processes are mostly irreversible: once an object is fabricated, it is detached from its original virtual model and cannot be physically changed. In this work, we propose WireTruss, a novel approach for rapid prototyping truss mesh structures, which can be rapidly fabricated, easily assembled and manually modified to support an intuitive design iteration. We developed a parametric design tool that first simplifies the object into a truss graph composed of multi-way joints, and then calculates the route of the wire which is inherently a Euler path. Furthermore, WireTruss can be demonstrated its practical usability through a series of application cases.

Eye contact is an important part of in-person communication. However, in modern remote communication - video conference calls, non-verbal cues through eye contact are lost. This is because it is not possible to make eye contact in a video call; to make it look like a person is making an eye contact, the user has to look at the camera directly, but then this means that the user is not looking at the visual of the interlocutor on the screen. In our research, we aim to build a hardware and software system that helps the users to make eye contact in video conference and change their perspective, placing a moving camera behind a semi-transparent screen. The system keeps the position of camera right behind the remote user's eye position on screen tracked by a computer vision algorithm, so users will be able to make eye contact during the video call. We believe the system will lead to better user experience in remote conference calls. In this work, we share our motivation, design choices, the current progress, and a plan to evaluate the system.

Textiles encompass a wide variety and rich characteristics that ranges from soft to tough, foldable to rigid, cuttable to stiff, elastic to plastic. On the other hand, 3D printing thermoplastic (e.g., PLA) materials exhibit controlled deformation at a certain transition temperature. In this paper, we present a method of fabricating form-fitting composite textiles by printing PLA on different fabric substrates, designing different structures and transforming the fabric-PLA composite to form different textures.

In this paper, we propose a method for gesture input which uses a sensor attached to a mask to detect the user's breathing and trajectory of his or her breath. To prevent the spread of infectious diseases, it is important to keep our hands clean and avoid spreading the virus to our homes. Using a smartphone after touching various objects in public facilities increases the risk of infection. Our research focuses on operating devices such as smartphones using breathing gestures rather than touching the devices directly. The gestures were performed while wearing a commercial mask with a breathing sensor and a 9-DoF IMU attached. The breathing sensor detects inhalation and exhalation patterns, and the 9-DoF IMU estimates the breath direction and movement trajectory by the user's head. The breathing gesture was estimated by combining the information from the two sensors. Through prototyping, we determined that our method was able to estimate the gestures. We also provide an example application of this interface.

MorphSensor is a 3D electronic design tool that enables designers to morph existing sensor modules of pre-defined two-dimensional shape into free-form electronic component arrangements that better integrate with the three-dimensional shape of a physical prototype. MorphSensor builds onto existing sensor module schematics that already define the electronic components and the wiring required to build the sensor. Since MorphSensor maintains the wire connections throughout the editing process, the sensor remains fully functional even when designers change the electronic component layout on the prototype geometry. We present the MorphSensor editor that supports designers in re-arranging the electronic components, and discuss a fabrication pipeline based on customized PCB footprints for making the resulting freeform sensor.

We demonstrate DepthLab, a playground for interactive augmented reality experiences leveraging the shape and depth of the physical environment on a mobile phone. Based on the ARCore Depth API, DepthLab encapsulates a variety of depth-based UI/UX paradigms, including geometry-aware rendering (occlusion, shadows, texture decals), surface interaction behaviors (physics, collision detection, avatar path planning), and visual effects (relighting, 3D-anchored focus and aperture effects, 3D photos). We have open-sourced our software at https://github.com/googlesamples/arcore-depth-lab to facilitate future research and development in depth-aware mobile AR experiences. With DepthLab, we aim to help mobile developers to effortlessly integrate depth into their AR experiences and amplify the expression of their creative vision.

Due to the COVID-19 pandemic, wearing a mask to cover one's mouth is recommended in public spaces to prevent the spread of the virus. Wearing masks hinders our ability to express ourselves, as it is hard to read facial expressions much less lips behind a mask. We present Unmasked, an expressive interface using lip tracking to enhance communication while wearing a mask. Unmasked uses three different methods - either accelerometers, LEDs with a camera tracking, or streaming video - to make speaking while wearing a mask more expressive. Unmasked aims to improve communication during conversations while wearing a mask. This device will help people express themselves while wearing a mask by tracking their mouth movements and displaying their facial expressions on an LCD mounted on the front of the mask. By enhancing communication while wearing a mask, this prototype makes social distancing less disruptive and more bearable, metaphorically closing some of the distance between us.

It is known that the pressure and contact area change contribute to hard and soft perception as a cutaneous sensation. In this study, we propose a novel method of haptic presentation based on the profiles of physical objects by using the load-displacement measurement. We fabricated a pneumatic haptic system with an elastic membrane that enables controlled pressure stimuli. We verified that the proposed method is capable of reproducing various profiles by comparing with physical objects. As a result, we found that our system could well reproduce the load-displacement profiles of a sponge and a button specimen.

Mobile Augmented Reality applications have become increasingly popular, however the possible interactions with AR content are largely limited to on-screen gestures and spatial movement. There has been a renewed interest in designing interaction methods for mobile AR that go beyond the screen. Mobile phones present a rich range of input, output, and tracking capabilities, and have been used as controllers for Virtual Reality and head-mounted Augmented Reality applications. In this project, we explore the use of a second phone as a controller for Mobile AR. We developed ARTWO, an application that showcases Handheld Dual Phone AR through a series of small demos in which a second phone can be used to perform basic tasks such as pointing, selecting, and drawing, in the context of real use cases. We believe that the Dual Phone AR approach can help address many of the issues faced when using conventional mobile AR interactions, and also serves as a stepping stone to the general use of phones with head-mounted AR systems in the near future.

Despite the marked improvements in image classification and recognition tasks over the past decade, affordances and interfaces for searching images have remained largely unchanged on the web. In this demo, we present Newspaper Navigator, an open faceted search system for 1.5 million historic newspaper photographs. In contrast to standard faceted search, which requires facets to be pre-selected and applied to images, Newspaper Navigator empowers users to specify their own facets in an open-domain fashion during the search process by selecting relevant examples and iteratively training a machine learner. With Newspaper Navigator, users can quickly sort 1.5 million images according to dynamically-specified facets such as "baseball player'' and "oval-shaped portrait.'' Newspaper Navigator also drives facet exploration by suggesting related keyword search queries for a user to perform. Our demo walks through examples of searching with Newspaper Navigator and highlights the facet learning and exploration affordances.

In the area of ubiquitous computing, there is research that use surfaces of elements in the environment such as windows and walls as screens. It is necessary to install an opaque screen and projectors on both sides of the screen in order to switch projection surfaces, i.e., front, back, or both sides. In this paper, we propose Janus Screen, a novel screen system with polarizers that allow switching of the projection surface to the front, back, or both sides using only projectors on one side. Janus Screen employs projectors equipped with polarizers and a screen with multiple layers. We implement a prototype and confirm that Janus Screen can selectively switch the projection surface. Moreover, we discuss unique characteristics and possible applications of the proposed system.

Prosthetic limbs are an extension of the human body and should allow for the same function and control that a natural limb provides. Most prosthetics aim to allow an individual to return to a baseline quality of life, but few allow the individual to return to sports. Sport prosthetics can be difficult to produce due to the specificity of the designs and associated costs. Some prosthetics are becoming more accessible with the availability of 3D-printing. In this work, we create a prototype basketball prosthetic hand to enable a more natural shot. This prototype uses 3D printed springs for energy return and 3D printed electronics for integrated strain sensing that can be mapped to haptic feedback for the user. Combining these technologies will lead to a sport prosthetic limb whose technology advancements can be applied to future prosthetic designs.

Virtual Assistants are becoming increasingly popular. However voice-only systems providing limited functionality and minimal variability are often unusable because the exposed behavior is either fully deterministic or essentially ambiguous for supporting human-like dialogues. This paper introduces a Nimble, solution that allows users to ask short questions and to get answers about objects selected from a scene by natural pointing gestures. With a modified Visual Question Answering model we have shown how the integration of gestures to the attention mechanism can reduce questions? ambiguity while sustaining the same accuracy level of the system. We performed this by modifying the model's attention scores using gestures fused with linguistic information.

The bias of news articles can strongly affect the opinions and behaviors of readers, especially if they do not consume sets of articles that represent diverse political perspectives. To mitigate media bias and diversify news consumption, we developed Counterweight---a browser extension that presents different perspectives by recommending articles relevant to the current topic. We provide a platform to encourage a more diversified consumption of news and mitigate the negative effects of media bias.

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.

Audio navigation tools have the potential to help visually impaired people (VIPs) navigate independently, but today's audio navigation tools merely guide users to destinations rather than give users the full freedom to explore and navigate environments themselves. To address this problem, we present NavStick, a self-directed audio navigation tool for virtual environments. NavStick repurposes a game controller's thumbstick to allow a user to survey their immediate surroundings by "scrubbing" the thumbstick in a circular fashion. We also describe a user study that we are performing to compare NavStick with today's guide-based audio navigation tools.

Making electronic gadgets that meet today's consumer standards has become a difficult task. Electronic gadgets are expected to have displays with visually appealing interfaces and, at the same time, be physically and screen-interactive, making the development process of these devices time-consuming and challenging. To address this problem, we have created Appliancizer, an online synthesis tool that can automatically generate sophisticated electronic devices from web pages. Appliancizer takes advantage of the similarities between software and physical interfaces to reduce development steps and allow the rapid development of electronic devices. By matching the interface of hardware components with the interface of graphical HTML elements found on web pages, our tool allows a designer to transform HTML elements from a digital to a tangible interface without changing the application source code. Finally, a modular design enables our tool to automatically combine the circuit design and low-level hardware code of selected hardware components into a complete design. Attendees can interact with our online tool and produce manufacturable PCBs from web pages.

In this paper, we propose a thin display panel system for converting an existing kitchen or vanity tabletop with an engineered marble into a smart touch display. The display system, which consists of an LED matrix panel including IR-LEDs and photodiodes, will be placed on the rear side of a tabletop of a kitchen or washbasins. The normal LED light is used for a visual display transmitted with a thick opaque resin, and the infrared light is used for detecting touch areas and objects using the same resin panel. This display panel can be used for various applications and can be touched with wet and/or dirty hands. In this paper, we describe the system overview, the optical characteristics of blurred display via an engineered marble plate, and an touch-sensing experiment. We also provide some applications of the smart touch display.

The increasing complexity of web pages has brought a number of solutions to offer simpler or lighter versions of these pages. The qualitative evaluation of the new versions is commonly carried out relying on user studies. In addition to the associated costs, running user studies might become extremely challenging due to health, travel, and financial constraints. Moreover, user studies are prone to subjectivity, which makes it difficult to compare the results of different studies. In this paper, we propose PQual, a tool that enables the automation of the qualitative evaluation of web pages using computer vision. Results show that PQual computes comparable similarity scores to the scores provided by human raters. In addition, it can effectively evaluate all the functionality of a web page, whereas humans might skip many of the functional elements during the evaluation.

In this paper, I introduce a concept of mechanical shells, which are physical add-ons that can adaptively augment, extend, and reconfigure the interactivities of self-actuated tangible user interfaces (TUIs). While a variety of research explores actuated and shape-changing interfaces for providing dynamic physical affordance and tangible displays, the concept of mechanical shell intends to overcome the constraint of existing generic actuated TUI hardware thereby enabling greater versatility and expression. This paper overviews the mechanical shell concept, describes project examples, outlines a research framework, and suggests open space for future research.

Laser-cut 3D models found in repositories tend to be basic and trivial-models build over long periods of time and by multiple designers are few/non existent. I argue that this is caused by a lack of an exchange format that would allow continuing the work. At first glance, it may seem like such a format already exist, as laser cut models are already widely shared in the form of 2D cutting plans. However, such files are susceptible to variations in cutter properties (aka kerf) and do not allow modifying the model in any meaningful way (no adjustment of material thickness, no parametric changes, etc.). My first take on the challenge is to see how far we can get by still building on the de-facto standard, i.e., 2D cutting plans. springFit [7] and kerf-canceling mechanisms [6] tackle the challenge by rewriting 2D cutting plans, replacing non-portable elements with portable ones (shown in Figure 1). However, this comes at a cost of extra incisions, reducing the structural integrity of models and impacting aesthetic qualities and rare mechanisms or joints may go undetected. I thus take a more radical approach, which is to move on to a 3D exchange format (kyub [1]). This eliminates these challenges as it guarantees portability by gener-ating a new machine-specific 2D file for the local machine. Instead, it raises the question of compatibility: Files already exist in 2D-how to get them into 3D? I demonstrate a software tool that automatically reconstructs the 3D geometry of the model encoded in a 2D cutting plan, allows modifying it using a 3D editor, and re-encodes it to a 2D cutting plan. I demonstrate how this approach allows me to make a much wider range of modifications, including scaling, changing material thickness, and even remixing mod-els. The transition from sharing machine-oriented 2D cutting files, to 3D files, enables users worldwide to collaborate, share, and reuse. And thus, to move on from users starting thousands of trivial models from scratch to collaborating on big complex projects.

Interactive task learning (ITL) allows end users to 'teach' an intelligent agent new tasks, the corresponding task conditions,and the relevant concepts. This paper presents my research on expanding the applicability, generalizability, robustness, expressiveness, and script sharability of ITL systems usinga multi-modal approach. My research demonstrates that a multi-modal ITL approach that combines programming by demonstration and natural language instructions can empower users without significant programming expertise to extend intelligent agents for their own app-based computing tasks.

Emerging 3D printing technology has enabled the rapid development of physical objects. However, 3D-printed objects are rarely interactive and adding interactivity to printed objects is inherently challenging. To boost 3D printing for a wider spectrum of applications, I introduce in-place augmented structures, a class of 3D printable parametric structures that can be integrated with physical objects and spaces for augmented behaviors. In my research, I explore how 3D printing can support interaction (e.g., sensing and actuation) by creating novel design techniques and building interactive design tools that enable end-users to design and control desired behaviors. With these techniques and tools, I fabricate the in-place structures with readily available fabrication techniques and demonstrate my approach with a suite of applications across different domains.

Hundreds of millions of users work with digital documents for their everyday tasks but the user interfaces have not fundamentally changed since they were first designed in the late seventies [10]. We focus on two examples of today's 'extreme users' of documents, legal professionals and scientists. Based on their work practices, we designed two document-centered tools: Textlets supports constraints and consistency within documents; FileWeaver automatically tracks dependencies across documents. I then describe two research directions that emerged from our empirical studies and recent literature: collaboration in documents, e.g. Jupyter notebook and direct manipulation for computational documents such as LaTeX. This dissertation will enhance our understanding of how today's users work with documents; demonstrate novel tools and expand the fundamental theory of interaction.

My research goal is to build practical and intuitive 3D free-hand interactions for augmented reality (AR) on smartphones and to explore relevant behavioral-data-driven interaction techniques. In service of that goal, I have developed two preliminary AR systems on the smartphone: a Portal-ble system powered by visual, auditory, and haptic feedback and a set of grabbing accommodations that allows the user's hand to interact with AR contents directly. I have also developed a Portalware system that incorporates a smartphone-wearable interaction scheme to expand visual feedback beyond the smartphone's display and have leveraged this system to improve 3D mid-air sketching. My next steps are to use behavioral data such as device motion, hand motion and user postures to create dynamic and personalized interaction systems that facilitate intuitive AR interaction. Ultimately, these individually-tailored systems unlock new interaction possibilities for the general public and expand the usage scenarios for smartphone AR applications.

Internet-of-Things (IoT) devices promise to enhance even the most mundane of objects with computational properties by seamlessly coupling the virtual world to the physical. However, IoT's associated costs and cumbersome setup limits its extension to many everyday tasks and objects, such as those in the home. To address these issues, my dissertation work will enable IoT Stickers'a book of inexpensive, battery-free sensors and composition patterns'to support customizing everyday objects with software and web services using stickers. Using RFID-based paper mechanisms, IoT Stickers integrates common sensors and web services with interactive stickers through a trigger-action architecture. This integration enables computational services to be tailored to everyday activities by setting parameters to be passed to the sticker's actions and composing the stickers together. Thus, IoT Stickers demonstrates a way to associate IoT services with a dramatically wider set of objects and tasks.