Koji Yatani, Ph.D.

  Associate Professor
Interactive Intelligent Systems Laboratory
Department of Electrical Engineering and Information Systems (EEIS)
School of Engineering
The University of Tokyo

I received my Ph.D. at University of Toronto under the supervision by Prof. Khai N. Truong at Dynamic Graphics Project.My research interests lie in Human-Computer Interaction (HCI) and ubiquitous computing. More specifically, I am interested in developing new sensing technologies to support user interactions in mobile/ubiquitous computing environments, and interactive systems with computational linguistics methods. I am also interested in deeply evaluating interactive systems through quantitative and qualitative approaches. Besides HCI and ubiquitous computing, I am interested in machine learning, statistical analysis, computational linguistics, psychology, and physiology.

Office: Faculty of Engineering Building 2, 7-3-1, Hongo, Bunkyo-ku, Tokyo, JAPAN.
E-mail: [my_first_name] "at" iis-lab.org

Research / Publication / Teaching / CV / HCI Stats Wiki

Professional Activities / Biography / Photos

About me    
Two honorable mention awards at MobileHCI 2014
Our ReviewCollage (in collaboration with Haojian Jin and Tetsuya Sakai) and TalkZones (in collaboration with Bahador Saket, Sijie Yang, Hong Tan, and Darren Edge) projects won Honorable Mention Awards at MobileHCI 2014. Our papers will be available on my website soon.
Two honorable mention awards at CHI 2014
Our PitchPerfect (in collaboration with Ha Trinh and Darren Edge) and TurningPoint (in collaboration with Larissa Pschetz and Darren Edge) projects won Honorable Mention Awards at CHI 2014. Our papers are available on my website (PitchPerfect and TurningPoint).
UbiComp 2014 Technical Program Committee
I am serving on UbiComp 2014 as a Technical Program Committee member. We are looking forward to seeing your cool sensors, devices, and applications in ubiquitous computing!
MobiSys 2014 Technical Program Committee
I am serving on MobiSys 2014 as a Technical Program Committee member. We are looking forward to seeing your cool mobile systems, applications and services!
Statistics for HCI research
I have published my wiki about some statistical methods useful for HCI research (with an emphasis on R). If you are using R and/or know about statistics well, your feedback would be greatly appreciated.

Hooked on Smartphones: An Exploratory Study on Smartphone Overuse among College Students  
The impact of smartphone addiction on young adults, such as sleep deprivation and attention deficits, are increasingly being recognized. This emerging issue motivated us to identify smartphone usage patterns relating to smartphone addiction. We investigate smartphone usage for 95 college students using surveys, logged data, and interviews. We first divide the participants into risk and non-risk groups based on self-reported psychometric scale data about smartphone addiction. We then analyze the usage data to uncover between-group usage differences, ranging from overall usage patterns to app-specific usage patterns. Our results reveal that compared to the non-risk group, the risk group has longer usage time per day and differences in diurnal usage. The risk group is more susceptible to push notifications, and tends to consume more online content. We identify a relationship between usage features and smartphone addiction with analytic modeling and provide detailed illustration of problematic usage behavior from interview data.
TurningPoint: Narrative-Driven Presentation Planning  
Once upon a time, people told stories unencumbered by slides. What modern presentations gain through visual slide support, however, is often at the expense of storytelling. We present TurningPoint, a probe to investigate the potential use of narrative-driven talk planning in slideware. Our study of TurningPoint reveals a delicate balance between narrative templates focusing author attention in ways that save time, and fixating attention in ways that limit experimentation.
PitchPerfect: Integrated Rehearsal Environment for Structured Presentation Preparation  
Rehearsal is a critical component of preparing to give an oral presentation, yet it is frequently abbreviated, performed in ways that are inefficient or ineffective, or simply omitted. We conducted an exploratory study to understand the relationship between the theory and practice of presentation rehearsal, classifying our qualitative results into five themes to motivate more structured rehearsal support deeply integrated in slide presentation software. In a within-subject study (N=12) comparing against participantsf existing rehearsal practices, we found that our resulting PitchPerfect system significantly improved overall presentation quality and content coverage as well as provided greater support for content mastery, time management, and confidence building.
Escape-Keyboard: A Sight-free One-handed Text Entry Method for Mobile Touch-screen Devices  
Mobile text entry methods traditionally have been designed with the assumption that users can devote full visual and mental attention on the device though this is not always possible. In this paper, we present the design and evaluation of Escape-Keyboard, a sight-free text entry method for mobile touch-screen devices. Escape-Keyboard allows the user to type letters with one hand by pressing the thumb on different areas of the screen and consequently performing a flick gesture. Our user study showed that participants reached an average typing speed of 14.7 words per minute (WPM) with 4.4% error rate in the sight-free condition and 16.8 WPM with 1.7% error rate in the sighted condition after 16 typing sessions. Our qualitative results indicate that the participants had difficulty learning the keyboard layout, which led to slow typing speed improvements over time. We thus implemented and evaluated features to mitigate this learnability issue. We also performed a theoretical analysis of sight-free performance of our keyboard, which predicts expert peak performance to be 39 WPM.
SidePoint: A Peripheral Knowledge Panel for Presentation Slide Authoring  
Presentation authoring is an important activity, but often requires the secondary task of collecting the information and media necessary for both slides and speech. Integration of implicit search and peripheral displays into presentation authoring tools may reduce the effort to satisfy not just active needs the author is aware of, but also latent needs that she is not aware of until she encounters content of perceived value. We develop SidePoint, a peripheral panel that supports presentation authoring by showing concise knowledge items relevant to the slide content. We study SidePoint as a technology probe to examine the benefits and issues associated with peripheral knowledge panels for presentation authoring. Our results show that peripheral knowledge panels have the potential to satisfy both types of needs in ways that transform presentation authoring for the better.
HyperSlides: Dynamic Presentation Prototyping  
Presentations are a crucial form of modern communication, yet there is a dissonance between everyday practices with presentation tools and best practices from the presentation literature. We conducted a grounded theory study to gain a better understanding of the activity of presenting, discovering the potential for a more dynamic, automated, and story-centered approach to prototyping slide presentations that are themselves dynamic in their ability to help presenters rehearse and deliver their story. Our prototype tool for dynamic presentation prototyping, which we call HyperSlides, uses a simple markup language for the creation of hierarchically structured scenes, which are algorithmically transformed into hyperlinked slides of a consistent and minimalist style. Our evaluation suggests that HyperSlides helps idea organization, saves authoring time, creates aesthetic layouts, and supports more flexible rehearsal and delivery than linear slides, at the expense of reduced layout control and increased navigation demands.
Communication and Coordination for Institutional Dementia Caregiving in China  
With a general trend worldwide towards greater life expectancies, interventions and tools that can help caregivers working in elder care are becoming increasingly important. In China, with a greater number and proportion of elders due to the long-term effects of the one-child policy, these interventions and tools are needed even more. Improved communication between care staff of an institutional home can reduce medical errors and improve coordination of care. At the same time, increased conversation with elders with cognitive impairments like dementia or Alzheimer's can help the elder to maintain their cognitive ability, and can reduce negative feelings like loneliness. Our qualitative study with eleven institutional caregivers in Beijing delved into the communication patterns that exist between caregivers and elders with dementia. We found that knowing more about each individual resident's disposition and personal history was helpful in maintaining quality care, that many care staff in China use placating talk as a means to calm or guide elders to a desired action, and that care staff found the topic of past careers or past 'glories' to be the most efficient in getting elders to chat. In addition, we also found that much of the information that is gleaned through working with an elder long-term is not recorded or shared in any official capacity with other care workers, an area where technology could be particularly helpful.
BodyScope: A Wearable Acoustic Sensor for Activity Recognition  
Accurate activity recognition enables the development of a variety of ubiquitous computing applications, such as context-aware systems, lifelogging, and personal health systems. Wearable sensing technologies can be used to gather data for activity recognition without requiring sensors to be installed in the infrastructure. However, the user may need to wear multiple sensors for accurate recognition of a larger number of different activities. We developed a wearable acoustic sensor, called BodyScope, to record the sounds produced in the user's throat area and classify them into user activities, such as eating, drinking, speaking, laughing, and coughing. The F-measure of the Support Vector Machine classification of 12 activities using only our BodyScope sensor was 79.5%. We also conducted a small-scale in-the-wild study, and found that BodyScope was able to identify four activities (eating, drinking, speaking, and laughing) at 71.5% accuracy.
SpaceSense: Representing Geographical Information to Visually Impaired People Using Spatial Tactile Feedback  
Learning an environment can be challenging for people with visual impairments. Braille maps allow their users to understand the spatial relationship between a set of places. However, physical Braille maps are often costly, may not always cover an area of interest with sufficient detail, and might not present up-to-date information. We built a handheld system for representing geographical information called SpaceSense, which includes custom spatial tactile feedback hardware-multiple vibration motors attached to different locations on a mobile touch-screen device. It offers high-level information about the distance and direction towards a destination and bookmarked places through vibrotactile feedback to help the user maintain the spatial relationships between these points. SpaceSense also adapts a summarization technique for online user reviews of public and commercial venues. Our user study shows that participants could build and maintain the spatial relationships between places on a map more accurately with SpaceSense compared to a system without spatial tactile feedback. They pointed specifically to having spatial tactile feedback as the contributing factor in successfully building and maintaining their mental map.
Investigating Effects of Visual and Tactile Feedback on Spatial Coordination in Collaborative Handheld Systems  
Mobile and handheld devices have become platforms to support remote collaboration. But, their small form-factor may impact the effectiveness of the visual feedback channel often used to help users maintain an awareness of their partner's activities during synchronous collaborative tasks. We investigated how visual and tactile feedback affects collaboration on mobile devices, with emphasis on spatial coordination in a shared workspace. From two user studies, our results highlight different benefits of each feedback channel in collaborative handheld systems. Visual feedback can provide precise spatial information for collaborators, but degrades collaboration when the feedback is occluded, and sometimes can distract the user's attention. Spatial tactile feedback can reduce the overload of information in visual space and gently guides the user's attention to an area of interest. Our results also show that visual and tactile feedback can complement each other, and systems using both feedback channels can support better spatial coordination than systems using only one form of feedback.
Design of Unimanual Multi-finger Pie Menu Interaction  
Context menus, most commonly the right click menu, are a traditional method of interaction when using a keyboard and mouse. Context menus make a subset of commands in the application quickly available to the user. However, on tabletop touchscreen computers, context menus have all but disappeared. In this work, we investigate how to design context menus for efficient unimanual multi-touch use. We investigate the limitations of the arm, wrist, and fingers and how it relates to human performance of multi-targets selection tasks on multi-touch surface. We show that selecting targets with multiple fingers simultaneously improves the performance of target selection compared to traditional single finger selection, but also increases errors. Informed by these results, we present our own context menu design for horizontal tabletop surfaces.
The 1Line Keyboard: A QWERTY Layout in a Single Line  
Current soft QWERTY keyboards often consume a large portion of the screen space on portable touchscreens. This space consumption can diminish the overall user experience on these devices. In this work, we present the 1Line keyboard, a soft QWERTY keyboard that is 140 pixels tall (in landscape mode) and 40% of the height of the native iPad QWERTY keyboard. Our keyboard condenses the three rows of keys in the normal QWERTY layout into a single line with eight keys. The sizing of the eight keys is based on users' mental layout of a QWERTY keyboard on an iPad. The system disambiguates the word the user types based on the sequence of keys pressed. The user can use flick gestures to perform backspace and enter, and tap on the bezel below the keyboard to input a space. Through an evaluation, we show that participants are able to quickly learn how to use the 1Line keyboard and type at a rate of over 30 WPM after just five 20-minute typing sessions. Using a keystroke level model, we predict the peak expert text entry rate with the 1Line keyboard to be 66-68 WPM.
Review Spotlight: A User Interface for Summarizing User-generated Reviews Using Adjective-Noun Word Pairs  
Many people read online reviews written by other users to learn more about a product or venue. However, the overwhelming amount of user-generated reviews and variance in length, detail and quality across the reviews make it difficult to glean useful information. In this work, we present the iterative design of our system, called Review Spotlight. It provides a brief overview of reviews using adjective-noun word pairs, and allows the user to quickly explore the reviews in greater detail. Through a laboratory user study which required participants to perform decision making tasks, we showed that participants could form detailed impressions about restaurants and decide between two options significantly faster with Review Spotlight than with traditional review webpages.
Sensing Foot Gestures from the Pocket  
Visually demanding interfaces on a mobile phone can diminish the user experience by monopolizing the user's attention when they are focusing on another task and impede accessibility for visually impaired users. Because mobile devices are often located in pockets when users are mobile, explicit foot movements can be defined as eyes-and-hands-free input gestures for interacting with the device. In this work, we study the human capability associated with performing foot-based interactions which involve lifting and rotation of the foot when pivoting on the toe and heel. Building upon these results, we then developed a system to learn and recognize foot gestures using a single commodity mobile phone placed in the user's pocket or in a holster on their hip. Our system uses acceleration data recorded by a built-in accelerometer on the mobile device and a machine learning approach to recognizing gestures. Through a lab study, we demonstrate that our system can classify ten different foot gestures at approximately 86% accuracy.
Pen + Touch = New Tools (also known as Manual Deskterity)  
We describe techniques for direct pen+touch input. We observe people's manual behaviors with physical paper and notebooks. These serve as the foundation for a prototype Microsoft Surface application, centered on note-taking and scrapbooking of materials. Based on our explorations we advocate a division of labor between pen and touch: the pen writes, touch manipulates, and the combination of pen+touch yields new tools. This articulates how our system interprets unimodal pen, unimodal touch, and multimodal pen+touch inputs, respectively. For example, the user can hold a photo and drag off with the pen to create and place a copy; hold a photo and cross it in a freeform path with the pen to slice it in two; or hold selected photos and tap one with the pen to staple them all together. Touch thus unifies object selection with mode switching of the pen, while the muscular tension of holding touch serves as the glue that phrases together all the inputs into a unitary multimodal gesture. This helps the UI designer to avoid encumbrances such as physical buttons, persistent modes, or widgets that detract from the user's focus on the workspace.
SemFeel: A User Interface with Semantic Tactile Feedback for Mobile Touch-screen Devices  
One of the challenges with using mobile touch-screen devices is that they do not provide tactile feedback to the user. Thus, the user is required to look at the screen to interact with these devices. In this paper, we present SemFeel, a tactile feedback system which informs the user about the presence of an object where she touches on the screen and can offer additional semantic information about that item. Through multiple vibration motors that we attached to the backside of a mobile touch-screen device, SemFeel can generate different patterns of vibration, such as ones that flow from right to left or from top to bottom, to help the user interact with a mobile device. Through two user studies, we show that users can distinguish ten different patterns, including linear patterns and a circular pattern, at approximately 90% accuracy, and that SemFeel supports accurate eyes-free interactions.
Understanding How and Why Open Source Contributors Use Diagrams  
Some of the most interesting differences between Open Source Software (OSS) development and commercial colocated software development lie in the communication and collaboration practices of these two groups of developers. One interesting practice is that of diagramming. Though well studied and important in many aspects of co-located software development (including communication and collaboration among developers), its role in OSS development has not been thoroughly studied. In this project, we investigate how and why OSS contributors use diagrams in their work. We explore differences in the use and practices of diagramming, their possible reasons, and present design considerations for potential systems aimed at better supporting diagram use in OSS development.
Understanding Mobile Phone Situated Sustainability: The Influence of Local Constraints and Practices on Transferability  
Mobile phones are the most prevalent example of pervasive computing technologies in use today, with phone subscriptions reaching 3.3 billion in 2007. According to a 2005 estimate, consumers discard roughly 125 million mobile phones into landfills every year. Although devices continue to proliferate, viable options for ecologically responsible solutions remain elusive, inaccessible, or unknown to users. We examine people's practices with mobile phones, particularly those surrounding end-of-use. We focus on the differences and commonalities between practices in North America, Japan, and Germany, and the impact of varying local constraints on mobile phone sustainability. Building upon previous research examining sustainability and mobile phone ownership decisions, we explore the notion of situated sustainability by looking at how mobile phone sustainability is affected by local and community factors.
Escape: A Target Selection Technique Using Visually-cued Gestures  
Many mobile devices have touch-sensitive screens that people interact with using fingers or thumbs. However, such interaction is difficult because targets become occluded, and because fingers and thumbs have low input resolution. Recent research has addressed occlusion through visual techniques. However, the poor resolution of finger and thumb selection still limits selection speed. In this paper, we address the selection speed problem through a new target selection technique called Escape. In Escape, targets are selected by gestures cued by icon position and appearance. A user study shows that for targets six to twelve pixels wide, Escape performs at a similar error rate and at least 30% faster than Shift, an alternative technique, on a similar task. We evaluate Escape's performance in different circumstances, including different icon sizes, icon overlap, use of color, and gesture direction. We also describe an algorithm that assigns icons to targets, thereby improving Escapefs performance.
An Evaluation of Stylus-based Text Entry Methods on Handheld Devices in Stationary and Mobile Scenarios  
Effective text entry on handheld devices remains a significant problem in the field of mobile computing. On a personal digital assistant (PDA), text entry methods traditionally support input through the motion of a stylus held in the user's dominant hand. In this paper, we present the design of a two-handed software keyboard for a PDA which specifically takes advantage of the thumb in the non-dominant hand. We compare our chorded keyboard design to other stylus-based text entry methods in an evaluation that studies user input in both stationary and mobile settings. Our study shows that users type fastest using the miniqwerty keyboard, and most accurately using our two-handed keyboard. We also discovered a difference in input performance with the mini-qwerty keyboard between stationary and mobile settings. As a user walks, text input speed decreases while error rates and mental workload increases; however, these metrics remain relatively stable in our two-handed technique despite user mobility.
A Multiplayer Whack-A-Mole Game Using Gestural Input in a Location-Sensitive and Immersive Environment  
ARHunter is a computer-enhanced multi-player whack-a-mole game. It creates an immersive entertainment environment combined with gestural input and location recognition technologies, which aims at increasing the level of players' engagement and excitement.
A Fast and Accurate Positioning Technique Using the Ultrasonic Phase Accordance Method  
We developed a positioning technique using ultrasonic signals. Our technique can accurately identify the relative distance and orientation between devices by using an one-time ultrasonic packet. The technique, which is named phase accordance method, uses two or more carriers in ultrasonic communication. A special ultrasonic burst signal, called a sync pattern in the header part of the communication packet gives the base point of the time measurement. The whole time difference calculation is then carried out using this base point. An experiment showed that the technique yielded errors of less than } 1 mm for 3 m distance measurements and less than 0.5 degree errors for smaller than 30 degree.
Toss-It: Intuitive Information Transfer Techniques for Mobile Devices  
Toss-It provides intuitive information transfer techniques for mobile devices, by fully utilizing their mobility. A user of Toss-It can send information from the user's PDA to other electronic devices with a toss or swing action, as the user would toss a ball or deal cards to others. Toss-It uses inertial sensors and optical markers to recognize the user's gestures and location.
An Interactive and Enjoyable Educational System in a Museum  
We developed a system called Pi_book to support children's exploration in a science museum. Pi_book provides additional contents about exhibitions with PDAs. Additional contents on the PDAs are designed to be interactive in order to increase the children's interests in exhibitions and contents on the PDAs. Our system help children have interests scientific phenomina which is often difficult to understand without any assistance.
Musex: A System for Supporting Children's Collaborative Learning in a Museum with PDAs  
Musex supports children's collaborative learning and exploration in a museum with PDAs (Personal Digital Assistants). Musex provides questions about exhibitions that are not interactive, such as explanatory panels and videos. In this manner, our system encourages children to look into these exhibitions. Our user study with Musex revealed that children interacted with exhibitions actively and were engaged in solving questions with Musex.

Professional Activities
Program Committee AH (2013);
CHI (Interaction Techniques and Devices subcommittee; 2013);
MobiSys (2014);
Ubicomp (2012 -- 2014);
UIST (2013);
World Haptics Conference (2013)
Conference Committee Video Chair: Ubicomp (2013);
Mentoring Chair: ITS (2012)
Reviewer (Journal) ACM Transactions on Computer-Human Interaction;
IEEE Transactions on Haptics;
International Journal of Human-Computer Studies (Elsevier);
Pervasive and Mobile Computing (Elsevier)
Reviewer (Conference) ACE (2010, 2011);
APCHI (2012);
APSIPA Annual Summit and Conference (2010, 2011);
CHI (2008 -- 2012);
CSCW (2010);
DIS (2012, 2014);
GI (2014);
Internet of Things Conference (2008);
ITS (formally TableTop; 2008, 2011, 2012, 2014);
IUI (2010, 2012);
MobileHCI (2008, 2009, 2011 -- 2013);
NordiCHI (2012);
Pervasive (2010);
TEI (2012);
Ubicomp (2009 -- 2011);
UIST (2008 -- 2012, 2014);
3DUI (2009)
Reviewer (Japanese Domestic Journal) IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences;
IEICE Transactions on Information and Systems;
Transactions of Human Interface Society
Student Volunteer CHI (2010); IJCAI (2011)

Dr. Koji Yatani (http://yatani.jp) is an Associate Professor in Department of Electrical Engineering and Information Systems (EEIS), School of Engineering at The University of Tokyo, where he leads Interactive Intelligent Systems Laboratory (http://iis-lab.org). His main research interests lie in Human-Computer Interaction (HCI) and Ubiquitous Computing. His current research focus is to develop new applications and redesign user experience on mobile and wearable devices to better support user's creativity and productivity. He also extends his research interests to a broader range of topics in HCI, including mobile interaction techniques, wearable sensing methods, quantitative and quantiative studies for understanding users, productivity tool design, and assistive technologies.

He received his B.Eng. and M.Sci. from University of Tokyo in 2003 and 2005, respectively, and his Ph.D. in Computer Science from University of Toronto in 2011. On November 2011, he joined HCI group at Microsoft Research Asia in Beijing, China. On October 2013, he had an appointment of a Visiting Associate Professor in Graduate School of Information Science and Technology, at The University of Tokyo. He then joined The University of Tokyo as a full-time Associate Professor on August 2014. He was a recipient of NTT Docomo Scholarship (October 2003 -- March 2005), and Japan Society for the Promotion of Science Research Fellowship for Young Scientists (April 2005 -- March 2006). He received five Best Paper Award and Honorable Mention Awards at CHI (2011 and 2014) and MobileHCI (2014). He served as a program committee on major international conferences in the field of HCI, Ubiquitous computing and Haptics, including CHI (2013), UbiComp (2012 -- 2014), UIST (2013), MobiSys (2014), and WHC (2013). He is serving as a Program co-Chair for UbiComp 2015.