EyeNav: Gaze-Based Code Navigation
- 8 minsSummary:
A detailed description of EyeNav, a code editor plugin for navigating source code using your gaze and keyboard shortcuts. The plugin is written in JavaScript, and the wrappers around the eye tracker SDKs are written in C# and Java. The end result is a plugin that allows you to scroll, click, select code, and more using your gaze (how cool is that). You can find the source code and more info on installation and usage here. You can see a demo video showing the capabilities and usage of EyeNav here. EyeNav has been published as a demo paper at NordiCHI 2016 conference (from where I shamelessly copied text and images).
Introduction
I can say with confidence that many developers try to keep their hands on the keyboard and avoid using the mouse as much as possible, and with a good reason - switching to the mouse is slow (and annoying). I had an eye tracker lying around, so I figured it will be cool to be able to do some basic navigation using your gaze without removing your hands from the keyboard. I tried it out, and it was working better than I expected. I started making a plugin for the editor I was using back then (Brackets.io), and EyeNav was born.
Simply said, EyeNav brings eye tracking to code editors. It allows for a more natural source code navigation, controlled by the developer’s gaze and keyboard shortcuts. You simply look somewhere, trigger a click shortcut for example, and it moves there, no cursor required.
To those unfamiliar with eye trackers, simply said, they are devices that track where you look on the screen. Although it sounds like the ideal input device, there are some accuracy issues that are explained in some details in the Eye Tracking Drawbacks section. Despite the issues, I still think it is good-enough for code navigation, especially as the technology advances and the prices drop. In particular, EyeTribe (seems to be down) and Tobii have some very affordable eye trackers, and their new trackers have gotten much better too. Tristan Hume has a nice review on few eye trackers that is worth checking. I think eye tracking has the potential to be embedded into computers and laptops in the near future, making tools like EyeNav much more accessible.
Eye Tracking Drawbacks
The main drawback of eye tracking is accuracy. Because of the physiology of the eye, even in ideal conditions, the error is ±0.5◦. That in turn means an error of around 6mm on a 21-inch display from 70cm viewing distance. Depending on the font size you use, the error usually won’t be more than ±1-2 characters. The second drawback is the need to calibrate as body movements happen. As most devices available on the market nowadays compensate for head movements, the need to calibrate and its impact on usability have dropped to virtually none.
Architecture
While building EyeNav I aimed at making it as flexible and extensible to different eye trackers as possible. In order to achieve flexible and loosely coupled architecture, I decided to base the architecture on WebSockets, separating device-specific logic to a thin server wrapping over the provided SDK, while keeping all the logic as an editor plugin, as shown below. This means EyeNav can be extended so it works with essentially any eye tracker on the market with minimal effort.
One cool benefit of having the tool built on top of WebSockets is that the eye tracker can run on a separate machine! You might think this is as useless as it gets, but some devices (Tobii EyeX for example) work only on Windows, so you can have it run in a Virtual Machine and it can send the gaze data to your editor on Linux or Mac. Another one is that you can theoretically have as many eye trackers as screens. So if you have, let’s say, two different files opened in the editor on two different screens, you can use EyeNav just looking at the respective screen, no need to drag a cursor around. You can also use it for focusing a screen based on gaze, and so on.
As for the source code, version 1.0 of EyeNav is very tightly coupled to the Brackets.io API, and I am currently working on separating the business logic from the editor API, so it can be easily shipped to any editor with a bit of wrapper code and no breaking changes. The next code editor I am aiming for is Atom.io, as it is the editor I currently use.
Functionalities
So, let’s talk how EyeNav works and what it has to offer in terms of functionalities.
As you can see from the Architecture section, EyeNav gets the data from a wrapper around the SDK of the respective eye tracker. The data contains the x and y coordinate of where the gaze is on the screen, plus some additional information on the status of the eye tracker. There is no need for explicit feedback on the gaze location (such as the cursor moving) as you always know where you are looking at, right? Whenever a shortcut is pressed, the respective command is executed. That’s it, simple as that. I think the demo video shows how everything works better than what I can explain in words, so head there and watch it! Head to the github page for more info on how to install and use EyeNav.
As mentioned before, commands are executed by shortcuts on the keyboard. The general shortcut pattern is Trigger Key + (Modifier Key) + Command Key, pressed in the listed order. The trigger key essentially switches the keyboard key meaning between standard key functions and EyeNav-related functions. More details about it on the github page.
Clicking
The clicking functionality doesn’t need much explanation as it does exactly what you would expect - when the shortcut is pressed, the caret moves to the gaze position. In order to mitigate some of the accuracy issues with eye tracking, some guesswork is done when you are doing a click. When clicking, the caret is placed at the nearest code in both x and y directions, as shown in the image below (the eye target represents the gaze location).
Page Scrolling
Once again, no need for extensive explanation. Currently, only vertical scrolling is supported as text wrapping is usually on for me, but one of the next tasks is to implement horizontal scrolling as well. As you can see from the image below, the scrolling speed increases by following a square function the further away you look from the center. There is a threshold of around 10% around the center where no scrolling happens. When trying to scroll to some code outside of the page, you would naturally look further away from the center, thus scrolling at a high speed. As the code of interest enters the screen, the eyes will naturally follow it, automatically decreasing the scrolling speed, until the scrolling halts. As a result, the scrolling feels very natural.
Code Scrolling
This is one of the functionalities that might not be clear just by reading the name. What code scrolling allows you to do is move the caret only in one direction, horizontally or vertically. This improves accuracy over a normal click because the error is minimized to only one axis. It is particularly useful for moving the caret horizontally to the middle of a long variable name, for example. As you can see on the image below, no matter how far you look along the y axis, the caret will move only in the current line of code.
Once Character Movements
As mentioned several times, accuracy is the main problem with eye tracking. This means that when you want to do a click using EyeNav, the caret may be positioned 1-2 characters away from your intended location. In order to fix this quickly, I mapped the arrow keys to WASD (well known keys to most gamers) for easier access.
Conclusion and Future Goals
Eye tracking has the potential to make software development smoother, and EyeNav is an attempt to make that available to developers. EyeNav is still in its youth as a project, and there is much more to be done before it becomes a mature tool. The first main goal is to do a complete refactor and ship EyeNav to Atom.io, as it is among the most popular code editors nowadays. Once refactored, a test suite will be written. Some of the planned improvements are smoothing of gaze data, implementing more heuristics for more accurate navigation, implement file navigation functionality, and much more. Stay tuned and try EyeNav out!
Stevche Radevski
Full-stack Software Engineer, with focus on JavaScript. Working at balena.io. Open-source supporter. Tinkerer. Trilingual.