Eye Gaze Outdoors, Wear a Baseball Cap

This week at the Microsoft Ability Summit 2015, we held another hackathon to work on technology for people with disabilities. The team I worked on focused on the problem of using Tobii's EyeX sensor in sunlight where IR interference often makes the sensor malfunction.

We wrote a simple application to measure noise in the eye gaze signal and then tried a number of different mitigation techniques:

  • Add an additional high end IR bandpass filter on top of the EyeX camera
  • Wear a baseball cap, tightly fitted to the brow
  • Add a 'duck bill' extension to the baseball cap, increasing the blockage of the sunlight
  • Use an umbrella to block direct light to the face and the EyeX sensor
  • Add tinted windshield (smoked polycarbonate) to the wheelchair

We then measured the effectiveness of each technique at rest. Keep in mind that vibration from a moving wheelchair also increases noise, so these measurements taken at rest may be optimistic.

  • A tightly fitting baseball cap makes a huge positive impact in the viability of eye gaze outdoors. Often it makes the difference between 'cannot see eyes' at all and 'working but noisy'.
  • Adding an umbrella is helpful when sunlight is falling directly on the EyeX sensor. It reduces noise by 10-20%.
  • Adding a tinted windshield or a supplemental IR filter did not improve signal quality.

Pics or it didn't happen

The Test Harness
Abhi and Yossi using our test harness to take measurements

IR Bandpass Filter
Attaching an IR filter to the Tobii EyeX using a 3d printed mount

Duck Billed Baseball Cap
The Duck Billed Baseball Cap (that did not work as well as a baseball cap)

The Full Monty Wheelchair
The Full Monty Wheelchair


Our measurement technique was to take 3 seconds of sample data (90 samples @ 30 frames/second) and then calculate the standard deviation from the mean (e.g. assume you're looking at a fixed point). This is measured in pixel distance as returned by the XY gaze data stream from the EyeX SDK. For the EyeX, these metrics look like:

Noise Result
  • Indoors, best conditions
  • No IR interference sources like halogen or sunlight through windows
  • Focused individual proficient at eye gaze
  • Indoors, light interference from halogen or indirect sunlight
  • Sufficient to use an eye gaze keyboard
  • Sufficient to drive a wheelchair
  • Indoors, bright lighting, direct sunlight
  • Outdoors with full cloud cover
  • Not sufficient to use an eye gaze keyboard
  • Sufficient to drive a wheelchair
  • Outdoors, light cloud cover
  • Outdoors, full sunlight, baseball cap, umbrella
  • Not sufficient to use an eye gaze keyboard
  • Sufficient to drive a wheelchair, *jerky*
  • Outdoors, full sunlight, baseball cap, no umbrella
  • Not sufficient to drive a wheelchair
  • Outdoors in full sunlight or partial cloud cover with no baseball cap, with or without an umbrella
  • Does not occur, EyeX loses track of eyes
  • Not sufficient to drive a wheelchair