Here is the source.

In this implementation, the mouse is used to update the cursor but ultimately this would of course be driven by the eye point. This demo also features buttons with preset messages that the computer can speak (using utilities from Cocoa) and a method for sending out the cursor data over a local network via OpenSoundControl, so that we can ultimately have the EyeWriter talk to other apps.

a quick update from the office with the new eyewriter 2.0 design in which ito tag’s “GRL” :

EyeWriterGRL0518

Keys Sized According to Use in English Language

Here are some of the earlier prototypes.

Qwerty Interface

Alphabetical Interface

I wanted to take some of the ideas from the design team and begin to put them into practice. Unfortunately, a lot of their design requires reworking the interface, but this is a proof of concept type model using the ofxControlPanel interface tools.

First Step in Interface Update

Next step in this area is to start from scratch and really think about how the system is put together and the different views/tools fit into it.

Here’s the latest circuitry for detecting “bright eye” and tracking glints!  “Bright eye” occurs when the eye is flooded with concentrated infrared light (we’ve placed these right by the camera lens) causing the pupil to turn white and making it easy for us to know where the eye is within the camera frame. By alternating between turning on/off circuit A (LEDs around the camera) and circuit B (two  hexagonal-shaped clumps of 10 infrared LEDs, one to the left and one to the right of the camera), we alternate between seeing the bright eye pupil and then the glint created from the left and right set of LEDs. Using subtraction between these two frames we can clearly find the pupil and detect where it’s looking.

Subtraction between Bright Eye and Glint (video)

A few notes/issues about the overall circuit/testing:

• We need to test this out under different lighting conditions. Most testing has been done in a low lit room at night. We need to test at different times of day with varying amounts of sunlight.
• We’d like to try to design the circuit without an Arduino to bring down the costs. Next step is to try making a Hackduino (instructable can be found via hackduino.org courtesy of Joe Saavedra).
• An issue we’ve working around but would like to fix is that the bright eye/glint detection is most precise when the eye is not in the top of the frame. The top of the frame are at risk of not lighting up the eye properly because the CMOS camera’s scrolling shutter interferes. The top 1/5th of frame is at risk of not lighting up the eye properly because the camera’s scrolling shutter interferes. If anyone has ideas for how to better account for the camera’s shutter and give us more real estate with which to detect the pupil, please let us know by commenting on this post.

DETAILS OF THE CIRCUITRY

Programmed using Arduino: For the code, see the bottom of our last post. Bottom left of this schematic shows how the parts of the circuit hook up to Arduino

Circuit A (top left): We’ve changed our camera lens to a narrower 25mm lens with a IR filter. Surrounding the lens are 16 IR LEDs, which are soldered together and wired (with resistors) to the circuit board.

Circuit B (right): Two sets of 10 IR LEDs. Each set of LEDs are soldered together and wired (with resistors) to the circuit board.

In order to control/ensure that the brightness of the face is the same in all frames (so that we can do a clean subtraction and not get any noise from the face) we’ve attached a potentiometer dial to the LED circuits so we can match the overall brightness of circuit A’s LEDs with the overall brightness of circuit B’s LEDs.

As mentioned in a previous post, we’re setting the VSYNC signal from the PS3 Eye Camera at 60 frames per sec (60 fps) and alternating between circuit A and circuit B every other frame. We’ve used openFrameworks to save 300 consecutive camera images. Stepping through the 3oo frames we see, for example, that the odd frames show the bright eye and the even frames show the glint from the right/left LED clusters. (Note: it’s not quite exact but very close. Out of 300 frames, about two did not have the right light source showing. However, we believe for our purposes that this is fine.)

The magnification is performed in gray scale which hopefully doesn’t strain the eyes.

The source code sections are in the below hosted files.
Source Code

We’ve been testing alternating the LEDs  with the camera’s frame rate to turn on IR every other frame. Along with this we’ve created a circuit to create enough IR light to give us the white “bright eye”. Ito did some legwork, and figured out the specifications: 10 infrared LEDs (Agilent HSDL-4220 880nm diodes) in a circle on a small (1 7/8″ x 2 1/4″) solderable perf-board MED copper pad circuit board that fits around the Eye Camera lens. (We had to use a drill to cut a hole in the middle of the board that’s big enough to fit around the lens.) And to be safe, we added resistors to the LEDs. After setting up the LED ring around the Eye Camera, we hooked up the LED ring and the wires attached to the camera’s VSYNC to Arduino.

Thanks to Kyle, we are continuing the test for alternating 2 LED light sources. We improved the Arduino code (below) to control the lighting duration and the timing for each frame. To check this, we made an application with openFrameworks to see successive 64 frames in one screen. Unfortunately there are still some bad frames – some frames both LEDs are off/not alternating, which might be caused by how we’re programming the capturing of the LEDs.

References:

Carlos Morimoto, Dave Koons, Arnon Amir, Myron Flickner. Real-Time Detection of Eyes and Faces. In Proceedings of 1998 Workshop on Perceptual User Interfaces, pages 117-120, San Francisco, CA, November 1998.
http://www.acm.org/icmi/1998/Papers/Morimoto.pdf

C.H. Morimoto, D.koons, A.Amir, M.Flickner. Frame-Rate Pupil Detector and Gaze Tracker. IEEE ICCV’99 FRAME-RATE Workshop, 1999.
http://sirl.stanford.edu/~bob/pdf/EyeTracking/Morimoto_pupilcam_preprint99.pdf

Craig Hennessey, Borna Noureddin, Peter Lawrence. A Single Camera Eye-Gaze Tracking System with Free Head Motion. Proceedings of the 2006 symposium on Eye tracking research & applications, March 27-29, 2006, San Diego, California.
http://portal.acm.org/citation.cfm?id=1117349

Y. Ebisawa. Unconstrained pupil detection technique using two light sources and the image difference method. Visualization and Intelligent Design in engineering and architecture, pages 79-89,1995.
http://library.witpress.com/pages/PaperInfo.asp?PaperID=10807

Wikipedia ”Red eye effect”
http://en.wikipedia.org/wiki/Red-eye_effect

Below is the latest Arduino code.

/// =====================
// Arduino code
// =====================

// Using Arduino MINI04
// LED1 is connedted to pin11 with 1k resistor.
// LED2 is connected to pin12 with 1k resistor.

// VSYNC signal from PS3EYE is connected to pin2.
// Port Registers are used to control LED accurately.

int currentTurn = 0;
int prevTurn = 0;
int count = 0;

int delarray[100];
int delcounter;
int delval;
int prevDelval;

int state = 0;        // 1: LED1, 2: LED2

int DURATION = 360;           //Duration of lighting
int DELAYTIME = 0;            //Delay time

void setup() {

  // Port D pin order : 7,6,5,4,3,2,1,0
  // Port B pin order : ?,?,13,12,11,10,9,8

  DDRD = DDRD & B00000011;                  // set pin 7-2 as INPUTs
  DDRB = DDRB | B00011000;                  // set pin 11,12 as OUTPUTs
  delcounter = 0;
  delval = 0;

}

void loop(){

   if (((PIND & B00000100) != 0) && prevTurn == 0) {  // check pin2 & prevTurn
     currentTurn++;
     currentTurn = currentTurn % 2;
     prevTurn = 1;
   } else if ((PIND & B00000100) == 0) {              // check pin2
     prevTurn = 0;
   }

   // Delay Process
   delarray[delcounter] = currentTurn;
   delval = delarray[(delcounter - DELAYTIME) % 100];

   prevDelval = delarray[(delcounter - DELAYTIME - 1) % 100];

   //LOW -> HIGH or HIGH to LOW
  if (delval == 1 && prevDelval == 0) {
    count = 0;
    state = 1;
  }

  if (delval == 0 && prevDelval == 1) {
    count = 0;
    state = 2;
  }

  //
  if (count < DURATION) {

    if (state == 1) {
      PORTB = B00010000;              // set pin12 ON, pin11 OFF
    } 

    if (state == 2) {
      PORTB = B00001000;              // set pin12 OFF, pin11 ON
    }

    count++;

  } else {
    PORTB = B00000000;                // set pin11 & 12 OFF
  }

  delcounter++;
  delcounter = delcounter % 100;

}

// ======================================================

Download the source code featuring this scene!

Tech notes

The message configuration XML file is “messagePresets.xml” in the bin/data folder. The layout of the buttons themselves can also be customized from that file (number of buttons per row, button dimensions in pixels, and the spacing between each button). The speech synthesis objective-c method is declared and implemented in src/speech.h and src/speech.m respectively. Before compiling, be sure that you have the Cocoa framework added to libs/core/core frameworks/ (I had to add it manually).

This is a PDF describing our suggestions for a big update to the UI. This update continues the work done on the HUD and incorporates different clicking options, navigation and changes to user settings in all parts of the EyeWriter software.

Download the design team’s UI suggestion in PDF