Cursor and Modal GUI Proof of Concept

I’m working on a new version of an eye-driven cursor that energizes as its being activated. As the cursor energizes, it draws the outline of a circle. The circle is completed as the cursor has reached its max energy and thereby gets activated. The cursor begins this activation routine either when it is over a button, or if it comes to rest on a certain spot for a few seconds. If the cursor is activated and it is not over a button, a modal with its own unique button will appear over where the cursor is. The modal can then be hidden by moving the cursor to any edge of the stage.

Here is the source.

In this implementation, the mouse is used to update the cursor but ultimately this would of course be driven by the eye point. This demo also features buttons with preset messages that the computer can speak (using utilities from Cocoa) and a method for sending out the cursor data over a local network via OpenSoundControl, so that we can ultimately have the EyeWriter talk to other apps.

Head Mount Update

For the last few weeks, I’ve been working in creating a kit for a more reliable head mount for the Eye Writer. For that matter, I designed some adjustable parts that can be assembled into a reliable support arm for the camera. The parts were laser cut (you can find the laser cutter file here) and glued together and then I made a mold out of urethane rubber and casted some copies with polyesther resin.
Original and Mold
I tested the mount with the resin copies and arrived to the following conclusions:
- The material is adequate for the function that is going to perform. It can hold the weight of the electronics while remaining light weight.
- The gear locks that support the joints need to have a smaller angle between the teeth to allow a more precise adjustability of each part with respect to the others.
- A way to adjust the depth of the camera is missing. Making the middle arm a slider rail might solve the problem.
I will continue working on perfecting the new head mount.
head mount assembly

Posted in Uncategorized | 1 Reply

Typing Interface Update

Played around with some different keyboard interfaces. I’m really excited about the possibilities with this one! Now just need to test it with the actual eyewriter system.

Keys Sized According to Use in English Language

Here are some of the earlier prototypes.

Qwerty Interface

Alphabetical Interface

I wanted to take some of the ideas from the design team and begin to put them into practice. Unfortunately, a lot of their design requires reworking the interface, but this is a proof of concept type model using the ofxControlPanel interface tools.

First Step in Interface Update


Next step in this area is to start from scratch and really think about how the system is put together and the different views/tools fit into it.

Bright Eye/Glint Update

We’ve made great process to improve the use of bright eye/glint to detect the eye and where it’s looking. For a bit of background, see our last post.

LED boardsHere’s the latest circuitry for detecting “bright eye” and tracking glints!  “Bright eye” occurs when the eye is flooded with concentrated infrared light (we’ve placed these right by the camera lens) causing the pupil to turn white and making it easy for us to know where the eye is within the camera frame. By alternating between turning on/off circuit A (LEDs around the camera) and circuit B (two  hexagonal-shaped clumps of 10 infrared LEDs, one to the left and one to the right of the camera), we alternate between seeing the bright eye pupil and then the glint created from the left and right set of LEDs. Using subtraction between these two frames we can clearly find the pupil and detect where it’s looking.

Subtraction between Bright Eye and Glint (video)

A few notes/issues about the overall circuit/testing:

  • We need to test this out under different lighting conditions. Most testing has been done in a low lit room at night. We need to test at different times of day with varying amounts of sunlight.
  • We’d like to try to design the circuit without an Arduino to bring down the costs. Next step is to try making a Hackduino (instructable can be found via hackduino.org courtesy of Joe Saavedra).
  • An issue we’ve working around but would like to fix is that the bright eye/glint detection is most precise when the eye is not in the top of the frame. The top of the frame are at risk of not lighting up the eye properly because the CMOS camera’s scrolling shutter interferes. The top 1/5th of frame is at risk of not lighting up the eye properly because the camera’s scrolling shutter interferes. If anyone has ideas for how to better account for the camera’s shutter and give us more real estate with which to detect the pupil, please let us know by commenting on this post.

DETAILS OF THE CIRCUITRY

Programmed using Arduino: For the code, see the bottom of our last post. Bottom left of this schematic shows how the parts of the circuit hook up to Arduino

Circuit A (top left): We’ve changed our camera lens to a narrower 25mm lens with a IR filter. Surrounding the lens are 16 IR LEDs, which are soldered together and wired (with resistors) to the circuit board.

Circuit B (right): Two sets of 10 IR LEDs. Each set of LEDs are soldered together and wired (with resistors) to the circuit board.

In order to control/ensure that the brightness of the face is the same in all frames (so that we can do a clean subtraction and not get any noise from the face) we’ve attached a potentiometer dial to the LED circuits so we can match the overall brightness of circuit A’s LEDs with the overall brightness of circuit B’s LEDs.

As mentioned in a previous post, we’re setting the VSYNC signal from the PS3 Eye Camera at 60 frames per sec (60 fps) and alternating between circuit A and circuit B every other frame. We’ve used openFrameworks to save 300 consecutive camera images. Stepping through the 3oo frames we see, for example, that the odd frames show the bright eye and the even frames show the glint from the right/left LED clusters. (Note: it’s not quite exact but very close. Out of 300 frames, about two did not have the right light source showing. However, we believe for our purposes that this is fine.)

amit pitaru visits class

we we’re quite lucky to have a visit from Amit Pitaru, who discussed his work creating interfaces and software to help students with various disabilities play video games.  It was quite an inspiring lecture about assistive technology, his research and process and how it could relate to the development of the eyewriter.

Magnifier Add-on

Magnifier add-on enables portion of the eye tracking console to a magnified size without interrupting the functions to be performed. It operates by toggling ‘M’/'m’ key. The magnification moves across with the cursor.

The magnification is performed in gray scale which hopefully doesn’t strain the eyes.

The source code sections are in the below hosted files.
Source Code

Bright Eye effect

We found some articles that informed us about how to capture alternating frames of bright eye/dark eye. The “bright eye” concept is something we first heard about from Mark at Eyegaze, a very precise commercial eye tracking system, which uses this bright eye/dark eye method to track eye movement. Similar to the “red eye” effect when taking photos with flash (the red is the light bouncing off the back of your eye and reflecting the color of the blood in your eye), the “bright eye” occurs when enough concentrated infrared light bounces off the back of your eye and reflects as white. (Below is a photo of the “bright eye” taken in a dark room making the pupil large.)

IR "bright eye"64 fps testing alternating LEDsWe’ve been testing alternating the LEDs  with the camera’s frame rate to turn on IR every other frame. Along with this we’ve created a circuit to create enough IR light to give us the white “bright eye”. Ito did some legwork, and figured out the specifications: 10 infrared LEDs (Agilent HSDL-4220 880nm diodes) in a circle on a small (1 7/8″ x 2 1/4″) solderable perf-board MED copper pad circuit board that fits around the Eye Camera lens. (We had to use a drill to cut a hole in the middle of the board that’s big enough to fit around the lens.) And to be safe, we added resistors to the LEDs. After setting up the LED ring around the Eye Arduino Circuit SchematicCamera, we hooked up the LED ring and the wires attached to the camera’s VSYNC to Arduino.

Thanks to Kyle, we are continuing the test for alternating 2 LED light sources. We improved the Arduino code (below) to control the lighting duration and the timing for each frame. To check this, we made an application with openFrameworks to see successive 64 frames in one screen. Unfortunately there are still some bad frames – some frames both LEDs are off/not alternating, which might be caused by how we’re programming the capturing of the LEDs.

References:

Carlos Morimoto, Dave Koons, Arnon Amir, Myron Flickner. Real-Time Detection of Eyes and Faces. In Proceedings of 1998 Workshop on Perceptual User Interfaces, pages 117-120, San Francisco, CA, November 1998.
http://www.acm.org/icmi/1998/Papers/Morimoto.pdf

C.H. Morimoto, D.koons, A.Amir, M.Flickner. Frame-Rate Pupil Detector and Gaze Tracker. IEEE ICCV’99 FRAME-RATE Workshop, 1999.
http://sirl.stanford.edu/~bob/pdf/EyeTracking/Morimoto_pupilcam_preprint99.pdf

Craig Hennessey, Borna Noureddin, Peter Lawrence. A Single Camera Eye-Gaze Tracking System with Free Head Motion. Proceedings of the 2006 symposium on Eye tracking research & applications, March 27-29, 2006, San Diego, California.
http://portal.acm.org/citation.cfm?id=1117349

Y. Ebisawa. Unconstrained pupil detection technique using two light sources and the image difference method. Visualization and Intelligent Design in engineering and architecture, pages 79-89,1995.
http://library.witpress.com/pages/PaperInfo.asp?PaperID=10807

Wikipedia ”Red eye effect”
http://en.wikipedia.org/wiki/Red-eye_effect

Below is the latest Arduino code.

/// =====================
// Arduino code
// =====================

// Using Arduino MINI04
// LED1 is connedted to pin11 with 1k resistor.
// LED2 is connected to pin12 with 1k resistor.

// VSYNC signal from PS3EYE is connected to pin2.
// Port Registers are used to control LED accurately.

int currentTurn = 0;
int prevTurn = 0;
int count = 0;

int delarray[100];
int delcounter;
int delval;
int prevDelval;

int state = 0;        // 1: LED1, 2: LED2

int DURATION = 360;           //Duration of lighting
int DELAYTIME = 0;            //Delay time

void setup() {

  // Port D pin order : 7,6,5,4,3,2,1,0
  // Port B pin order : ?,?,13,12,11,10,9,8

  DDRD = DDRD & B00000011;                  // set pin 7-2 as INPUTs
  DDRB = DDRB | B00011000;                  // set pin 11,12 as OUTPUTs
  delcounter = 0;
  delval = 0;

}

void loop(){

   if (((PIND & B00000100) != 0) && prevTurn == 0) {  // check pin2 & prevTurn
     currentTurn++;
     currentTurn = currentTurn % 2;
     prevTurn = 1;
   } else if ((PIND & B00000100) == 0) {              // check pin2
     prevTurn = 0;
   }

   // Delay Process
   delarray[delcounter] = currentTurn;
   delval = delarray[(delcounter - DELAYTIME) % 100];

   prevDelval = delarray[(delcounter - DELAYTIME - 1) % 100];

   //LOW -> HIGH or HIGH to LOW
  if (delval == 1 && prevDelval == 0) {
    count = 0;
    state = 1;
  }

  if (delval == 0 && prevDelval == 1) {
    count = 0;
    state = 2;
  }

  //
  if (count < DURATION) {

    if (state == 1) {
      PORTB = B00010000;              // set pin12 ON, pin11 OFF
    } 

    if (state == 2) {
      PORTB = B00001000;              // set pin12 OFF, pin11 ON
    }

    count++;

  } else {
    PORTB = B00000000;                // set pin11 & 12 OFF
  }

  delcounter++;
  delcounter = delcounter % 100;

}

// ======================================================

Preset messages scene prototype

I added a simple scene to the drawing portion of the EyeWriter app, which displays an array of buttons with preset phrases on them. The phrases themselves are customizable –> they’re specified in an XML file and then a button per phrase is generated in the scene. When a button is triggered, the computer speaks the phrase out loud. I used OSX’s built in speech synthesizer via the Cocoa framework. There should hopefully be analogous ways to do it on other operating systems.

screenshot of preset messages scene prototype

Download the source code featuring this scene!

Tech notes

The message configuration XML file is “messagePresets.xml” in the bin/data folder. The layout of the buttons themselves can also be customized from that file (number of buttons per row, button dimensions in pixels, and the spacing between each button). The speech synthesis objective-c method is declared and implemented in src/speech.h and src/speech.m respectively. Before compiling, be sure that you have the Cocoa framework added to libs/core/core frameworks/ (I had to add it manually).

Posted in Uncategorized | 1 Reply

Design Team Ideas for the User Interface

The design team has been working on ideas for improvements to the EyeWriter software’s user interface, and combining our ideas with the different updates and features that other teams have already started developing into one interface.

This is a PDF describing our suggestions for a big update to the UI. This update continues the work done on the HUD and incorporates different clicking options, navigation and changes to user settings in all parts of the EyeWriter software.

Download the design team’s UI suggestion in PDF