[nigel92]

Hi all,
I am new to team-bhp. I have been a silent reader since 2009. I am currently pursuing my B.E. in electronics and communications and have decided to design a sleepy eye detection system as my final year project.

We( our project group) have approached a guy who has offered to guide us through. We have chosen MATLAB for the purpose of image processing. So far we have extracted the driver's face from the image and blackened out the background. We plan on triggering an alarm when the blink frequency of the driver's eyes fall below a certain level. I do not have much knowledge on image processing as I've never studied the subject before. Would really like to know the opinions of fellow members'.

Below is the original image and the image with the face extracted:

[Kaumad]

Good initiative Nigel.

I am not an expert in electronics but this..

Originally Posted by nigel92

We plan on triggering an alarm when the blink frequency of the driver's eyes fall below a certain level.

might not be the best method to evaluate.

Many OEMs have been successful in developing Driver Alert Systems.
To being with, please google for "PERCLOS" - it is a "Psychophysiological Measure of Alertness As Assessed by Psychomotor Vigilance". There are many articles related to this. Maybe you can read up on them and post your ideas.

All the very best!!

[NetfreakBombay]

This is an interesting project. Approach that you have taken seems feasible for a BE project.

Implementation will be based on "Feature Extraction" and analysis of those features.

To implement this, you need to:

1. Extract all "features" from image. Sample image is later in post.
2. Identify Eyes among the features. (Initially you can train the system for specific users).
3. Keep doing that in a loop for every "frame" in video.
4. If eyes change from open to close between frame, tag that as blink.

To do that, you can use few open source libraries. Examples are:


http://www.chrisevansdev.com/compute...-opensurf.html (Read "Notes on the OpenSURF Library" document)
Open CV : http://opencv.willowgarage.com/wiki/


Example image:

[muditgrover]

You can use a Raspberry Pi and use its RaspiCam module which can be mounted on the windscreen. You can use OpenCV to detect when the drivers eyes are on the road and have a beep when they are not..

I have bought a CubieTruck ARM board for a project, let me know if you need any help.

[DerAlte]

Good project!

Feature Extraction, as @netfreakBombay has described it, is the way to go. But, it is neither the only step, nor the starting step. Matlab is not the right environment, or rather - Matlab is only the programming environment. You have to have an independent executable for dynamic analysis, and you have to write a lot of code. Don't assume someone else has already written the code which you can conveniently use as modules or library calls. Anyhow, this being a college project, coding is not the objective (availability of time in a semester program will be an issue for you, apart from resources - TANSTAFL) nor the greatest achievement - as long as you prove your algorithm is sound.

The starting point is the physical algorithm, as @kaumad has pointed out. There is plenty of information available on how sleepiness is detected - some of which only humans can figure out, and some which can be distilled to a mathematical algorithm. To give you a clue, what you are looking for is not blinking 'rate'. Some people blink not so often, yet others do it rapidly - something to do with dryness of the cornea. You should look for blinking 'behaviour': *how* the eyelids open/close - sometimes fast, sometimes slowly, and sometimes not at all (open/shut). Your first endeavor has to be detection of eyelid, after detecting the eye region (that is relatively simple).

[NetfreakBombay]

nigel92,

Also look at this paper. It describes ways to extract Eye contours from images. A solution can be built around it (If your prof / guide agrees with approach).

http://citeseerx.ist.psu.edu/viewdoc...=rep1&type=pdf

[nigel92]

Mod Note : Please use the EDIT or MULTI-QUOTE buttons instead of typing one post after another on the SAME THREAD!.

To know how to multi-quote, click here.

Originally Posted by muditgrover

You can use a Raspberry Pi and use its RaspiCam module which can be mounted on the windscreen. You can use OpenCV to detect when the drivers eyes are on the road and have a beep when they are not..

I have bought a CubieTruck ARM board for a project, let me know if you need any help.

As of now we have planned to use a FPGA kit to dump the code and trigger the alarm.

@netferakbombay,Our guide is making it look relatively simple. I have no prior knowledge of image processing. I hope he's guiding us in the right direction. Our progress so far as seen from the first post is extracting the face. Now, our guide's approach is that the eye region is the only region in the image where the sequence of skin-white-black-white will exist. So he says that if this combination exists that means that the eye is open, if not, the eye is closed.

@Kaumad yeah i get your point. Maybe the speed of the blink will be better.

Appreciate the help

Nigel

[scopriobharath]

Here is a brief logic that i would use - by the by i used to design robots in my college days

• Capture face from a camera.
• Process two parts of the image namely eye lid movement (use amount of white color of eye as a baseline for identifying whether eyelids are closed or open).
• Secondly, capture sudden facial movement, similar to that of a person falling asleep and waking up.
• Process these images and make a microprocessor send out a beep / sound an alarm.

Of course, a nutty professor at college might ask you, how would this system work if one is driving with sun glasses.

The ones Mercedes / volvo use are through high end electronic systems, that detect, suddent throttle changes, lane movement (without indicators), steering veer and a host of other things like car approaching median in a turn etc and apply brakes. There is a sensor at every logical junture like steering rod, brake, bumpers etc which send signals to a central "Computer/"Chip" and that chips does the secondary task of sounding an alarm or slaming the brake, or readying the airbag for deployment.

[nigel92]

Originally Posted by scopriobharath

Here is a brief logic that i would use - by the by i used to design robots in my college days

• Capture face from a camera.
• Process two parts of the image namely eye lid movement (use amount of white color of eye as a baseline for identifying whether eyelids are closed or open).
• Secondly, capture sudden facial movement, similar to that of a person falling asleep and waking up.
• Process these images and make a microprocessor send out a beep / sound an alarm.

Of course, a nutty professor at college might ask you, how would this system work if one is driving with sun glasses.

The ones Mercedes / volvo use are through high end electronic systems, that detect, suddent throttle changes, lane movement (without indicators), steering veer and a host of other things like car approaching median in a turn etc and apply brakes. There is a sensor at every logical junture like steering rod, brake, bumpers etc which send signals to a central "Computer/"Chip" and that chips does the secondary task of sounding an alarm or slaming the brake, or readying the airbag for deployment.

@scorpiobharath, the part about the driver wearing sunglasses did cross our mind but we still went ahead with the idea. Compared to the sophisticated systems used in Mercs/Volvos this one will be much cheaper if successful. Our guide has assured us that he will complete the project by 31st March. Have my next class on Monday, till then have to try and detect the skin-white-black-white combination in the image.

Nigel

[scopriobharath]

Originally Posted by nigel92

@scorpiobharath, the part about the driver wearing sunglasses did cross our mind but we still went ahead with the idea. Compared to the sophisticated systems used in Mercs/Volvos this one will be much cheaper if successful. Our guide has assured us that he will complete the project by 31st March. Have my next class on Monday, till then have to try and detect the skin-white-black-white combination in the image.

Nigel

This logic should work - When the face is captured on turn of the key or slotting into gear, the wide open eyes would be captured. Due to fatiugue, they eyes close over the period of the drive. This picture could be used as a reference to capture fatigued eye. If the initial eye itself is blurry, it is probably because the person is under influence.

[NetfreakBombay]

Originally Posted by nigel92

@netferakbombay,Our guide is making it look relatively simple. I have no prior knowledge of image processing. I hope he's guiding us in the right direction.

Given that he is looking for things to be done by March 31st, MATLAB looks like the only solution.

MATLAB has bunch of plugins for video processing and feature extraction from frames. Do you have access to any of those in institute or elsewhere?

[nigel92]

Originally Posted by NetfreakBombay

Given that he is looking for things to be done by March 31st, MATLAB looks like the only solution.

MATLAB has bunch of plugins for video processing and feature extraction from frames. Do you have access to any of those in institute or elsewhere?

I have Matlab installed on my computer. I'll have to purchase the FPGA kit later. A webcam will be used for capture. Any idea of how to make this work in the night?

Nigel

[scopriobharath]

Originally Posted by nigel92

I have Matlab installed on my computer. I'll have to purchase the FPGA kit later. A webcam will be used for capture. Any idea of how to make this work in the night?

Nigel

Night vision IR camera can capture the "eye" as a bright white streak /ball

use this as a reference, compared to a sleepy eyelid.

[charanreddy]

My 2 cents on this interesting project.

1. In less than ideal light conditions or when there is some shaking, like it is bound to be when you are driving, the images captured from webcam are not like those in your first post. I had faced this problem practically when working on my final year project (Object tracking using Vision algorithms esp. Camshift algorithm) on a mobile robot moving at very slow speed, so I can only imagine how it would be on a fast moving vehicle on uneven roads.

2. Assuming you are able to capture usable pictures, Do check if mugshot/ feature detection algorithms can help. I.E Lets say in 20 to 30 successive frames captured 0.2 secs apart ( 4 to 6 seconds), if for majority of the time, you are not able to find the eyes open in majority of the cases, you can trigger an alarm.

A couple of pointers from one such algorithm I know is

Viola, Paul and Michael J. Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. Volume: 1, pp.511–518.

http://www.mathworks.in/help/vision/...ctorclass.html

Hope this is useful.

[nigel92]

Originally Posted by charanreddy

My 2 cents on this interesting project.

1. In less than ideal light conditions or when there is some shaking, like it is bound to be when you are driving, the images captured from webcam are not like those in your first post. I had faced this problem practically when working on my final year project (Object tracking using Vision algorithms esp. Camshift algorithm) on a mobile robot moving at very slow speed, so I can only imagine how it would be on a fast moving vehicle on uneven roads.

We never thought of this condition. Thanks a lot for pointing it out. Since our final demonstration won't be in a car we can get away with it unless the external examiner reads this thread.

2. Assuming you are able to capture usable pictures, Do check if mugshot/ feature detection algorithms can help. I.E Lets say in 20 to 30 successive frames captured 0.2 secs apart ( 4 to 6 seconds), if for majority of the time, you are not able to find the eyes open in majority of the cases, you can trigger an alarm.

Hope this is useful.

I tried to execute some codes from the link you mentioned but i was unsuccessful as i don't have the image processing toolbox installed. Is this addition available free of cost? I have my next class on Monday. At the moment we're stuck at detecting the eyes. Our guide told us to detect whether the eye is open or closed using only 'if else' statements. We haven't yet succeeded.


Nigel