Play all

Intro

Introduction Blind recognition of touched keys Evaluation Countermeasures Conclusion

Use computer vision techniques to recognize possible touched keys and use a language model to correct the prediction. Poor success rate for passwords.

Taking Videos Use sneaky cameras including Google Glass, web cameras, smartphone cameras, even smartwatch! - Factors: camera angle, distance, lighting Adjust the camera angle at a distance to record …

Preprocessing Keep the area of moving hand on screen. - Use Deformable Part-based Model (DPM) - an object tracker - to track the area of interest for a moving target

Detecting Touching Frames Derive a pattern of the touching finger movement Finger moves downward, stops and then upward. Track feature points on the hand by optical flow. Al fingers keep the same ges…

Deriving Homography Matrix Derive touch screen corners, intersection of the four edges of the touch screen. Canny edge detector to detect edges Hough line transform to get the lines. Use these four p…

Locating Touching Fingertip Use the DPM object detector to locate the touching fingertip in touching frames. Derive a large box bounding the touching fingertip.

Estimating Touched Area Deriving the fingertip contour Use k means clustering to cluster pixels in a small bounding box to get the fingertip contour. Two groups bright fingertip and dark screen. Deri…

Recognizing Touched Keys Which pixels are the touched points in this estimated tiny touched area? If the touched point is found, map the estimated touched point to the reference image of the software…

Step 7. Recognizing Touched Keys (Cont'd) Apply k-means clustering to estimated touched area. -5, because of illumination and shadowing Use a point in the upper part of the darkest cluster as the tou…

Description:

Discover a novel computer vision-based attack that automatically discloses touch screen inputs, including passcodes, from a distance of three meters using spying cameras like Google Glass. Learn how the attack tracks fingertip movements, analyzes shadow formations, and applies advanced computer vision techniques to recognize over 90% of tapped passcodes, even when the screen is not visible to the naked eye. Explore the methodology behind this stealthy attack, including optical flow, deformable part-based model object detection, k-means clustering, and planar homography. Understand the potential impact of this security threat in public spaces and conferences. Examine the proposed countermeasure: a context-aware Privacy Enhancing Keyboard (PEK) for Android systems that randomizes the keyboard layout for sensitive information input.

My Google Glass Sees Your Passwords

Black Hat

Add to list

#Conference Talks #Black Hat #Information Security (InfoSec) #Cybersecurity #Computer Science #Artificial Intelligence #Computer Vision #Object Detection #Optical Flow

0:00 / 0:00