Our program can currently detect mouse clicks on some menu items in Google Chrome. It then displays an alert in the corner of the screen, notifying the user that a button was clicked and also showing the shortcut for that button (currently in keycodes).
All mouse events and keypresses are currently also logged in the console for debugging purposes.
We detect clicks using OpenCV. By comparing pictures in our database with the screenshot taken at the time of a click we can find out what button was clicked.