Interactive Mirror

Alix Katague, ack242

Jeffery Lee, jyl58

Objective

  • Build a smart mirror with a variety of display modules that can be controlled with a user touch interface
  • User interface detects taps and swipes on the mirror with a matrix of IR break beam sensors
  • “Passcode” swipe required to “unlock” the mirror protects the user’s private information (calendar display module, etc)

Introduction

Our project is based on the popular Magic Mirror project that is available online here. We decided to improve this product by adding user interaction. With the incorporation IR break beam sensors, users can now interface with the mirror and control the displays. The IR sensors elevate this mirror to a fully fledged product that users can interface with and use on a daily basis.

Design and Testing

Overall Design


Since this was based on the pre-existing project, the first step was to get this base running. We began this process by obtaining the necessary materials, as listed in the Bill of Materials below. For the base of this project, we needed a monitor, a one-way mirror, and a Raspberry Pi. The first step was to strip the monitor of the shell so that it only had the display. Next, we had to create the one-way mirror. Due to cost constraints, simply buying a one-way mirror was too expensive, so we elected to buy a glass plane and cover with reflective material. The glass was standard glass that we cut to the size of the monitor. The reflective material was intended as a privacy screen to be applied to windows which served our purposes since it essentially made a one-way mirror. We used epoxy to glue the mirror to the display and at that point, we had everything we needed for the mirror base code.


To get the mirror running, in terms of software, it was a simple task of installing the software since the Github implementation provided an easy installer. We used the provided Magic Mirror installer on a fresh installation of NOOBS on the RPi. From here, it was just following the directions to get the mirror up and running.


Next, we evaluated the capabilities and specs of the IR break beam sensors. We wrote a simple program in Python that printed whether the sensors were connected or not. Our primary objective was to test the sensors to determine the maximum distance between the transmitter and receiver. We found that the sensors still worked at 21 inches, which was the width of our monitor so we could use the sensors on either axis of our mirror.


We then had to incorporate the IR break beam sensors into the existing Magic Mirror code. There were some issues with this part since the pre-existing Magic Mirror code was written in Javascript. As such, we needed to find a sufficient package that could check the GPIO pins. We did find one, but we had issues importing the package. As such, we opted to code this in Python to utilize the RPi.GPIO library. The Python code would write to a file that the Javascript would read from. As such, the Javascipt would get information from the IR break beam sensors by monitoring the file.


We then added the sensor functionality. After this, the user would be able to interact with the screen. There were three main functionalities that we added: unlocking the screen, swiping between pre-determined pages, and hiding and showing modules.


For the unlock screen, we required specific beams to be broken in a certain order, or else the mirror would be unresponsive. Once unlocked, the user can utilize the other to features to interact with the mirror. The unlock screen was implemented for a matter of privacy. One of the modules is a calendar, and so to keep that private and not available to every passerby, the unlock screen is essential to privacy.


Next, we implemented swipes so that the user can go between pre-designed screens. The user can swipe either up or down, and it just cycles through the different layouts. This allows the user to control the screen display and lets them customize it to their liking. These swipes are detected when the two sensors that span the width of the mirror are broken and the direction of the swipe is determined by the order that they are broken.


The last feature is the ability to hide certain modules, specifically the ones in the top corners. The top left has the calendar and clock and the top right has the current weather and weather forecast. They can tap these modules to either hide or show them, and allows for another layer of privacy, especially in terms of the calendar. These taps are detected with a pair of IR sensors where one is on the horizontal axis, and the other on the vertical. If both sensors are broken at the same time, then we have a tap in that location.


Software Design


This is a picture


There were two main parts to our code: getting the information from the break beam sensors and altering the display as necessary.


The first part, interacting with the sensors, was coded in Python. The main reason for this was the ease of the RPi.GPIO library that allowed us to easily read information from the GPIO pins. The code would first loop and look for the specific passcode gesture. Once this gesture was detected, it would proceed on to taking in user interactions. The code would read the IR sensors and based on which sensors were broken and when they were broken relative to the other sensors, write either ‘l’, ‘r’, ‘u’, or ‘d’ into ir.txt. These stood for ‘left’, ‘right’, ‘up’, and ‘down’ and signaled what type of interaction was signaled. After these signals are detected, we take a 1 second delay to prevent multiple detections of the same motion. The python continually monitors the IR sensors and writes to the file whenever something is detected.


The second part is to alter the display as necessary. The Magic Mirror is set up as a set of modules, each essentially running independently of each other and can be placed in certain locations. These modules were managed by a server that started them up and monitored their status. We elected to make our code another one of these modules with the ability to hide and show other modules. It would read the file written by the python code, and depending on the contents, react accordingly. If it was a tapping gesture, the module would either hide or show the necessary modules and if it was a swiping motion, it would go to the next screen and show the necessary modules and hide the others. We did run into some issues with the modules and importing packages but we were able to resolve this by creating a helper_node module and using socket notifications to send data. The helper_node module was able to import the necessary packages to read local files and send the gestures to the main module.


Hardware Design


This is a picture


We used the Raspberry Pi 3 Model B microprocessor, with the Adafruit Pi Cobbler and Breakout Cable to connect to a standard breadboard. The IR break beam transmitters are connected directly to the 5V and ground rails on the breadboard. The IR break beam receivers are connected to the 5V and ground rails as well as to individual signal pins (GPIO 12, 21, 23, and 25) in series with 10k Ω resistors in order to protect the system in the event of a software error. Each of the wires was extended with soldering and heat shrinking in order to ensure that the sensors would reach all sides of the mirror. The horizontal beams are relatively close together so the sensors were attached with one receiver and one transmitter on each side so as to prevent the receivers reading incorrect signals.


Results and Conclusions

We were able to meet our goals for our project. We were able to take the Magic Mirror project and bring it to a new level. The users can now interact with the mirror and have a far greater degree of control over the display. It provides privacy and security in terms of the calendar and unlock screen, and lets the user decide what information they want to see.


Mostly everything did end up working as planned. We were unable to implement the usage of a camera as we originally intended, but perhaps this turned out to be a positive since a camera on a mirror in a bathroom could prove to be... controversial. In terms of the IR sensors, we went beyond our initial goals and was actually able to implement more features than originally intended.


Future Work

Next steps for this project would be to add more functionality, both with the IR sensors and perhaps other methods of interacting with the screen. Currently, there are four IR sensor pairs on the mirror but if this number were to be increased, we could have far more complex interactions.

Code Appendix

checkForSwipes.py 


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
#Alix Katague (ack242) and Jeffery Lee (jyl58). ECE 5725 SP17. Cornell University.
#output writes "d" "u" "l" or "r" char to file to be read by javascript interface
import time,RPi.GPIO as GPIO

beamT = 23 #horizontal beam on top
beamB = 12 #horizontal beam on bottom
beamL = 25 #change later. vertical left
beamR = 21 #change later. vertical right

GPIO.setmode(GPIO.BCM)
GPIO.setup(beamT,GPIO.IN)
GPIO.setup(beamB,GPIO.IN)
GPIO.setup(beamL,GPIO.IN)
GPIO.setup(beamR,GPIO.IN)

swipeDelay = 100000
timeDelay = 0.25

#Look for passcode
loopbreak = False
unlocked = False
while not unlocked:
		loopbreak = False
		if(not GPIO.input(beamL)):
				print("1")
				for i in range(swipeDelay):
						if(not GPIO.input(beamB)):
								print("2")
								f = open('/home/pi/ir.txt', 'w')
								f.write("u")
								f.close()
								unlocked = True
								print("UNLOCKED")
								loopbreak = True
								time.sleep(1)
								break
						if(i == 99999 or loopbreak or not GPIO.input(beamT) or not GPIO.input(beamR)):
								break
#Monitor breakbeam sensors
while True:
	#check for a tap at the intersection of beam1 (horizontal) and beam3 (vertical)
	if(not GPIO.input(beamT) and not GPIO.input(beamL)):
		time.sleep(timeDelay)
		#if one of the beams becomes solid
		if(not GPIO.input(beamT) and not GPIO.input(beamL)):
			f = open('/home/pi/ir.txt', 'w')
			f.write("l")
			f.close()
			print "TAP LEFT"
			time.sleep(1)
	#check for a tap at the intersection of beam1 (horizontal) and beam3 (vertical)
	elif(not GPIO.input(beamT) and not GPIO.input(beamR)):
		time.sleep(timeDelay)
		#if one of the beams becomes solid
		if(not GPIO.input(beamT) and not GPIO.input(beamR)):
			f = open('/home/pi/ir.txt', 'w')
			f.write("r")
			f.close()
			print("TAP RIGHT")
			time.sleep(1)
	#check for down swipe
	elif(not GPIO.input(beamT)):
		for i in range(swipeDelay):
			if(not GPIO.input(beamB)):
				f = open('/home/pi/ir.txt', 'w')
				f.write("d")
				f.close()
				print("SWIPE DOWN")
				time.sleep(1)
				break
	#check for up swipe
	elif(not GPIO.input(beamB)):
		for i in range(swipeDelay):
			if(not GPIO.input(beamT)):
				f = open('/home/pi/ir.txt', 'w')
				f.write("u")
				f.close()
				print("SWIPE UP")
				time.sleep(1)
				break

breakbeam.js 


  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
/* global Log, Module, moment */

/* Magic Mirror
 * Module: breakbeam
 */

Module.register("breakbeam", {

	// Module config defaults.
	defaults: {
		updateInterval: 1000,
		remoteFile: null,
		fadeSpeed: 1000,
		visible_screens: [
			'', 'calendar clock compliments currentweather weatherforecast newsfeed', 'clock calendar', 'clock compliments', 'clock currentweather weatherforecast', 'clock newsfeed' 
		], 
		hidden_screens: [
			'calendar clock compliments currentweather weatherforecast newsfeed', '', 'compliments currentweather weatherforecast newsfeed', 'calendar currentweather weatherforecast newsfeed', 'calendar compliments newsfeed', 'calendar compliments currentweather weatherforecast'
		] 
	},

	// Define required scripts.
	getScripts: function() {
		return ["moment.js"];
	},


	// Define start sequence.
	start: function() {
		Log.info("Starting module: " + this.name);	
		this.index = 0;
		var self = this;
		setTimeout(function() {
			var modules = MM.getModules().withClass('calendar clock compliments currentweather weatherforecast newsfeed');
			var i;
			for (i = 0; i < modules.length; i++) {
				modules[i].hide()
			}
			self.update();
		}, 5000);
	},

	update: function() {
		var self = this;
		setInterval(function() {
			self.sendSocketNotification("CHECK_SWIPES", {});
		}, this.config.updateInterval);
	},

	// Override socket notification handler.
	socketNotificationReceived: function (notification, payload) {
		if (notification === "TAP_LEFT") {
			var cal = MM.getModules().withClass('calendar');
			this.toggleModule(cal[0]);
		} else if (notification === "TAP_RIGHT") {
			var weather = MM.getModules().withClass('currentweather weatherforecast');
			this.toggleModule(weather[0]);
			this.toggleModule(weather[1]);
		} else if (notification == "SWIPE_UP") {
			if (this.index + 1 == this.config.visible_screens.length) {
				this.index = 0;
			} else {
				this.index = this.index + 1;
			}
			this.updateMirror();
		} else if (notification == "SWIPE_DOWN") {
			if (this.index - 1 < 0) {
				this.index = this.config.visible_screens.length - 1;
			} else {
				this.index = this.index - 1;
			}
			this.updateMirror();
		}

		this.updateDom(this.config.animationSpeed)
	},

	toggleModule: function(m) {
		if (m.hidden) {
			m.show();
		} else {
			m.hide();
		}
	},

	updateMirror: function() {	
		//get hidden and visible modules
		var hidden = MM.getModules().withClass(this.config.hidden_screens[this.index]);
		var visible = MM.getModules().withClass(this.config.visible_screens[this.index]);

		// hide all hidden modules
		var i;
		for (i = 0; i < hidden.length; i = i + 1) {
			hidden[i].hide();
		}

		// show all visible modules
		for (i = 0; i < visible.length; i = i + 1) {
			visible[i].show()
		}

		//return this.index; 
	},

	// Override dom generator.
	getDom: function() {
		//var emptyString = this.updateMirror();

		//var empty = document.createTextNode(emptyString);
		//var wrapper = document.createElement("div");
		//wrapper.className = this.config.classes ? this.config.classes : "thin xlarge bright";
		//wrapper.appendChild(empty);

		//return wrapper;
	},

});

node_helper.js 


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
/* Magic Mirror
 * Node Helper: Breakbeam
 */

var NodeHelper = require("node_helper");
var fs = require("fs");

module.exports = NodeHelper.create({
	// Override start method.
	start: function() {
		console.log("Starting node helper for: " + this.name);

	},

	// Override socketNotificationReceived method.
	socketNotificationReceived: function(notification, payload) {
		var self = this;
		if (notification === "CHECK_SWIPES") {
			// read the file
			fs.readFile('/home/pi/ir.txt', "ascii", function (err, data) {
				if (err) throw err;
				if (data.charAt(0) == 'r') {
					self.sendSocketNotification("TAP_RIGHT", {});
					console.log(data.charAt(0));
				} else if (data.charAt(0) === 'l') {
					self.sendSocketNotification("TAP_LEFT", {});
					console.log(data.charAt(0));
				} else if (data.charAt(0) == 'u') {
					self.sendSocketNotification("SWIPE_UP", {});
					console.log(data.charAt(0));
				} else if (data.charAt() == 'd') {
					self.sendSocketNotification("SWIPE_DOWN", {});
					console.log(data.charAt(0));
				}
				fs.truncate('/home/pi/ir.txt', 0, function(){})
			});
		}
	},
});

Bill of Materials

  • Monitor: Donation (estimated $50)
  • 4 IR break beam sensors: $6.50 x 4 = $26
  • Glass sheet: $5
  • Mirror Film: $15

Contributions

Both Alix and Jeffery worked on the construction of the mirror together. Their individual contributions are listed below.


Alix Katague (ack242)


Designed and created the python code for this project as well as configured and tested the IR sensor setup.


Jeffery Lee (jyl58)


Jeffery worked primarily on the Magic Mirror base code and created the new module that would alter the display in response to the user's gestures. This included reading the file with the node_helper function, send the socket notification to the main breakbeam module, interpreting the gesture, and altering the display according to the gesture.

Thanks

We would like to thank our Professor, Joe Skovira, for all his guidance and suggestions.