Introduction

This project is split up to a number of different stages, first being data acquisition. The project uses the Raspberry Pi camera module due to the ability of three Raspberry Pis through the University of Derby for the purpose of this project. Any IP CCTV cameras would have sufficed, but through using with a standardized platform such as the Raspberry Pi improved the setup and monitoring aspects of this project.

The test environment will consist of three televisions in three different rooms, the living room, kitchen and bedroom. The kitchen and living room are adjacent to each other and the bedroom is on the next floor. Face detection will be recorded as defined below, allowing for an infield analysis how effective the face detection is before the impletion stage as well as any anomalies that exist with the technology or cameras usage.

Research Strategy

Due to the optical detection methods to detect the rooms are occupied or vacant status there will be variations in the quality of the data that is collected due to changes in the light levels caused by time of day. For example, once the sun has set and user turn light on artificial lighting the main light source is generally dimmer, and in most cases mounted on the ceiling. This leads to shadows being cast over the users’ faces, affecting the accuracy of face detection. Conditions such as rain storms can also have the same effect on the accuracy of face detection as already outlined by (Belaroussi & Milgram (2012) whose research outlines the importance of lighting conditions as well as angles in faces when detecting faces.

In order for the system to take this into account, an externally mounted CCTV camera without night vision has been used to calculate the lighting conditions in real time, allowing the system to drop into night mode if the external light levels become too dark. See image below:

Every 10 seconds a photo of each room is taken using Raspberry Pi camera modules, the images ares then uploaded to the central server for processing. The image is then passed onto a Python script which uses OpenCV to process the image and identify the number of faces that exist in the image. The results are then passed onto to PHP, which then looks at the GRB values to estimate the brightness of the image. The results are stored in the database against a time stamp; some of the images are stored for debug purpose.

From this a separate PHP service runs every 10 seconds, which is offset by 1 second to establish which rooms are occupied based on the data in the database, unless night mode is enabled where the room light levels as used to determining the occupancy status. Night mode can also be enabled by the time of day a using a PHP built in feature called ‘date_sunset’, which takes the longitude and latitude (PHP, 2015) to establish dawn and dusk times. Each media centre is then queried to see which are currently playing media and which are occupied. If any of the rooms are occupied but not playing media the television in that room is turned on and content is transferred. After a room has been unoccupied for the time out period of 5 minutes the media is stopped and the TV is turned off.

Each room has been calibrated to take into account the strength of the artificial lighting, for example the install for this project considers the living room as a 80, the bedroom as a 90 and the kitchen as 100. For further information this data can be viewed in the PHP file, under the setting directory.

The below graph is an example 2 hour window with 2 individuals watching a TV program under artificial light during the evening. At the end of the program the TV is left on for a while before the TV and room lights are turned off. The graph below shows wide fluctuations in light levels as well as well as fluctuations in the number of faces detected during ideal conditions where subjects are looking directly towards the camera.

This 2 hr data set shows how clearly the light being turned off is depicted in the data, and how under artificial light the accuracy the face detection decreases.

below shows wide fluctuations in light levels as well as well as fluctuations in the number of faces detected during ideal conditions where subjects are looking directly towards the camera.

This 2 hr data set shows how clearly the light being turned off is depicted in the data, and how under artificial light the accuracy the face detection decreases.

This data supports previous assertions that problems are amplified in low lighting and under artificial light because of the non-directional nature of artificial lighting whereby shadows are cast over the subjects 3d human faces. In the diagram below, the two locations were the green faces are denoted on the floor plans show where the system struggles to detect faces because the light source is directly behind the subjects face. The middle location on the bottom row does return detection because it’s illuminated by the TV screen assuming it is on.

This evidence is further supported by the data collected in the bed room whereby only around 10% of face detection occurs when the users are watching TV in the location under artificial lighting conditions. This is caused by the layout of the light sources, the location of the TV in relation to the users.

The second floor plan of the bedroom demonstrates that users sitting on the bed

watching TV have light source on the right of them. This in turn casts a show reducing the chances of face detection. It was not possible to move the location of the light or the windows to improve rates of detection.

Software – Myth TV

To provide live TV for the user MythTV’s backend server has been implemented. XBMC contains built in supported plugins that allow XBMC to access MythTV frontend allowing live play back and access to the recorded TV programs. Live TV will be provide to the Media Centre (XBMC) from the MythTV backend which is utilises two Freeview tuners (Amazon, 2015)

This allows two separate channels to be recorded at the same time, One of the reasons that MythTV was selected for this project is because of its ability to use a feature called viral tuners. Viral tuners in theory can allow MythTV to record up to 12 separates channels at the same time using only two physical Freeview tuners (MythTV, 2015).

This is achieved because Freeview is divided up into a number of multiplexes (ref http://www.digitaluk.co.uk/industry/Multiplexes). At present there are 8 different multiplexes in the UK, coverage dependent. When a user selects a channel the turner tunes to the relevant multiplex and ignores everything but channel requested by the user. MythTVs viral tuners feature allow MythTVs to uses all of the multiplex that is it is currently tuned to. An example of this would be that all of the BBC channels are on the BBC multiplexes (BBC A) and all of the ITV channels are on the ITV multiplexes (SDN). If tuner 1 was receiving multiplexes (BBC A) and tuner 2 was receiving ITV multiplexes (SDN) during the time you couldn’t watch channel 4 but users could receive BBC 1,BBC2, BBC 3, BBC News, ITV , ITV2,ITV4. There are usability downsides to the setup, there are often times when certain channels aren’t available this is because they exist outside of the currently recorded multiplexes and there is not a free tuner available to capture that particular multiplex. This can become confusing for end users to understand although this feature is perfect for this project, because more than one media centre will be required to view the same channel without having without add physical tuners.

Control:

CEC:

CEC, consumer electronics control, also needs to be taken into consideration in the design of this project to allow XBMC to utilise the built in TV remote, as well as allowing XBMC to control the TV. CEC is part of the HDML 1.3 [ CITATION Hit06 \l 2057 ] specification, but manufacturers have their individual implementations that often contain a subset of features. An example of this would be Samsung equipment where the CEC has been branded as Anynet+ Control Pass, the CEC control device can not adjust the volume due to the implementation not supporting this feature.

The media centre (XBMC) can be controlled directly in a number of way using a standard keyboard and mouse, remote control, CEC via the HDMI as well as JSON API which can be accessed using a Mobile app or Web interface.

Other feature of XBMC that are relevant to this project is that many different XBMC instances can share a common database as well as the same media (Kodi, 2015), this is key for our project otherwise each instance of XBMC would index the media folder and potency give the same media different unique identifier. Because XBMC can share its library it means that the Movie 66 happens to be A-Team on every ALL of the XBMC instances. The XBMC Android app for currently controls Media Centres using the topologies as depicted in the below diagram. The app itself is not aware of its proximity to any of the media centres; it is only aware that it is on the same network:

If a user scans a static NFC tag attached to the TV to, for example, start BBC 1 the proposed logical topology can be adjusted by the system to:

This is achieved because the tag states the content on the relevant media centre then instructs the Android XBMC Remote App to change media centres and to display itself on the phone. This information is determined by the NFC tag itself. The static NFC tag simply contains a URL such as:

• http://none.simon-everett.co.uk/play/LiveingRoom/channle/1

which informs the system where it is and what the user expects. An example of a movie would be:

• http://none.simon-everett.co.uk/play/LiveingRoom/movie/66

NFC tags can be written using a simple smartphone app such as NFC Tools by and tags are at an affordable price for the constraints of the project and easily available from amazon for around 3 pounds (Amazon, 2015b). The following table shows the four images of the process to scan a static NFC, this tag is located on the bedroom TV and is set to play channel 1.

The project has further extended the physical control by using surface mounted NFC tags that can be programed to instigate and display TV channel. This physical control surface can is used in a number of two distant ways, either attaching an tag on a wall near the media centre allowing a user to select BBC 1 or by attaching tags to DVD cases and reading the TAGS using an static reader. This could be an NFC enabled phone or a Raspberry a project named Raspberry Kiosk could provide an suitable platform as recommended by Raspberry Kiosk (2013).

For the purposes of this project static NFC tags are used with smart phones, this has the added benefit of once the Media centre started the playback commands can be issued to the smart phone to display and setup the relevant media remote control (Yatse, 2015) for the relevant media centre on the user’s phone. It is also possible to query the data stored in the library, it would be possible to setup and NFC tag to play the most recently recorded episode of “Marvel's Agents of Shield” assuming that XBMC has indexed the media.

The first image demonstrates that the wrong media centre is selected.
The Second image shows the NFC being read and loading the webpage.
Third image demonstrates the XBMC remote app loading and changed to the correct Media Centre.
Last image shows the TV turned on, and displaying the required channel. BBC 1 in this case.

This last image shows the interaction between the XBMC remote app and Firefox, Firefox looks like the XBMC remote in the preview because Firefox has called the XBMC app.

Introduction

In the late 1980’s Bill Gates founder of Microsoft designed and implanted an integrated media centre solution into his newly built home near Lake Washington, which used small pin badges to detect the presence of users and re-direct the media accordingly to the nearest appliance as the user moved around the home. Unfortunately there it is little literature to support how Gates’ achieved his project, however Gates (1995) published a video documenting the media centre, which is available on YouTube[ CITATION Nig07 \l 2057 ]. This project was coved in a number of publications at the time. U.S News (2012) reported that Bill Gates had completed construction of his house at cost of $100 million, and detailed that it had a number of features such as high-definition television monitors that displayed art digitally, computer controlled music, lighting and climate settings that are linked to individual users location within the mansion. Again, the article did not state how this was achieved.

Since 1997, when the Gates home was completed, technology has significantly advanced in a number of linked technologies such as computing, televisions, radio-frequency tags, CCTV cameras as well as media centre technology. Alongside these developments many users have cited privacy concerns, relating to some detection methods, such as technologies processing images taken, tracking movement and even logging played content on the home automation platform. A further question is whether a user would want the fact that they’re out of the house recorded by others.

Part of the literature review will focus on possible ways of reliably detecting room occupancy. This can be achieved in a number of ways, such as radio sensors which users must carry, face detection, movement or co2 sensors. This Literature review will offer a critique each approach, as well as looking at how the technologies are applied in different settings. The literature review will then go on to focus on media centre technology, possible hardware solutions and effective ways to synchronise all aspects of the project to allow the Media Centre to act as one.

Ways of Detecting Room Occupancy:

Radio

Sensors:

There are existing projects similar to the Bill Gates project (Gates, 1995) that use pins or devices to track room occupancy for the purpose of redirecting media content to the nearest appliance. Linux MCE (2015) is a whole home automation suite that uses also uses radio sensors, namely Bluetooth, as a method to track users around the home from mobile devices. There are limitations associated with Bluetooth, namely the inability to obtain signal strength between two devices around without first paring them together using standard hardware. On the other hand, there are some devices that are able to communicate to all Bluetooth devices without paring, such as Ubertooth (Chai, Deardorff and Wu, 2012) which allows the interception and transmission of Bluetooth packets from and to any mac address, as well as to detecting the single stretch of remote devices. Ossman (2015) has developed the product Ubertooth to allow users to directly access the Bluetooth protocol Hardware such as the Ubertooth One makes tracking Bluetooth without paring possible without excessively expensive hardware Hay and Harle (2009) discuss the process involved, in their paper, and conclude that connection-based tracking which permits tracking of a previously identified handset within a field of fixed base stations is a viable alternative to inquiry-based Bluetooth tracking. Although the paper is from 2009 the research is still relevant today.

Although Bluetooth tracking could be seen as a convenient approach for tracking individuals, this would not be suitable for this project due to usability issues. Users would be required carry a phone or an object to be detected as occupying a room. Although this approach suitable for the project discussed in this paper, there are commercially available products.

Tracking in Retail:

There are a number of products that have designed to allow companies to track shoppers uniquely using the signals from their Bluetooth device such as phone. Specially designed hard and software is used to triangulate users as they move around an environment such as a shopping centre. Solutions like Rapidblue Solutions (2014) offer real world analytics about to customer shopping patterns and their product uses a number of listening devices to triangulate a mobile phone as they move around a retail outlet. Case studies by Rapidblue show that through using the shopper analytics platform companies can increase revenue for shopping centres clients by understanding the flow of consumers between retailers in a shopping centre. Again, this approach will not be adopted in this project as it requires the user to carry their mobile phone at all times.

The problem of identifying individual customers with in the retail space has existed for a number of years and has been addressed in a number of ways. Retailers have historically used loyalty cards to track the buying habits of their customers. If a customer presents his or her loyalty card during the transaction they receive an incentive, the data collected is then calculated to make very accurate projections on the demand for personal items. This data allows retailers to better reduce spoilage. Companies such have Tesco’s have used incentives cards such as the club card from as early as 1995 to collect buying habits [ CITATION Roh06 \l 2057 ]. Through using buying habits data retailers such as Tescos, are able to make informed decisions about targeting promotions to a selected group of people or predicting how much skimmed milk is needed for the upcoming day. However this method of tracking does not aid in the process of tracking customers around the store. Tesco are moving to further innovations in the retail space by using “facial recognition software at some 450 of its petrol stations in the UK” [ CITATION Mil15 \l 2057 ] with the directly targeted adverts on the video screen in the store changing based on their gender, age or another unspecified characteristics. Tesco’s are using an Amscreen product called OptimEyes (Huffington Post, 2013) that only collects statics and allows targeted adverts, addressing concerns that users may have when it comes to privacy in terms of recording images.

Optical

Detection:

Following in the footsteps of retailers the literature review will move away from looking at radio sensors as a way of tracking users to looking optical recognition. There are a number of different approaches that can used process an optical images to detect room occupancy within an image. These methods include tracking movement, lighting conditions or using a process to count the number of human faces in an image taken. There are a number of commercial products that perform the task of processing human faces, as well as Opensource library which is able to detect shapes and patterns. In the following paragraphs existing products will be critiqued to see if a suitable solution can be found for this project.

OpenCV

OpenCV is an Open Source project that offers a number of simple really time image processing libraries, it was first developed by an Intel Research Initiative (Bradski and Kaehler, 2008). and is written in C++. There os also an API that can be accessed using Python, Perl and Ruby (Bradski and Kaehler). OpenCV can be used in a number of ways from stereo correspondence to eye and mouth tracking and even gesture recognition. OpenCV also allows for face recognition similar to OptimEyes (Huffington Post, 2013). OpenCV is open source which allows developers are able to modify and extend the sourcecode allowing for innovation after innovation.

When compared to other comparable products this would be a suitable way of detecting room occupancy for the purpose of this project, as researchers Belaroussi and Milgram (2012) have already investigated the success and failure rate of OpenCV’s face tracking and facial recognition libraries and came to the conclusion that OpenCV only had a small, acceptable percentage failure rate. The study also investigated the processing power required to detect faces in a real time environment. One of the key issues highlighted by Belaroussi and Milgram is that “lighting conductions can dramatically change a faces tracking process” and the orientation of the subjects face also has an effect on the success of positive face detection.

Existing

work (Bill

Gates project)

Linking back to the original inspiration for this project, due to limited literature on Bill Gate’ (1995) project, it can only be speculated as to the technologies used in the original project based on available products of the time. For comparison purposes it would be interesting to know what technology was used for the Gates project for the purpose of using similar the modern day alternatives to create a system that works in a similar way. The only different comparison we can conclusively draw is that we need a way to track unique individuals around a large environment. The next part of the literature review will compare possible modern approaches and technologies that could be used in a modern system of this kind, which are within the budget constraints of this project. Following on from this a conclusion will be drawn as to how the improved media centre automation system that will be developed should be deployed and what hardware, software and technologies should be used to fulfil the requirements outlined in the project aims and objectives.

Existing

work (LinuxMCE Home

Automation Project

One comparable product to the project that will be created is LinuxMCE (MCE) which is based on Ubuntu. The product contains everything needed to setup a home automation system. This system supports many of the modem automation protocols such as X11 and Z-Wave. What makes LinuxMCE noteworthy is that is that is outdated thanks to developments in both hardware and software. The system uses Windows CE or XP tablets that allow the user to control their home automation system and media centre from a single location. MCE has also an implementation that allows the computer or tablet to detect the proximity of Bluetooth device. This feature is called “follow you” and allows lights and media devices to be turned on and off, but this does required Bluetooth to be turned on and the computer or tablet to be on your person. As mentioned previously with other Bluetooth devices these requirements make it unsuitable for this project.

RFID/NFC

In relation to the Gates project (1995) it can be hypothesised that some form of RFID technology such as a product called HITAG by Philips that was commercially available around the time could have been used in the home automation system at the Gates mansion. As early as 1996 the HITAG [ CITATION Phi98 \l 2057 ] used protocol and technology simpler to modern RFID/NFS. These small tags could have small amounts of data written to them and contained unique identifiers that could be read from a distance of 200 mm to 1000 mm according to the data sheet [ CITATION Phi98 \l 2057 ]. It’s fair to theorise that a network of devices such as these could have been deployed in a grid layout to accurately track the movements of users around a large environment, this theoretical approach would most likely be expensive but the resulting accuracy and redundancy would most like offset the cost when taking into account that money wasn’t an obstacle in the Bill Gates mansion, but cost is a consideration in this project.

At present products available include RFID tags (Amtel, 2005) and NFC tags (ECMA International, 2013). These are identification tags that can be read using passive tags using different low frequencies, both forms of these tags are powered using an inductive style. RFID contains a hardcoded ID, which would make them unsuitable for this project, whereas some NFC tags are writeable or can implement cryptography. NFC tags can contain a number of different types of data such as phone numbers, email/addresses, business cards even URL addresses (ECMA International, 2013).. This feature of NFC is useful for the project outlined in this paper because the NFC tags could be read by a mobile device and points a user’s device (Phone, tablet or reader) to a specially designed URL that could instigate playbacks of a certain film or TV channel. NFC and RFID tags need to be within a range of 20cm to be able to be read, although there are products that use specially designed to be read as far as up to 10 meters away. IPICO have developed a product called Sports Tag (Active Network, 2015) that have been developed for the purpose of communicating timings for sporting events by reading RFID tags attached to runner shoes

Another point for consideration is that researchers have been able found ways to extend the range of NFC and RFID tags. Paget demonstrated one of theirs approaches at (Defcon, 2009) tags by creating “RFID Door Frame Skimmer” with attached the RFID reader antenna to a door frame allowing every tag that passes through door to be read. Although this would be an accurate way of tracking user movement around the home, the price of the components would be out of the budget for this project.

Media Centres

Although the purpose of this project is not to simply create a media centre, there will be a high level of integration required to control the media centre, therefore the software used needs to be taken into consideration. The end product needs to query information from the media centre as well as compare this information with sensor/data, then instruct the media centre to change its state.

Samsung

Samsung currently offer a number of unorthodox control surfaces or their latest range of smart TV that includes voice control and hand gestures (Samsung, 2009). Although theirs approach works sufficiently well under some circumstances, it still requires direct user input, as well as requiring users to see and read the feedback from the media centre.

XBMC

As the media centre is not the main focus for the project there the literature review will only briefly mention the research studied to underpin the decision to XBMC (Kodi, 2015). From the literature provided about XBMC on the Kodi website it is suggested that media centre supports an extensive set of API integration using JSON, as well as having the ability to run on a wide range of devices from a PC to an Apple TV and on Raspberry Pis.

Frontend Hardware

Another consideration is that the hardware for this project must be durable as a home is a twenty-four hour setting. Power consumption and stability are important factors as well as a responsive user experience. HD Video playback is also sensible requirement as a majority of modern televisions support HD content, as well as supporting audio over HDMI to allow for the use of the built in speakers. Following research a further key requirement for the media centre hardware is that the frontend must have CEC support (HDMI Licensing, 2013) to allow the automated media centre to control the television. The below list contains 3 different options for frontend platforms with their specifications:

• Raspberry Pi with an 800 MHz processor and 512mb memory costing approximately £35.
• Shuttle XS35GS V3 with Intel® Atom™ Processor D2700 with 128 GB SSD and 4GB of ram built in Wi-Fi (+ USB - CEC Adapter + £30) = £600.
• M8 Amlogic S802 Quad Core Android TV BOX, with built in wi-Fi (+ USB - CEC Adapter + £30) = £129.99.

All of the options apart from the Raspberry Pi require an extra device (USB - CEC Adapter) to allow for interaction with the television CEC. The Shuttle and M8 both require power adaptors ranging from 12-19 Volts at 2 amps. The Raspberry Pi only requires between 700-1000 mA, some power supplies that in theory can be supplied by the TV USB service port, or even PoE using (POE: http://www.raspberrypioneer.com/2013/06/13/how-to-run-your-raspberry-pi-from-poe-for-under-15/) as developed by Peter (year) using off the shelf parts allowing a Media Centre to be powered using only one network cable and a PoE injector on the other end. The Raspberry Pi is the only device from the selection that does not have Wi-Fi built-in as well as benign slowest device when compared to the other devices under consideration although all devices have the same speed 10/100 mbits network devices.

From the comparisons drawn the Raspberry Pi hardware platform is a well suited platform for this project, even considering its drawback in regards to connectivity and processing power. This is because it meets the budget constraints of the project, without compromising on meeting key project outcomes.

Conclusions

After reviewing a range of modern, available technologies and approaches a selection will be made identified based on cost and accuracy. The main goal of this project is to instigate similar approach to that of Gates (1995) which is affordable to the mainstream consumer, whilst still having the same sense of automation. From the research discussed in the course of the literature review it can be concluded that there are number of ways to detect occupancy of a room. Having critically reviewed the positive and negative aspects of ways of tracking room occupancy this project will use face detection as a means of establishing room occupancy. A further positive aspect of this method is that it does not require direct interaction from the user. From the research discussed there can be a number of considerations to ensure effective face detection these will be evaluated in line with research in the methodology section of this documentation. From research into Bluetooth communication, RFID and NFC tags this project will focus on using NFC tags for the user to control the media centre. The aim is to mitigate problems and better the user experience for those with eyesight or literacy issues.