What We Learned from Using Azure Computer Vision at HackBytes

HackBytes is a hackathon jointly sponsored by Loblaws Digital and Microsoft. The goal was to build something that used computer vision in a retail environment. Right away the creative juices started flowing, and some incredible concepts emerged from this 24-hour coding spree.

Team BlackBerry once again dominated with a strong showing of the capabilities of BlackBerry Athoc. Our demo took a 3-pronged, microservices-based approach to enhancing the capabilities of security cameras in a retail space. The project, titled ‘Sight for Store Eyes’ accomplished three things:

1.     Real-time detection of blacklisted individuals with Azure computer vision, followed by an automatic alert to security teams at the store.

2.     Real-time detection of a customer showing the peace sign to a camera, broadcasting an alert to customer service staff to let them know a customer in aisle ‘x’ needs help.

3.     Analytics on the data collected from these AI driven alerts.

In the United States alone, retailers lose $50 billion a year due to theft. A big chunk of this loss is often due to a small handful of repeat offenders. To address this problem, our team built a system to detect blacklisted individuals, and then alert security staff to their presence. We leveraged the Azure FACE API and BlackBerry Athoc to bring the idea to life. An iOS app was used to simulate a security camera feed, and then the FACE API was used to both train an AI model and detect criminals as they were entering the store. When a blacklisted individual was detected, the iOS app sent a request to the team’s Athoc server, which sent an SMS to the store’s security team, with the identity of the individual in the payload of the text.

Next, our team looked inward to solve a personal problem, “how many times have I needed help in a big grocery store, but couldn’t find anyone?” Rather than hire more staff, we leveraged computer vision and AI to detect customers in need. In our project, a custom model to detect ‘peace signs’ from customers was trained using Azure’s CustomVision API. Once the model could accurately detect customers showing a peace sign to the security camera, we exported the model to CoreML, for use on our simulated iOS security camera. Then, when our model detected a customer requesting help, the iOS app sent a request to our Athoc Server, which then sent an SMS to customer service staff saying, ‘A customer has requested help in aisle x’. The tool has broad applications in the retail sector and can be used to dramatically enhance the customer service experience.

Finally, our team took this epic AI driven alert system and created a web analytics dashboard. From the dashboard, upper management can gain insight into things like what aisle customers are having the most questions in, and what locations repeat offenders are targeting.

For a high-level view of our architecture, check out the image below:

In summary, our project addressed a clear opportunity to dramatically enhance security cameras in stores, which currently lack smart features like threat detection and customer service. At the end of the day though, the idea was only among the runner ups to the top prize at HackBytes. Kudos to the first-place project that utilized computer vision and a customer’s cart to recommend more products to them, and the second-place prize that scans and quickly locates a product on a grocery shelf.

Azin Taqizadeh

About Azin Taqizadeh

Enterprise Solutions Development Student - IoT

Gurjit Ghangura

About Gurjit Ghangura

Enterprise Solutions Developer, Love BlackBerry, Developer at Heart.

Jeff J.

About Jeff J.

As a part of the Enterprise Solutions Team, I work to bring the latest BlackBerry software and security features to life on the Android platform.