There are a number of options if you are looking to implement barcode scanning in your HTML5/WebWorks applications.
The original community extension available in GitHub leverages native BlackBerry functionality to scan barcodes. However, it has two caveats:
- Data transfer relies on saving/loading images from the filesystem, resulting in a viewfinder (renders camera stream) refresh rate around 2-3FPS.
- Each render creates a new image from the filesystem, resulting in memory issues when the viewfinder displays for an extended period of time.
Our WebWorks team will soon be integrating BlackBerry 10 support into a cross-platform PhoneGap plugin for barcode scanning, based on the existing community extension. For short-burst scanning, the existing community extension can fill that role.
Recently though, I was involved in a project that required extended barcode scanning functionality and, due to the caveats with the original community extension, I opted to explore a purely HTML5 solution to the issue. I want to share my experiences in implementing a solution that would satisfy the following requirements:
- Viewfinder: We need to display what is being scanned.
- Performance: Barcode processing should have minimal impact on viewfinder rendering.
- Continuous Scanning: The approach should be capable of scanning multiple items in a row without requiring any sort of reset, but also allow for one-offs.
Scanning With getUserMedia
There are only a handful of freely distributed barcode scanning libraries out there. After some initial evaluation, I went with Barcode Reader by Eddie Larsson for 1D processing and jsqrcode by Lazar Laszlo for 2D processing, which seemed to be the most popular.
For this example, we will focus on a webkit implementation of getUserMedia. However it’s important to note the following considerations:
- iOS does not currently support getUserMedia
- Android supports getUserMedia as of 4.4 via Chrome for Android
Source: http://caniuse.com/stream
To also target iOS and earlier versions of Android, a Cordova/PhoneGap plugin would be the recommended approach.
Viewfinder
The main job of the viewfinder is to simply render what the camera sees. We’ll then pass that image data for barcode processing.
We’ll start by callingnavigator.webkitGetUserMediawhich takes three arguments:
- An object that specifies whether we want to streamvideo, audio, or both.
- A success callback function that will contain ourLocalMediaStreamobject.
- A failure callback function if aLocalMediaStreamobject cannot be obtained.
In ouronSuccessfunction, we’ll immediately configure our<video>element to begin streaming content from the camera.
The code for configuring the viewfinder is as follows:
/* These are made available in the global scope of the application. */
var stream, video;
navigator.webkitGetUserMedia(
{ video: true,
audio: false
},
function onSuccess(localMediaStream) { stream = localMediaStream;
video = document.querySelector('video'); video.addEventListener('loadedmetadata',function onloadedmetadata() { video.removeEventListener('loadedmetadata', onloadedmetadata, false); video.play();
/* We can now use the video to initialize our canvas. */
initCanvas();
}, false);
video.src = window.URL.createObjectURL(stream);
},
function onFailure() { /**
* LocalMediaStream was not available. This could
* be due to another application already having
* a lock on the camera.
*/
}
);
We’ve defined two variables,streamandvideo, in a wider scope so that they can be used in other key areas of our application.
We are assuming one (1)<video>element contained on our documentwhich will receive the stream. If you have multiple<video>elements, you can easily modify thequerySelectorcall to be more specific, or usegetElementByIdinstead.
We add a listener for theloadedmetadata event and call play once that triggers. We’ll also be using the videostream’s dimensions to configure our<canvas>, which is used to scale our image data.
Performance
Even with optimization, we in the HTML5 world will not enjoy the same 60FPS viewfinder that you get with a native implementation, at least with how things are today. What we can expect is around 15-20FPS. Although this isn’t ideal, it’s good enough for the task at hand.
Performance – Rescaling
We could leverage the data from our<video> element as-is and pass that for barcode processing. However, passing 480×640 pixels (or more) produced long processing time, particularly for the 1D barcode analyzer. I found better performance, with minimal impact to usability, by rescaling the data to 256×256 which is roughly 4.7x fewer pixels to analyze.
During re-scaling, it may also be beneficial to maintain the same aspect ratio on the<canvas>as in the originalvideo stream. However, I have not experienced any usability impact so far.
With that, ourinitCanvas function that is invoked following the<video> element’sloadedmetadata event looks as follows:
/* These are made available in the global scope of the application. */
var canvas, context, vx, vy, vw, vh, cx, cy, cw, ch;
// Previously Defined:
// video
function initCanvas() { canvas = document.querySelector('canvas'); canvas.width = 256;
canvas.height = 256;
context = utils.canvas.getContext('2d'); vx = 0;
vy = 0;
vw = video.videoWidth;
vh = video.videoHeight;
cx = 0;
cy = 0;
cw = canvas.width;
ch = canvas.height;
}
Above, we’ve defined a handful of variables that will also be available in the larger scope of the application since they will be used by other functions. Both canvas and context are self-explanatory, and then we have eight (8) variables that denote position and size.
Those beginning with av are in reference to thevideostream and those beginning with acare in reference to thecanvas. Each has four (4) properties:
- X-coordinate to start reading/writing data.
- Y-coordinate to start reading/writing data.
- Width of the element.
- Height of the element.
To avoid runtime re-calculations we initialize these variables and use them throughout the remaining code.
Performance – Web Workers
There are a handful of additional issues we need to address.
If you look at the chosen barcode libraries you’ll notice that Barcode Reader has already been written to function as a Web Worker. This is critical because it allows the processing to occur off the main UI thread which would otherwise block operations and make our stream rendering quite choppy.
Unfortunately, jsqrcode is not designed for this out-of-the-box.
The second issue we need to be mindful of is the timing of processing. If both 1D and 2D libraries are constantly processing, our main UI thread can get bombarded by activity, impacting rendering performance. Our goals then are:
- Both libraries should function as Web Workers.
- We must maintain a rapid rate of processing, but not overwhelm the main UI thread with messages.
With this in mind, the workers were configured as follows.
/* These are made available in the global scope of the application. */
var workers;
// Previously Defined:
// video, context, vx, vy, vw, vh, cx, cy, cw, ch
function initWorkers() { workers = { '1D' : new Worker('js/BarcodeReader/DecoderWorker.js'), '2D' : new Worker('js/worker2d.js') };
workers['1D'].onmessage = function (message) { if (message.data.success === true) { /* Take action here. */
}
context.drawImage(video, vx, vy, vw, vh, cx, cy, cw, ch);
workers['2D'].postMessage(context.getImageData(0,0, cw, ch));
};
workers['2D'].onmessage = function (message) { if (message.data.success === true) { /* Take action here. */
}
context.drawImage(video, vx, vy, vw, vh, cx, cy, cw, ch);
workers['1D'].postMessage(
{ 'pixels': context.getImageData(0,0, cw, ch).data,
'width': cw,
'height': ch,
'cmd':'normal'
}
);
};
}
First, we define a workers object with 1D and 2D Workers. The 1D Worker leverages DecoderWorker.js, which is provided with the Barcode Reader library. The 2D worker leverages a custom JavaScript file on which we’ll go into detail shortly.
Next, we configure the onmessage functions of each worker very similarly.
- When a message is received, check whether a barcode was successfully decoded. If so, we can take specific actions (i.e. display the barcode value, etc.).
- Following that, we make a call tocontext.drawImagewhich copies our <video> stream data to our<canvas> element.
- Finally, we call the alternate worker to process the new data.
In this manner, we are flipping between the 1D and 2D workers. This prevents us from relying on methods such assetTimeoutto maintain processing and avoids the case where processing is happening to slowly (or too quickly) based on an arbitrary time delay.
With this approach, we simply process as soon as the previous processing is complete. By alternating between 1D and 2D, we’re also ensuring that the main UI thread isn’t getting constantly bombarded by the quicker library.
By leveraging workers in this way (and scaling the image through the canvas) I was able to eliminate almost all rendering lag in the<video>.
The piece that we haven’t yet discussed is how to convert the 2D library into something that is worker-friendly. It actually ends up being quite simple and this is what my worker2d.js file looks like:
/*global self, importScripts, qrcode */
(function (self) { 'use strict';
importScripts(
'jsqrcode/grid.js','jsqrcode/version.js',
'jsqrcode/detector.js','jsqrcode/formatinf.js',
'jsqrcode/errorlevel.js','jsqrcode/bitmat.js',
'jsqrcode/datablock.js','jsqrcode/bmparser.js',
'jsqrcode/datamask.js','jsqrcode/rsdecoder.js',
'jsqrcode/gf256poly.js','jsqrcode/gf256.js',
'jsqrcode/decoder.js','jsqrcode/qrcode.js',
'jsqrcode/findpat.js','jsqrcode/alignpat.js',
'jsqrcode/databr.js'
);
self.addEventListener('message',function onmessage(message) { var result, success;
qrcode.imagedata = message.data;
qrcode.width = 256;
qrcode.height = 256;
try { success = true;
result = qrcode.process();
} catch (e) { success = false;
result = e.toString();
}
self.postMessage({ 'result': result,
'success': success
});
}, false);
}(self));
As you can see, the jsqrcode library only requires four (4) things:
- The imagedata, passed in from our main UI thread.
- Thewidth of theimagedata, matches our canvas dimensions.
- Theheight of theimagedata, matches our canvas dimensions.
- A call toqrcode.process.
With this, the library will be able to process off the main UI thread and we can simply return the success/failure of the decoding attempt.
Continuous Scanning
In the previous section, I discussed how I leveraged the 1D and 2D workers to alternate. With that setup, we already have continuous scanning since we were alternating workers on both success and failures.
What I didn’t show yet was how to actually start that scanning cycle.
/* These are made available in the global scope of the application. */
var scanningTimeout;
// Previously Defined:
// video, context, workers, vx, vy, vw, vh, cx, cy, cw, ch
function startScanning(timeout) { scanningTimeout = window.setTimeout(function () { context.drawImage(video, vx, vy, vw, vh, cx, cy, cw, ch);
workers['2D'].postMessage(context.getImageData(0,0, cw, ch));
}, timeout || 0);
}
We can callstartScanningwithin our application to kick off the scanning cycle. I chose to start with the 2D worker as it appeared to be returning more quickly than the 1D worker, but in reality the choice is arbitrary.
There’s also the ability to define atimeoutvalue which will delay the start of the scan. I found this particularly useful following a successful decode. Chances are the person holding the device isn’t immediately scanning another item so I used this to introduce a two second delay between scans. The scanningTimeout variable allows us to cancel the start should the user do something in those two seconds that would require scanning to stop, like navigate to another page.
And finally, we have the same process inside thesetTimeout, where we are rendering the currentvideo data to the canvas and then allowing our 2D worker to process that data, thus beginning the scanning cycle.
Thanks For Reading!
If you have any questions at all, please don’t hesitate to leave a comment on this blog, reach out via Twitter (@BlackBerryDev or @WaterlooErik), or send me an email directly (eoros@blackberry.com).