Conference App Secrets Part 3: JSON vs XML

BlackBerry Jam

Guest post by Ekkehard Gentz, BlackBerry Elite Developer

This is Part 3 of my series, “Conference App Secrets.”

This time I want to discuss how to persist data to guarantee a fast-responding app.

I was asked why my conference apps are performing so incredibly fast while retreiving sessions, speakers, or searching (it got many five star reviews because of this).

You should download the BBJam Asia Conference App to see it there in action. The answer isn’t easy and will cover some more articles. Let me start to explain…

BlackBerry Jam Asia uses a JSON Data Model

I’m using JSON to persist the data of my conference apps and there are many reasons. Using REST interfaces, you still get much of the data from responses in XML format, and in most cases, you have to consume such data and cannot change the backend.

The first thing I’m always doing is to convert the XML data into JSON data before proceeding.

You’ll find all used datafiles of this sample app and sources at Github.

I cannot tell you details about the BBJam Asia Conference API, so I generated some mockup data:

  • 10,000 random addresses to have a large file to experiment with
  • 119 public available speaker data to have some conference-relevant data

The addresses I’m using are generated by a free service from fakenamegenerator.com. I highly recommend this service to developers if you have to provide prototype apps with data or if you want to test using large data sets. They even generate different data for the country you selected as well as GeoCoordinates.

The speaker data is a WordPress export file from another conference app and contains real data from publicly available sites.

Addresses and speaker data are in XML format. I’ll show you how to convert these files into JSON and explain why you should do this.

Both datasets can be found here:
app secrets 1

While developing and testing, you often have to take a look at the data, and JSON is much easier to read than  XML. Here’s one address in XML:

<Address>
    <City>Lincoln</City>
    <Company>Record Bar</Company>
    <Country>US</Country>
    <CountryFull>United States</CountryFull>
    <Domain>AffordableShow.com</Domain>
    <EmailAddress>SharonSMcCall@cuvox.de</EmailAddress>
    <GivenName>Sharon</GivenName>
    <Latitude>40.722909</Latitude>
    <Longitude>-96.678577</Longitude>
    <Number>1</Number>
    <State>NE</State>
    <StreetAddress>3002 Poling Farm Road</StreetAddress>
    <Surname>McCall</Surname>
    <TelephoneNumber>402-310-0424</TelephoneNumber>
    <Title>Mrs.</Title>
    <ZipCode>68501</ZipCode>
</Address>

And here, the same address in JSON:

{
   "City" : "Lincoln",
   "Company" : "Record Bar",
   "Country" : "US",
   "CountryFull" : "United States",
   "Domain" : "AffordableShow.com",
   "EmailAddress" : "SharonSMcCall@cuvox.de",
   "GivenName" : "Sharon",
   "Latitude" : "40.722909",
   "Longitude" : "-96.678577",
   "Number" : "1",
   "State" : "NE",
   "StreetAddress" : "3002 Poling Farm Road",
   "Surname" : "McCall",
   "TelephoneNumber" : "402-310-0424",
   "Title" : "Mrs.",
   "ZipCode" : "68501"
}

So improved readability is one reason to use JSON.

If you’re using QML then you can easily copy and paste objects from JSON directly into your QML code like:

var myAddress = {  …. the JSON object … } … myDataModel.insert(myAddress) ….

Sometimes this is useful for mock data where you simulate inserting objects into list DataModels.

In most cases JSON datafiles are smaller then XML:

  • 10,000 addresses in XML: 6.7 MB
  • 10,000 addresses in JSON: 5.9 MB

If the XML is using many long property names the file size can be reduced more, but sometimes if the XML structure is very complicated it may be that a JSON file is larger then XML (take a look at the speaker data).

Perhaps you already know that Cascades supports JSON very well. For example, if using QVariants from C++ in QML, where QVariantMap was mapped automatically to JavaScript Objects (== JSON). Perhaps this deep integration of mapping is the reason for ….

Speed Speed Speed :-)

conference app 4© David Castillo Dominici | Dreamstime.com

Most of the time, inside a conference app I’m reading data from the cache and this is where JSON clearly wins over XML. Here are some values I measured using the 10,000 addresses:

  • READ JSON:  2,392 ms
  • READ XML:  71,342 ms

Rule #1: if caching data, always use JSON – this can be up to 30 times faster then XML

It’s curious: writing data to JSON is somewhat slower then XML. Here’s what I measured for the 10,000 addresses:

  • Write JSON: 5,255 ms
  • Write XML:   3,567 ms

The write speed differences are marginal, and you can do the write async and continue working while you’re waiting to get the data from read.

In some cases, you’re getting more data than you need. Take a look at the speaker data exported from WordPress as RSS in XML format. Both file formats, XML and JSON, aren’t easy to read and contain too much data in a complex structure, but it’s easy to workaround. As soon as you get the data from HTTP Response, transform it and remove unneeded properties. This will give you even more speed. Here’s the measurement using the 119 speakers:

  • Read XML:     378 ms
  • Read JSON:     76 ms
  • Read JSON v2: 23 ms

JSON v2 contains only the attributes you need in a less complex structure.

Rule #2: if getting complex data remove unused properties to get even more speed

It’s boring only to compare numbers, so let’s use Cascades to demonstrate the speed of JSON-based persistence or caching and see how easy it is to convert data between XML and JSON.

UI (QML)

The sample application is only using two QML files:

  • main.qml
  • MeasureSpeakerPage.qml

Main.qml contains a NavigationPane displaying a page to visualize the measurements of reading and writing the 10,000 addresses.

MeasureSpeakerPage.qml will be pushed on top to visualize the measurements of reading speaker data from XML, JSON and JSON-less properties.

Let’s take a look at the main.qml first. There are some attachedObjects:

  • SystemToast
  • ComponentDefinition
  • Qtimer
attachedObjects: [
    SystemToast {
        id: infoToast
        ....
    },
    ComponentDefinition {
        id: measureSpeakerPageComponent
        source: "MeasureSpeakerPage.qml"
    },
    QTimer {
        id: myTimer
        property int usecase
        interval: 100
        singleShot: true
        onTimeout: {
            switch (myTimer.usecase) {
                case 0:
                    app.compareJSONandXMLaddresses()
                     eturn
                case 1:
                    app.compareJSONandXMLspeaker()
                    return
                ....
            }
        }
    }
]

We’re using the SystemToast as an easy way to let the user know if a process was started or finished.

While the first Page is defined directly inside the main.qml, the MeasureSpeakerPage will only be pushed on demand if the users starts an Action. To avoid unnecessary creation of UI Controls, we only define the component and create it dynamically before pushing on top.

The QTimer is used as a workaround to have a small delay between hitting the Action item and calling the process in C++. As we’ve seen above, it takes a long time to read the 10,000 addresses from XML; over one minute! While executing this task, we want to disable the Action items what was done async and needs more ms.

Without using the QTimer there will be not enough time to disable the Action items. Why? Because we’re doing something you never should do in a real application: executing a long-running task on the UI Thread. While Network requests are async operations out of the box, reading a file isn’t and will block the UI.

The goal of this demo application is to let you feel the long execution time

So I’m using the QTimer workaround. Now disabling the ActionItems works as expected. Here’s how I did it:

NavigationPane {
    id: navPane
    property bool running: false
    .....
    Page {
        id: measureAddressesPage
        ....
        actions: [
            ActionItem {
                title: "Measure Addresses"
                enabled: ! navPane.running
                ...
                onTriggered: {
                    navPane.running = true
                    ...
                    myTimer.usecase = 0
                    activityIndicator.start()
                    myTimer.start()
                }
            },
            ... more ActionItems
        ]

    }
}

All Action Items will be disabled if a Task is running by using a property from NavigationPane. As soon as an ActionItem was triggered, the property ‘running’ was set to ‘true‘. Then, the QTimer was started with a delay of 100 ms. Before the QTimer starts, the System has enough time to disable all Action Items. Because I only want to use one QTimer Object, I added a property ‘usecase’ to know which task from C++ should be started.

How do you get the information now that the task has finished? We’re using Qt Signals and Slots; C++ emitted a signal and in QML we connected a function (Slot) to this Signal. The connection itself was done in onCreationCompleted{}.

You’ll find different Slots (functions) connected in the sample app. Here’s a simple function showing a Toast, stopping the ActivityIndicator and setting the property ‘running’ to false:

function processFinished() {
    activityIndicator.stop()
    navPane.running = false
    switch (myTimer.usecase) {
        case 0:
            infoToast.body = "10'000 Addresses processed.\nread JSON ..."
            infoToast.button.label = "OK"
            infoToast.icon = "asset:///images/ca_done.png"
            infoToast.exec()
            return
        case 1:
             ........
            default:
                return
    }
}
onCreationCompleted: {
    app.conversionDone.connect(processFinished)
}

There are more complex Slots getting result values from C++ if measuring addresses or speakers.

Here are some screenshots to see how the results from measuring addresses are displayed.

We start with an empty Screen:
conference app 8

Starting the Action “Measure Addresses” means the ActionItems will be disabled, a Toast will be shown, and an ActivityIndicator is running:
conference app 9

This is a long running task and can take up to two minutes!

Getting the signal that the task is finished, the Toast changes the text and icon, the ActivityIndicator is stopped, a bar chart is drawn and the ActionItems are enabled:
conference app 10

The BarChart should also work in Landscape:
conference app 11

…and also on the Q10:
conference app 12

From this BarChart you easily understand how fast reading a JSON file is directly compared with XML.

You also see that the Z30 is much faster than the Q10 (Z10 is similar); 81 seconds vs 117 seconds to read a XML file with 10,000 addresses.

As you know, Cascades provides no BarChart diagram as UIControl out of the box, but it can be done using plain Containers.

To get all of this working smoothly, there are some Cascades tricks, so let’s take a look at the code step-by-step.

I’m using a StackLayout, but you can also use a DockLayout – it’s up to you. For me, in this case a StackLayout seems to fit better. Here’s how the Containers are layed out:
conference app 13

Everything is contained in an outer Container with Top->Bottom orientation:

  • the ActivityIndicator: only visible and consuming space if running
  • two Containers using StackLayout Left->Right for the Header Labels using spaceQuota to adjust them to be correct
  • one Container using StackLayout Left->Right for the Footer Labels
  • in the middle the most difficult Container: the valueContainer – also using a StackLayout Left->Right containing 4 Containers: one for each bar

The valueContainer should consume all available space, so the footer Labels will always be positioned at the bottom. The available space will change if changing the orientation or starting and stopping the ActivityIndicator.

The 4 Containers inside the valueContainer represent the bars and are filled with a background color. The height of the bars must correspond to the values we’re getting from C++, where the largest value should fill out the complete height. All other bars will get a height relative to their values. This means every time the height changes we have to recalculate the height of these Containers.

The trick to get it done is using LayoutHandlers. A LayoutHandler will be notified if the Layout changes and you can ask the handler about the height, which is the value we need. We attach a LayoutHandler to the valueContainer and if the height changes, we recalculate the values.

Page {
    id: measureAddressesPage
    property int barHeight: 1200
    onBarHeightChanged: {
        calculateBarChart()
    }
    property int max: -1
    property int readJson: 0
    property int writeJson: 0
    property int readXml: 0
    property int writeXml: 0
    ......
    Container {
        id: valueContainer
        topPadding: 10
        preferredHeight: measureAddressesPage.barHeight
        layout: StackLayout {
            orientation: LayoutOrientation.LeftToRight
        }
        attachedObjects: [
            LayoutUpdateHandler {
                id: valueContainerLayoutHandler
                onLayoutFrameChanged: {
                    measureAddressesPage.barHeight = layoutFrame.height
                }
            }
        ]
        ......
     }
}

We store the values as properties at the Page (max, readJson, writeJson, readXml, writeXml) and there’s also a ‘barHeight’ property.

The ‘barHeight’ property is bound to the preferredHeight of the valueContainer and initially set to a high value: 1200. Cascades tries to set this value as preferredHeight, but it is smart to only use a height which fits into the available space. As we already discussed, the available space can change. For example, starting the ActivityIndicator will reduce the height, and stopping it will enlarge the height. In this case the LayoutHandler will get a notification and we’ll use the actual assigned height to recalculate the values.

There are some special situations where the LayoutHandler from valueContainer won’t be notified about Layout changes, but the outerContainer will be notified. So as a little trick, we also add a LayoutHandler to the outerContainer, and if the Layout changes we simply set the preferredHeight of the valueContainer to a high value (1200), causing Cascades to grab all the available space and so on.

Another situation is coming back from Landscape to Portrait, where we also set this height value to let Cascades do the calculating of the available space.

The TitleBar was set as sticky. This is a trick to get the correct height on the Q10. Otherwise, the Q10 will use a floating TitleBar, which means the height of the TitleBar will be ‘available’ and then your footer Labels will be only visible if scrolling the TitleBar away. Setting the TitleBar as sticky solves this.

Please take a deeper look at the sample code:

Business Logic (C++)

Now let’s see what happens on the C++ side.

We need the ‘app’ context property to get access to the app. Remember, all methods must be invokable from QML. For more information, take a look at the sources or the previous articles of this series.

We now want to concentrate an JSON and XML. You must include:

#include <bb/data/JsonDataAccess>
#include <bb/data/XmlDataAccess>

Now we can convert. Here’s an example of how to read the content of a File, convert from XML to JSON, and save the File:

void ApplicationUI::convertXMLtoJSONAddresses() {
    XmlDataAccess xda;
    QVariant data;
    QString filenameXML;
    filenameXML = "addresses_us.xml";
    QFile dataFile(assetsXMLPath(filenameXML));
    if (!dataFile.exists()) {
        // do something
        return;
    }
    bool ok = dataFile.open(QIODevice::ReadOnly);
    if (ok) {
        data = xda.loadFromBuffer(dataFile.readAll());
        dataFile.close();
    } else {
        // do something
    }
    QString filenameJSON;
    filenameJSON = "addresses_us.json";
    JsonDataAccess jda;
    jda.save(data, dataPath(filenameJSON));
    emit conversionDone();
}

Removing the File-Open-Read-Write code converting is as simple as:

XmlDataAccess xda;
QVariant data;
...
data = xda.loadFromBuffer(dataFile.readAll());
...
JsonDataAccess jda;
jda.save(data, dataPath(filenameJSON));

You can load from Files or from a ByteArray you’re getting from a Network request. JsonDataAccess or XmlDataAccess will give you a QVariant. This QVariant can be a QVariantMap if the content is a single Object, or a QVariantList if it’s an Array.

In the sample app, you’ll find samples of how to convert from XML to JSON, or from JSON to XML. You’ll also see how Signals are emitted and how to convert the speakers to a much smaller JSON file with really fast read access. Here’s the MeasureSpeakerPage:
conference app 18

Don’t forget
Please don’t forget to add the libraries to your .pro file

LIBS +=  -lbbsystem -lbbdata

Summary

From this sample you have learned:

  • HowTo create a Custom BarChart diagram
  • Enable / Disable ActionItems
  • Use of SystemToast in QML
  • Use of ActivityIndicator
  • Use of LayoutHandler to provide flexible Layouts for all Devices
  • Connect QML functions to C++ Signals
  • How to read and write JSON and XML files
  • How to convert JSON to XML or XML to JSON

The main goal of this article was to demonstrate the speed of JSON and the easy way to deal with JSON.

The next part of this series will explain:

  • How I’m using JSON to persist the data in a conference app
  • How to attach JSON data to ListView DataModels
  • How to search and filter JSON data.

We’ll use the same data as used in this sample.

Download and Discuss

The Sample APP is available at GitHub Open Source (Apache 2 License).

I also created a Thread in the Forums, where you can ask or discuss this.

Have fun with the sample app and copy and paste what you need to implement Custom BarCharts into your own apps or to use JSON for your data files!

Join the conversation

Show comments Hide comments
+ -
blog comments powered by Disqus