Guest post from Anders – Ed.
Since the first release of the Cascades™ Beta, we have received a lot of feedback, suggestions and questions from developers trying it out. Thank you for all of this — we are very grateful! It really helps us get perspective on what we are doing, how it is perceived, and where we still have work to do.
Some of the most common questions are about custom UI: How do I create a custom UI component? How do I extend the existing components? How do I implement my own paint() method?
Custom UI works very differently in Cascades compared to how it used to work in the old BlackBerry® Java® platform, and in this blog post I want to try to bring some clarity to that — how it works and why it works in the way it does.
I’ll start off by briefly comparing how Cascades renders its UI compared to traditional widget toolkits such as the BlackBerry Java UI. Understanding the differences explains why custom UI has to be approached differently in the two architectures.
Traditional UI rendering – Immediate mode rendering
In the BlackBerry Java platform and in many UI toolkits for desktop and mobile, UI is rendered using “widgets” drawn with “paint()” calls. The exact naming of the building blocks and functions differs between implementations, but the principle is the same and is often called immediate mode rendering. The application developer constructs a hierarchy of widgets, with each one representing a specific area on the screen. A more advanced widget can consist of a parent widget and a number of child widgets. For example, a list could consist of a widget drawing the background, widgets representing each item and a widget representing the scroll bar. Together they represent a List Widget. The position of each individual widget is decided by the system using a layout engine. The positions are based on where the application wants them to be, and often it is possible to add completely custom layout code as well. However, the actual content – what the widget is drawing on the screen – is completely up to the application to draw when the rendering engine asks it to.
The application uses low level drawing commands such as “DrawLine”, “DrawCircle”, “SetColor”, and so on to draw the widget on a pixel buffer. Finally, the system composites the whole screen by going through the set of widgets, making sure their position is updated (based on animations, scrolling, etc.) and asks the widgets to draw themselves on the piece of screen real estate which is theirs. The callback from the rendering thread, where the widget should issue their draw commands, is typically called the “paint()” method.
(As a side note, there are usually other steps in the rendering where the application gets callbacks, such as layout() to do custom layouts. The same reasoning can be applied for those cases as well.)
From a threading perspective it can be represented as the picture below.
Often the UI Framework contains a predefined set of UI widgets for the application to use. If the developer wanted to customize a widget, it could selectively overload the paint() method and replace it with custom drawing commands, or call the old paint() method but decorate it with new custom code.
Immediate mode rendering is easy to work with and works well in many cases, but it certainly has limitations and drawbacks:
- The application can block the rendering thread. Since the rendering thread is calling app code in the paint() method, slow code in one widget could easily slow down the whole UI.
- Overdraw. It’s fairly easy for the rendering engine to understand which widgets are totally obscured by others (although transparency makes it a little complicated), but for partly obscured or clipped widgets, usually the drawing code can only draw a complete widget, hence many pixels gets drawn and then overdrawn several times.
- Hard to optimize for GPUs. Modern GPUs have been designed mainly for gaming, where most of the geometry, textures, and so on is known beforehand. As a result, GPUs like to have their work nicely organized for them in terms of drawing states and textures. “First let’s draw everything of this kind then let’s draw everything of that kind” – that’s how your GPU likes to roll! Optimizing the drawing commands in an immediate mode rendering is inherently difficult because there is no single place where the complete rendering work is known. The rendering is spread out in autonomous widgets. The result is that the GPU needs to change state constantly and shuffle textures in and out of graphics memory.
- Overloading overload. Overloading paint() methods and UI behavioral methods works for simpler widgets, but when widgets get complex with lots of features and lots of dynamics, it gets really hairy. You more or less have to expose every single behavior as an overloadable function, and it has to work no matter what combinations of function overloads the developer chooses to implement. This puts great pressure on the internal design of the widgets and a wide API to test and maintain. It also makes it cumbersome to add new features and behaviors without messing up something else. In the end, for complex controls, you just can’t make any assumptions about anything, because anything may be overloaded with unknown application code.
Of course, there are ways of working around these problems, but immediate mode rendering has a hard time performing well on modern hardware, and more modern kinds of rendering engines are taking over using retained mode rendering.
Modern UI rendering – Retained mode rendering
Cascades uses a fundamentally different way of rendering known as retained mode rendering. Instead of letting the application be an active part of the rendering loop, the application only manages the ui tree. When the rendering engine needs to render, it takes the current state of the ui tree and renders it. To use a gaming analogy, in retained mode rendering the UI becomes a “scene” that’s being rendered by the rendering engine.
Since the application is not directly involved in the rendering, slow application code cannot block the rendering, which generally means better performance. There are ways to get a slow UI anyway, generally by making the scene big or complex. Also, if the application can’t feed the data of the scene fast enough – for example, not providing list items to a list – the UI can be perceived slow by the user, but the actual rendering is never delayed.
As an example, you can try to create a really slow ListItemManager in Cascades that sleeps for a second every time it provides an item. Even if the list populates slowly, you will be able to scroll the list smoothly.
Another great advantage with retained mode rendering is that the UI engine can get a holistic view of the whole UI and optimize textures, states, and so on for the GPU.
Needless to say, using function overloading for customization does not work very well with retained mode rendering since it would introduce callbacks to the application from the rendering thread. It could be used for overloading behaviors that are completely disconnected from the rendering of the control, but not a lot for UI features that are completely disconnected from the rendering.
A retained mode renderer has one inherent quirk that is good to know about: Since the application cannot block the rendering thread, the application can never know exactly where things are on the screen. You can listen to layout updates, but once you get the updates in your app, you cannot be 100% sure that the position is up to date. A new frame could have been rendered and moved your object. That may sound like a huge limitation, but in real life it’s usually not a problem. Also, writing code that depends on where things are on the screen is often a recipe for bugs, especially when moving between different form factors and orientations.
This explains why the paint() method is gone in Cascades. There is also another aspect to consider: UI consistency.
There is also a design consistency aspect to customization. With BlackBerry® 10, the RIM® designers have spent a lot of time designing, prototyping and testing out the core UI controls. They look and feel as they do because of deeply thought-through considerations and design decisions. They should be considered a unit together rather than individual classes spread out in the class hierarchy.
Hence, we are sensitive to letting developers change them too much. It would break the unity and cause inconsistencies, and BlackBerry device users would have a hard time finding their way in applications. They might start to wonder why a radio button behaves in one way in one app but another way in another app.
We definitely could have chosen a different path and focused on making the controls generic and easy to customize, making it simple for each individual developer to make their own drop down, picker or progress bar. However, we really wanted a consistent look and feel of the core UI. Does that mean every app has to look the same? Of course not. Firstly, some controls are more prone to being customized such as buttons. We chose to make them more generic and support greater customization options or generally more usable for compositions. The ImageButton and Container represent such cases.
Custom UI in Cascades
The toolbox features you should be looking for are:
- Generic, basic controls such as ImageView, Label, Containers, ScrollView, ImageButton
- Animations and layouts
- Event and gesture handling
When you have reached the look and feel you want, you can package it up into something easily reusable in QML and C++ by exposing its features as QML properties and Q_INVOKABLEs.
So does that mean that you hardly can reuse anything in the existing controls?
No. In Cascades you reuse by using a control as part of your custom control composite; for example, by including a label or ImageButton as part of a more advanced custom list item. Usually you would use the more generic controls, but nothing is stopping you from including any control in your composite.
We know there are some important layouts missing that would make some Custom UI easier to build — we are planning to add them soon.
Will I be able to implement any UI in Cascades?
Cascades has its limitations just like any other framework. We have been focusing on core UI for the first release of BlackBerry 10 because we believe that is what is most useful for most developers. We are aware of some limitations in certain areas of the framework that can make some custom UI tricky to implement. This is also why we created the “Stump the Developers” challenge. We want to know what is messy to do and we want to understand what kind of UI you are creating.
There is a classic rule of thumb for APIs: “If in doubt, leave it out.” Some APIs we have left out intentionally in the first release to make sure they are well thought-through before being published. That being said, Cascades in its first version is more than capable of handling most custom UI.
This is nice and all, but I really need a drawLine() to draw my graph.
There is nothing stopping a retained mode renderer from having lower level vector features such as drawLine. We don’t use vector graphics to draw our UI internally and we just didn’t have time to add a vector library in Cascades yet. For now you will have to use a ForeignWindowControl and draw into it using OpenGL (something along the lines of this tutorial) or perhaps an open source graph renderer. This would be useful for many developers, so it’s a great opportunity for a friendly developer to contribute this to one of our repositories on github
Okay I got it, you don’t want us to change your core controls. But I just want to change color to make them match the color scheme of my app. Why can’t I do that?
The next release of Cascades will support two color themes — light theme and dark theme. Adding free colorization of controls is trickier than you would think. A control in Cascades is not just a couple of different colored texts and boxes, but instead composites of several layers of visuals with subtle patterns, shading and transitions. Translating that into simple parameters where random colors can be assigned is not easy. A simple color based themeing system is perhaps not the way forward. Again, we are in doubt, so we have left it out — at least for now.
We are still looking for feedback in this area and have already seen some areas of improvement. We love to see great custom UI so this is an area where you can expect more features coming in the future.