GUI Frameworks

The GUI framework is the software library which allows applications to put pixels onto the LCD. It handles contention between applications, deals with windows and usually provides an abstraction away from pixel-by-pixel drawing through widget libraries.

On a desktop system, this is separated into several components: An X-server which allows client applications to connect and draw things in the client's window. Then there are the widget toolkits, where you generally have a choice between Motif, GTK or QT (there are many others, but these are the more common ones). On an embedded system, this separation is not always as clear cut.

NOTE: LinuxDevices has a more detailed guide to embedded graphics here, although this is quite old now.

Back Ends

Frame buffer (FBDev)

Most Linux systems on embedded devices only provide a frame buffer device. This allows an application to memory-map the pixels on the screen and write to them as if it was a regular array. This frame buffer interface is universal across all architectures and all devices, allowing applications utilising the frame buffer to be ported easily. The disadvantage is that quite often the Application Processor has more advanced graphics hardware for rendering which is not utilized by the frame buffer interface. Another disadvantage is when contention for the frame buffer exists. What happens when two applications want to write to the framebuffer at once? How do they know not to overwrite each other's contents?

By the way, Matt has a page on getting the Frame Buffer working with a GumStix ConsoleLCD-VX.

DirectFB

DirectFB addresses the issue of not utilising hardware acceleration by providing applications with a more advanced programming interface. Common graphics operations such as line drawing, blit'ing and basic windowing are provided. When the underlying hardware is able to accelerate these operations, this acceleration is used. When hardware acceleration is not available, the operation is performed in software, providing maximum compatibility for applications. While this is an excellent project, the number of hardware-accelerated devices supported are small and does not include any CPUs you'd find on a phone. This may, of course, change in the future. DirectFB also addresses the contention issue by providing windowing. Applications request a window from DirectFB, which ensures no applications overwrite each other's contents (Similar to X).

Also... the page describing how to draw pixels on the screen with the pxafb frame buffer also has instructions on how to get DirectFB working on the GumStix ConsoleLCD-VX.

Nano-X (Formally Microwindows)

An alternative to using a frame buffer device is to use an x server as you would on a desktop PC. There are many (MANY) applications out there which use X. The problem with x servers are their footprint. Both memory and storage requirements of a normal x server are too large for small embedded systems. To address this, an x-like server has been produced called Nano-X. While not conforming to the X windows API, the API is similar, allowing X windows applications to be ported. The last item of news on the nano-x site is dated 2005, so this project may be dead.

KDrive (formally Micro-x)

Unlike nano-x, KDrive is a complete X server and even supports the RENDER X windows extensions. Although KDrive is a full X Server, it goes to great lengths to not become bloated and is designed with embedded systems in mind. As KDrive supports the RENDER extensions, accelerating drawing operations in hardware is possible, but requires modification for the target Application Processor's graphics core. As with DirectFB, the list of supported hardware accelerated devices does not include graphics cores found in phones. There is an x server for the HP IPaq, however this simply uses the standard frame buffer and adds support for the VGA-Out device.

OpenKODE

OpenKODE is a new specification for mobile devices and includes OpenGL ES, OpenVG, OpenMAX and similar into a single, integrated API. The emphases here is hardware acceleration. OpenKODE is cross platform and is gathering a lot of industry support. One reason why this might be of interest is rather nicely demonstrated by nVidia in this user interface demo which uses an OpenKODE 1.0 implementation: Next-Gen Phone Interface. There are currently no open source implementations of the OpenKODE specification, however there are OpenGL ES & OpenVG (binary) Linux drivers available for PowerVR MBX & SGX cores, found in higher-end [Application Processors]. These drivers are not available from Imagination Technologies (who designed the PowerVR series) directly but are instead supplied with the BSP for your chosen CPU.

Widget Toolkits

A widget toolkit provides useful widgets such as buttons, menus, text boxes etc. The application programmer doesn't care what a button looks like, just what happens when it's pressed. On embedded systems, you generally have two choices: GTK+ or QT. Recently however, people working on the E17 (Enlightenment) window manager project have started looking at running enlightenment on embedded devices. Also emerging are more advanced, Flash based user interfaces. Sadly though, there are currently no open source equivalents at present.

GTK+

GTK+ has excellent support on embedded systems. Ports of GTK+ have been made for rendering directly to the frame buffer, to a DirectFB device and of course KDrive. For more information about GTK+, visit [1].

QT/Qtopia

QT is a commercial product written by a company called Trolltech. However, Trolltech uses a dual license scheme where it releases QT under the GPL. This means that if you also use the GPL to license your application, you can use QT for free. QT as it stands is unsuitable for embedded devices due to it's large size and large dependency tree. However, a few years ago, Trolltech launched an embedded version of QT, cleverly named Qt/Embedded. While this is still used for the OPIE application framework, Troltech have updated Qt/Embedded and produced their own framework called Qtopia. Out of the box, both Qt/Embedded and Qtopia render directly to the frame buffer. In Qtopia, it is possible to accelerate this rendering by re-implementing several of the classes the library uses to perform all drawing operations. The Both QT/Embedded & the Qtopia framework also handles windowing internally when multiple applications are running. More recently, Trolltech has incorporated OpenGL ES into Qtopia, which is used for hardware accelerated drawing operations and for compositing multiple windows. The Trolltech documentation also hints at visual effects similar to those on Beryl/Compiz, although there are no screen-shots avaliable.

Enlightenment

Enlightenment started life as a window manager for X11. Over the years, it has transformed into a very rich, full desktop environment similar to KDE and Gnome. Enlightenment has always had a very slick user interface with lots of eye-candy. The latest development version of Enlightenment, E17 is highly modularised. E17 has its own widget toolkit called the Enlightenment Widget Library (Ewl). The Ewl draws its widgets using the Enlightenment Foundation Library (ELF) which renders onto a canvas component called EVAS. EVAS supports compositing with true alpha-blending allowing some very attractive user interfaces. EVAS also supports multiple back-ends including X11, OpenGL, the Linux framebuffer and DirectFB.

Quantum Step

QuantumStep is essentially OpenStep for mobile devices. It's layered on top of X11 so is (in theory) fairly portable. Developers may be familiar with another product derived from NeXTStep... MacOS X. QS uses the MacOS X development chain, including Xcode and Objective-C. So... if you're a Mac-o-phile, then there is an open alternative to the iPhone.

Application Frameworks

The widget toolkits provide facilities for applications to draw buttons on the screen, but what if they want to find a telephone number? On a typical phone, lots of different applications are going to need a contact list. Having a different contact list for each application is going to get silly. Instead, there are several application framesworks available. These can be split into 3 groups, those based on QT, those based on GTK+ and those based on Enlightenment. This is similar to the KDE vs Gnome vs Enlightenment dilemma, just on embedded devices. In general, GTK+ frameworks use Kdrive as an X server and the matchbox window manager. Frameworks based on QT (there are only 2 of them) use the windowing system built into QT/Embedded (now Qtopia Core). As such, an application using a GTK+ framework can't run simultaneously with an application using a QT based framework.

The last 12-months has seen interest in Embedded Linux explode. As a result, the market has become fragmented, with lots of different competing application frameworks. All of these newer frameworks (OpenMoko, GPE, GPE Phone Edition, Hiker, Sato, Sugar & Hildon) are based on the GTK+/KDrive/Matchbox combination. All of these projects have forked GTK+ and develop on separate branches, which is a lot of duplicated and wasted effort. To address the problem of fragmentation, the Gnome Mobile Initiative was founded to bring these different frameworks together and introduce some consistency & compatibility. The Gnome Mobile website is at http://www.gnome.org/mobile.

GTK+ Based Frameworks

OpenMoko

OpenMoko is an open source project sponsored by FIC. It aims to create a complete open source software stack for mobile phones. As the work is being sponsored by FIC, the first phone OpenMoko is designed to run on is the FIC Neo1973. OpenMoko uses KDrive to provide the back end for graphics. All the widgets used on the phone are designed specifically for OpenMoko and written using the GTK+ toolkit.

GPE Palmtop Environment

From GPE's homepage: "The GPE Palmtop Environment aims to provide a Free Software GUI environment for palmtop/handheld computers running the GNU/Linux™ operating system. GPE uses the X Window System, and the GTK+-2.6 widget toolkit.

GPE is not a single piece of software, but an entire environment of components which make it possible to use your GNU/Linux handheld for standard tasks such as Personal Information Management (PIM). GPE makes it easy for developers to create powerful programs, by giving them the infrastructure they need.

Besides providing core software such as shared libraries, and perhaps more importantly, the GPE environment fixes standards for program interaction, such as SQL, XML, and other data schemas."

A recent development has been the launch of GPE Phone edition. This aims to implement a complete LIPS compliant software stack. For more information on GPE in general, visit [2]

Hiker (Access Linux Platform)

The Access Linux Platform contains an application framework called Hiker. Hiker is built on top of GTK+ and provides facilities to allow applications to exchange data etc. More information at the Access Linux Platform's website: www.access-company.com/products/linux/alp.html

Sato

Sato is another framework based on GTK+ and is produced by OpenedHand for their "Poky" Linux distribution. Sato includes lots of applications such as games, a web browser, contact list etc. See pokylinux.org for details.

Hildon (Maemo)

Hildon is user interface part of the Maemo distribution used on Nokia web tablets. It has started to be ported to other platforms such as Ultra-Mobile computers (E.g. Samsung Q1) as part of the Ubuntu Mobile & Embedded project. It, again, is based on GTK+ and contains widgets for menus, status bars etc. The best place to look into Hildon is through the http://maemo.org website.

Sugar (OLPC)

Sugar is the user interface used in the One Laptop Per Child project. It takes a different approach to the "desktop" metaphor and uses it's own unique user interface. The Sugar wiki can be found at http://wiki.laptop.org/go/Sugar.

QT/Qtopia Based Frameworks

Qtopia

from Qtopia's homepage: " Qtopia is unrivaled as the application platform and user interface for Linux, allowing efficient creation of mobile and embedded devices."

Qtopia is split up into different versions, including a Qtopia Phone Edition. Qtopia core is name given to the actual application framework. Apart from several demos and examples, Qtopia cores does not come with any applications. Applications are included in the "Open Source Edition" and "Phone Edition". For more information, visit [3].

Open Palmtop Integrated Environment (OPIE)

Originally started as an alternative GUI for the Sharp Zaurus, OPIE has now spread across several devices and has a large collection of applications. Opie uses an older version of QT/Embedded which lacks many of the feature of the more modern Qtopia. However, due to it's stability & large application base it remains a very interesting prospect. More information at [4]

Enlightenment Based Frameworks

Currently, there is only one distinct project using Enlightenment on embedded devices and that is Fancypants. However, the Enlightenment developers have been working with embedded devices in mind and there are numerous blog posts about things which have been prototyped.

Fancypants

Fancypants is an "advanced graphics and high-performance multi-media platform for developing embedded applications in consumer, commercial and industrial devices." It is developed commercially by Fluffy Spider Technologies. It uses EVAS and Ecore from the Enlightenment project and includes a very attractive user interface. See http://www.fancypants-graphics.com/ for details, including videos showing off some of the features.