becoming a cog

I spent a couple weeks more or less out of the loop, on purpose, to take some personal time. I did make a few commits (including a large set of maintenance work on the QGraphicsProxyWidget based widgets in libplasma), met with some local KDE and Free software enthusiasts, answered emails and started a few blog entries (before stoppng myself before they consumed too much of the day I was supposed to not be doing such things in ;). In general, though, I stayed away and allowed myself to breath in the air of the city of what will be my new home.

One great thing that I enjoyed during that time had nothing to do with me: I watched the activity that kept on rolling even without my constant full time involvement with Plasma. Some 300 commits to Plasma, dozens of mails to the list, several review requests ... all concrete signs of good healthy activity. Amidst the bug fixes, new features, performance improvements was the sense that this community is alive and breathing and happily busy. This is why I became a cog in KDE in the first place, and it's stunning to see that same kind of environment thriving to this day, and not just in Plasma but all over the KDE community.

Another example of this is how the KDE Commit Digest is back again. Thank you to everyone involved, both those helping Danny crank the new issues out and Danny himself for putting together new infrastructure for it and getting a team put together.

One more example is how the Git services for the KDE community keep improving, from Git integration in KDevelop to the rapidly maturing infrastructure the sysamdin team have been tooling up for us for some time now. projects.kde.org continues to get better and better and Tom is doing an awesome job of keeping everyone informed about that process.

I had told myself this morning that I wouldn't blog today, but I couldn't help it. I caught up on news and a bunch of email communication and after all I read and saw, I just had to race over to share the great feeling I was left with.

Read More

plasma in 18-24 months?

In my last blog entry, I mentioned QtComponents, QML and the new Qt scene graph and then vaguely alluded to profound implications for Plasma. I will not tease you, dear reader, for longer than necessary: this blog entry contains my current preliminary thinking on the what this could all mean for Plasma going forward.

First, I'd like to note that none of this would be possible without the fantastic work going on at Nokia's Qt development offices. They are tackling hard and interesting problems with gusto and producing some very nice results in the process. QtComponents is being developed very much in the open right from the start: an open mailing list for all dev discussion, a public git repo that even contains experiments and early code sketches, a set of use cases and open tasks in Jira. Outreach to community members such as myself, which allowed me to join their design sessions last week, is just one more piece of this. This open from end-to-end, right from the beginning development model is part of the "new Qt" ecosystem that is the culmination of years of consistent effort on the part of many individuals involved with Qt. It's paying off now, and I hope that all new Qt components undertake a similar, or even the exact same, type of approach.

Enough about that, however... what does all this new stuff mean for Plasma?

There are two things that I'm really not very happy with when it comes to Plasma right now:

• libplasma provides some very elementary UI components that really belong in Qt, but because they aren't in Qt have ended up in libplasma out of necessity where they are more of a distraction (in terms of code clarity and design) than anything else


• QGraphicsView, particularly QGraphicsProxyWidget, is not nearly as performant as it could be. It's gotten pretty good in terms of being quite usable even on modest hardware with Qt 4.7 and KDE Platform 4.5, but it could be so much better.

The new scene graph would allow every paint operation in Plasma to be hardware accelerated on the GPU using OpenGL. Not only would effects like blur become cheap (the expense is why we don't use blur on the canvas itself, but only on top level window backgrounds where we can use KWin's OpenGL driven compositing effects to achieve blurring) but things like nice halos around text should be able to be rendered at high quality and low cost. Pixmap usage should drop, framerates skyrocket and client-side image manipulations all but disappear.

This implies moving away from QPainter, which only gets in the way of proper acceleration since the painter can do "anything" in "any" order. Which means moving all of our user interface to the declarative model, specifically QML. In turn that means needing proper declarative style widgets, specifically QtComponents. Finally, it means leaving QGraphicsView behind and moving to a scene graph only system.

This is not a small amount of work: every Plasmoid, Containment, popup and window dressing (e.g. the add widgets interface, the panel controller, the activity manager ..) that does direct painting needs to be moved over to QML. Thankfully we can do these one at a time with the results working very nicely with the current QGraphicsView based libplasma. This means we don't need to do a "massive porting effort with a huge pause between releases"; each release can contain more and more QML driven elements and fewer and fewer uses of QPainter. The result for the user while this happens is likely to be nicer looking interfaces, as QML usually makes this easier to achieve.

Marco has been working extremely hard on the first thing we need to make this process possible: Plasma <-> QML integration. This work is being merged into trunk right now and will debut in KDE Platform 4.6. This means we can start the process of QML-ification of Plasma now. Plasmoids written in C++ can create Plasma::DeclarativeWidgets, Javascript Plasmoids can load QML files from their package and Plasmoids can be written entirely in QML (using the Plasma Javascript API from within the QML).

We will also need to replace our current QGraphicsProxyWidget based pushbuttons, sliders, etc. with QtComponent based ones. This is mostly an issue for Plasmoids written in C++; Plasmoids written using Javascript should shielded from this set of changes. Other scripting languages such as Ruby and Python expose more of the implementation details and so those Plasmoids may require some adjustments. However, moving to QtComponents means waiting for QtComponents to be "ready" and dealing with the baggage of our existing C++ QGraphicsProxyWidget subclasses in libplasma.

Of course, we're not simply waiting for QtComponents to magically mature on their own. Several of us (Artur, Marco, myself) are engaged with the QtComponents project and its team already. Marco has also already provided the first QtComponent push button whose text is set using a source from a Plasma::DataEngine, all done in QML. So we know the basic ideas work in a very practical manner.

Here's the big bomb-shell, however: to fully complete the migration process, we'll need to create a "libplasma2" which is binary and source incompatible to the current libplasma. Corona will cease to be a QGraphicsScene subclass and instead become a scene graph manager; Containment and Applet will become some kind of QML item; the QGraphicsProxyWidget subclasses will be dropped.

The good news is that this is an almost completely internal set of changes. The design of Plasma lends itself very nicely to this set of changes, and the C++ classes in libplasma are nicely aligned as well. For instance, Corona right now is the QGraphicsScene and manages Containments and provides "global" management of things like configuration data, screen geometry hooks, etc. None of that would change, except it being a QGraphicsScene. DataEngines, Plasma::Services, Plasma::Svg, etc. would not change in the least, either. So the external impact looks like it will be surprisingly small for anything written in Javascript or which uses QML for its user interface. Note that Plasma Mobile is already 100% QML, modulo the components it inherited from other Plasma workspaces, consisting mostly of individual Plasmoids.

All of this means that it isn't going to happen in the next release, or even in the release after that. It will be a measured set of changes over the course of many releases to reach the final goal of "going scene graph". User disturbance will therefore be kept to a minimum and we'll be able to continue to make regular releases with noticeable improvements while we do this.

The end result of full hardware acceleration, a fully QML driven user experience and a much improved resource footprint are, in my estimations, worth all of this effort.

There are some potentially very exciting things this could mean beyond simply improving what we already do, though. One of the main ones involves threading. The scene graph is capable of rendering the scene in multiple threads. QML needs some adjustments to take advantage of this, but if/when that work gets done it would allow Plasma to run each Plasmoid in its own thread. (Note: this is different from multi-process.) That would mean that any pausing in a given Plasmoid or other user interface components would cause no annoyances in the rendering or interaction with any other part of the user interface. It would also enable fairly trivial resource management: we could then track per-Plasmoid memory and CPU consumption, for instance. Achieving such a multi-threaded Plasma will require some additional work on top of everything already discussed above, including things such as DataEngine access (which is not currently designed for multi-threaded access) and sharing of Svg renderers between Plasma::Svg objects (something that lets us gain significant performance benefits right now). My initial scan over things says that it isn't insurmountable: we can make rules about things like Plasma::Svg (can only be used in the same thread as the Plasmoid; so no sharing Plasma::Svg's between different instances of Plasmoids) and probably adjust things like DataEngine successfully (the idea of putting the DataEngines out of process, something I've toyed with conceptually since the 4.0 release, becomes even more attractive under this scheme).

It's an exciting set of possibilities, though none of this is certain yet. It's all still "work in process" research and none of the above may happen. I'm hoping it will, however, and the pieces are actually falling into place. Huzzah.

Read More

oslo was, zurich is

I was in Oslo for a few days last week to meet with various people in the Qt offices there. The main purpose was to collaborate a bit on the design of QtComponents, which is a framework that provides a modern widget abstraction for users of QML. The problem to solve is simple to state: create a way to have "MeeGo widgets", "Symbian widgets", "Plasma Desktp widgets", etc. that developers using QML can easily hook into and use so they are both native looking as well as don't have to re-implement buttons and other basics themselves over and over. It's not a very easy problem to solve given the various constraints we face. I'm increasingly happy with the direction QtComponents is going, however, and think the combination of "a reusable core" with "enabling easily rewritable components for a given platform", while perhaps non-conventional, is very compelling.

I also met with Gunnar to talk about the new scene graph he's working on, as well as with Thiago to discuss MeeGo / KDE strategy and Aron Kozak to discuss community integration and how to ramp up KDE and mobile success. Not bad for 2.5 days, even if it left me a little tired at the end of it all.

The scene graph thing is particularly interesting, though, as it allows full and proper hardware acceleration for Qt interfaces. Think "gaming style performance and graphics capabilities" for Qt applications. To really take full advantage of this, the UI will need to either use the scene graph directly (unlikely and undesirable) or else be written with QML which will use the scene graph internally.

Combine this with QtComponents and the ramifications for Plasma workspaces (or any other application that decides to go this route) is absolutely profound. But that's something for my next blog entry. This one is about my traipsing about the European sphere this month. ;)

Which brings me to Zurich, my future home. I received my first piece of mail addressed to Herr Aaron Seigo, which was rather exciting and a little surreal all at once. We went to see the new apartment, which is well on its way to being finished. Between all the "yes, this is really happening" reinforcement, we also took time out to wander about Zurich a bit, stopping in at a book store (where I picked up a cool book for P. about Zurich for young people), cafes and generally enjoying the city. This includes biking about everywhere (Zurich is, despite protestations otherwise by some of its locals, very biking friendly) and wandering the streets with bags of heisse maroni (hot, fresh roasted chestnuts) in hand, which is such a lovely pre-winter time thing to do.

Today I am hanging out at ETH (where S. works), the same university that Albert Einstein (one of my personal heroes) attended, catching up on communication and getting down to some work for the week. I'll probably do similarly tomorrow.

It's feeling more and more natural, home-ish, to be here. :)

Read More

activities as homonyms

I've been reading around the blogosphere about activities in both KDE Plasma and the upcoming GNOME Shell. There's an unfortunate thing at play here: Plasma and GNOME Shell use the word differently. Yes, it's the same spelling and pronunciation and, worst of all, they are used to describe something in the same area of the interface (the desktop shell), but they aren't used to describe the same thing.

In GNOME Shell an "activity" is a virtual desktop, the same thing we've all come to know and love since X got support for them in 1989 based on work done earlier at Xerox PARC. Virtual desktops rock, and GNOME Shell has added an enforced overview (with +/- buttons to easily add and remove virtual desktops, something also in KWin these days) along with an integrated application and document launcher sidebar to the idea. They call this "activities" in an attempt to make the abstract and geekish "virtual desktops" more approachable to people. It is not, however, what most humans would call an activity in every day conversation. It's just a more recognizable name for an old concept that they gave some polish.

In KDE software, an "activity" is actually much closer to what we'd call an activity in the real world: it's a collection of tools with a task based theme. What the theme is remains up to the person using the computer; Plasma just provides an interface with which to create an activity and start defining what belongs to it.

The concept of activities in KDE software is not limited to Plasma, however, as activities are in GNOME Shell. KDE's activities are orchestrated behind the scenes, transparently to the user, by Nepomuk. Other applications can hook into this information and adjust their presentation accordingly. For instance, KWin recently gained support to not only hide and show windows associated with a given activity, but also to stop and start applications using session management. This means that when I switch to my "personal coding project" activity, I get a Kate window with the files loaded in it that I was last editing related to that activity.

Any application can talk to Nepomuk and get notified of activity changes or even create new activities. The idea is to create a task centric workflow allowing one person to use the same laptop or desktop system for multiple (potentially very different) tasks, keeping each one out of the way of the other.

Virtual desktops are really not particularly related to this concept at all, as should be fairly obvious. It's unfortunate that GNOME Shell developers decided to use the same word we are using for something so very different. Ah, well.

In Plasma Mobile and Netbook, we also have activities though they are slightly different beasts. Here they are sets of mini-tools and related applications. So using Plasma Mobile, there are different activities for different common tasks on such a device, such as communication (e.g. "making a phone call") and playing games. When switching between these activities, different sets of home screen applications and tools are presented and run (user customizable, of course).

Again, no relationship between virtual desktops and activities there, where activities map directly to "what I am doing with this device".

It is completely possible to use virtual desktops as a way to separate tasks into similar activity sets. However, it doesn't map very nicely to common use cases such as "I always have my email client (perhaps also always on desktop 4) regardless of what I'm working on". In such (rather common) cases, it becomes clear that virtual desktops and activities (as used in KDE software) are orthogonal, though they can be used as complimentary tools.

Using virtual desktops also doesn't give the applications any useful information as to what to do about the information inside of them. Being on virtual desktop 4 doesn't give KMail, for instance, any information as to what email accounts to start checking automatically. Knowing that the current activity is "Office" does, though.

One of the downsides of taking the approach we are in Plasma is that it is a new set of ideas. There is nothing pre-existing for infrastructure (e.g. we had to create all of the Nepomuk management and application notification from scratch) and we don't have the luxury of looking at other mature systems from which to copy when it comes to things like "what is the best way to switch between system-wide activities?" This means it is more work and a slower path to final results. We feel, however, that it is a slightly more interesting result, in terms of providing a new set of tools that address hard real world challenges in modern computer usage, than creating a full screen virtual desktop pager with integrated app launcher.

As activities take more and more defined shape and become more solid in KDE's Plasma workspaces, it will probably be wise for us to communicate clearly and simply what they really are and how to use them.

Otherwise people will confuse "activities" for "activities" and perhaps even miss out on this next generation of tools without ever understanding what they are missing.

Read More

on the impending future of ui greatnesses

What a title for a blog entry, huh? Well, it's rather tongue in cheek on the one hand, and almost serious on the other.

Lions, Tigers and Cloud HTML5 .. oh my!

I've written about this from various points of view before, and I just love Larry Ellison's response to a question about "the cloud", but it bears repeating and looking at the core issues from different angles because people continue to find themselves confused on the matter. This time we'll look at it from the perspective of "form factors are not programming languages".

Reality is that well over 300 million new desktops and laptops are sold every year. That number is even growing, albeit slowly as one would expect from a mature market segment that is also seeing displacement pressure from new (or revived) form factors. So the "large screen, got a keyboard and a pointing device" form factor is really not going anywhere. It's a good form factor that works for a lot of real world use cases and it will be with us for a while.

HTML5 and the "cloud" do not change that one bit. That's because those technologies are not drivers of physical form factor. I could use those technologies equally well on a phone or a laptop or a T.V. or a fancy billboard at a sports stadium. That is why coming to the conclusion that "Our idea of the desktop is gone." is so wrongheaded.

Now, our way of writing applications for "the desktop" may change over the next decade, but the desktop will still be with us. People will still want a way to launch their apps, manage the shapes they appear in on the screen (aka "windows", since I assume that HTML5CloudAwesomeness doesn't mean "everything is fullscreen with one app at a time" for most people), will want to place these HTML5CloudAwesomenesses around their screen (aka "desktop widgets"), etc. That could, indeed, be written in HTML and Javsacript, but it will still exist.

So what appears inside of our windows may change in the form of where some or all of the data being manipulated is stored and/or what language is used to write them .. but it will still be a lot like a laptop computer.

Just like a tablet doesn't become a desktop just because you run a desktop application on it. If you don't get that one, try running one of the games that are available for the N900 that are just re-compiles of games made for a desktop. It's pretty evident that the language used doesn't magically change the form factor or render it useless.

Network centric computing is awesome and is changing the landscape. How it will change the landscape in the future is not how it is changing it now in terms of creating all these walled gardens that require you to pay with your attention (ads) or with your wallet (rental sites), and I think it is imperative that Free software does provide network storage solutions that work well in this new reality to prevent all the technology from aggregating behind those walled gardens, such as Google.

We will change how we write apps

On that note, I am fairly confident that we will radically change how we write applications with a graphical interface over the next few years. This will not just be on the desktop, but across all kinds of devices. The most exciting thing won't actually be the language used (e.g. HTML vs C++ vs $WHATEVER), but in the level of modularity and subsequent re-use that happens as a result.

Ryan Rix published a screen cast about the Plasma Dashboard KPart that he worked on, particular with regards its use in Kontact. The first comment on his blog asked why use Plasma there at all. The answer is wrapped up in a new way of thinking about application development. This is something I've also blogged about in the past, but that we're starting to actually see happen now.

Instead of writing applications as monolithic things, or at best as collections of components that work with each other only, we're starting to see the emergence of applications that are written as collections of self-contained components that are then collected together into a single interface.

In the specific case of Kontact and it's summary view, it needed to be rewritten for Akonadi anyways. So it was decided to do it in a way that (a) improved what is already there and (b) made new things possible at the same time.

The version shown in Ryan's screencast isn't finished, and the layout and graphic design still has tweaking left to do so it looks a bit nicer in certain places, but even in its current state it gives Kontact's summary dashboard some nice improvements.

I'm not sure how many people new that you could actually move items around in the summary. This is both more obvious now (which is mostly a factor of the graphic design), but also has the ability to provide multiple columns in addition to rows. Horizontal scrolling is there when needed, another improvement.

To extend the summary page requires a plugin system of some sort that everyone can use. Plasma provides just such a component model that fits that perfectly. So an existing widget that is related to email, contacts, events, etc. can now be used in Kontact without any additional work being required by the Kontact team. The email summary widget seen in Ryan's screencast, with all of its great features, is Lion Mail. This was developed for Plasma Desktop in mind originally by sebas, but can now be used in Kontact, saving the Kontact team a ton of work.

What about a weather widget? It was going to have to be re-written, too, to use the new weather data fetching in KDE Platform 4 (which happens to be driven by a Plasma DataEngine now). Well, with the dashboard it doesn't need to be re-written at all: the existing weather widgets can be used there.

Conversely, components written for the Kontact dashboard can also be used in Plasma Desktop, Netbook, other apps using Plasma Dashboard, etc. So the nifty "upcoming events" Plasmoid, written for Kontact is now also available wherever you want.

This is a remarkable, if seemingly subtle, change in how we are writing applications. It's a floating mix of components that we can re-arrange, mash-up and re-use .. at runtime.

We see this not only in Plasma derived work, but also in Akonadi itself as well as KDE Telepathy which renders the functions of Kopete into a similar raft of run-time components.

Not only does this save all of us developers time, it increases consistency between applications, raises the number of features that are easily available to integrate seamlessly into your application (or mash-up) and reduces the runtime overhead of doing so.

Componentizing the Chrome

Another sort of components (don't you love it when we use the same words for different things?) is the user interface chrome itself: the buttons, the knobs, the dials, the menus, the sliders, the web canvas, the print dialog ..

The status quo has been that everyone writes a set of such components for each project type: one for this phone, one for that phone, one for desktop apps, one for PIM apps, one for T.V.s, etc. These components have traditionally been a horrible mix of logic ("when the user presses on this area and then releases on that area, then ...") and display ("paint this gradient here when it is in a pressed state ...").

The display of these elements has been further compounded by the need and desire to make things look and work differently in different contexts. So we have the Oxygen style along with dozens of other Qt styles, each of which is a behemoth of complex C++ that can style all kinds of things in all kinds of situations. There are very real limits to this approach, mostly due to the complexity it introduces and the limitations one runs into quickly due to this.

We also have a different presentation style on a small screen than we do a large screen. Dialogs may look completely different, for instance.

What would be great, however, is if we could get the C++ guy out of the room as much as possible, break up the logic and the display into different pieces so that changing one doesn't impact the other (and removes so much need for subclassing!), make it easy to define new components that overlay just those components from the defaults (so I don't have to write a whole new style just to get a different kind of toggle button semantics, e.g. checkboxes in a mouse driven UI and a slider toggle on a finger driven UI).

If that was possible, app developers could write their UI quicker and easier than they do now (thanks to logic/display separation) and platform developers could mold those application UIs much more cleanly when run on their platform.

Now imagine if this was all done using a proper scene graph allowing for proper hardware acceleration (the sort you usually only get with games these days), built upon a declarative model for user interface (lowering costs and shortening devel times), allowed designers to directly create the elements which developers could then directly use (and vice versa) and easily allowed one to step away from compiled code where desired and use things like QML and JavaScript .. though not be forced to do so, either.

This is the future that QtComponents is attempting to bring us. There is a development hot-house happening next week in Oslo (that I hope to attend for a couple of days as I just happen to be nearby that week, as it turns out) that will be critical to the future shaping of this potentially ground breaking approach.

It should, done right, result in it being possible to write UIs which are as or more powerful than the desktop interfaces we have now that look better, work faster, cost less to make and without being bound to a web browser or the technologies they provide (or don't, as the case may be). All with write-once-run-anywhere-there-is-Qt, even without a native compile if done right.

The Device Spectrum

Something that has struck me about what the Linux kernel did that was groundbreaking for the time was to view all computing devices as a spectrum upon which the kernel, albeit in different configurations, should run well on. A wrist watch, a smartphone, a desktop, a super computer .. they are all just collections of CPUs and storage (volatile and non-volatile) coupled with form factor specific stuff, like graphical displays and networking, with processes running on them, right? Hey, it's all the same! Sort of. So why not one (sort of) kernel that runs on all of them? So came the Linux kernel which now is both running rampant in mobile and owns the super computing world, two opposite sides of the spectrum.

Why not our graphical applications?

With decomposable applications and componentized interface elements we begin to approach the age of the device spectrum with our graphical applications. Right now Marco and I are helping a company in Asia working on a tablet device running MeeGo with Plasma providing the UI. They are able to simply re-use work done for the desktop and the tablet we've already done, re-arrange it, put it on MeeGo and profit. (Well, ok, that last point is yet to be proven: the product is still in R&D. :)

We can now take pieces of Kontact, be it at the level of Akonadi data delivery or in Kontact widgets written with Plasma, and use them on a phone, in a Plasma Desktop panel or in Kontact where we'd traditionally expect them to be stuck. Today, we can even pull that widget live from Kontact's dashboard and show it on my smartphone with Plasma. Tomorrow, with QtComponents, we'll go even further with what we can do with the presentation on these devices.

In Summary: It's Doing Less For More Reward, Not Just Doing It Different

I really don't care if someone writes their app using HTML5 or Javascript or C++ or Brainfuck. That's highly uninteresting to me. What's interesting is this:

• In how many environments and form factors can I run it?


• Does it look good and work well wherever I run it?


• How much does it cost to write it?


• Does it provide me with all the facilities I want during usage?

(Before you lob security and privacy as additions to the above, realize that those are actually facets of the fourth point.)

It becomes apparent that HTML5 (to continue to pick on it :) is one attempt to fill the above four points with varying degrees of success. HTML5 itself isn't very interesting at all, viewed that way. It is of high interest to groups like Google who have run themselves down the narrow alley of "web browser", but not because HTML5 is, per se, that great but because it addresses more of the above in their delivery channel (a web browser) than HTML4 does. While it certainly doesn't hit 100% on all the above points, it moves the metrics closer in that direction.

Still, HTML5 doesn't do a whole hell of a lot for my delivery channels, and it won't for many others as well. QtComponents, QML, Plasma ... these things do a lot for my delivery channels and does so remarkably better than HTML5 on every metric save one: ease of deployment. This is solely because the web browser is ubiquitous these days, even if HTML5 isn't just yet. When "Qt Everywhere" is achieved (or close enough to it, and terrific progress is being made on that goal), even this will no longer be the case. On all the other metrics, we're already at or headed shortly to a much better solution that is more than just "hey, what language can we write applications in for easier deployment?" Of course, that better solution also works seamlessly with HTML5 content thanks to QtWebKit.

This set of technologies will provide (some pieces aren't done yet, e.g. QtComponents) convincing answers to the device spectrum challenges, the component re-use issues and cost factors while opening doors to deployment. Best of all: without sacrificing existing or impending capability. (Aka "we already have OpenGL, what's so exciting about that?")

This is a set of changes that is being seen reverberating across our UI stack even now. Plasma is built around these ideas today, and whether it's a tablet or a smartphone shell or Kontact, KDevelop and Skrooge using it for a summary dashboard or the good ol' laptop with Plasma Desktop on it, we're getting to see the early days of this shift in how applications are being made and delivered right here in KDE.

Read More