Friday, December 7, 2012

JavaFX: Designing for Multiple Screen Resolutions

Throughout the last couple of years we have seen significant improvements in screen resolutions. On cell phones, on tablets, on laptops – everywhere. Gone are the days when resolutions like super VGA (800 × 600), or later full HD (1920 × 1200) would be the de facto standard for years in a row. Now, the only thing we know is that the next device coming out will have an even more impressive screen resolution.
Obviously, not knowing which screen resolution to design for, presents a challenge to JavaFX application designers. The solution is to design your application in a resolution independent manner, and this article shows you how.

Cupertino Pixel Doubling

Apple has been a major driver of innovation in the area of screen resolutions, since they coined the term "Retina Display" when introducing the iPhone 4 on June 7, 2010. Everyone seemed thrilled over this new phone, because it literally quadrupled the number of pixels of the previous model, leading to an impressive pixel density of 326 pixels per inch.
But there was a catch. Apple's very successful App Store already contained hundreds of thousands of apps specifically designed for the old, lower resolution. So to ease the transition to retina, Apple introduced a feature in iOS that allowed rendering each pixel of an old app onto a 2 × 2 pixel area of the new display, effectively stretching the app to full screen size.
While this was an intelligent, pragmatic solution to an immediate problem, it was also a perfect example of how you should never, ever approach the challenge of a heterogeneous end-user environment. Because what Apple did, was to allow the lowest quality graphics of the old apps to become first class citizen on the new phones. Google it, and you will find literally thousands of articles and forum posts on how terribly pixelated this looked.

A Better Approach with JavaFX

The application I am creating for wefend.com will be launched globally by the end of 2013 (as requested, you can now click "Like", if you would like to test this JavaFX app during the beta). This means it has to support virtually any thinkable end-user screen resolution. So from the beginning in 2011, I did a number of tests to figure out which approach would work best. I had five potentially conflicting requirements to meet:
  1. High readability: On no device should any part of the user interface be too small or hard to read.
  2. Pixelation free: The application should never render scaled up, pixelated images.
  3. Crispness: The graphics should always snap to pixels, so that a 1 pixel wide vertical line renders crisply, without anti-aliasing artefacts.
  4. Size and aspect ratio flexibility: To ensure optimal use of screen real estate, the application should be able to layout perfectly in both 16:9, 16:10, and 4:3 aspect ratios, by showing more content instead of just empty space.
  5. Animation smoothness: Animations of the application should never decline below 24 prames per second.
These requirements were a great help in shaping my approach to multi-resolution support. First of all, simply stretching the application to fit the user's screen would not suffice, since it would lead to pixelated images (conflicting with requirement 2), and it would not make users of 24 inch or 30 inch monitors able to take full advantage of their screen real estate (requirement 4).
Also, rendering everything at a very high resolution and subsequently scaling that user interface down to fit the user's screen, would also not meet the requirements. Downscaling the scene would mean texts would scale down accordingly leading to less crispness, and thereby less readability. Tests also proved, that such an approach would most likely conflict with animation smoothness (requirement 5).
And finally, designing for an average screen resolution, say, full HD, would lead to low readability (conflicting with requirements 1) on very high pixel density screens, such as the new 13 inch Mac Pro which fits a whopping 2560 × 1600 pixels into a its tiny screen.

Introducing Wide Quad Full HD

Instead, I decided to take advantage of JavaFX's support for vector graphics, scalable images, and switchable style sheets.
First, I specifically chose to design for a so-called wide quad full HD resolution. That is a wide screen display of 3840 × 2160 pixels – the resolution you would get, if you took four full HD television screens and stacked them up in a 2 × 2 configuration. I chose this resolution because it has some really nice properties:
  1. Manageable: It is small enough to be manageable in graphics design applications like Photoshop without killing the design process, with having to constantly access the hard drive when working on the design.
  2. It has a 16:9 aspect ratio: More and more monitors are wide screen, either 16:9 or 16:10. Designing for the narrowest of the two only means that the 16:10 users get to see a little more content at the bottom of each user interface, which is no disadvantage.
  3. Crisp: It ensures that when scaling down to the most commonly used resolutions, graphics still look crisp, as long as the designer tries to place and size design elements following a 12 × 12 pixel grid.
The crispness results from the fact that the width of other, smaller resolutions are factors of 3840 (or of 3840 × 2). Examples:
  • Width of 2008 PC laptop in my office: 1280 = 3840 ÷ 3
  • Width of common 21 or 24 inch PC LCD screen: 1920 = 3840 ÷ 2.
  • Width of late 2012 13 inch MacBook Pro: 2560 = 3840 × 2 ÷ 3
Consequently, we can be certain that a 12 × 12 rectangle drawn in a wide quad full HD design will measure an integral number of pixels on smaller resolution screens:
  • 2008 PC laptop: 4 × 4 (one third)
  • Common 21 or 24 inch PC LCD screen: 6 × 6 (half)
  • MacBook Pro: 8 × 8 (two thirds)
As an example, if I draw a TextField in Photoshop measuring 60 × 180, it will end up on the end-user's MacBook Pro measuring exactly 40 × 120 – not 40.123 × 120.369, which would render with anti-aliasing artefacts and look blurred.

Choosing Target Resolutions

Even though we could deliver an application perfectly optimised for almost any modern day resolution this way, less is more in this case – less resolutions means more time for assuring the quality of each scaled version of the graphics.
Exactly which specific resolutions to support depends on your target audience. If you are developing a company intranet application, you might be able to export your graphics to just one resolution known to work reasonably well on the company bought monitors. But by designing for quad full HD from the beginning, you ensure forwards compatibility, anticipating the day when everyone switches to better resolution screens.
With our coming global software release in mind, three target resolutions were chosen for the wefend.com application:
  • HD – 1280 × 720
  • Full HD – 1920 × 1080
  • Quad HD – 2560 × 1440
Of course, other resolutions are supported as well – if your screen is 1600 × 1200, you will most likely chose our HD settings, which renders labels at 10px. In other words, not matching our specifically targeted resolutions gives you the same experience Windows users have had for years, unless they chose scaled fonts.

Using Switchable Stylesheets

With the overall screen sizes in place, we move on to looking at layouts. JavaFX is no less than a sublime layout engine, and when used right, every end-user gets a perfect user experience.
To achieve this, we built three different style sheets for the resolution specific style settings:
  1. hd.css
  2. fhd.css (short for full HD)
  3. qhd.css (short for quad HD)
To ensure that we would not have to rewrite all styles from scratch in each of these files, we created a file called common.css containing all style rules that can be shared. So, for buttons for instance, common.css contains
.button, .toggle-button {
  -fx-font-family: "Open Sans Regular";
  -fx-text-fill: -wf-text-fill;
  -fx-background-color:
        -wf-edge-color, -wf-body-color;
  -fx-effect: -wf-pushable;
}
Here all the "-wf-" constants are just aliases for regular hsb(…) color definitions to avoid having to repeat them all over the place and to ensure consistency across the application. Also, notice how the support for advanced CSS selectors allows us to style both buttons and toggle buttons with a single rule.
Since all this is shared, all we have to define in fhd.css is
.button, .toggle-button {
  -fx-padding: 3 9 3 9;
  -fx-background-insets: 0, 1;
  -fx-background-radius: 5, 4;
}
These settings then have to be repeated in hd.css and qhd.css, and tested to see which exact pixel sizes look best. Always use integral sizes, as coordinates like 3.5 means you will get anti-aliasing, which again means your application will look bad.
When the user changes his or her preference to Full HD, we simply remove all style sheets and load common.css and fhd.css. It works like a charm, and it executes almost instantaneously.

From Photoshop to JavaFX

Even though buttons and panels can easily be expressed using just JavaFX vector graphics and CSS, other visual elements like icons, diagrams, or pictures, cannot. Ideally, you would want to just export 1 set of graphics, to ease the burdon on the graphics designer, and to lower the time you spent managing all the files. But what if you have multi resolution support?
Because we are using JavaFX, exporting just the designed wide quad full HD version will suffice in most cases. That is because JavaFX is capable of scaling images on-the-fly, just like a modern browser does when the user magnifies the page. The only key issue here is exporting enough pixels to avoid upscaling.
On-the-fly scaling can be achieved with both the ImageView and the Image class, and we have used the latter. The Image class contains the following constructor
public Image(
    InputStream is,
    double requestedWidth,
    double requestedHeight,
    boolean preserveRatio,
    boolean smooth
)
Using this constructor, we can just point to an image file containing the design size image, while telling which actual width and height we want. In the current prototype of my app, this looks just fine for photographs. However, when dealing with digitally fabricated drawings like a company logo or a diagram, this on-the-fly scaling will sometimes look too crude, even with smooth set to true. To solve this problem, I created a utility class which lets you load images without knowing if the designer has provided a resolution specific version of that image. It works as follows:
  1. Look up the currently selected resolution.
  2. Check if a file named <image name>.<resolution name>.<file extension> exists (e.g. "logo.fhd.png").
  3. If so, load it and return it.
  4. If not, default to loading <image name>.<file extension> (e.g. "logo.png") which contains the full design size image, and use the Image constructor above to downsize it.
This has turned out to work quite well. In fact, it even gives the graphics designer the option of supplying completely different contents for each resolution. As we all know, a large poster can contain many fine details, but a small stamp can only contain a few. So, if you have a complex graphic that looks great in quad HD, but messy when scaled down to HD, this approach allows you to load a simplified version for the HD resolution.

Conslusion

JavaFX contains a wide range of features that help you support multiple screen resolutions. Designing for a very high resolution, and then using JavaFX's on-the-fly image scaling, ensures all your customers get a perfect end-user experience. Creating a utility class for automatic downscaling of images helps lowering the burden on the graphics designer's export process. Using CSS while favouring style rule reuse helps lower the cost of maintaining your application's look and feel. All in all, adding multiple screen resolution support is not for free, but using JavaFX wisely lets you lower your development costs significantly.
I will end this article by returning to the question of which resolution to design for? Will wide quad full HD with its 3840 × 2160 pixels be enough in the foreseeable future?
While we have seen many new screen resolutions throughout the last year, I consider it likely, that the pixel density will stabilise under less than 400 pixels per inch. Yes, we are used to RAM sizes and CPU clock frequencies doubling again and again, but there are a couple of reasons why screens are different:
  • Natural human vision is not able to distinguish individual pixels on a 300 pixels per inch display held at normal reading distance. Especially with modern day anti-aliasing and subpixel rendering improving the visual appearance.
  • Higher density means more work for the chip rendering the graphics, which leads to lower frame rates (a complaint raised by multiple reviewers of the 300 pixels per inch Nexus 10 tablet of fall 2012).
  • More load on the graphics chip means more power consumption which, at least on mobile devices, drains the battery.
  • More load requires a faster graphics chip which increases the price of the device.
Consequently, I believe you are reasonably well covered if you design for a wide quad full HD resolution. Mind you that this resolution is currently being marketed under the names "4K" and "Ultra HD" in state of the art movie theaters which have invested in brand new equipment to be able to project new movies like Peter Jackson's "The Hobbit". In other words: If it is enough for Peter Jackson and 480 inch projection screens, it will probably suffice for most 15 inch laptops in the foreseeable future.

7 comments:

  1. An extremely well thought through, informative and well presented post. Thank you!

    ReplyDelete
  2. You are most welcome, DMV. And congratulations by the way – it seems you were visitor number 2.000 (ish) of this blog, which is only 5 weeks old. I am very pleased. Please do share this article with anyone you know, who could benefit from reading it. Happy (upcoming) holidays.

    ReplyDelete
  3. Nice approach, but how do you handle 4:3 screens. You explain that 16:10 will get more information, but 4:3 screens are very different. Do you plan to create new layout, or will your layout adapt automatically. If so, how do you plan to do this ?

    We're working on a commercial app and our customers have a very heterogeneous range of screen resolutions. We have at least, requirement 4 in common.

    ReplyDelete
  4. Interesting question, Grégory Fernandez. As mentioned, the smallest width we support is 1280. If you have a 16:9 screen only 1280 pixels wide, this gives you a height of 720. If you have a 4:3 screen only 1280 pixels wide, your height is 960. So again, you just get to see a little more content than the what we design for, because you have more height.

    As an example, we have a UI where people can see a map of their country on the left and a scrollable list of information on the right. If you switch from 16:9 to 4:3, your screen becomes higher, and the map becomes larger, while the scrollable list is stretched so it shows more items at once. Again, not a disadvantage to anyone.

    Of course, depending on your target audience, setting the bar at 1280 may not be an option, but in our case, that is a system requirement.

    ReplyDelete
  5. Nice reading but there remains one thing to know. How do you tell a Mac with a Retina display to switch off the automatic "Cupertino Pixel Doubling" for Java applications (with Oracle Java 7 or above).

    One more question. Do you see a chance to make your style sheets available to the public?

    ReplyDelete
  6. Hi mipa. According to a post from Richard Bair on the openjfx-dev mailing list, Mac retina support is work in progress. According to Jira, this will be fixed and is scheduled for Lombard, which is the next big feature release (see http://javafx-jira.kenai.com/browse/RT-24009).

    I expect parts of our stylesheets to be shared as part of the Intuism JavaFX application framework, I hope to be releasing after product launch. Stay tuned.

    ReplyDelete
  7. Trees get often enjoyed a crucial part inside individuals lives, through delivering shade and also keeping the sweetness of the destination for a fruit and also flowers that attract fauna! Once you are about bushes you're feeling a sense involving serenity and also relaxed plus more notably, safeguard. For anyone who is worried about saving the earth, next sapling properly is actually what you need to be considering number one priority.Træfældning

    ReplyDelete