A Simple 3D Renderer in HTML5, Part 2

Continuing from where we left off last time, we took some time early on to research existing (free) HTML5 3D rendering engines.  After going over that, we'll take a more in-depth look at our own rendering tech.

Researching Existing Engines

We looked at a number of pre-built Javascript frameworks, but none of them quite did what we needed.  There are a number of matrix libraries out there, though they're either slow (Sylvester http://sylvester.jcoglan.com/) or buried within a more generalized scientific math library (Numeric Javascript http://www.numericjs.com/).  Writing our own library proved much faster.

We also spent some time researching existing HTML5 3D renders, which again, did not do everything we needed.

http://sixrevisions.com/web-development/how-to-create-an-html5-3d-engine/

This is nice and fast, but it uses a lot of assumptions, like a fixed camera angle.

http://www.netmagazine.com/tutorials/build-your-own-html5-3d-engine

This was probably one of the most enlightening articles, both for 3D design and general Javascript class design.  Again, the camera is fixed.  It also uses focal length for its rasterization, whereas our engine would prefer to rely on field of view.  It was easier for us not to worry about converting between the two.

Most of the actual HTML5 3D engines are done in WebGL, which is unfortunately not supported in many places, especially Android web browsers.

Rendering Specifics

Screen from the final game

Back in the 3D renderer, as mentioned the previous post, we did our 2D sprites as HTML elements that we moved and scaled around the screen.  This was largely done because attempting to draw the images directly onto the HTML5 canvas was even slower, especially on Android browsers.  That being said, it's still expensive to manipulate DOM objects - both getting and setting properties on them - so we decided to keep track of all the sprites' values in Javascript variables and only modify the actual DOM object if the values changed.  This helped a lot with speed.

Another issue we ran into was handling objects that move and scale beyond the bounds of our screen.  Depending on the particular browser, this could cause the entire screen to shift (especially when trying to accommodate negative values), or to simply have parts of the sprite draw off to the side when it should be clipped.  After running through a number of solutions, including provide a masking HTML element so we could just hide the overflow, we eventually stumbled on the idea (via random Internet forums) of having the sprites be composed of two elements: an enclosing DIV element and the actual IMG element.  We could then move the DIV to the extents of the screen, and if the sprite needed to go further, we could alter the IMG's margins, which would contain its effects inside the DIV and also hide the overflow (with the CSS overflow: hidden property).

For the HTML5 canvas itself that we used for 3D rendering, we confined our actual rendering to flat polygons.  The canvas allows for drawing arbitrary paths along a set of points, and then either rendering just the path itself, or filling in the enclosed region.  This gave us the flexibility to draw a bunch of arbitrary shapes for the road, but limited us in drawing actual 3D objects, because we did not build in any z-sorting.

Coming up next, architecting the source code and dealing with the game loop and performance.

A Simple 3D Renderer in HTML5

We mentioned this a couple months back, and now we're finally getting around to it.  Over the next few weeks, we'll be posting parts of a postmortem we wrote for our first HTML5 project.  We've done a couple HTML5 projects since that have allowed us to refine our techniques, but this is still an interesting look back at the research and development process.

HTML5 holds a lot of promise as a "write-once, run-everywhere" standard.  The trick then becomes to get it running decently.  And we tackled one of the harder problems: a real time 3D racing game.  We simplified it a great deal so that it behaves more like Rad Racer from the original NES days, but even still, there were a lot of challenges.  Here's the first one.

Rendering Technologies

Note: The screenshot is an early (thought not the earliest) test of our canvas/sprite rendering.

In order to accomplish the 3D environment, we looked at three methods: HTML5 Canvas, SVG, and CSS3.

SVG seemed like the immediate obvious choice.  It would automatically give us nice vector graphics based on a known standard, so we could design art in Inkscape and import it pretty much verbatim into the game.  After some basic experimentation, we found two major flaws: SVG isn't supported on every browser (Android versions 2.3.3 and below are notable in that regard), and when it is supported, it's often comparatively slow to update.

Our next idea was to go to the HTML5 Canvas.  We could do some relatively cheap math to fake a 3D background, especially since the initial concept of the game had players looking from directly behind the car.  This worked well enough, but Android (at least versions before 4.0) tended to be slow to update/refresh.

We looked at a number of benchmarks for the canvas as well.  This proved a very good article about the state of things:

http://www.codefessions.com/2012/03/what-is-fastest-mobile-browser-html5.html

Microsoft also has a pretty nifty performance test:

http://ie.microsoft.com/testdrive/Performance/FishIETank/

CSS3 seemed to be the best bet.  Its transition animations are all nicely accelerated in browsers, delivering a very good framerate.  The trouble, though, is that all the animations are fire-and-forget.  For things like games where player actions will interrupt movement, we couldn't find a good way to make CSS3 behave properly within the limited time we had to research and pick a technology.

In the end, we ended up with a hybrid of HTML5 Canvas and DOM objects that we mostly manipulate through normal CSS.  We wrote a basic 3D renderer and rasterizer for the HTML5 Canvas that would take care of drawing the road, and we treated all the game objects as 2D sprites, which were actually HTML image objects that we moved around and scaled as needed.  This gave us a reasonable framerate on higher-end Android devices, and of course iOS devices, where HTML5 is well-supported (except for audio, which we'll touch on in an upcoming post).

We pulled most of the guts of the 3D renderer from our OpenGL renderer that we use for Live Wallpapers.  This gave us the flexibility to manipulate the camera without having to worry about faking any math, plus gave us a pipeline to not only do the 3D road on the HTML5 Canvas, but also figure out how to place and scale the 2D sprites.  We did have to write a custom matrix class for Javascript, and we had to write a custom rasterizer to figure out how to actually draw points in the 2D space of the canvas/browser window.

Up next, researching existing HTML5 3D engines.

Prismatic Live Wallpaper v1.2

  - New Feature: Daydream support on Android 4.2+
  - New Feature: Simulated scrolling on some devices

Again, primarily a maintenance update that brings some nice side effects along with it.  We had one report about custom images not working, this build brings with it a framework update that may stabilize that some.

Blue Skies Live Wallpaper v2.1

  - New Feature: Daydream support on 4.2+ devices
  - New Feature: Simulated scrolling on some devices
  - Update: "None" wind speed

Again, this is primarily a maintenance release with the associated improvements.  However, you can now select 'none' as your wind speed, as some folks don't like the lateral motion of the clouds.  Enjoy!

Galactic Core Live Wallpaper v2.3

  - New Feature: Daydream support on 4.2+ devices
  - New Feature: Simulated scrolling on some devices
  - Update: Much better accelerometer behavior

This started out as a maintenance update, but turned out to actually be a pretty nice one.  In addition to getting Daydream support and simulated scrolling by virtue of being ported to our most recent framework, this version drastically improves the behavior of the orientation camera.

Previously we were trying to get away with just using the orientation sensor for this, as it's cheaper, but honestly it never worked all that well.  Starting with this version we're more properly using the accelerometer, and as a result it feels much more correct.

Opaque Daydreams and Live Wallpapers

As Jeremy mentioned, Daydream support for devices running Android 4.2+ is starting to roll out to select Live Wallpapers. We're starting out slowly, but the intent is to eventually have pretty much all of our wallpapers (including the free versions!) work as a Daydream once any initial issues are sorted out.

Speaking of implementation issues, there was a subtle one that we caught internally which I haven't seen discussed elsewhere: if your DreamService uses a full-screen SurfaceView (just about anything OpenGL-based), Android will see its window as transparent. Because of that, the application underneath the Daydream is never paused and the two will compete for resources. That's especially problematic if both are using reasonably complex OpenGL scenes (a Daydream over a view of the desktop/lock screen with a view of a Live Wallpaper, for example).

One way to work around the problem is to launch an Activity from the Daydream service. New Activities, even using a SurfaceView, will generally pause what's underneath them unless using a translucent style. While it did pause the wallpaper below, the transition between the Daydream and the Activity was noticeable and the whole thing felt a bit fragile to use. So that approach was quickly discarded.

After browsing the Android SDK source for a while and comparing application window properties in Hierarchy Viewer, the difference between Daydreams that did properly pause apps (no SurfaceView) and ones that didn't (contained a vanilla SurfaceView) came down to a single private flag that signals a window to report how much (if any) of it is transparent. The flag is set in an unavoidable call to requestTransparentRegion() that is made by a SurfaceView as soon as it's added to the view hierarchy.

After some trial and error, our shipping solution is to create a custom RelativeLayout container for the SurfaceView that overrides requestTransparentRegion(). It's an empty method in our version, which effectively blocks the request to gather the transparent region from making its way up the view hierarchy. The window is now totally opaque to Android, but that opens up a new problem: SurfaceViews are added underneath the current window by default, in case you want to draw standard UI controls above them. In other words, the SurfaceView is totally hidden by the blank DreamService window! To get around that, call SurfaceView.setZOrderOnTop( true ).

The end result is a DreamService window that displays your SurfaceView scene and pauses whatever is below it. Optional support for transparency can be easily added, if ever needed.

A minor downside is that I'm pretty sure the above-mentioned solid blank rectangle is still being drawn every frame below the SurfaceView. I haven't confirmed this yet with dumpsys, but because the contents of the rectangle never changes, it should never invalidate and so hopefully end up in a hardware overlay (fast) rather than drawing into a frame buffer (slower). Either way, it's certainly faster than drawing another complicated OpenGL scene below it, and I haven't noticed an actual real-world hit to performance.

I did try an approach that overrode the container's gatherTransparentRegion() method to always return true (opaque) and a 0x0 transparent region. Despite seeming like the Right Way to do this with public APIs, it outright failed to pause the application below the DreamService. It seems that even if the transparent region is reported to not exist, just the presence of the request flag disqualifies DreamService's window from being opaque to the view hierarchy. In the end, what we came up with gives a nearly-identical end result and has so far been extremely reliable in practice.

Clock Tower Live Wallpaper v1.1

  - New Feature: Daydream support on 4.2+ devices
  - Update: New clock face: Blue Star

The primary goal of this update is to integrate our new Daydream support, as a clock seemed like a very appropriate use of the feature!  Assuming things go well here we'll be looking to propagate Daydreams out to our other products as well.  :)

In addition, this update includes a new clock face!  This one uses Latin numbers, so as to grant an alternative to the whole roman numerals controversy. :P

Halloween Live Wallpaper v1.55

  - New Feature: Daydream support on 4.2+ devices!
  - Bug Fix: Color picker crash on some devices

I've been putting off a few updates waiting for us to get fully working Daydream support in, and now that it's complete I'll be enabling it on more or less all our products as we issue updates.  Halloween and Clock Tower will be the first, and assuming nothing comes up I'll try to hit a bunch more shortly.

Beyond Daydream support, this update also fixes a crash related to choosing the candle color, which happened only in certain locales.

Bamboo Forest Live Wallpaper v2.01

  - Update: Light rays now use 'light color' pref
  - Bug Fix: Color picker crash in some locales

This update is primarily to fix a crash -- the same locale-specific one that was affecting Autumn Tree a little while back.  I'd overlooked Bamboo being affected, but it's taken care of now.  In testing it I noticed that the light rays in the background always stay white, so updated them to make use of the current light color.  This makes a touch easier to customize your scene.

Aquarium Live Wallpaper v3.05

  - New Feature: Fizzy style bubbler!

Had a user point out a few days ago that their tank has one of those bubblers that produces a ton of little tiny bubbles instead of a small number of big bubbles.  I said we'd look into that, and so here it is!  Screenshot included, though honestly it looks much better in motion than it does here...

Java Strings, toLowerCase(), and Locale

I fixed an interesting bug last night, and figured I'd mention it here for anybody who happens to be searching.

Basically, we got a customer service e-mail from a fellow who was having one of our products crash on his Galaxy S2.  I asked him to send over some logs, we went back and forth a few times, and eventually he discovered that if he set his phone to English, the product worked no problem.  That's strange, we don't do anything special for different languages, do we?  As it turns out, we do.

Replicating the bug, I discovered it was in the midst of Mesh.readFromText().  This is a function that reads the text version of our model format -- we have both a working binary and text-based path, since for some problems having a human-readable model format is extremely useful.  Almost everything we use is binary regardless, but in this case we're reading a text formatted model.  The format is simple, and the top of the file looks like so:

TMDL 7
NUMFRAMES 1

TC 4

0:    1.0    1.0
1:    0.0    1.0
2:    0.0    0.0
3:    1.0    0.0

WINDING 2

0:    0    1    3
1:    1    2    3

FRAME 0

VERTEX 4

0:    1.0    1.0    0.0
1:    -1.0    1.0    0.0
2:    -1.0    -1.0    0.0
3:    1.0    -1.0    0.0

Now, what's happening is when it gets to the first set of windings, it tells me I've overrun an array and throws an exception.  After a bunch of experimentation, it turns out that when I have the device set to Turkish, it never recognizes the "winding" key as being hit, so it's still trying to read in texture coordinates.  Except the array is already full of those.  That's weird, why is it missing that key?

The reason why is that I made the file parsing non-case-sensitive, and I did it by sending the current line though toLowerCase() before parsing it.  It's in no way obvious, but toLowerCase() makes decisions based on your current locale, so it's possible for your lower-case version of a string to have accents, hard-spaces instead of soft-spaces or what have you inserted.  I'm still not sure which character does it, but when it was forcing "WINDING" to lower-case, one of those characters ended up different from the ascii "winding".  Thus the key was never recognized.

This was a pretty harsh lesson to learn.  Luckily, it's easy to fix, as you can call toLowerCase with an argument specifying a locale, like so: myString.toLowerCase(Locale.ENGLISH).  That means you can at least count on it being consistent if you're doing something like parsing a file.  This last week or so has had more than one lesson about this.

So, here's the TL;DR take away: If you're using toLowerCase() or a similar operation on a user-visible string, let it default to the current locale.  If you're using it when parsing, tell it which one to use to ensure consistency.

Snowfall Live Wallpaper v2.2

  - New Feature: Influence snow direction by tilting device!
  - Update: Simulated scrolling on some devices

I didn't want to radically change the behavior of the product, as there's a ton of folks using it, but this version adds a fun new feature.  Basically, you can influence the direction that new snowflakes will fall by tilting your device!  This doesn't affect existing flakes currently so it takes a second, though I could look at doing that if folks want it.  :)

NA Flags v1.81 & EU Flags v1.81

  - Bug Fix:  Issues with custom images

We recently changed out underlying framework to support mipmaps much more generally, and as a result a couple defaults changed.  Unfortunately, we apparently broke custom images, as while they don't support mipmaps properly we still had the texture mode for mipmapping set.  As a result, these were showing up as black on a lot of devices. Sorry about that folks, it's fixed now.

NA Flags & EU Flags v1.8

  - New Feature: Simulated scrolling on relevent devices

This is primarily a maintenance update that gets a couple of minor fixes in place, and more importantly supports simulated scrolling on Samsung and other devices.

Autumn Tree Live Wallpaper v1.1

  - New Feature: Oak Tree!
  - Bug Fix: Custom color crash in some locales

This should fix an issue some folks were seeing  when trying to customize the light colors -- basically, we updated our pref saving a while back to use String.format when saving color values.  Sounds fine, except that some locales use commas instead of periods to indicate decimal places.  This means when you try to read the values back in, parseFloat can't handle it.  This took a while to realize.

In addition, the oak tree leaves are finally in shippable shape, so you get an additional tree option!