HTML5 video brings cool new features, same old fragmentation

I was excited to spend some time digging into the new video features of HTML5 recently.  My mindset has been that once HTML5 becomes the standard, browsers are finally going to behave better and conform to the web standards more closely than ever.  The reality is that the same old problems that have long plagued web developers still exist, and may become more prominent than ever with HTML5.

After the launch of a new website the other week, we had reports of the embedded video not playing correctly on some iPhones, even when using the latest iOS4.  I hadn’t done the research or development on the video feature for this project, so I ramped up on the implementation details quickly with the help of one of the original developers, and then set off to diagnose the problem.

Chiefly, I needed to define the bug (aka an “unexpected feature”) a little better.  The HTML5 video element (using .m4v and .ogv sources) is displayed in a modal window when the thumbnail is clicked, and while the desktop browsers all pass beautifully, the various mobile devices (iOS devices specifically) were hit or miss.

I tested on the devices that were used by the reporters of the bug, as well as with some of the other mobile devices laying around the office for better context.  All of the desktop browsers worked fine, as multiple codecs were used along with the HTML5 video tag to properly support Chrome, FireFox, and Safari, and an elegant Adobe Flash fallback was done for IE (IE7 and IE8 do not support HTML5 at all).  Only iOS was correctly displaying the modal window, but failing to play the video on some devices.

The devices I tested with include:

  • iPad (iOS 3.2.2)
  • iPhone 3G  (iOS 4.0)
  • iPhone 3GS  (iOS 4.1)
  • iPhone 4  (iOS 4.1)
  • iPod Touch  (iOS 3.2)

Here’s how I worked through it.

The Problem

  • iPhone 4  (iOS 4.1) works fine.
  • All other iOS versions and devices showed the poster thumbnail for the video, but the play button was crossed out, indicating the video was not playable.

The Analysis

  • The page was using textbook HTML5 code to declare a video element along with 2 separate source elements, an MPEG4 and an OGG, each with mime types and codecs declared.
  • The HTML5-compatible desktop browsers were running the video, so the code was at least syntactically correct.
  • The web server was sending the correct mime-type for both the .mp4 file and the .ogv file.
  • The root cause is likely a codec or player issue, not a code or mime-type issue.
  • Either the file was not encoded in an acceptable format, or the set of failing iOS devices were not able to correctly detect the (correct) format of the video file.

The Solution

Re-encode the file!  I know from some basic video production experience that file formats, video players and transcoders can be finicky.  Since a few code changes weren’t doing the trick, and the mime-type HTTP header was already being sent, the video file itself just seemed to be the next likely culprit.

Here’s a great article on mobile file formats.  I was most interested in the MP4 knowledge, and since I didn’t know which profile was used when the video was encoded, I decided to re-encode the video using the “baseline” profile.  The H.264 Baseline Profile is compatible with a broader range of devices, while the High Profile has better quality.  It made perfect sense that iPhone 4’s would play the newer, better quality video format, and that my 3GS (though running iOS 4.1) would be more likely to support only the Baseline Profile.

I used HandBrake to go back to my original .mp4 source file, and it quickly churned out my new video file in no time at all (they give us some pretty rad hardware at Springbox!).  With my new .m4v file (notice the different file extension) in hand, I updated our staging server and got the QA team started on approving the fix for production.  I had also removed the “type” attribute from the .m4v file’s source tag for better Android support, even though we weren’t officially supporting Android.

The fix was confirmed by QA, the new video file was pushed to production, and the client was happy.  All in a good day’s work.  However, in troubleshooting this issue, I’ve found that the HTML5 spec is really not locked down very well at all, and the fragmented support of current browsers (desktop or mobile) make development arduous and troublesome, especially across the ever-growing array of mobile devices.  This is nothing new for web developers, who’ve fought some hard battles with their software tools and technologies, even as some have matured into standards.  For now, those of us who publish video to the masses should get used to encoding videos in 2 or 3 formats (at least) and hacking code to make it work on everybody’s device.

Vendors, developers and clients alike aren’t waiting for HTML5 to be finalized.  In fact, they’re chomping at the bit because it’s new and cool (and they are right!).  An official with the W3C has been quoted as saying that they “want to be feature-complete by mid-2011”, which seems like a long way off.  But while HTML5 and it’s features were originally sold as a solution to the compatibility nightmare that is cross-browser web development, I’m finding that developers will need to wait a little bit longer for our jobs to get easier.  In the meantime, all of us code ninjas will need to continue doing redundant development to make things work for everybody – as usual.  But the results will be rewarding, and the client will be happy!

Could Apple be making nice with developers?

Apple has released a surprising statement that seems to relax restrictions on both developers and the tools they use to create apps.  BlackBerry has long been an open and easy environment to write code, and Google Android‘s approach is even more so.  Apple risks losing market share if they can’t make developers happy, and I’m one of them.  They also released more information on the guidelines they use to approve apps in the App Store.  It’s hard to believe they were so secretive about the rules they used to judge new applications and were notorious for rejecting apps with no concrete broken rule given.

The following is the statement from Apple, as you must have a developer account to access the actual URL:

We are continually trying to make the App Store even better. We have listened to our developers and taken much of their feedback to heart. Based on their input, today we are making some important changes to our iOS Developer Program license in sections 3.3.1, 3.3.2 and 3.3.9 to relax some restrictions we put in place earlier this year.

In particular, we are relaxing all restrictions on the development tools used to create iOS apps, as long as the resulting apps do not download any code. This should give developers the flexibility they want, while preserving the security we need.

We are continually trying to make the App Store even better. We have listened to our developers and taken much of their feedback to heart. Based on their input, today we are making some important changes to our iOS Developer Program license in sections 3.3.1, 3.3.2 and 3.3.9 to relax some restrictions we put in place earlier this year.
In particular, we are relaxing all restrictions on the development tools used to create iOS apps, as long as the resulting apps do not download any code. This should give developers the flexibility they want, while preserving the security we need.

This has 2 big impacts on my world as an iPhone developer and user.  First, it gives me back the tools to create mobile apps across platforms faster, and with better quality and stability.  Code toolsets like Appcelerator Titanium and PhoneGap are amazing examples of integrating languages, platforms and technologies to produce mobile software for multiple devices!

The second impact is on myself as a user, and the fact that my iPhone web browsing experience is often crippled by error messages where Adobe Flash content should be running.  While Apple is still leaving its restriction on Flash within the Safari Mobile browser in place, Adobe tools like Packager – which can convert Flash apps into iPhone apps – are now being approved for the App Store.  The product’s web page up until now has had a “discontinued” message after Apple started playing some really hard ball.

So, assuming this huge announcement is genuine and represents a real change in my relationship with them, I’ll say thanks to Apple.  Keep it up – this is how we want you and the entire mobile market to behave.  We’ll see where this all goes, of course, but given your previous history with developers, I’ll keep an eye on them for future threats to my choice of toolsets.  But without a doubt, my heart grew 3 sizes after reading about this move.

Apple releases iOS 4.1 to iPhone and iPod

Today, Apple released the latest version of the operating system behind the iPhone, iPod and iPad, iOS v4.1.  Of course, the iPad still only runs iOS 3.2 which is a little disappointing.  However, also announced was a status update on the future iOS 4.2, which will bring multitasking, folders, and other functionality to the iPad, will be here in November.  Finally!

I hadn’t downloaded iTunes 10 yet, so I had to sit and wait a bit.  iTunes 10 is an improvement in the look n’ feel, so that’s a plus.  I’m not the biggest iTunes fan in the first place after often losing photos, overwriting contacts, and other disasters that never happened with my BlackBerry Pearl.  Not even once.  And with this latest version also comes support for Apple’s new music-related social networking service, “Ping”.  More on that as it evolves.

iOS 4.1 has a few new features:

 

  • Updated Game Center
  • HDR photos
  • HD video uploads over WiFi
  • TV show rentals

I can’t wait to try out the TV show rentals feature, though mostly just to test out the technology.  My eyes wouldn’t last long staring at any phone’s relatively small screen for 30 mins or more at a time.  The HDR photos addition is arguably one of the cooler features, however I can’t give it a test drive with my 3GS, it’s for iPhone 4 owners only.  There’s a great tutorial on the subject of HDR photography at Ars Technica and how to use the new feature.  Some may argue that the method in which iOS implements the feature is “not true HDR”, which is an interesting point.  I’ll take a look at some of the third party apps for comparison before I render judgement on the quality of the resulting image, though.

The iOS 4.1 installation to my iPhone went smoothly, just as I expected.  There are rarely huge problems updating the iPhone operating systems, but always remember to do a separate backup of your device before you start any update as a best practice!  The best improvement with the update was the extra speed on my 3GS.  From applications to animations, and even resuming from it’s “locked” state, my phone runs perceivably faster.

I do recommend that everybody grab this latest update to your phone, it’s a safe and quick installation.