about standards, webdesign, usability and open source

Problems with HTML5 video codec detection

In case you haven’t heard yet, Microsoft released a new preview release of Internet Explorer 9 with all kinds of great goodies we have been waiting for, including HTML5 video support. I did notice that this new preview didn’t score any bonus points on the HTML5 test for its video and audio support. This was pretty strange, because it should have scored bonus points for the H.264 codec. The article below is the result of a little investigation about why IE9 doesn’t pass the H.264 codec test and the problems I found in other browsers.

Codecs, containers and mime types

First a little background on audio and video. Both the audio and video are encoded by a codec. A codec turns a large number of images into a compressed video stream or a large number of audio samples into a compressed audio stream and vice versa. There are many codecs to choose from with different qualities, but some are more popular than others. That is just half of the story. When you want to merge the compressed audio and video streams in a single file, you also need a container format. Again, there are many container formats to choose from, but not all container formats support all codecs. Currently there are 2 popular combinations for HTML5 video:

  • H.264 video and AAC audio in a MP4 container
  • VP8 video and Vorbis audio in a WebM container

The following two combinations are also used, but are less popular:

  • Theora video and Vorbis audio in an Ogg container
  • MPEG-4 video and AAC audio in a MP4 container

Each combination of codecs and containers can be described by an identifier. That identifier contains the MIME type of the container and the codecs parameter, a combination of unique strings for each of the codecs that have been used.

For example, the MIME type for the Ogg container is video/ogg. The unique string for the Theora codec is theora and the string for the Vorbis codec is vorbis. The identifier would look like this: video/ogg; codecs="vorbis, theora"

H.264, MPEG-4 and AAC are a bit more complicated, because these codecs use profiles which determine which features are used by a codec. This only makes the unique string for each codec a bit more lengthy, but the principle is the same. For example Baseline Profile H.264 video has a unique string of avc1.42E01E and High Profile H.264 video has a unique string of  avc1.64001E.

The MIME type for the MP4 container is video/mp4. The unique string for Low-Complexity AAC audio is mp4a.40.2. Combined with the unique string for Baseline Profile H.264 video you would get the following identifier: video/mp4; codecs="avc1.42E01E, mp4a.40.2".

Detecting which codecs are supported

The HTML5 specification does not specify which combination of codecs and containers should be supported in a browser, but it does provide a way to detect which actually are. This has become quite important because not every browser supports the same codecs. The reasons for this are complex and beyond the scope of this article. What is important, is that by calling the canPlayType() function with either the MIME type of the container or the complete identifier for a specific combination you can ask the browser if it supports that MIME type or combination. The browser then tells you one of three things:

  • an empty string: it does not support it,
  • the string maybe: it does not know if it is supported, but it is not sure it isn’t supported either,
  • the string probably: it supports the combination of container and codec.

Especially if you only specify the MIME type of the container the browser may not be entirely sure it can play it. The reason for this is that some containers support many codecs and the browser may only support one of them. So if you specify only the MIME type you should expect only an maybe answer at best.

The string probably is the best answer you can get. The reason why it isn’t a simple ‘yes’ is that there are other factors that determine whether or not a file can be played by a browser. This answer is as definitive an answer as you will get. The only way to be even more sure is to actually try to load the file and see if it fails or succeeds.

The rest of this article is a detailed explanation of how the different browsers respond to various canPlayType() requests. You’ll notice there are still quite some problems and Internet Explorer isn’t the only one affected. It’s just the first to be broken in such a way that actually prevents proper detection of codecs. I’ve only looked at the results for video codecs, so each codec parameter given to the canPlayType() function only contained the unique identifier of the video codec. I used a special testcase for determining the results: the HTML5test.com video testcase, which you try for yourself.

The results

Internet Explorer 9

Let’s start with the browser that started this little investigation: Internet Explorer 9 preview release 3. It is the first version of IE that supports the video element: it currently has only support for H.264 in an MP4 container, but in the future they may also support WebM on systems where this codec has been manually installed.

Without a codec parameter, IE will report back maybe for MIME types of containers it supports. If it doesn’t recognize the MIME type, it simply returns an empty string. This is absolutely correct.

However IE does not seem to do anything with the codec parameter. If I specify the codecs parameter for the H.264 codec it still returns maybe while it should have returned probably. Also if I specify the codecs parameter for a codec that it does not support, such as MPEG-4, then it still returns maybe, while it should return an empty string.

This makes it impossible to reliably detect which codecs it supports, without actually loading a video file and waiting for it fail or not. This is a real problem and should be fixed as soon as possible.


Safari uses the QuickTime media framework for playing back video instead of shipping any build-in decoders. This means it will support all formats supported by QuickTime. On Macs this isn’t a problem, but when Safari 5 was released, I did receive a couple of reports from users who installed Safari on a Windows system without QuickTime. The result was that Safari failed the video and audio tests of html5test.com.

QuickTime supports the H.264 and MPEG-4 codecs by default. It also supports the MP4 file format.

If a codecs parameter is absent, Safari will report back maybe for all MIME types that it recognizes. That also includes all MIME types for containers that are not normally used for HTML5 video. The good news is that QuickTime lets you manually install additional codecs such as Ogg Theora which are also properly detected.

If a codecs parameter is present, Safari will report back probably for all MIME types that it recognizes, regardless of the contents of the codec parameter. Even if the codec is not supported or simply gibberish, Safari will report back probably. This is a huge bug and should be fixed.


Chrome performs slightly better than Safari, even though both are based on the same rendering engine. Chrome actually has decoders build in for H.264 and Theora and natively supports both the Ogg and MP4 container. Future versions will also support WebM.

Without a codecs parameter, Chrome will report  maybe for all MIME types of containers that it recognizes. This is correct behavior.

However if a codecs parameter is present, and Chrome recognizes both the MIME type and the codec parameter, it will return probably. If recognizes the MIME type, but not the codecs parameter, it will report back maybe which seems reasonable, although it would probably be better to simply return an empty string in this case. Because Chrome ships with build in decoders, it knows exactly which codecs it supports and which won’t work.

There is one more thing that can be improved though. Chrome also reports probably for combinations of containers and codecs that it does not support, for example Theora in a MP4 container does not make sense, but Chrome does report probably. This is just a minor problem that does not lead to any real world problems, but it should be fixed nevertheless.


For Firefox I used the latest nightly which includes support for both Ogg Theora and WebM. The build I tested has the best support for canPlayType() so far and I could only find a simple, small problem.

Without a codecs parameter, Firefox will report back maybe for the Ogg MIME type and probably for the WebM MIME type. That last one is something that the specification specifically discourages. If the codecs parameter is not specified, it should report back maybe at best.

However when I did specify the codecs parameter, Firefox passed all tests. It reports back probably if it recognizes the MIME type and the codecs parameter, but only when the combination actually makes sense. Also if it doesn’t recognize the codecs parameter, it will simply return an empty string. Great!


Finally a browser where I couldn’t find anything wrong. I tested the beta version of Opera 10.6 on Mac which supports both Ogg Theora and WebM. I haven’t been able to test Opera on Linux which uses the system GStreamer framework which might also support other codecs.

Without a codec parameter, Opera will report back maybe for all MIME types it recognizes. This is correct.

When a codec parameter is specified, Opera will report back probably for all MIME types and codecs it recognizes, but only when the combination makes sense. So a VP8 stream in an Ogg container simply returns an empty string. Perfect!

Safari for the iPhone

The iPhone should support both H.264 and MPEG-4 in a MP4 container. Safari on the iPhone seems to suffer from similar problems as Safari on the desktop, but in a worse way.

Unlike its desktop equivalent it will report back probably for containers that it supports when only a MIME type is specified. This is something that is explicitly discouraged by the specification. When no codecs parameter is present, it should report maybe at best.

When a codecs parameter is present, Safari will strangely report back maybe for all MIME types it recognized, regardless of the contents of the codecs parameter. The proper response would be to report back probably for supported codecs and an empty string for codecs it doesn’t recognize. This needs to be fixed as soon as possible.

Palm WebOS 1.4

WebOS should support both MPEG-4 and H.264 in a MP4 container, but I haven’t found a way to actually play the video files. canPlayType() does seem to work though and what it reports seems actually quite good.

If only a MIME type is specified, it will return maybe if it supports the container. If it doesn’t recognize the container, it will return an empty string.

If the codecs parameter is present and it supports the codec, it returns probably which is good. However if the parameter is not recognized, it still returns maybe which should probably be and empty string in this case. PalmOS knows exactly which codecs it supports, so it should also report back with certainty when it encounters a codec which it doesn’t support.

Android 2.2

The Android browser seems to be the worst of all. While it does support the video element, it didn’t want to play any of my test files and canPlayType() always returned an empty string regardless of mime type or codecs parameter. Perhaps it is because I used the Android emulator and not a real phone.

13 Responses to “Problems with HTML5 video codec detection”

  1. badger wrote on June 25th, 2010 at 9:21 pm

    Great article, Niels! Now, does this mean you’ll be removing the workaround for the iPhone? I know you said you were contemplating it. I think it’s the only fair thing to do. Also, it would make sense to then change the wording above the test results to say something like, “If a browser claims support for one or more video codecs, two bonus points are awarded for each codec.”

    Otherwise, good analysis. This stuff is important to hash out in the name of “same markup.”

  2. Gaurav wrote on June 25th, 2010 at 11:23 pm

    This is a very interesting and informative article. Thanks. I can see now there is no quick fix and evangelism is necessary.

    I am actually more bothered by Safari’s behavior than IE9′s.

    I have another suggestion in addition to badger’s: on your test results, produce a ? or a * or something instead of an X for the ‘maybe’ results (no bonus points of course). This way we can see which browsers are reporting ‘maybe’.

  3. Andrés Delfino wrote on June 27th, 2010 at 1:00 am

    Please note that WebM is a container that only contains VP8 video and Vorbis audio. That might be why Firefox returns “probably” when no video/audio codec is specified, since it does support VP8 video and Vorbis audio.

  4. Niels Leenheer wrote on June 27th, 2010 at 8:21 am

    @Andrés: Chris Double from Mozilla has stated in a bug report I filed about this issue that it was indeed deliberate. I do still think it should be changed though, because there is no guarantee that the WebM spec will never change. It is not unthinkable a new audio codec will be added to WebM. Or perhaps a new version of VP8 that is not backwards compatible. In that case Firefox will answer “probably” while it can’t play back the content. And finally, the WebM spec itself also says ‘maybe’ is the right answer.

  5. f.aa wrote on June 29th, 2010 at 4:43 pm

    How come WebGL is included in the tests? It’s not part of HTML5. At best it should be bonus points, I’d think.

  6. grigio wrote on July 4th, 2010 at 7:34 pm

    What about Opera Mini? It’s different from Opera an it is very common on Windows Mobile/Phone and Symbian.

  7. Killer_man wrote on July 14th, 2010 at 12:39 am

    Mozilla Firefox 4 Beta 1 supports Youtube captions but the HTML5 test says that it’s not supported… just try with a video like this:


    Is that a problem of the test or what?

  8. Bernard wrote on July 15th, 2010 at 3:47 pm

    Hi Niels,
    Long time, no see :) If you wish, I can run the tests against a real Android 2.2 browser on my Nexus One. Just contact me.

  9. Niels Leenheer wrote on July 21st, 2010 at 12:09 pm

    @Killer_man YouTube does not use the HTML5 subtitle feature. Instead they created their own using JavaScript and layers which are positioned on top of the video.

  10. Killer_man wrote on August 15th, 2010 at 7:06 pm

    @Niels Leenheer So… in the Firefox 4 Beta videos the subtitles are in the same way like Youtube created them? Using JavaScript and layers?

  11. cristcost wrote on August 23rd, 2010 at 3:20 pm

    Hi Niels,
    What about to make a test (as bonus point) for support of video by other protocols than HTTP (and file)?
    It is compliant to HTML5 standard to access video but no-one is taking care of it…

    In particular, RTSP protocol (UDP based, has URL based resource localization, is better for live video) could be compliant with HTML5 and RTSP URLs could be used in video tags’ src attributes if browsers support it.
    Another video transport protocol available is the RTMP, used for Flash video from Flash media streaming servers but RTSP is more in line with www philosophy.

  12. George wrote on August 24th, 2010 at 3:53 am

    Also, on the topic of detection, I noticed that the test doesn’t seem to pick up on a couple of other things (at least). I built the latest WebKit from svn source this morning and fired it up through Safari on Mac OS X 10.6 and it missed awarding bonus points for “SVG in text/html” and “MathML in text/html”. I went to a sample page with inline MathML and SVG and it seemed to work fine.

  13. Niels Leenheer wrote on August 24th, 2010 at 9:13 am

    @George Webkit currently does have a HTML5 parser and supports MathML and SVG inline in the document. However, they only support this in natural documents, not document fragments and when using innerHTML. I believe they are still working on this and when innerHTML is supported it will be properly detected.