More jheora stuff

By now I patched Wikimedia's Cortado/jheora fork to play back any valid Theora bitstream with 4:2:0 subsampling. This is mostly a matter of properly consuming bits from the bitstream used for non-VP3 features (e.g. per-block qi indices).

Also, I hardened jheora and the jst-component using jheora to not simply stop playback whenever an exception occurs within the decoder. Upon losing sync of the bitstream (e.g. when the bitstream is damaged) corrupted values may end up getting used as array indices later on - which of course usually lead to an ArrayOutOfBoundsException. The decoder now catches all exceptions and returns appropriate error codes (well, in fact just BADPACKET). The playback component can then decide to continue (the next packet may be fine anyway) or not. The new default behaviour is to just continue decoding. I tested this "error recovery" by randomly consuming bits in the decoder during various stages of decoding, thus wrecking the bitstream on purpose. This results in colorful block-storms and all imaginable sorts of corruption - but playback won't stop, which I consider a much better behavior than just giving up.

Anyway: Thusnelda, the Java world is prepared for you, so don't hesitate to replace the old encoder! ;-)

jheora updates

As many of you know there's a Java applet ("Cortado") for playback of Ogg audio and video which originally was developed at Fluendo. Wikimedia is perhaps the best known "customer" of this applet and they since have forked it ( ) as apparently Fluendo isn't really doing active development on Cortado anymore.

The Java Theora decoder used in Cortado is called "jheora". It is a ported version of an old C reference decoder, meaning it doesn't support the whole Theora 1.0 specification. This means that it won't decode Theora-streams created by "advanced" encoders like Monty's "Thusnelda" encoder.

Thusnelda is most likely going to become the new reference encoder, so we have a problem here.

I had a look into many options to somehow get an updated Java Theora decoder (e.g. port the new C reference decoder, which supports the complete Theora specification), but in the end I decided that just tacking missing Theora features onto the old jheora decoder may actually be the quickest way to get a robust decoder.

I'm positive that this may be the way to go and I was able to write a (biggish) patch to enable jheora to correctly decode Thusnelda streams:

This patch is basically a complete rewrite of how quantization matrices are computed (interpolation, scaling etc. etc.) and implements proper handling of 4MV for partially coded macroblocks.

Still missing from jheora, but currently not used by any relevant encoder:

  • support for subsampling other than 4:2:0 (personally I think that this isn't relevant for web-video, but it may be a nice-to-have).

  • support for per-block quantizers. The patch already doing some decoding related to that (decode what quantizers are used within a frame) and it "just" needs finishing (decode what block is using what quantizer). Should be pretty straightforward to implement and debug once I have a fittingly encoded file.

To put it into a nutshell: jheora isn't completely unmaintained anymore ;-)

With an updated Cortado applet it should be pretty easy to bring HTML5 video (and audio) support to browsers not supporting Ogg codecs (e.g. Apple is shipping Java by default on OS X, so we can enable Ogg Theora/Vorbis playback with a tiny bit of JavaScript embedding the Cortado-applet instead).

Video in Firefox

As the work on finding a consensus on a baseline codec for HTML5 <video> is either stalled or at least not visibly (from the outside) moving along, browser vendors have the difficult task to ensure <video> is not an interoperability nightmare.

One frequent idea how to cope with the multitude of formats is to have a codec download system, which would link into the system's media framework (Gstreamer, DirectShow, Quicktime) and install a fitting codec to deal with any media format. This is e.g. discussed over at In case of Mozilla it seems the idea of shipping at least one set of free codecs may be dropped. This would make Mozilla completely depend on the media framework of the underlying system.

Now, I see several problems with this approach:

  • Not every user has the permission to install additional codecs.
  • There are quite a few systems out there with broken media frameworks (some users apparently love to install about every "codec pack" they can find, which can break things horribly).
  • There are widespread formats for which no free (and legal everywhere) codecs are available for certain environments. The classic example would be Windows Media, for which Linux users would have to buy Gstreamer components with Microsoft's blessing or resort to unlicensed components, which may be illegal depending on where you live.
  • People wouldn't know if using a free playback component for a given format is legal where they live, so any "click here to install if you're aware of the legal risks"-dialog would mostly read "Make page work? y/n". One shouldn't push the nastyness of codec licensing onto the users and expose them to risks they cannot evaluate.
  • It opens the door for any proprietary format. Windows users will most likely produce Windows Media content (after all it would work on the "majority of systems"), while Mac users may use an old toolchain to produce content which only Quicktime may be able to decode.
Without at least one codec (Theora, Dirac, OMS Video (if latter actually surfaces and is completely free) or any other free alternative) being shipped as default one cannot even credibly point out that there is at least one free alternative which is worth producing content for that ensures compatibility at least within the Firefox population. Given that Firefox is by far the most popular browser amongst those that implement <video> this would have some weight.
  • Current Mood
    worried worried
  • Tags

Yet another recylced FOMS post on Theora

In my ongoing series of recycling semi-smart posts that'd otherwise bitrot in non-public archives, I hereby present my view on Theora's bitrate management. The question originally asked sorta boils down to "The last comparisons are a couple of years old, why not do new comparisons?".

Those tests were done with "DVD backups" in mind. That is: The transcoded video was supposed to fit onto a single CD, which means bitrate management has to be enabled.

(Overstatements following for dramatic effect)

Theora's VP3-inherited bitrate management to my knowledge is what one may call a "disaster area". From what I hear it works more by luck and by consuming kittens than by design. Even if it gives you a file of fitting size it'll have its revenge by climbing onto a nearby hill and screaming "Ha Ha, I misplaced bits here and there and royally screwed up quality!".

So albeit there have been bugfixes in other areas things will still look pretty nasty if one dares to activate bitrate management.

General consensus is that the bitrate management has to be ripped out and replaced with something completely different. Here, again, Thusnelda is supposed to come to our rescue, by doing a proper management based on rate distortion theory (IIRC).

Now, there are two lessons to be learned:
  1. If you want maximum Theora quality possible *now* you should avoid using the bitrate management and try a quality setting matching your bitrate target (if possible).
  2. Once the new encoder is actually in a releasable state Theora may become much much more suitable for web video, which often carries bitrate constraints.
I for sure am curious how much better the new bitrate management will actually be, but until the new encoder is actually available to the public sceptics have every right to remain sceptic. It would be no fun otherwise, I guess.

Theora and quality and FOMS

Well, I think I did never mention how great FOMS is, did I? Nope, I wasn't there. But I'm subscribed to the FOMS mailing list, which is a very nice "place" with a lot of cool people trying to push open/free multimedia technology to the next level. One actually nice property of this list is isn't just populated by us "the usual suspects" volunteers - there are also people from ffmpeg or Mozilla and other great communities.

Perhaps mostly thanks to the HTML5 <video> tag there's renewed interest in free video codecs. The two best known free video codecs would be Dirac and Theora. And when it comes to Theora sooner or later there's the "so, how good is it?" question. Some claim it's bad, others will say it's the best thing ever.

The FOMS mailing list archive is non-public, so I'll just copy & paste what I wrote on that topic:

Well, depending on where you come from you reach different conclusions - e.g. that *all* those mentioned codecs [H.261, H.263, H.264 and Theora] are in fact old technology. They're all DCT based codecs with block based motion compensation with different levels of sophistication to squeeze some more compression out of the same basic scheme, which dates back to the 1980ies.

As for Theora, I feel people like to confuse Theora (the reference implementation as shipped in alpha and beta versions) and Theora (the bitstream specification). The current reference implementation in terms of coding efficiently is (from my evaluations, which were not conducted in a scientific way, so anyone can trust or distrust my findings) superior to H.261 and/or MPEG-1. It is somewhere in the same league as H.263 and/or MPEG-4 Part 2, with robust and well-tuned MPEG-4 Part 2 encoders (e.g. XviD) beating Theora. There are different sweetspots (for example I tend to think Theora does sorta catch up at low bitrates as it doesn't tend to get too blocky) but in general good MPEG-4 Part 2 encoders are more polished. H.264/MPEG-4 Part 10 is yet another notch better. It doesn't do miracles, but it's pretty nice and the most powerful video compression scheme currently available.

Now, comparing bitstream features I don't see why Theora wouldn't be able to catch up with the best MPEG-4 Part 2 encoders. In fact, it has a few (as of now unused) tricks which may make a nice difference.

As for the comparison with H.264 I'll leave that to the more experienced coders [what I meant to say: People with more experience in coding technology]. I feel that Theora overall is the "lighter" format, being able to deliver good quality at modest computational complexity, while some feel that H.264's complexity isn't always justified by the gains in compression efficiency. H.264 seems to have more headroom overall, though.

As for VP3 being "old technology" because of VP6 and VP7 being there: Although this suggests VP3 is three or four generations behind it seems that in fact On2 just released finer tunings of "the same theme" at a regular intervals. VP4 is not a new "generation" over VP3. It appears to be a better tuned and rearranged VP3 (in the doom9 codec shootouts VP4 did mostly show the same strengths and weaknesses as VP3, but an overall  a tad-bit better tuning). I'd guess VP3 to VP7 would roughly be equivalent to the transition of H.263 to H.264, but with more evolutionary steps in between (after all every new iteration of On2's technology will bring new licensing income).

That'd be similar to DivX. Version 3.11 was a hacked MPEG-4 Part 2 derivate, Verion 4 a "proper" MPEG-4 Part 2 codec and the now current version 6.8 is.... still MPEG-4 Part 2. Still of course each version improved in quality - and XviD (also MPEG-4 Part 2) is considered to be even better and remains competetive to the "new" H.264 generation depending on what you're searching for. When VP3 was released it was competetive with the MPEG-4 Part 2 of those days (there was mostly just DivX 3.11 and/or the rather lousy DivX 4), but while the MPEG-4 Part 2 encoders improved (without changing the bitstream!) Theora up to this day is still using the ca. 1999 VP3 encoder without major changes. Work is underway to change just that, but as of now the outdated VP3/Theora encoder (which has plenty of well-identified implementation weaknesses)  continues to distract from what Theora as a format is capable of.

Thus, in my opinion, deducing that a "new generation" of bitstreams automatically outclasses "previous generation" bitstreams can be misleading.

So yeah, I think that Theora can be pretty valuable codec, provided the encoder receives some attention. It's not a secret that there's a new encoder under development (Thusnelda). We'll see how big the actual improvements are once the new encoder is ready - I have big confidence that Theora quality will be noticably better, but until then people of course have every right to remain sceptical.