Now before I even start this post let me make something clear. I am trying to do some research on HTML 5 video and have some questions that I’m hoping someone can help me with. I’m planning on doing some demos as I am genuinely interested in trying to build some players. Any comments that try to rekindle the flame war from the last post will be deleted. Let’s keep this one about the technology, not whether HTML 5 video is better than Flash, etc. OK so here are my questions. Any help would be much appreciated.
- Is there a definitive reference for both the video tag and the JS API used to communicate with it?
- My assumption is that each browser provides a default set of controls. Can these be modified or do you need to start from scratch?
- How do you do true streaming with the video tag (i.e. not progressive download)?
- Is it possible to provide some form of DRM protection for video in HTML 5?
- Is there a tool that is optimal for encoding Ogg videos for Firefox?
- I remember reading that you can seek to any part of a video even if it hasn’t been downloaded yet. Is that true and if so, how is that achieved?
- Is there no hope for IE ever supporting the video tag?
Thanks in advance for the respectful commenting 
Lee
Those are good questions. I don’t have any answers for you but Ryan Christensen posted this this on twitter today to a very slick looking HTML 5 player. A very large issue I see with HTML 5 is the inability to actually do true full screen. It may not be a deal breaker for most but one huge advantage Flash has in the video department is delivering a full, seamless, take over video experience.
The points you raise above are also incredibly valid as well, especially the IE one. I have a feeling that IE will eventually have to give in. I’ll be watching this post to find out what you hear back. I would love to see a blending of Flash into HTML 5 somehow. I realize that this isn’t possible but there may be a crossover somewhere that can make both parties happy. Like a full featured Flash Video player that degrades into a limited but usable HTML 5 one. A lot of the work I have been doing with my CSS engine and Decal Sheets (Sprite Sheets) would allow Flash and HTML5 to use the same assets and skin. I would love to help on a project like that and really think that is the future of the web.
I got this site from twitter-sphere, Might hold some answers.
http://camendesign.com/code/video_for_everybody
@Martin great site. Thanks dude!
The thing that I’m having a hard time finding is the full JavaScript reference. Found some sample code on the Apple website but not a full reference.
One thing I noticed when checking out these two examples were some quality issues. The video seem to be blocky(this is most likley the compression), not very good aliasing(http://jilion.com/sublime/video), and some jitters at the bottom(http://jilion.com/sublime/video).
@Lee
While you are looking into the details of HTML 5 Video could you also look into the performance and quality options.
I am really exited to see more option for pushing video to the web.
I got a “browser not supported” on the first site.
The second site works great.
The video quality isn’t close to what I need.
This may help a little bit if you have not already got it:-)
http://www.youtube.com/watch?v=siOHh0uzcuY
Great idea. Checked some links and so far am not satisfied… Firefox eat up almost 300Mb watching http://jilion.com/sublime/video But I guess those are early days.
Howdy Lee, I don’t know, but here’s what I’ve seen, some leads:
1. Is there a definitive reference for both the video tag and the JS API used to communicate with it?
The WhatWG document describes some things, the W3C document based upon that is changing too fast for me to follow, and then there are the actual vendor implementations that content developers will eventually have to deal with. Best current docs may be at WhatWG spec, or one of the intro articles from Mark Pilgrim of Google at diveintomark.org.
2. My assumption is that each browser provides a default set of controls. Can these be modified or do you need to start from scratch?
I haven’t seen much conversation from the browser vendors on default UI, as in Windows Media Player or Real Player or QuickTime Player. Most of the discussion has been at the “chromeless video” level, where SWF is integrated with the page, and the content provider makes their own interface to stream down or cache with each viewing.
3. How do you do true streaming with the video tag (i.e. not progressive download)?
I’d defer to each individual browser vendor on this about what they’ve done, and what they say they’ll do. The main emphasis I’ve seen in conversation has been about a YouTube-style site. I haven’t seen much conversation yet about serverside video capability.
4. Is it possible to provide some form of DRM protection for video in HTML 5?
Rights-management techniques vary, and an all-HTML site could use tactics like membership requirements. I don’t recall seeing discussion of metadata-based tactics or ongoing server-based authentication or other techniques. A browser vendor could conceivably add their own rights-management techniques for use on their own video marketplaces. The topic of rights-management and creators’ rights is still very volatile in these communities.
5. Is there a tool that is optimal for encoding Ogg videos for Firefox?
Chris Blizzard of Mozilla has been tracking this… I’m not sure of current recommendations, but bet it’s in here:
http://www.0xdeadbeef.com/weblog/
6. I remember reading that you can seek to any part of a video even if it hasn’t been downloaded yet. Is that true and if so, how is that achieved?
I remember that being from the Vimeo materials…? I don’t have details, sorry, but would also like to know the mechanism.
7. Is there no hope for IE ever supporting the video tag?
Never say never.
It’s conceivable future Internet Explorer could handle VIDEO/H.264 by rewriting the tag Greasemonkeyishly and calling up their existing decoder. For VIDEO/Theora, it may depend on how their lawyers perceive “submarine patent” risk. A video site could conceivably content-sniff and serve alternate HTML versions to different browsers. And Google’s SVGWeb already uses Flash to provide SVG capability to IE. Could be a couple of ways to go. Conceivably. 
jd/adobe
@JD thanks man!
Here’s the link the the WHATWG spec on the video tag:
http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html
Of course, as we’ve seen in the past just because it is in the spec doesn’t mean that all browser vendors will implement a tag that way.
For DRM in HTML5, it’s my understanding that all the browsers that support the video tag, give user’s the option to save the source of the video, by right-clicking on the video element. I seriously doubt that we will ever see DRM implemented in the video tag, because of the different views on the matter from browser vendors.
Which is why I don’t think we will ever see a HTML5 version of Hulu or any other video website that wants to protect their content.
The html5 video link in the first post put my dual G5 Mac CPU to about 130% processing power (after a long delay to start). I find this CPU use is about the same as the average HQ (non-HD) YouTube video. Not very acceptable, IMO.
Interestingly, I am currently experimenting with Google’s Chromeless YouTube API, and found that when building my own player interface, a video which uses 130% CPU at YouTube.com drops close to around only 40% when called to my built-from-scratch Flash player.
PS. I learned to build the flash player calling the Google Chromeless AS3 player I mention above largely from your GoToAndLearn videos. Thanks for the great repository of info, Lee!
Thanks everyone. But I’d like to prevent this from being about performance, CPU, etc. Just looking for the best info on HTML 5 video. Thanks
I think I’ve seen adaptive http streaming as the proposed way forward for the html5 video.
http://iphone.akamai.com/
http://arstechnica.com/web/news/2009/07/apple-proposes-http-streaming-feature-as-a-protocol-standard.ars
Possible answer for your question #4:
Chris’ Ars link above leads to Apple’s IETF HTTP Live Streaming proposal. Section 3.1.3. EXT-X-KEY covers AES-128 as optional encryption for media files. There’s more info in section 5. “Key files”. Section 6.2.5. “Playing encrypted media files” is also of interest as is 7.3. “Playlist file with encrypted media files.”
Lee, you are my hero. =D
iBrent
Regarding question 3, a media resource is identified by URL. There’s no theoretical reason why that URL couldn’t be an rtsp://, mms:// or rtmp:// URL, or even a multicast session descriptor. In practice, I don’t think any browser supports something like that yet. Still, that’s how you would do “true streaming” given browser support.
Regarding question 4, the answer is mostly the same: it depends entirely on browser support. If you find a browser that supports fooble://-URLs as a media source, and fooble happens to be your hyper-encrypted extra-fancy protected protocol, then you can do DRM. The HTML5 standard will be agnostic about this.
When support for webfonts was introduced into Firefox recently, Mozilla agreed to enforce a same-origin policy for font files, obey HTTP access control headers for font files, and support a new obfuscated font format (WOFF) which has the sole purpose of making the fonts more difficult to use with other software. Obviously, there is no technical reason to do any of this: they did it purely because font vendors begged them too. The situation is similar for other browsers.
This suggests to me that browser vendors would probably be open to supporting some DRM for media elements, if content owners lobby them enough. (Which is unfortunate. I’d much rather they tell DRM pushers to GTFO because DRM is idiotic and nasty.)
As for question 6, the underlying protocol (HTTP, RTSP, …) needs to support requesting specific parts of the video. This is true for real streaming protocols, they all have some way of telling the server “give me the video starting at 23 seconds” or something like that.
With basic HTTP, you can only ask the server for a byte offset into the video file (instead of a time offset), but that doesn’t make seeking impossible, only more annoying to implement. Doesn’t Flash support seeking in MP4 over HTTP? A browser would have to do it the exact same way.
@iBrent oh no that might not be a good idea
@Mark thanks so much for that detailed info!
@lee, here is a good reference I found some time ago outlining the current drawbacks to implementing a pure HTML5 video experience:
http://diveintohtml5.org/video.html
We’re still in the early phases of HTML5 adoption, but with competition being as strong as it is in the browser market these days I fully expect HTML5 to be fairly well supported in IE9; making a lot of sites interested in offering a non-flash solution. However, your points are quite valid. I can see sites wanting to protect their content utilizing Flash as their delivery model at least for the next five years; and by then who knows what crazy things will be made reality by Adobe?
My hope is now that we’re well past the Macromedia/Adobe switch over the Flash API settles down a bit; and starting in CS5 we see a clear path through the settling dust as to what direction Adobe plans to take this app.
Lee,
We’ve had it working in IE6+, Firefox, Chrome, Safari for awhile (using H.264). Our source code is open (MIT-based license) and we’ve said that it shouldn’t be hard to do the exact same thing with Flash (we used Silverlight; Nishant has the Flash experience, but we have more expertise in SL here).
http://visitmix.com/Opinions/HTML5-Video-Youll-Shoot-Your-Eye-Out
A Flash version would be really cool, and I’d be happy to help you get it working. Pretty sure you have my e-mail; if not, you can contact us through the “contact us” link.
We implemented straight from the draft HTML5 spec, which is pretty straightforward. The JS API is defined there, and things like controls, etc. as well. We added some extra functionality and controls, and made it so people can skin the player with an external XAML file, but it’s basically just proving the concept.
Oh, and Mark Pilgrim has step-by-step on Theora, which is similar to what we used when doing Theora (though, as I mentioned, we’re doing H.264 only now).
http://diveintohtml5.org/video.html
And one last thing. It would be unwise to assume that IE9 will do things anything at all like what we did — whether it be our use of Silverlight, our support for H.264, or the use of doc rewriting to enable cross-browser. Our project is a stopgap solution which can be used for browsers that don’t support H.264 HTML5 (like firefox or IE7+8). The IE9 team are smart folks and will talk more about *their* plans sometime in the future.
Having said that, if you’re doing stopgap Flash-based player for HTML5 H.264, you’ll probably want to use our architecture.
Is there no hope for IE ever supporting the video tag?
ANSWER:
There is NOT no hope. Microsoft say they may support html5 video tag:
http://arstechnica.com/microsoft/news/2009/09/ie-program-manager-endorses-html-5-multimedia-tags.ars
Here are my suggestions for Flash to improve itself so it can not lose out to the html5 video tag.
1)Read the following criticisms about Flash and fix the problems and add the features that Flash doesn’t currently have:
http://www.smashingmagazine.com/2009/05/09/flash-vs-silverlight-what-suits-your-needs-best/
2)Include Flash in the next version of Internet explorer. Currently IE versions from 7 and up don’t have flash included in the installer because i think of some royalty issues. To make people happy, allow Flash to be uninstalled from Add/remove programs without affecting Internet explorer, and ask Microsoft to include an option in the next version of Internet explorer that lets users disable/enable flash from their browser settings(it should be ENABLED by default). If Flash was included in IE lots more people will get used to using it and when they go to html5 video tag websites and see that flash is better, they will probably email website owners and get them to support Flash. Better yet, you should Open Source Flash so other browsers like Chrome,Firefox, etc would probably include Flash in their installers.
3)Flash needs to use much less CPU resources. You guys should get a video file, play it in Windows Media player, then play it in Flash, and you will see Flash uses way more CPU power than windows media player.
4)Develop some sort of compression technology where the player can compress a 100MB .flv file into 10MB using very little CPU power and without losing any quality. Videos will load faster and good for slow broadband connections.
5)Create a new Flash version of .flv that supports 3D video. So people can play videos in 3D on their computer’s crt/lcd/oled/plasma screen. It should work WITHOUT the person needing to wear 3D glasses. It should work with all video cards.
Can you pass these suggestions onto the Flash team?
There are many great answers to the questions you have posted that helped me too (thanks guys – recently I thought of building an HTML5 video player, but I postponed it because of the codecs – I don’t want to encode my videos twice).
Here’s a few words for what I know about the questions Lee posted:
1. Is there a definitive reference for both the video tag and the JS API used to communicate with it?
Since Mozilla is the one truly developing JavaScript, I would use this refference: https://developer.mozilla.org/En/Using_audio_and_video_in_Firefox – it may not be the same for all other browsers but I believe it would help.
2. My assumption is that each browser provides a default set of controls. Can these be modified or do you need to start from scratch?
You start from scratch.
3. How do you do true streaming with the video tag (i.e. not progressive download)?
Probably what Maik posted is the only way – I would have said “no” if I hadn’t read his comment.
4. Is it possible to provide some form of DRM protection for video in HTML 5?
No – at least for now (probably some hacky way in the future).
5. Is there a tool that is optimal for encoding Ogg videos for Firefox?
Try http://firefogg.org/ (open it with Firefox). It’s an easy-to-use way for encoding. I believe it uses the ffmpeg2theora encoder: http://v2v.cc/~j/ffmpeg2theora/download.html
6. I remember reading that you can seek to any part of a video even if it hasn’t been downloaded yet. Is that true and if so, how is that achieved?
I think that what has been meant in what you read is that when you seek somewhere in the video, the player starts buffering from that place – that’s all.
7. Is there no hope for IE ever supporting the video tag?
.
We could wait for IE9 and see if it at least tries to do so
You asked wether Microsoft would support HTML5 or not. Looks like they do support it, see: http://news.cnet.com/8301-1001_3-10305822-92.html
@Maik
“With basic HTTP, you can only ask the server for a byte offset into the video file (instead of a time offset), but that doesn’t make seeking impossible, only more annoying to implement. Doesn’t Flash support seeking in MP4 over HTTP? A browser would have to do it the exact same way.”
Unfortunately flash doesn’t support seeking in MP4 over HTTP. I dont see why it doesnt, from what I understand, once you’ve downloaded the MP4 header you have all the keyframes with time and byte offset which would enable you to seek the video file.
There are solutions to the problem out there, eg:
Maybe this will be possible in 10.1? I hope so! *crosses fingers*
http://h264.code-shop.com/trac/wiki/FlashPlayer
This will make the flash player request a new video file when seeking, and the webserver will dynamically generate a new MP4-header with the video sliced from the time offset requested. Unfortunately the size of the header is not insignificant so there will always be some buffering when seeking – especially for longer video files (with more keyframe data). Sorry lee, but I think it’s really sad that Adobe didnt bother to implement this the proper way
Thanks again for all the great info. One thing that I can’t seem to find out about is the default controls that popup in Firefox. Is this JS and if so is there a way to get at the code?
DRM; Flash have some protection but there’s so many tools to download videos these days. When I want to download a video, I don’t use any plugin at all. In Safari I just click in WINDOW/ACTIVITY and give a double click in the video file (can be seen easily).
Other websites can make it more difficult but if the user want to download protected content like Hulu videos, he’ll simply capture the screen and encode later. It is more difficult and takes more time. Bottom line is; DRM won’t protect your content.
Thanks again for all the great info
@lee
I recently have been working at creating an HTML5 version of my Flash FLV Player product, and was also recently looking at how to get at the default player controls.
The best way to accomplish this, as far as I can tell so far, is to create your own controls using the new HTML5 markup and custom css/javascript. Then you simply turn off the original controls and skin the object with your own interface.
The only thing that I have not exactly found how to do yet is turn off the right click menu for the …or to edit it. Currently, users have the ability to turn the default controls on/off using the right click context menu.
Another note, I’m also guessing that each browser could/will have it’s own default controls, unless if one becomes a standard.
The video controls are implemented in XUL and Javascript, so you couldn’t just drop their code into a webpage and expect it to work. The main code is in ./toolkit/content/widgets/videocontrols.xml in the Firefox source. Browse it online at http://hg.mozilla.org/mozilla-central/file/tip/toolkit/content/widgets/videocontrols.xml.
Hi Lee,
About encoding Ogg Theora videos for Firefox, I think the VLC Media player can do the trick! Web sites like Inkscape Screencasters (http://screencasters.heathenx.org/) are using Ogg Theora videos for their tutorials.
5. Ogg is not a video or audio format. It’s a suite of file formats, container formats, that can host a whole bunch of different video and audio formats, not just “Theora” video and “Orbis” audio which I believe are the data formats you’re implying.
You can put MPEG-4 AVC (aka H.264) video in Ogg. You can put MPEG-4 ASP (aka “xvid/divx”) video in Ogg. You can put MPEG-2 (aka the stuff used on DVDs) video in Ogg. The list goes on.
there is no optimal tool for encoding the theora video format (which i guess is what you mean when you erroneously say “ogg video”), as there is only one code base for creating theora video. the problem is that the theora video format along with the theora encoder, are the worst possible options for video when it come to quality of the product – in this aspect theora is worse than everything else available. yes, i know it’s free and open, and i know this matters, but so does the resulting product, and here theora falls behind everything else available.
keep in mind that you don’t need to pay any licensing fees for providing h.264 video and aac audio over the net as long as you’re doing it for free, in an entirely uncommercial context.
Most of these seem to have been answered, but just in case my answers below (I was one of the developers of support in Firefox):
> 1. Is there a definitive reference for both the video tag and the JS API used to communicate with it?
The WHATWG spec is what I based development from. It has the JavaScript API and the reference for the tag:
http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#video
2. My assumption is that each browser provides a default set of controls. Can these be modified or do you need to start from scratch?
You need to start from scratch. The source for the Firefox controls is in a file called videocontrols.xml. There are other players out there that people have written though that are probably easier to follow.
3. How do you do true streaming with the video tag (i.e. not progressive download)?
What is ‘true streaming’ vs ‘progressive download’? For Ogg playback we use HTTP to retrieve the file and use byte range requests for seeking.
4. Is it possible to provide some form of DRM protection for video in HTML 5?
Not in the spec, no.
5. Is there a tool that is optimal for encoding Ogg videos for Firefox?
ffmpeg2theora is the tool I used when I was operating tinyvid.tv (a video site using the video element)
6. I remember reading that you can seek to any part of a video even if it hasn’t been downloaded yet. Is that true and if so, how is that achieved?
The server hosting the Ogg file must support byte range requests. See Chris Pearce’s blog for more details:
http://pearce.org.nz/2009/08/configuring-web-servers-for-html5-ogg.html
7. Is there no hope for IE ever supporting the video tag?
There is hope in that they haven’t said they won’t. And they have discussed the element on the W3C list.
For ogg conversion I use ffmpeg2theora – never tried the Mac/ Win versions though: http://v2v.cc/~j/ffmpeg2theora/
I posted my reply/thoughts on HTML 5 and Flash here:
http://richardleggett.co.uk/blog/index.php/2010/02/01/the-world-is-moving-to-html5
Mark Pilgrim’s draft of his upcoming book may be a good starting point on some of these questions:
http://diveintohtml5.org/
> 1. Is there a definitive reference for both the video tag and the JS API used to communicate with it?
Yes, the spec.
> 2. My assumption is that each browser provides a default set of controls. Can these be modified or do you need to start from scratch?
The latter.
> 3. How do you do true streaming with the video tag (i.e. not progressive download)?
If your server supports streaming video then the browser should be able to play it. I haven’t looked into this much so I don’t know the details.
> 4. Is it possible to provide some form of DRM protection for video in HTML 5?
To the same extent as with images.
> 6. I remember reading that you can seek to any part of a video even if it hasn’t been downloaded yet. Is that true and if so, how is that achieved?
With HTTP byte range requests. The server needs to support this for it to work.
hey lee,
here is some relevant info on fullscreen playback
http://ajaxian.com/archives/fullscreen-api-coming-to-browsers-near-you
Since there is a standard for choice of codec to go along with the HTML5 video tag standard, I think Adobe should jump in and yet again provide ubiquity, by adding flv support to every browser that supports HTML5 video (assuming there is a way to add a codec to the browser through a plugin). Why not?
@Kevin well Flash Player supports playing H.264 video so we actually are already doing that. FLV adds advanced stuff like cue points and alpha channels which can not be done in HTML 5.
The video element creates a normal CSS box, that can be styled with the whole set of CSS transformations, including rgba color w/ transparency. See eg this demo: http://people.mozilla.com/~prouget/demos/round/index.xhtml
If you’re referring to embedding an alpha channel in the video stream itself, this is currently not implemented, but not at all prohibited by the HTML5 specs. It’s also quite pointless for simple video players, which is where HTML5 Video is supposed to replace Flash, not RIA.
I think Flash enthusiasts are missing the point when they’re whining about all the things HTML5 Video cannot do, even when they’re not lying and find things HTML5 actually cannot do. People hate Flash-based video players because they can do all these things. The infrastructure needed to support them (color space conversions, indirect rendering, two full virtual machines, …) is what makes Flash so slow and inconvenient when all you want to do is simply play a video.
Lee, I actually just found this site today for encoding to Ogg format for FF and it worked in my testing. I have been playing around with HTML 5 and I had been stumped about the OGG format until I found this. Hope this helps you, or one of your readers.
http://www.online-convert.com/
This site has an app that seems to convert most formats to .ogg
http://www.erightsoft.com/SUPER.html
Fast conversion VOB -> AVI I then edit the AVI
and convert again AVI -> OGG
HTML 5 and OGG on a disk with Google Chrome portable is much smaller than DVD
Early days but works on a DVD RW
Answer to 7: IE9 will have HTML5, so they’ll have the video tag!
Thanks for this contribution, i think i will use and quote you in my next presentation !