Java movie playback: JOGL + Fobs4JMF
Recently I had to integrate video playback on my job’s Java OpenGL engine, which uses JOGL.
Java has a support to media playback through it’s Java Media Framework, which unfortunately, on it’s current version (2.1.1e) does not support many formats for video playback.
So I quickly looked for alternatives, including IBM Toolkit for mpeg4, that hadn’t a sufficient production performance I was looking for, and didn’t offer an easy option for frame grabbing or plugin extensions as JMF does.
Next was Fobs4JMF, which is JMF + ffmpeg. This solution was much more interesting, since it offers a wide variety of codecs (ogg, mp3, m4a, divx, xvid, h264, mov, etc) and is based on the solid ffmpeg solution to decode audio and video.
My implementation, uses the plug-in capabilities of JMF to extend a custom renderer that does a pixel type conversion and rendering to a texture:
This custom renderer works with RGB textures, a type I seemed to made work on my two test machines:
- MacBook with a Integ GMA x3100 – Leopard;
- PC with a Radeon x600 – Debian.
You might wanna try different pixel types to increase the performance on different target machines.
First, lets describe how the Renderer works:
It got to be an implementation of a javax.media.renderer.VideoRenderer since it will be installed as a plugin on JMF.
For the different methods we need to implement, there are a few we need to take proper care of:
- process: this is the method JMF calls passing the movie’s current frame buffer, here we process the buffer so that we can latter render it in OpenGL;
- getSupportedInputFormats: we return RGBFormat, our target texture format;
- setInputFormat: here we simply tell JMF that the format it chooses is the one we want. Since RGB was the only one we returned as supported, there is not much to do here as well.
- getName: returns the renderer neat name!
Next we need a way to access this renderer outside of the JMF world, so that we can get the texture to render it on the teapot. For this purpose our class must also be a javax.media.Control, then we can easily get it through an getControl call, such as:
player.getControl(“javax.media.renderer.VideoRenderer”);
So we implement:
- getControl: returns it’s instance;
- getControls: returns an array containing only it’s instance as a valid control.
The renderer implementation is org.pirelenito.multimedia.jmf.plugin.RGBGLTextureRenderer.
And also, to make further development easy, there is an IGLTextureRenderer interface with the public methods called by the Canvas:
- render: that plots the buffer on the texture surface;
- getTexture: to retrieve the texture instance.
Last but not least important, you will need to register the renderer on JMF, this is done through the JMFRegistry application. The easiest way to start it is inside Eclipse, where our custom renderer is already on the class path:
- Create a new run configuration;
- Set the main class as: JMFRegistry;
- Start it, and go to the Plugins tab, then Renderer;
- Add the org.pirelenito.multimedia.jmf.plugin.RGBGLTextureRenderer;
- Move it to the top of the list;
- Push Commit, and you are good to go!
To test it out there is also a helper class to instantiate the movie player, and a Main class which is an OpenGL canvas used to render the teapot with the texture on its surface.
I am not very experienced with OpenGL, so there might be more effitient ways to do this using, for instance, PBO (Pixel Buffer Object). If you have any question or suggestions on how to improve this solution, don’t hesitate on drooping a comment!
Here is the project’s repository. You will need to download a few dependencies, so check the Readme file for more information.
Update: I wasn’t properly initializing the Texture. and since I want to add support for more pixel formats, it is easy for you to checkout the updated Eclipse project at: http://labs.pirelenito.org/experiments/svn/java/MovieGL/ latter this afternoon now.
Update 2: Added the repository link on the image above.
Update 3: I’ve migrated my server to a new location, and the Subversion wasn’t up until today. Sorry.
Update 4: Small review on the post.
Update 5: Moved the repository to github.
Cheers! 
August 21st, 2008 at 6:02 am
Hey pal!
Interesting way to solve the problem =]
Using PBO would certainly increase performance, since you wouldn’t need to ‘draw frames to a texture’, but would be writing directly to the buffer. One thing I think (THINK) that would still make my knees shake is the conversion from the media data format to RGB texture format (that ‘for’ statement made me feel a huge fear… look at http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=35, the ‘for’ thing is also needed but was done through assembly code [REAL PROGRAMMERS CODE IN BINARY, aeouahaeo, or NOT!], don’t know if Java’s JIT is capable of increasing the ‘for’ perfomance), it (format conversion, if you got lost at the last ‘parenthesis’ statement.. hehe) is needed in both methods.
Congratulations! It’s a pretty code chunk!!!
August 21st, 2008 at 7:45 am
Thanks man!
That conversion part is something I want to avoid by using popper pixel types, that is why I didn’t look on doing it through JNI (:p), I hope I can get rid of it.
Check latter for more updates on the code… I am still messing with it!
August 21st, 2008 at 4:27 pm
Cool nito!
Keep posting good stuff
November 22nd, 2008 at 1:41 am
Hi, I’m trying to run your code but I get this error:
Something got wrong!
java.lang.ClassCastException: com.omnividea.media.renderer.video.Java2DRenderer cannot be cast to org.pirelenito.multimedia.jmf.plugin.IGLTextureRenderer
at org.pirelenito.multimedia.jmf.MoviePlayer.getRenderer(MoviePlayer.java:48)
at org.pirelenito.movieGL.Main.(Main.java:104)
at org.pirelenito.movieGL.Main.main(Main.java:74)
I don’t understand what the problem is…
November 27th, 2008 at 9:19 am
Hi lodestar,
The problem is that you haven’t registered the custom renderer I’ve created on JMF, so when you are executing the getRenderer method it gets the default JMF renderer (Java2DRenderer) that obviously cannot be cast to my custom IGLTextureRenderer interface.
To fix this, simply follow the steps above on the paragraph that contains “register the render on JMF using the JMFRegistry”
Drop me by if you have any more problems.
Thanks for passing by!
November 30th, 2008 at 6:25 am
Hi! Thanks for taking the time to write this, it’s been most helpful during my reading up on JOGL and video. A simple improvement I’d like to suggest is changing the conversion loop in the process method of your VideoRenderer to:
byte[] line = new byte[width * 3];
int srcN = 0;
for (int y = height; y > 0; y–) {
int dstN = 0;
for (int x = width; x > 0; x–) {
int rgb = conversionBuffer[srcN++];
line[dstN++] = (byte) (rgb >> 16);
line[dstN++] = (byte) (rgb >> 8);
line[dstN++] = (byte) (rgb >> 0);
}
byteBuffer.put(line);
}
I get go from 50% CPU usage to 30% with this.
November 30th, 2008 at 7:25 am
Thank YOU! I am glad I helped.
Already applied your modifications on the code.
thanks again!
December 8th, 2008 at 8:43 am
Yo, this has been very useful!
after getting it running however (with jmf/jogl/etc…).. I see the teapot (or whatever object I make glut cubes, etc..) and I hear the movie, yet the movie that plays is just a continuous set of solid color that changes… obviously it’s getting some kind of data from the video, but it’s rendering solid colors on the teapot rather than all the different pixels
why might this be?
December 17th, 2008 at 12:48 pm
Hey dudee, long time.. now, lets learn JavaFX and do the video on a cube on my Firefox
http://javafx.com/samples/
February 8th, 2009 at 10:52 am
thanks for the example.
I found that this example did not work with all the JMF compatible video files, whereas Sun’s Java3D equiverlent application did.
I found that by removing the VM argument
-Djava.library.path=lib/jogl
and loading the JOGL DLL directly
before running JMF, i.e
static void loadLibs() {
try {
String anAppPath =
System.getProperty(“user.dir”);
System.load(
anAppPath +
“\\lib\\jogl\\jogl.dll”);
} catch (Exception theE) {
}
}
resolved this problem;
strange – possibly a ClassPath or ClassLoader issue.
February 26th, 2009 at 4:21 am
Following on from my later entry and more experiments.
Don’t use -Djava.library.path=lib/jogl.
Don’t use loadLibs.
Do ensure that jogl’s DLL directory is located near the beginning of your system’s PATH.
March 5th, 2009 at 8:00 am
Hi Paolo,
very nice work. This was exactly what I was looking for.
But I have a (hopefully) minor problem: I apply video to a cube and no matter what kind of source I use it is always flipped vertically.
renderer.getTexture().getMustFlipVertically() always returns TRUE.
Do you have a hint on how to display videos not flipped vertically?
Thanks in advance,
Steff
March 9th, 2009 at 8:12 pm
I am glad I’ve helped you.
Regarding your issue, you can fix it in two different ways:
First, by changing the way you map your texture.
Or, you can change the MustFlipVertically value to false by going inside the renderer’s createTexture method and changing the call to create a new TextureData:
from this:
to:
Good luck!
June 15th, 2009 at 12:41 pm
Hi Paulo!
thanks for the codes! It is very useful.
I’m trying to implement your code into mine. So far it works when I only have one textured object.
If I have two or more, the textures overlaps so that it flicker.
I don’t understand how JOGL works
July 9th, 2009 at 2:46 pm
Hello Paulo,
Your work on a video texture renderer is very interseting.
I tried to download your code but it seems to be no more available…
Possible to get it ?
Thanks !
Franck
July 10th, 2009 at 8:42 am
Nice….
I posted a message yesterday to ask about the source code that are no available but yet referenced in your blog (http://labs.pirelenito.org/experiments/svn/java/MovieGL/).
Result: my post have been zapped without any message.
Waouuuhhhh: kind, elegant … words are missing.
As you said Paulo: “thanks for passing by”…
– Franck
July 10th, 2009 at 11:42 am
Sorry if you misunderstood that your comment didn’t show up on the website until today, but unfortunately due to spam I have to moderate the comments that are posted here, which I don’t do all the time.
I’ve seen your comments today and have fixed the link to the repository. I hope you find it helpful.
And I’ll say it again: Thanks for passing by
July 16th, 2009 at 5:03 am
Nito,
Sorry for this misunderstanding…
I thought you have zapped my first message.
Thanks again for your (kind) reply and for restoring your svn link.
I’ll try to port your work on JOGL+Gstreamer-java.
– Franck
October 26th, 2009 at 2:54 pm
hi there. im trying to work the lib with Processing. It works just fine with java, just not with processing.
here’s the error:
Fobs4JMF – Native shared library found
14.36First Position: 0, 0 Duration: 14360
Frame Rate: 25
Opening Thread[JMF thread: com.sun.media.PlaybackEngine@1e16483[ com.sun.media.PlaybackEngine@1e16483 ] ( configureThread),9,system]
Fobs Java2DRenderer: setInputFormat
Fobs Java2DRenderer: setInputFormat
java.lang.ClassCastException: com.omnividea.media.renderer.video.Java2DRenderer cannot be cast to org.pirelenito.multimedia.jmf.plugin.IGLTextureRenderer
Fobs Java2DRenderer: start
Exception in thread “Animation Thread” java.lang.NullPointerException
at MovieGL$CustomScene.Render(MovieGL.java:262)
at MovieGL.draw(MovieGL.java:192)
at processing.core.PApplet.handleDraw(PApplet.java:1425)
at processing.core.PApplet.run(PApplet.java:1327)
at java.lang.Thread.run(Thread.java:619)
any ideas?
October 27th, 2009 at 8:14 am
i have fixed it.. thanks
October 28th, 2009 at 6:18 pm
[...] my research i came across this website with a great example on using FOBS+JMF, exactly what i needed. So i downloaded it and started [...]
November 5th, 2009 at 3:26 pm
hey… do you have any idea on how to be able to get the first frame of the video and save it in the texture, so i can have videos that show on the screen but dont start playing until i click a button ?
i’ve been playing with the events but with no success, hopefully you know a way?
thanks
November 27th, 2009 at 6:38 pm
Sorry for the very delayed reply, but as you can see I am a little bit off this blog.
I guess you could use this: http://java.sun.com/javase/technologies/desktop/media/jmf/2.1.1/apidocs/javax/media/control/FrameGrabbingControl.html
I am sure by now you shold already have figured it out.
February 17th, 2010 at 10:43 am
Hello ,
I really interesting about your code ,Actually I would like to do something like this but with some differentiate . I would like to project video on ball and camera view be looking inside of ball , something like you are in 3d cinema and looking to curve screen .
So i would like to know can I have your source code as reference to work on it and modify to what i need .
Please let me know .
Thanks.
February 17th, 2010 at 11:10 am
Hi, first thanks for the interest in the code, I am really glad it can be of good to someone.
And sure you can have the code, I have a reference to a source repository on the post, that you can use to download it.
I would appreciate if you reference me and maintain my name of the source files you are about to modify.
Also it wold also be cool if you come back here latter and post your improvements/changes.
Thanks, and good luck with your project!
February 17th, 2010 at 11:28 am
Thanks you so much , I did download your source code , first i have to understand your code because i am not so strong on programing but i would love to make something new as you did .Also if i finish that i will post here to your blog .
In case have some difficulty to find my project i will take your advice .
With best regards
April 10th, 2010 at 1:22 am
I am having trouble getting a video file to load in. I’ve tried avi, mov, ogg, and mp4 video formats.
For the avi file, the error is
Unable to handle format: CRAM, 320×240, FrameRate=15.0, Length=96002 0 extra bytes
Failed to realize: com.sun.media.PlaybackEngine@2bbd86
Error: Unable to realize com.sun.media.PlaybackEngine@2bbd86
Something got wrong!
javax.media.CannotRealizeException
at javax.media.Manager.blockingCall(Manager.java:2005)
at javax.media.Manager.createRealizedPlayer(Manager.java:528)
at org.pirelenito.multimedia.jmf.MoviePlayer.(MoviePlayer.java:47)
at org.pirelenito.movieGL.Main.(Main.java:117)
at org.pirelenito.movieGL.Main.main(Main.java:108)
The .mov error message is similar. Any guesses on what I’m doing wrong? Is there an example video that the code is known to work on? Thanks!
April 10th, 2010 at 11:58 pm
Hi Robert,
Regarding your problem, fobs4jmf is a wrapper on top of ffmpeg and it supports a wide range of formats:
http://en.wikipedia.org/wiki/Libavcodec
So the problem might be on the configuration of your environment. Have you follow the steps on the README file to download all the dependencies?
http://labs.pirelenito.org/experiments/svn/java/MovieGL/README.txt
I googled for a sample MPEG1 file and found this website:
http://www.jhepple.com/support/sample_movies1.htm
If you still having trouble, don’t hesitate on dropping another comment.
Good luck.