| Subcribe via RSS

Java movie playback: JOGL + Fobs4JMF

August 20th, 2008 Posted in Java

Recently I had to integrate video playback on my job’s Java OpenGL engine, which uses JOGL.

Java has a support to media playback through it’s Java Media Framework, which unfortunately, on it’s current version (2.1.1e) does not support many formats for video playback.

So I quickly looked for alternatives, including IBM Toolkit for mpeg4, that hadn’t a sufficient production performance I was looking for, and didn’t offer an easy option for frame grabbing or plugin extensions as JMF does.

Next was Fobs4JMF, which is JMF + ffmpeg. This solution was much more interesting, since it offers a wide variety of codecs (ogg, mp3, m4a, divx, xvid, h264, mov, etc) and is based on the solid ffmpeg solution to decode audio and video.

My implementation, uses the plug-in capabilities of JMF to extend a custom renderer that does a pixel type conversion and rendering to a texture:

This custom renderer works with RGB textures, a type I seemed to made work on my two test machines:

  • MacBook with a Integ GMA x3100 – Leopard;
  • PC with a Radeon x600 – Debian.

You might wanna try different pixel types to increase the performance on different target machines.

First, lets describe how the Renderer works:

It got to be an implementation of a javax.media.renderer.VideoRenderer since it will be installed as a plugin on JMF.

For the different methods we need to implement, there are a few we need to take proper care of:

  • process: this is the method JMF calls passing the movie’s current frame buffer, here we process the buffer so that we can latter render it in OpenGL;
  • getSupportedInputFormats: we return RGBFormat, our target texture format;
  • setInputFormat: here we simply tell JMF that the format it chooses is the one we want. Since RGB was the only one we returned as supported, there is not much to do here as well.
  • getName: returns the renderer neat name!

Next we need a way to access this renderer outside of the JMF world, so that we can get the texture to render it on the teapot. For this purpose our class must also be a javax.media.Control, then we can easily get it through an getControl call, such as:

player.getControl(“javax.media.renderer.VideoRenderer”);

So we implement:

  • getControl: returns it’s instance;
  • getControls: returns an array containing only it’s instance as a valid control.

The renderer implementation is org.pirelenito.multimedia.jmf.plugin.RGBGLTextureRenderer.

And also, to make further development easy, there is an IGLTextureRenderer interface with the public methods called by the Canvas:

  • render: that plots the buffer on the texture surface;
  • getTexture: to retrieve the texture instance.

Last but not least important, you will need to register the renderer on JMF, this is done through the JMFRegistry application. The easiest way to start it is inside Eclipse, where our custom renderer is already on the class path:

  • Create a new run configuration;
  • Set the main class as: JMFRegistry;
  • Start it, and go to the Plugins tab, then Renderer;
  • Add the org.pirelenito.multimedia.jmf.plugin.RGBGLTextureRenderer;
  • Move it to the top of the list;
  • Push Commit, and you are good to go!

To test it out there is also a helper class to instantiate the movie player, and a Main class which is an OpenGL canvas used to render the teapot with the texture on its surface.

I am not very experienced with OpenGL, so there might be more effitient ways to do this using, for instance, PBO (Pixel Buffer Object). If you have any question or suggestions on how to improve this solution, don’t hesitate on drooping a comment!

Here is the project’s repository. You will need to download a few dependencies, so check the Readme file for more information.

Update: I wasn’t properly initializing the Texture. and since I want to add support for more pixel formats, it is easy for you to checkout the updated Eclipse project at: http://labs.pirelenito.org/experiments/svn/java/MovieGL/ latter this afternoon now.

Update 2: Added the repository link on the image above.

Update 3: I’ve migrated my server to a new location, and the Subversion wasn’t up until today. Sorry.

Update 4: Small review on the post.

Update 5: Moved the repository to github.

Cheers! ;)

Tags: , , , , , , , , , , ,
This entry was posted on Wednesday, August 20th, 2008 at 4:37 pm and is filed under Java. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

28 Responses to “Java movie playback: JOGL + Fobs4JMF”

  1. tinsukE Says:

    August 21st, 2008 at 6:02 am

    Hey pal!

    Interesting way to solve the problem =]

    Using PBO would certainly increase performance, since you wouldn’t need to ‘draw frames to a texture’, but would be writing directly to the buffer. One thing I think (THINK) that would still make my knees shake is the conversion from the media data format to RGB texture format (that ‘for’ statement made me feel a huge fear… look at http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=35, the ‘for’ thing is also needed but was done through assembly code [REAL PROGRAMMERS CODE IN BINARY, aeouahaeo, or NOT!], don’t know if Java’s JIT is capable of increasing the ‘for’ perfomance), it (format conversion, if you got lost at the last ‘parenthesis’ statement.. hehe) is needed in both methods.

    Congratulations! It’s a pretty code chunk!!!


  2. admin Says:

    August 21st, 2008 at 7:45 am

    Thanks man!

    That conversion part is something I want to avoid by using popper pixel types, that is why I didn’t look on doing it through JNI (:p), I hope I can get rid of it.

    Check latter for more updates on the code… I am still messing with it!


  3. Jardel Weyrich Says:

    August 21st, 2008 at 4:27 pm

    Cool nito!
    Keep posting good stuff :)


  4. lodestar Says:

    November 22nd, 2008 at 1:41 am

    Hi, I’m trying to run your code but I get this error:

    Something got wrong!
    java.lang.ClassCastException: com.omnividea.media.renderer.video.Java2DRenderer cannot be cast to org.pirelenito.multimedia.jmf.plugin.IGLTextureRenderer
    at org.pirelenito.multimedia.jmf.MoviePlayer.getRenderer(MoviePlayer.java:48)
    at org.pirelenito.movieGL.Main.(Main.java:104)
    at org.pirelenito.movieGL.Main.main(Main.java:74)

    I don’t understand what the problem is…


  5. admin Says:

    November 27th, 2008 at 9:19 am

    Hi lodestar,

    The problem is that you haven’t registered the custom renderer I’ve created on JMF, so when you are executing the getRenderer method it gets the default JMF renderer (Java2DRenderer) that obviously cannot be cast to my custom IGLTextureRenderer interface.

    To fix this, simply follow the steps above on the paragraph that contains “register the render on JMF using the JMFRegistry”

    Drop me by if you have any more problems.

    Thanks for passing by!


  6. CW Says:

    November 30th, 2008 at 6:25 am

    Hi! Thanks for taking the time to write this, it’s been most helpful during my reading up on JOGL and video. A simple improvement I’d like to suggest is changing the conversion loop in the process method of your VideoRenderer to:

    byte[] line = new byte[width * 3];
    int srcN = 0;
    for (int y = height; y > 0; y–) {
    int dstN = 0;
    for (int x = width; x > 0; x–) {
    int rgb = conversionBuffer[srcN++];
    line[dstN++] = (byte) (rgb >> 16);
    line[dstN++] = (byte) (rgb >> 8);
    line[dstN++] = (byte) (rgb >> 0);
    }
    byteBuffer.put(line);
    }

    I get go from 50% CPU usage to 30% with this.


  7. admin Says:

    November 30th, 2008 at 7:25 am

    Thank YOU! I am glad I helped.

    Already applied your modifications on the code.

    thanks again!


  8. Aaron Beach Says:

    December 8th, 2008 at 8:43 am

    Yo, this has been very useful!

    after getting it running however (with jmf/jogl/etc…).. I see the teapot (or whatever object I make glut cubes, etc..) and I hear the movie, yet the movie that plays is just a continuous set of solid color that changes… obviously it’s getting some kind of data from the video, but it’s rendering solid colors on the teapot rather than all the different pixels :) why might this be?


  9. Soro Says:

    December 17th, 2008 at 12:48 pm

    Hey dudee, long time.. now, lets learn JavaFX and do the video on a cube on my Firefox :D
    http://javafx.com/samples/


  10. malcolm binstead Says:

    February 8th, 2009 at 10:52 am

    thanks for the example.

    I found that this example did not work with all the JMF compatible video files, whereas Sun’s Java3D equiverlent application did.

    I found that by removing the VM argument

    -Djava.library.path=lib/jogl

    and loading the JOGL DLL directly
    before running JMF, i.e

    static void loadLibs() {
    try {
    String anAppPath =
    System.getProperty(“user.dir”);
    System.load(
    anAppPath +
    “\\lib\\jogl\\jogl.dll”);
    } catch (Exception theE) {
    }
    }

    resolved this problem;
    strange – possibly a ClassPath or ClassLoader issue.


  11. malcolm binstead Says:

    February 26th, 2009 at 4:21 am

    Following on from my later entry and more experiments.
    Don’t use -Djava.library.path=lib/jogl.
    Don’t use loadLibs.
    Do ensure that jogl’s DLL directory is located near the beginning of your system’s PATH.


  12. Steff Says:

    March 5th, 2009 at 8:00 am

    Hi Paolo,

    very nice work. This was exactly what I was looking for.

    But I have a (hopefully) minor problem: I apply video to a cube and no matter what kind of source I use it is always flipped vertically.
    renderer.getTexture().getMustFlipVertically() always returns TRUE.
    Do you have a hint on how to display videos not flipped vertically?

    Thanks in advance,
    Steff


  13. admin Says:

    March 9th, 2009 at 8:12 pm

    I am glad I’ve helped you.

    Regarding your issue, you can fix it in two different ways:

    First, by changing the way you map your texture.

    Or, you can change the MustFlipVertically value to false by going inside the renderer’s createTexture method and changing the call to create a new TextureData:

    from this:

    Texture texture = TextureIO.newTexture(new TextureData (GL.GL_RGBA8,…, true, byteBuffer, null));

    to:

    Texture texture = TextureIO.newTexture(new TextureData (GL.GL_RGBA8,…, false, byteBuffer, null));

    Good luck!


  14. Xenoid Says:

    June 15th, 2009 at 12:41 pm

    Hi Paulo!
    thanks for the codes! It is very useful.

    I’m trying to implement your code into mine. So far it works when I only have one textured object.
    If I have two or more, the textures overlaps so that it flicker.

    I don’t understand how JOGL works :(


  15. fbommeau Says:

    July 9th, 2009 at 2:46 pm

    Hello Paulo,

    Your work on a video texture renderer is very interseting.
    I tried to download your code but it seems to be no more available…
    Possible to get it ?
    Thanks !

    Franck


  16. fbommeau Says:

    July 10th, 2009 at 8:42 am

    Nice….
    I posted a message yesterday to ask about the source code that are no available but yet referenced in your blog (http://labs.pirelenito.org/experiments/svn/java/MovieGL/).
    Result: my post have been zapped without any message.

    Waouuuhhhh: kind, elegant … words are missing.
    As you said Paulo: “thanks for passing by”…

    – Franck


  17. nito Says:

    July 10th, 2009 at 11:42 am

    Sorry if you misunderstood that your comment didn’t show up on the website until today, but unfortunately due to spam I have to moderate the comments that are posted here, which I don’t do all the time.

    I’ve seen your comments today and have fixed the link to the repository. I hope you find it helpful.

    And I’ll say it again: Thanks for passing by ;)


  18. fbommeau Says:

    July 16th, 2009 at 5:03 am

    Nito,
    Sorry for this misunderstanding…
    I thought you have zapped my first message.
    Thanks again for your (kind) reply and for restoring your svn link.
    I’ll try to port your work on JOGL+Gstreamer-java.

    – Franck


  19. vitor Says:

    October 26th, 2009 at 2:54 pm

    hi there. im trying to work the lib with Processing. It works just fine with java, just not with processing.

    here’s the error:

    Fobs4JMF – Native shared library found
    14.36First Position: 0, 0 Duration: 14360
    Frame Rate: 25
    Opening Thread[JMF thread: com.sun.media.PlaybackEngine@1e16483[ com.sun.media.PlaybackEngine@1e16483 ] ( configureThread),9,system]
    Fobs Java2DRenderer: setInputFormat
    Fobs Java2DRenderer: setInputFormat
    java.lang.ClassCastException: com.omnividea.media.renderer.video.Java2DRenderer cannot be cast to org.pirelenito.multimedia.jmf.plugin.IGLTextureRenderer
    Fobs Java2DRenderer: start

    Exception in thread “Animation Thread” java.lang.NullPointerException
    at MovieGL$CustomScene.Render(MovieGL.java:262)
    at MovieGL.draw(MovieGL.java:192)
    at processing.core.PApplet.handleDraw(PApplet.java:1425)
    at processing.core.PApplet.run(PApplet.java:1327)
    at java.lang.Thread.run(Thread.java:619)

    any ideas?


  20. vitor Says:

    October 27th, 2009 at 8:14 am

    i have fixed it.. thanks


  21. MovieGL – Rendering movies with JOGL in Java/Processing Says:

    October 28th, 2009 at 6:18 pm

    [...] my research i came across this website with a great example on using FOBS+JMF, exactly what i needed. So i  downloaded it and started [...]


  22. vitor Says:

    November 5th, 2009 at 3:26 pm

    hey… do you have any idea on how to be able to get the first frame of the video and save it in the texture, so i can have videos that show on the screen but dont start playing until i click a button ?

    i’ve been playing with the events but with no success, hopefully you know a way?

    thanks


  23. nito Says:

    November 27th, 2009 at 6:38 pm

    Sorry for the very delayed reply, but as you can see I am a little bit off this blog.

    I guess you could use this: http://java.sun.com/javase/technologies/desktop/media/jmf/2.1.1/apidocs/javax/media/control/FrameGrabbingControl.html

    I am sure by now you shold already have figured it out.


  24. Ali Says:

    February 17th, 2010 at 10:43 am

    Hello ,

    I really interesting about your code ,Actually I would like to do something like this but with some differentiate . I would like to project video on ball and camera view be looking inside of ball , something like you are in 3d cinema and looking to curve screen .

    So i would like to know can I have your source code as reference to work on it and modify to what i need .
    Please let me know .

    Thanks.


  25. nito Says:

    February 17th, 2010 at 11:10 am

    Hi, first thanks for the interest in the code, I am really glad it can be of good to someone.

    And sure you can have the code, I have a reference to a source repository on the post, that you can use to download it.

    I would appreciate if you reference me and maintain my name of the source files you are about to modify.

    Also it wold also be cool if you come back here latter and post your improvements/changes.

    Thanks, and good luck with your project!


  26. Ali Says:

    February 17th, 2010 at 11:28 am

    Thanks you so much , I did download your source code , first i have to understand your code because i am not so strong on programing but i would love to make something new as you did .Also if i finish that i will post here to your blog .

    In case have some difficulty to find my project i will take your advice .

    With best regards


  27. Robert Says:

    April 10th, 2010 at 1:22 am

    I am having trouble getting a video file to load in. I’ve tried avi, mov, ogg, and mp4 video formats.

    For the avi file, the error is

    Unable to handle format: CRAM, 320×240, FrameRate=15.0, Length=96002 0 extra bytes
    Failed to realize: com.sun.media.PlaybackEngine@2bbd86
    Error: Unable to realize com.sun.media.PlaybackEngine@2bbd86
    Something got wrong!
    javax.media.CannotRealizeException
    at javax.media.Manager.blockingCall(Manager.java:2005)
    at javax.media.Manager.createRealizedPlayer(Manager.java:528)
    at org.pirelenito.multimedia.jmf.MoviePlayer.(MoviePlayer.java:47)
    at org.pirelenito.movieGL.Main.(Main.java:117)
    at org.pirelenito.movieGL.Main.main(Main.java:108)

    The .mov error message is similar. Any guesses on what I’m doing wrong? Is there an example video that the code is known to work on? Thanks!


  28. nito Says:

    April 10th, 2010 at 11:58 pm

    Hi Robert,

    Regarding your problem, fobs4jmf is a wrapper on top of ffmpeg and it supports a wide range of formats:
    http://en.wikipedia.org/wiki/Libavcodec

    So the problem might be on the configuration of your environment. Have you follow the steps on the README file to download all the dependencies?
    http://labs.pirelenito.org/experiments/svn/java/MovieGL/README.txt

    I googled for a sample MPEG1 file and found this website:
    http://www.jhepple.com/support/sample_movies1.htm

    If you still having trouble, don’t hesitate on dropping another comment.

    Good luck.


Leave a Reply

  • about me

    I'm Paulo Ragonha, a brazilian hobbyist game developer, who enjoys playing with technology on my free time, my (current) main language is Java so you will probably see a lot of stuff about it in here, I also occasionally talk abut random stuff... and will probably post a "game" every once in a while.
    Thanks for passing by!


  • twitter