Can I use the Gstreamer API to merge 2 videos?

gstreamer, video, video-processing

Solution

It turns out gstreamer can merge two videos, placing them side by side into an output video using the videomixer filter.

A basic pipeline that takes two input files, scales them to be the same size, then merges them and encodes them into a theora video might look like this:

filesrc -> decodebin -> ffmpegcolourspace -> videoscale ->  videobox -> videorate
                                                                                  \
filesrc -> decodebin ->  ffmpegcolourspace  ->  videoscale  ->  videorate   ->    videomixer -> ffmpegcolourspace -> theoraenc -> oggmux -> filesink

How you implement this pipeline depends on the language. I prototyped with the Ruby bindings, and it works really well.

Problem

I'd like to write a simple linux CLI application that can take 2 video sources (1 of a presenter talking and 1 with their slides and no audio) and merge them. I'd like the entire output video to be the two original videos, side by side. Failing that, my second best option would be a "picture in picture" style video, with the presenter in a small frame in the corner. From a few hours research, GStreamer looks like it might be able to do this. Can anyone confirm it before I spend more time trying it out? If it can't, are there other APIs out there that I might be able to use?

Original source