Post 03: Creating a video with gstreamer and C++/Rust

In the previous post, we created an MKV video file containing H.264 encoded video using gst-launch-1.0.

While gst-launch-1.0 is an excellent tool for experimenting with pipelines and validating ideas, real-world applications use the GStreamer API directly. In this post, we will build the same pipeline using C++, taking our first step toward developing multimedia applications with GStreamer.

Prerequisites

If you have followed the previous posts in this series, the runtime packages should already be installed. Calling the API from our own program needs two more things on top of that.

The first is the development headers, which is what pkg-config reads when we compile. The second is the plugins our pipeline actually asks for: x264enc lives in plugins-ugly and h264parse lives in plugins-bad. Post 00 described both of those as optional, and for gst-launch-1.0 experiments they are, but this pipeline will not run without them. If an element is missing, gst_element_factory_make simply returns NULL and the program stops at Failed to create elements.

C++

sudo apt-get install -y gcc pkg-config curl \
    libgstreamer1.0-dev \
    libgstreamer-plugins-base1.0-dev \
    gstreamer1.0-plugins-bad \
    gstreamer1.0-plugins-ugly

sudo dnf install -y gcc pkgconf-pkg-config curl \
    gstreamer1-devel \
    gstreamer1-plugins-base-devel \
    gstreamer1-plugins-bad-free \
    gstreamer1-plugins-ugly

sudo pacman -S --needed gcc pkgconf curl \
    gstreamer \
    gst-plugins-base \
    gst-plugins-bad \
    gst-plugins-ugly

Rust

The Rust bindings link against the same GStreamer development packages listed above, so install those first.

sudo apt-get install -y rustc cargo

Project source code

mkdir gstreamer-cpp
cd gstreamer-cpp
code main.cpp

You can write the following in main.cpp

gstreamer/post03/cpp/main.cppView on GitHub

#include <gst/gst.h>

int main(int argc, char *argv[]) {
    gst_init(&argc, &argv);

    GstElement *pipeline = gst_pipeline_new("test-pipeline");

    GstElement *src = gst_element_factory_make("videotestsrc", "src");
    GstElement *capsfilter = gst_element_factory_make("capsfilter", "capsfilter");
    GstElement *enc = gst_element_factory_make("x264enc", "enc");
    GstElement *parse = gst_element_factory_make("h264parse", "parse");
    GstElement *mux = gst_element_factory_make("matroskamux", "mux");
    GstElement *sink = gst_element_factory_make("filesink", "sink");

    if (!pipeline || !src || !capsfilter || !enc || !parse || !mux || !sink) {
        g_printerr("Failed to create elements\n");
        return -1;
    }

    g_object_set(src, "num-buffers", 90, nullptr);
    g_object_set(sink, "location", "test.mkv", nullptr);

    GstCaps *caps = gst_caps_new_simple("video/x-raw",
        "width", G_TYPE_INT, 1280,
        "height", G_TYPE_INT, 720,
        "framerate", GST_TYPE_FRACTION, 30, 1,
        nullptr);
    g_object_set(capsfilter, "caps", caps, nullptr);
    gst_caps_unref(caps);

    gst_bin_add_many(GST_BIN(pipeline), src, capsfilter, enc, parse, mux, sink, nullptr);

    if (!gst_element_link_many(src, capsfilter, enc, parse, mux, sink, nullptr)) {
        g_printerr("Failed to link elements\n");
        gst_object_unref(pipeline);
        return -1;
    }

    GstStateChangeReturn ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);
    if (ret == GST_STATE_CHANGE_FAILURE) {
        g_printerr("Failed to set pipeline to PLAYING\n");
        gst_object_unref(pipeline);
        return -1;
    }

    GstBus *bus = gst_element_get_bus(pipeline);
    GstMessage *msg = gst_bus_timed_pop_filtered(bus, GST_CLOCK_TIME_NONE,
        (GstMessageType)(GST_MESSAGE_ERROR | GST_MESSAGE_EOS));

    if (msg != nullptr) {
        GError *err;
        gchar *debug_info;
        switch (GST_MESSAGE_TYPE(msg)) {
            case GST_MESSAGE_ERROR:
                gst_message_parse_error(msg, &err, &debug_info);
                g_printerr("Error: %s\n", err->message);
                g_clear_error(&err);
                g_free(debug_info);
                break;
            case GST_MESSAGE_EOS:
                g_print("End of stream\n");
                break;
            default:
                break;
        }
        gst_message_unref(msg);
    }

    gst_object_unref(bus);
    gst_element_set_state(pipeline, GST_STATE_NULL);
    gst_object_unref(pipeline);

    return 0;
}

Now you run the code using

g++ main.cpp -o gstreamer-cpp `pkg-config --cflags --libs gstreamer-1.0`
./gstreamer-cpp

cargo new gstreamer-rust
cd gstreamer-rust
cargo add gstreamer
code src/main.rs

You can write the following in src/main.rs

gstreamer/post03/rust/src/main.rsView on GitHub

use gst::prelude::*;
use gstreamer as gst;

fn main() {
    gst::init().unwrap();

    let pipeline = gst::Pipeline::new();

    let src = gst::ElementFactory::make("videotestsrc")
        .property("num-buffers", 90)
        .build()
        .expect("Failed to create videotestsrc");

    let capsfilter = gst::ElementFactory::make("capsfilter")
        .property(
            "caps",
            gst::Caps::builder("video/x-raw")
                .field("width", 1280)
                .field("height", 720)
                .field("framerate", gst::Fraction::new(30, 1))
                .build(),
        )
        .build()
        .expect("Failed to create capsfilter");

    let enc = gst::ElementFactory::make("x264enc")
        .build()
        .expect("Failed to create x264enc");

    let parse = gst::ElementFactory::make("h264parse")
        .build()
        .expect("Failed to create h264parse");

    let mux = gst::ElementFactory::make("matroskamux")
        .build()
        .expect("Failed to create matroskamux");

    let sink = gst::ElementFactory::make("filesink")
        .property("location", "test.mkv")
        .build()
        .expect("Failed to create filesink");

    pipeline
        .add_many([&src, &capsfilter, &enc, &parse, &mux, &sink])
        .unwrap();

    gst::Element::link_many([&src, &capsfilter, &enc, &parse, &mux, &sink])
        .expect("Failed to link elements");

    pipeline
        .set_state(gst::State::Playing)
        .expect("Unable to set pipeline to Playing");

    let bus = pipeline.bus().unwrap();

    for msg in bus.iter_timed(gst::ClockTime::NONE) {
        use gst::MessageView;

        match msg.view() {
            MessageView::Eos(..) => {
                println!("End of stream");
                break;
            }
            MessageView::Error(err) => {
                eprintln!(
                    "Error from {:?}: {} ({:?})",
                    err.src().map(|s| s.path_string()),
                    err.error(),
                    err.debug()
                );
                break;
            }
            _ => (),
        }
    }

    pipeline
        .set_state(gst::State::Null)
        .expect("Unable to set pipeline to Null");
}

cargo add gstreamer writes the dependency for you, so your Cargo.toml should look like this:

gstreamer/post03/rust/Cargo.tomlView on GitHub

[package]
name = "gstreamer-rust"
version = "0.1.0"
edition = "2021"

[dependencies]
gstreamer = "0.23"

Now you run the code using

cargo run

Initializing GStreamer

The first thing every GStreamer application must do is initialize the library by calling gst_init.

gstreamer/post03/cpp/main.cpp

gst_init(&argc, &argv);

gstreamer/post03/rust/src/main.rs

gst::init().unwrap();

This function initializes the internal GStreamer infrastructure and prepares the library for use. It must be called before using any other GStreamer APIs.

Creating a Pipeline

A pipeline is the top-level container that holds all elements used by an application.

Technically, GstPipeline is a specialized type of GstBin that provides additional functionality such as state management, clock management, and a message bus.

A pipeline can be created using:

gstreamer/post03/cpp/main.cpp

GstElement *pipeline = gst_pipeline_new("test-pipeline");

gstreamer/post03/rust/src/main.rs

let pipeline = gst::Pipeline::new();

Next, we create the elements required for our application, just as we did previously with gst-launch-1.0.

Adding Elements to the Pipeline

Once the elements are created, they must be added to the pipeline.

gstreamer/post03/cpp/main.cpp

gst_bin_add_many(GST_BIN(pipeline), src, capsfilter, enc, parse, mux, sink, nullptr);

gstreamer/post03/rust/src/main.rs

pipeline
    .add_many([&src, &capsfilter, &enc, &parse, &mux, &sink])
    .unwrap();

At this point, the pipeline owns and manages these elements, but it still does not know how they should be connected.

Linking Elements

To connect elements together, we use gst_element_link_many. It returns whether the whole chain was linked successfully, so it is worth checking rather than ignoring.

gstreamer/post03/cpp/main.cpp

if (!gst_element_link_many(src, capsfilter, enc, parse, mux, sink, nullptr)) {
    g_printerr("Failed to link elements\n");
    gst_object_unref(pipeline);
    return -1;
}

gstreamer/post03/rust/src/main.rs

gst::Element::link_many([&src, &capsfilter, &enc, &parse, &mux, &sink])
    .expect("Failed to link elements");

This function links the pads between elements, allowing data to flow from one element to the next.

Configuring Element Properties

Many elements expose configurable properties that control their behavior.

Our program uses two of them. The num-buffers property on videotestsrc tells the source to produce exactly 90 frames and then stop, which at 30 frames per second gives us a three second video. The location property on filesink decides where the file is written.

gstreamer/post03/cpp/main.cpp

g_object_set(src, "num-buffers", 90, nullptr);
g_object_set(sink, "location", "test.mkv", nullptr);

gstreamer/post03/rust/src/main.rs

let src = gst::ElementFactory::make("videotestsrc")
    .property("num-buffers", 90)
    .build()
    .expect("Failed to create videotestsrc");

Notice the difference in style between the two. In C++ we build the element first and configure it afterwards with g_object_set, which works because most GStreamer elements are built on top of the GObject type system. The Rust bindings instead let us set properties on the builder before the element is created, so filesink receives its location at construction time rather than in a separate step.

videotestsrc has many more properties than the one we use here. pattern, for example, changes the generated test pattern, and it is worth experimenting with.

Starting the Pipeline

After creating, configuring, and linking all elements, we can start the pipeline by changing its state to PLAYING.

gstreamer/post03/cpp/main.cpp

GstStateChangeReturn ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);
if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr("Failed to set pipeline to PLAYING\n");
    gst_object_unref(pipeline);
    return -1;
}

gstreamer/post03/rust/src/main.rs

pipeline
    .set_state(gst::State::Playing)
    .expect("Unable to set pipeline to Playing");

Once the pipeline enters the PLAYING state, data begins flowing through the pipeline and the application starts performing its intended task.

Waiting for Messages

While running, GStreamer sends messages through a bus. Common messages include:

EOS (End of Stream)
ERROR
WARNING
STATE_CHANGED

Applications typically listen for these messages to monitor the pipeline and react to events.

We will not dive deeply into the bus in this article. It is an important topic that deserves its own dedicated post later in this series.

Summary

In this post, we learned the basic structure of a GStreamer application:

Initialize GStreamer with gst_init.
Create a pipeline.
Create the required elements.
Add the elements to the pipeline.
Link the elements together.
Configure element properties when needed.
Start the pipeline.
Monitor messages through the bus.

These steps form the foundation of nearly every GStreamer application, regardless of its complexity.

Exercises

Exercise 1

Create a program that displays a test video using the following pipeline:

videotestsrc ! autovideosink

Exercise 2

Experiment with different values of the pattern property in videotestsrc and observe how the output changes.

Exercise 3

Create an MKV video file using the same pipeline from the previous article, but this time implement it using C++ instead of gst-launch-1.0.

Exercise 4

Add error handling by listening for ERROR messages on the bus and printing the error details to the console.