Using Gstreamer in your Flutter project Pt.2: Linux

In the previous tutorial, we created a C program that outputs a sine wave using GStreamer. In this tutorial, we'll take it a step further by creating a Flutter application that utilizes GStreamer to produce a sine wave.

This tutorial is designed for Linux, so we'll be using Linux throughout the process.

Prerequisites

Before proceeding with this tutorial, ensure you have the following:

  • Basic knowledge of the C programming language.

  • Basic knowledge of Dart/Flutter.

  • Basic knowledge of CMake.

  • Basic knowledge of GStreamer.

Required Tools and Software

The instructions in this part of the series are demonstrated on a Linux PC. You can adapt them for other operating systems if needed. Below are the essential tools and software required:

  • GStreamer Libraries: Install GStreamer and its development libraries. Follow the GStreamer documentation for installation guidelines.

  • Code Editor: Any code editor will work, but I use VSCode.

  • CMake: Download from cmake.org. On Linux, you can install it via your package manager.

  • Flutter: Install Flutter by following the instructions on the official Flutter website.

GStreamer Pipeline

We'll use the GStreamer pipeline from the previous tutorial:

gst-launch-1.0 audiotestsrc ! audioconvert ! autoaudiosink

Creating the Flutter Project

We'll structure our project to include a Flutter app and an FFI (Foreign Function Interface) plugin.

  1. Create the project directories and files:

    mkdir app
cd app
flutter create fltgst # create app
flutter create -t plugin_ffi native_binding --platforms linux,windows,macos,android,ios # create ffi plugin
code fltgst # open app project in VSCode
code native_binding # open ffi project in VSCode

Start coding

First, let's modify the native_binding.c file in the FFI plugin project. Initially, it looks like this:

#include "native_binding.h"

// A very short-lived native function.
//
// For very short-lived functions, it is fine to call them on the main isolate.
// They will block the Dart execution while running the native function, so
// only do this for native functions which are guaranteed to be short-lived.
FFI_PLUGIN_EXPORT int sum(int a, int b) { return a + b; }

// A longer-lived native function, which occupies the thread calling it.
//
// Do not call these kind of native functions in the main isolate. They will
// block Dart execution. This will cause dropped frames in Flutter applications.
// Instead, call these native functions on a separate isolate.
FFI_PLUGIN_EXPORT int sum_long_running(int a, int b) {
  // Simulate work.
#if _WIN32
  Sleep(5000);
#else
  usleep(5000 * 1000);
#endif
  return a + b;
}

Next, copy everything except the main function from main.c in the previous tutorial and paste it into native_binding.c. You'll notice the functions in native_binding.c have a FFI_PLUGIN_EXPORT macro. This macro marks functions as externally accessible so they can be called from Dart.
Here’s how native_binding.c looks after modification:

#include <gst/gst.h>
#include "native_binding.h"

// gst-launch-1.0 audiotestsrc ! audioconvert ! autoaudiosink

typedef struct _FltGstData
{
    GstElement *pipeline;
    GstElement *audiotestsrc;
    GstElement *audioconvert;
    GstElement *autoaudiosink;

    GMainLoop *mainloop;
} FltGstData;

FltGstData *data;

FFI_PLUGIN_EXPORT void init(void)
{
    // init
    gst_init(NULL, NULL);
    data = (FltGstData *)malloc(sizeof(FltGstData));
}

FFI_PLUGIN_EXPORT void setup_pipeline(void)
{
    // setup pipeline
    data->audiotestsrc = gst_element_factory_make("audiotestsrc", NULL);
    data->audioconvert = gst_element_factory_make("audioconvert", NULL);
    data->autoaudiosink = gst_element_factory_make("autoaudiosink", NULL);
    data->pipeline = gst_pipeline_new(NULL);

    data->mainloop = g_main_loop_new(NULL, FALSE);

    if (!data->audiotestsrc || !data->audioconvert || !data->autoaudiosink || !data->pipeline)
    {
        g_printerr("Elements could not be created.\n");
        gst_object_unref(data->pipeline);
        return;
    }

    gst_bin_add_many(GST_BIN(data->pipeline), data->audiotestsrc, data->audioconvert, data->autoaudiosink, NULL);
    if (!gst_element_link_many(data->audiotestsrc, data->audioconvert, data->autoaudiosink, NULL))
    {
        g_printerr("Elements could not be created.\n");
        gst_object_unref(data->pipeline);
        return;
    }
}

FFI_PLUGIN_EXPORT void start_pipeline(void)
{
    // start pipeline
    GstStateChangeReturn ret;
    ret = gst_element_set_state(data->pipeline, GST_STATE_PLAYING);
    if (ret == GST_STATE_CHANGE_FAILURE)
    {
        g_printerr("Elements could not be created.\n");
        gst_object_unref(data->pipeline);
        return;
    }
}

FFI_PLUGIN_EXPORT void run_mainloop(void)
{
    // run main loop
    g_main_loop_run(data->mainloop);
}

FFI_PLUGIN_EXPORT void free_resource(void)
{
    // free resources
    gst_element_set_state(data->pipeline, GST_STATE_NULL);
    gst_object_unref(data->pipeline);
}

// A very short-lived native function.
//
// For very short-lived functions, it is fine to call them on the main isolate.
// They will block the Dart execution while running the native function, so
// only do this for native functions which are guaranteed to be short-lived.
FFI_PLUGIN_EXPORT int sum(int a, int b) { return a + b; }

// A longer-lived native function, which occupies the thread calling it.
//
// Do not call these kind of native functions in the main isolate. They will
// block Dart execution. This will cause dropped frames in Flutter applications.
// Instead, call these native functions on a separate isolate.
FFI_PLUGIN_EXPORT int sum_long_running(int a, int b) {
  // Simulate work.
#if _WIN32
  Sleep(5000);
#else
  usleep(5000 * 1000);
#endif
  return a + b;
}

Next, update the header file native_binding.h with the function definitions and the FFI_PLUGIN_EXPORT macro:

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

#if _WIN32
#include <windows.h>
#else
#include <pthread.h>
#include <unistd.h>
#endif

#if _WIN32
#define FFI_PLUGIN_EXPORT __declspec(dllexport)
#else
#define FFI_PLUGIN_EXPORT
#endif

FFI_PLUGIN_EXPORT void init(void);
FFI_PLUGIN_EXPORT void setup_pipeline(void);
FFI_PLUGIN_EXPORT void start_pipeline(void);
FFI_PLUGIN_EXPORT void run_mainloop(void);
FFI_PLUGIN_EXPORT void free_resource(void);

// A very short-lived native function.
//
// For very short-lived functions, it is fine to call them on the main isolate.
// They will block the Dart execution while running the native function, so
// only do this for native functions which are guaranteed to be short-lived.
FFI_PLUGIN_EXPORT int sum(int a, int b);

// A longer lived native function, which occupies the thread calling it.
//
// Do not call these kind of native functions in the main isolate. They will
// block Dart execution. This will cause dropped frames in Flutter applications.
// Instead, call these native functions on a separate isolate.
FFI_PLUGIN_EXPORT int sum_long_running(int a, int b);

Update the CMakeLists.txt file in the native_binding/src/ directory to link against GStreamer. Refer to the CMakeLists.txt from the previous tutorial:

# The Flutter tooling requires that developers have CMake 3.10 or later
# installed. You should not increase this version, as doing so will cause
# the plugin to fail to compile for some customers of the plugin.
cmake_minimum_required(VERSION 3.10)

project(native_binding_library VERSION 0.0.1 LANGUAGES C)

find_package(PkgConfig REQUIRED)

pkg_search_module(GST REQUIRED gstreamer-1.0)

include_directories(${GST_INCLUDE_DIRS})

add_library(native_binding SHARED
  "native_binding.c"
)
target_link_libraries(native_binding PRIVATE ${GST_LDFLAGS})

set_target_properties(native_binding PROPERTIES
  PUBLIC_HEADER native_binding.h
  OUTPUT_NAME "native_binding"
)

target_compile_definitions(native_binding PUBLIC DART_SHARED_LIB)

FFI Bindings

Generate the FFI files using ffigen. In the native_binding directory, run:

dart run ffigen --config ffigen.yaml

This generates function pointers in native_binding/lib/native_binding_bindings_generated.dart. Map these pointers to Dart functions in native_binding/lib/native_binding.dart:

void init() => _bindings.init();
void setupPipeline() => _bindings.setup_pipeline();
void startPipeline() => _bindings.start_pipeline();
void runMainloop() => _bindings.run_mainloop();
void freeResource() => _bindings.free_resource();

The work on the native_binding project is complete. Now, let’s update the app project.
Add native_binding as a dependency using a relative path. In app/pubspec.yaml, add the following under the dependencies section:

  native_binding:
    path: ../native_binding

Run flutter pub get to update the dependencies.
When you build the app, you should see the files in the native_binding project being compiled. Use the following command to build:

flutter build linux -vvv --debug

You'll notice output similar to:

$ flutter build linux -vvv --debug
...
[  +11 ms] ninja: Entering directory `build/linux/x64/debug'
[ +380 ms] [1/8] Building C object plugins/native_binding/shared/CMakeFiles/native_binding.dir/native_binding.c.o
[ +135 ms] [2/8] Linking C shared library plugins/native_binding/shared/libnative_binding.so
...

Implementing in Flutter

Update app/lib/main.dart to call the functions in native_binding/lib/native_binding.dart:

import 'dart:io';

import 'package:flutter/material.dart';
import 'package:native_binding/native_binding.dart' as native_binding;

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
        useMaterial3: true,
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key, required this.title});

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  void initState() {
    super.initState();
    native_binding.init();
    native_binding.setupPipeline();
    native_binding.startPipeline();
  }

  @override
  void dispose() {
    native_binding.freeResource();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    // This method is rerun every time setState is called, for instance as done
    // by the _incrementCounter method above.
    //
    // The Flutter framework has been optimized to make rerunning build methods
    // fast, so that you can just rebuild anything that needs updating rather
    // than having to individually change instances of widgets.
    return Scaffold(
      appBar: AppBar(
        // TRY THIS: Try changing the color here to a specific color (to
        // Colors.amber, perhaps?) and trigger a hot reload to see the AppBar
        // change color while the other colors stay the same.
        backgroundColor: Theme.of(context).colorScheme.inversePrimary,
        // Here we take the value from the MyHomePage object that was created by
        // the App.build method, and use it to set our appbar title.
        title: Text(widget.title),
      ),
      body: const Center(
        child: Text("FltGst"),
      ),
    );
  }
}

When you run the app, you should hear a sine wave.

Debugging

To debug our native code, open VSCode and navigate to the debug tab of the app project, click create a launch.json file, then add the following section to the configurations section:

        {
            "name": "Linux Native",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/build/linux/x64/debug/bundle/fltgst",
            "args": [],
            "environment": [],
            "cwd": "${workspaceFolder}"
        },

Then select Linux Native, head back to the files tab, expand DEPENDENCIES -> direct dependencies -> native_binding, then open native_binding.c, add a breakpoint at any line, and hit F5 to run the app. You should see the breakpoint being hit.

GitHub Repository

Find the repository and files updated in this tutorial at:

CMake Gstreamer Flutter Linux