Using Gstreamer in your Flutter project Pt.4: Android

In the previous tutorials, we successfully ran the app on Linux, Windows, and macOS. This tutorial will guide you through the process of making it run on Android.

Prerequisites

Ensure you have the following foundational knowledge before proceeding:

Basic understanding of the C programming language.
Familiarity with Dart/Flutter.
Basic knowledge of CocoaPods.
Basic knowledge of GStreamer.

Required Tools and Software

The instructions in this part of the series are demonstrated on a
Linux PC. You can adapt them for other operating systems if needed.
Below are the essential tools and software required:

GStreamer Libraries For Android: Install GStreamer and its development libraries. Follow the GStreamer documentation for installation guidelines.
Code Editor: Any code editor will work, but I use VSCode.
CMake: Download from cmake.org. On Linux, you can install it via your package manager.
Flutter: Install Flutter by following the instructions on the official Flutter website.
Android Studio: As well as Android sdk and ndk, you can download them using sdk manager inside Android Studio.

Before We Start

This tutorial is the most challenging part of this series, so brace yourself!

Building the App Without Modifications

First, let's try to build the app without any modifications. Navigate
to the app folder and run the build command targeting Android with
verbose output flutter build apk -vv --debug. You should see an error similar to this:

[        ] /usr/include/x86_64-linux-gnu/sys/cdefs.h:64:6: error: function-like macro '__GNUC_PREREQ' is not defined
[        ] # if __GNUC_PREREQ (4, 6) && !defined _LIBC
[        ]      ^
[        ] /usr/include/x86_64-linux-gnu/sys/cdefs.h:78:10: error: function-like macro '__GNUC_PREREQ' is not defined
[        ]      && (__GNUC_PREREQ (3, 4) || __glibc_has_attribute (__nothrow__))
[        ]          ^
[        ] /usr/include/x86_64-linux-gnu/sys/cdefs.h:84:31: error: function-like macro '__GNUC_PREREQ' is not defined
[        ] #  if defined __cplusplus && (__GNUC_PREREQ (2,8) || __clang_major >= 4)
[        ]                               ^
[        ] /usr/include/x86_64-linux-gnu/sys/cdefs.h:146:34: error: function-like macro '__glibc_clang_prereq' is not defined
[        ] #if __USE_FORTIFY_LEVEL == 3 && (__glibc_clang_prereq (9, 0)                  \
[        ]                                  ^
[        ] /usr/include/x86_64-linux-gnu/sys/cdefs.h:202:5: error: function-like macro '__GNUC_PREREQ' is not defined
[        ] #if __GNUC_PREREQ (4,3)
[        ]     ^
...

Like before, this was expected, and this errors appeared four times, this was because android will build for 4 different architecture:

x86
x86_64
armv7
arm64

Each architecture runs its own build. The errors occur because the compiler is trying to link against libraries on the host (Ubuntu), but the Android NDK does not have those functions. To resolve these errors, we need to link libraries built for Android.

Extract the downloaded GStreamer binaries for Android and place them in native_binding/third-party/gst-android, and the project structure should look like this:

native_binding
	- ...
	- third-party
		- gst-android
			- x86
			- x86_64
			- armv7
			- arm64

And add a file named .gitignore in native_binding/third-party/, with the content like this:

gst-android/

This ensures that the GStreamer binaries for Android are not committed to the repository.

Next, open native_binding/src/CMakeLists.txt. Remove the line include_directories(${GST_INCLUDE_DIRS}) and add an if block for handling Android-specific compiling and linking logic. Set variables to point to the GStreamer Android binaries so we can use them later. Add back include_directories(${GST_INCLUDE_DIRS}) to the else block. The modified CMakeLists.txt should look like this:

IF(ANDROID OR __ANDROID__)
  SET(GST_FOLDER ${CMAKE_CURRENT_SOURCE_DIR}/../third-party/gst-android)
  SET(ABI_SUFFIX ${ANDROID_ABI})

  IF(${ANDROID_ABI} STREQUAL "armeabi-v7a")
    SET(ABI_SUFFIX armv7)
  ELSEIF(${ANDROID_ABI} STREQUAL "arm64-v8a")
    SET(ABI_SUFFIX arm64)
  ELSEIF(${ANDROID_ABI} STREQUAL "x86")
    # skipped
  ELSEIF(${ANDROID_ABI} STREQUAL "x86_64")
    # skipped
  ENDIF()

  SET(GST_ROOT ${GST_FOLDER}/${ABI_SUFFIX})

  # -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include
  include_directories(
    ${GST_ROOT}/include/gstreamer-1.0
    ${GST_ROOT}/include/glib-2.0
    ${GST_ROOT}/lib/glib-2.0/include
  )

  link_directories(
    ${GST_ROOT}/lib
    ${GST_ROOT}/lib/gstreamer-1.0
  )
ELSE()
  include_directories(${GST_INCLUDE_DIRS}) # we need this for other systems
 ENDIF()

Now, let's try building again. We will encounter different errors:

[        ] ld: error: undefined symbol: libintl_bindtextdomain
[        ] >>> referenced by gst.c:552
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/arm64/gstreamer-1.0-1.22.9/_builddir/../gst/gst.c:552)
[        ] >>>               gst.c.o:(init_pre) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/arm64/lib/libgstreamer-1.0.a
[        ] >>> referenced by ggettext.c:109
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/arm64/glib-2.74.4/_builddir/../glib/ggettext.c:109)
[        ] >>>               ggettext.c.o:(ensure_gettext_initialized) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/arm64/lib/libglib-2.0.a
[        ] ld: error: undefined symbol: libintl_bind_textdomain_codeset
[        ] >>> referenced by gst.c:553
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/arm64/gstreamer-1.0-1.22.9/_builddir/../gst/gst.c:553)
[        ] >>>               gst.c.o:(init_pre) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/arm64/lib/libgstreamer-1.0.a
[        ] >>> referenced by ggettext.c:112
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/arm64/glib-2.74.4/_builddir/../glib/ggettext.c:112)
[        ] >>>               ggettext.c.o:(ensure_gettext_initialized) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/arm64/lib/libglib-2.0.a
...

We are getting linker errors this time, which means the compilation stage passed, but the linking stage failed.

Linking and Initializing GStreamer

On mobile platforms, GStreamer needs to be linked and initialized manually. To link and initialize GStreamer for Android, we can reference files provided by the GStreamer binaries.

Open native_binding/third-party/gst-android/x86_64/share/gst-android/ndk-build/ (any targeted architecture will have these files, but we’ll use x86_64 for this tutorial). There’s a gstreamer_android-1.0.c.in file, which we need to generate an actual file that will be compiled into the native_binding library. However, we don’t need gio for this project, so create a file in native_binding/src/gst_android.c.in with the following content:

#include <gst/gst.h>

/* Declaration of static plugins */
 @PLUGINS_DECLARATION@;

/* This is called by gst_init() */
void
gst_init_static_plugins (void)
{
  @PLUGINS_REGISTRATION@;
}

And open native_binding/third-party/gst-android/x86_64/share/gst-android/ndk-build/gstreamer-1.0.mk, we search GSTREAMER_PLUGINS_LIBS, and we see for each plugin required for our pipeline, we link against then with a gst prefix. And few lines below, we see for each plugin, a function call to GST_PLUGIN_STATIC_DECLARE and GST_PLUGIN_STATIC_REGISTER was added.

So how do we know what plugin was required for our pipeline? Remember the pipeline we use: gst-launch-1.0 audiotestsrc ! audioconvert ! autoaudiosink, we can search the element name on the GStreamer Plugins List, and follow the link, we can see what plugin the element belongs to. For example, audiotestsrc was belongs to plugin audiotestsrc, according to GStreamer audiotestsrc page.

So with the explaination from above, we make some changes to the native_binding/src/CMakeLists.txt, so it looks like this:

# The Flutter tooling requires that developers have CMake 3.10 or later
# installed. You should not increase this version, as doing so will cause
# the plugin to fail to compile for some customers of the plugin.
cmake_minimum_required(VERSION 3.10)

project(native_binding_library VERSION 0.0.1 LANGUAGES C)


find_package(PkgConfig REQUIRED)

pkg_search_module(GST REQUIRED gstreamer-1.0)


IF(WIN32)
  find_program(CMAKE_PKGCONFIG_EXECUTABLE pkg-config)
  IF(CMAKE_PKGCONFIG_EXECUTABLE)
  # pkg-config.exe gstreamer-1.0 --libs --msvc-syntax
    EXEC_PROGRAM(${CMAKE_PKGCONFIG_EXECUTABLE}
    ARGS " --libs --msvc-syntax gstreamer-1.0"
    OUTPUT_VARIABLE GST_LDFLAGS)
    # replace spaces with semicolons so that we don't have quotation marks in command line option
    string(REPLACE " " ";" GST_LDFLAGS ${GST_LDFLAGS})
    message("GST_LDFLAGS: ${GST_LDFLAGS}")
  ENDIF()
ENDIF()

IF(ANDROID OR __ANDROID__)
  SET(GST_FOLDER ${CMAKE_CURRENT_SOURCE_DIR}/../third-party/gst-android)
  SET(ABI_SUFFIX ${ANDROID_ABI})

  IF(${ANDROID_ABI} STREQUAL "armeabi-v7a")
    SET(ABI_SUFFIX armv7)
  ELSEIF(${ANDROID_ABI} STREQUAL "arm64-v8a")
    SET(ABI_SUFFIX arm64)
  ELSEIF(${ANDROID_ABI} STREQUAL "x86")
    # skipped
  ELSEIF(${ANDROID_ABI} STREQUAL "x86_64")
    # skipped
  ENDIF()

  SET(GST_ROOT ${GST_FOLDER}/${ABI_SUFFIX})

  # -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include
  include_directories(
    ${GST_ROOT}/include/gstreamer-1.0
    ${GST_ROOT}/include/glib-2.0
    ${GST_ROOT}/lib/glib-2.0/include
  )

  link_directories(
    ${GST_ROOT}/lib
    ${GST_ROOT}/lib/gstreamer-1.0
  )

  SET(PLUGINS_DECLARATION)
  SET(PLUGINS_REGISTRATION)

  LIST(APPEND GST_PLUGINS coreelements coretracers adder app audioconvert audiorate audiotestsrc gio autodetect opensles)
  foreach(GST_P ${GST_PLUGINS})
    LIST(APPEND LINK_LIBS "gst${GST_P}")
    LIST(APPEND PLUGINS_DECLARATION "\nGST_PLUGIN_STATIC_DECLARE(${GST_P});")
    LIST(APPEND PLUGINS_REGISTRATION "\nGST_PLUGIN_STATIC_REGISTER(${GST_P});")
  endforeach()
  
  configure_file(gst_android.c.in ${CMAKE_CURRENT_SOURCE_DIR}/gst_plugin_init_android.c)

  LIST(APPEND APPENDED_SOURCE gst_plugin_init_android.c)

ELSE()
  include_directories(${GST_INCLUDE_DIRS}) # we need this for other systems
ENDIF()

add_library(native_binding SHARED
  "native_binding.c"
  ${APPENDED_SOURCE}
)

IF(WIN32)
  target_link_options(native_binding PRIVATE ${GST_LDFLAGS})
ELSE()
  target_link_libraries(native_binding PRIVATE ${GST_LDFLAGS} ${LINK_LIBS})
ENDIF()

set_target_properties(native_binding PROPERTIES
  PUBLIC_HEADER native_binding.h
  OUTPUT_NAME "native_binding"
)

target_compile_definitions(native_binding PUBLIC DART_SHARED_LIB)

This is because for the plugins we linked against, they have dependencies as well, so for each undefined symbol, we go to native_binding/third_party/x86_64/lib and then run grep [missing symbol] -r .. For example, the missing symbol is libintl_bindtextdomain, the result is:

$ grep -r 'libintl_bindtextdomain' .
grep: ./libintl.a: binary file matches
grep: ./gio/modules/libgioopenssl.a: binary file matches
grep: ./libgdk_pixbuf-2.0.a: binary file matches
grep: ./libgstreamer-1.0.a: binary file matches
grep: ./libgstpbutils-1.0.a: binary file matches
grep: ./gstreamer-1.0/libgstladspa.a: binary file matches
grep: ./gstreamer-1.0/libgstavi.a: binary file matches
grep: ./gstreamer-1.0/libgstwavpack.a: binary file matches
grep: ./gstreamer-1.0/libgstflac.a: binary file matches
grep: ./gstreamer-1.0/libgstplayback.a: binary file matches
grep: ./gstreamer-1.0/libgstmidi.a: binary file matches
grep: ./gstreamer-1.0/libgstasf.a: binary file matches
grep: ./gstreamer-1.0/libgstaiff.a: binary file matches
grep: ./gstreamer-1.0/libgstisomp4.a: binary file matches
grep: ./gstreamer-1.0/libgstrtsp.a: binary file matches
grep: ./gstreamer-1.0/libgstlame.a: binary file matches
grep: ./gstreamer-1.0/libgstencoding.a: binary file matches
grep: ./gstreamer-1.0/libgstsoup.a: binary file matches
grep: ./gstreamer-1.0/libgstogg.a: binary file matches
grep: ./libsoup-2.4.a: binary file matches
grep: ./libgstaudio-1.0.a: binary file matches
grep: ./libglib-2.0.a: binary file matches
grep: ./libgsttag-1.0.a: binary file matches

Then we need to link against intl, and we add those libraries to the LINK_LIBS list, so for our pipeline, we add the following line to the end of the android block:

  LIST(APPEND LINK_LIBS intl ffi iconv gmodule-2.0 pcre2-8 gstbase-1.0 gstaudio-1.0 orc-0.4 gstapp-1.0 gio-2.0 log z OpenSLES)

Then we build the app again, this time we got functions that we can't find in the GStreamer binaries:

[        ] ld: error: undefined symbol: __FD_SET_chk
[        ] >>> referenced by gstpoll.c:485
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/armv7/gstreamer-1.0-1.22.9/_builddir/../gst/gstpoll.c:485)
[        ] >>>               gstpoll.c.o:(gst_poll_wait) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/armv7/lib/libgstreamer-1.0.a
[        ] >>> referenced by gstpoll.c:487
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/armv7/gstreamer-1.0-1.22.9/_builddir/../gst/gstpoll.c:487)
[        ] >>>               gstpoll.c.o:(gst_poll_wait) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/armv7/lib/libgstreamer-1.0.a
[        ] >>> referenced by gstpoll.c:489
(/home/nirbheek/projects/repos/cerbero.git/1.22/build/sources/android_universal/armv7/gstreamer-1.0-1.22.9/_builddir/../gst/gstpoll.c:489)
[        ] >>>               gstpoll.c.o:(gst_poll_wait) in archive
/home/tin/Projects/fltgst/fltgst/app/native_binding/src/../third-party/gst-android/armv7/lib/libgstreamer-1.0.a

And this is a function from standard library. Because the GStreamer binaries for android I use is verrsion 1.24.0, which was compiled against sdk version v21, so we need to update the minimumSdk for the android project, both on the native_binding and the app project.
For both native_binding/android/build.gradle and app/android/app/src/build.gradle, set android -> defaultConfig -> minSdkVersion to 21.
And the app build success, and we can hear sine wave!

Debugging

To debug the native code, we need Android Studio, open the android folder of the app project, and from the top left drop down select Android, then expand native_binding -> cpp , then you can open the native_binding.c file, and insert break point anywhere you want.

GitHub Repository

You can find the repository and the updated files for this tutorial at:

Repository: https://github.com/fengjiongmax/fltgst
Files after changes: https://github.com/fengjiongmax/fltgst/tree/69a07c012ab5227e26dfbc157562cb2959f462c7