Using Gstreamer in your Flutter project Pt.5: iOS

In the previous tutorials, we successfully ran the app on Linux,
Windows, macOS and Android. This tutorial will guide you through the process of making it run on iOS.

Prerequisites

Ensure you have the following foundational knowledge before proceeding:

Basic understanding of the C programming language.
Familiarity with Dart/Flutter.
Basic knowledge of CocoaPods.
Basic knowledge of GStreamer.

Required Tools and Software

The instructions in this part of the series are demonstrated on a
Linux PC. You can adapt them for other operating systems if needed.
Below are the essential tools and software required:

GStreamer Libraries For iOS: Install GStreamer and its development libraries. Follow the GStreamer documentation for installation guidelines.
Code Editor: Any code editor will work, but I use VSCode.
Flutter: Install Flutter by following the instructions on the official Flutter website.
CocoaPods: Install CocoaPods by following the guidelines on the CocoaPods website.
Xcode: Required for building projects on iOS. Install it through the Mac App Store.

Before proceeding, please ensure that you have Xcode, CocoaPods, and GStreamer for iOS installed.

Building the App Without Modifications

First, let's try to build the app without any modifications. Navigate to the app folder and run the build command targeting iOS with verbose output:

flutter build ios -vv --debug

You should see an error similar to this:

[   +7 ms] Lexical or Preprocessor Issue (Xcode): 'gst/gst.h' file not found
           /Users/max/Projects/fltgst/app/native_binding/src/native_binding.c:1:9

This error indicates that the GStreamer header file cannot be found. Now, let's fix that. We can reference the macOS port for guidance.

Open native_binding/ios/native_binding.podspec and add the compile and linking options to s.pod_target_xcconfig. Since the GStreamer for iOS is installed in ${HOME}/Library/Frameworks/GStreamer.framework, we need to modify them accordingly:

  'HEADER_SEARCH_PATHS' => '"${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Headers"',
  'LD_RUNPATH_SEARCH_PATHS' => ['"${HOME}/Library/Developer/GStreamer/iPhone.sdk/"','"${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Libraries"'],
  'OTHER_LDFLAGS' => '" -L${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Libraries -F${HOME}/Library/Developer/GStreamer/iPhone.sdk/ -framework GStreamer "',

Now, if we build again, we will encounter the following error:

[ +343 ms] Failed to build iOS app
[   +8 ms] Error (Xcode): Undefined symbol: _iconv
           
[   +5 ms] Error (Xcode): Undefined symbol: _iconv_close
           
[        ] Error (Xcode): Undefined symbol: _iconv_open
           
[        ] Error (Xcode): Linker command failed with exit code 1 (use -v to see invocation)

This indicates that the iconv library is missing. We can resolve this by adding it to the native_binding.podspec, so it looks like this:

#
# To learn more about a Podspec see http://guides.cocoapods.org/syntax/podspec.html.
# Run `pod lib lint native_binding.podspec` to validate before publishing.
#
Pod::Spec.new do |s|
  s.name             = 'native_binding'
  s.version          = '0.0.1'
  s.summary          = 'A new Flutter FFI plugin project.'
  s.description      = <<-DESC
A new Flutter FFI plugin project.
                       DESC
  s.homepage         = 'http://example.com'
  s.license          = { :file => '../LICENSE' }
  s.author           = { 'Your Company' => '[email protected]' }

  # This will ensure the source files in Classes/ are included in the native
  # builds of apps using this FFI plugin. Podspec does not support relative
  # paths, so Classes contains a forwarder C file that relatively imports
  # `../src/*` so that the C sources can be shared among all target platforms.
  s.source           = { :path => '.' }
  s.source_files = 'Classes/**/*'
  s.dependency 'Flutter'
  s.platform = :ios, '11.0'

  # Flutter.framework does not contain a i386 slice.
  s.pod_target_xcconfig = { 
  'DEFINES_MODULE' => 'YES',
  'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386 arm64',
  'HEADER_SEARCH_PATHS' => '"${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Headers"',
  'LD_RUNPATH_SEARCH_PATHS' => ['"${HOME}/Library/Developer/GStreamer/iPhone.sdk/"','"${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Libraries"'],
  'OTHER_LDFLAGS' => '" -L${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Libraries -F${HOME}/Library/Developer/GStreamer/iPhone.sdk/ -framework GStreamer "',
  }
  
  s.libraries= 'iconv' # <- we added this
  s.swift_version = '5.0'
end

The build command now runs successfully, but when we try to run the app, we get the following error:

Error (Xcode): Building for 'iOS-simulator', but linking in object file (/Users/max/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Versions/1.0/GStreamer[arm64][3656](libffi_a-types.c.o)) built for 'iOS'

We need to add arm64 to s.pod_target_xcconfig under EXCLUDED_ARCHS[sdk=iphonesimulator*], so it looks like this:

s.pod_target_xcconfig = { 
  'DEFINES_MODULE' => 'YES',
  'EXCLUDED_ARCHS[sdk=iphonesimulator*]' => 'i386 arm64',
  'HEADER_SEARCH_PATHS' => '"${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Headers"',
  'LD_RUNPATH_SEARCH_PATHS' => ['"${HOME}/Library/Developer/GStreamer/iPhone.sdk/"','"${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Libraries"'],
  'OTHER_LDFLAGS' => '" -L${HOME}/Library/Developer/GStreamer/iPhone.sdk/GStreamer.framework/Libraries -F${HOME}/Library/Developer/GStreamer/iPhone.sdk/ -framework GStreamer "',
  }

After making these changes, if we build the app again, we no longer encounter the error. However, upon running the app, we notice that there is no audio output.

Linking and Initializing GStreamer

On mobile platforms, GStreamer needs to be linked and initialized manually. To achieve this, we can reference the official GStreamer iOS projects. Each project will have gst_ios_init.h and gst_ios_init.m files in the Supporting Files directory. We can copy these two files and add them to our project.

Paste the gst_ios_init.m file into native_binding/ios/Classes/ so it will be compiled without any modifications. For gst_ios_init.h, paste it into native_binding/src/, and have our native_binding.c file reference this header when compiling for iOS. Add the following to native_binding.c after the include macros:

#ifdef IOS
    #include "gst_ios_init.h"
#endif

And call the gst_ios_init function in the init of native_binding.c:

FFI_PLUGIN_EXPORT void init(void)
{
#ifdef IOS
    gst_ios_init();
#endif
    // init
    gst_init(NULL, NULL);
    data = (FltGstData *)malloc(sizeof(FltGstData));
}

Since we changed the location of gst_ios_init.h, we need to update the reference in gst_ios_init.m:

#include "../../src/gst_ios_init.h"

Next, enable plugins for the elements we need by modifying gst_ios_init.h:

#ifndef __GST_IOS_INIT_H__
#define __GST_IOS_INIT_H__

#include <gst/gst.h>

G_BEGIN_DECLS

#define GST_G_IO_MODULE_DECLARE(name) \
extern void G_PASTE(g_io_, G_PASTE(name, _load)) (gpointer module)

#define GST_G_IO_MODULE_LOAD(name) \
G_PASTE(g_io_, G_PASTE(name, _load)) (NULL)

/* Uncomment each line to enable the plugin categories that your application needs.
 * You can also enable individual plugins. See gst_ios_init.c to see their names
 */

#define GST_IOS_PLUGIN_AUDIOTESTSRC
#define GST_IOS_PLUGIN_AUDIOCONVERT
#define GST_IOS_PLUGIN_AUTODETECT
#define GST_IOS_PLUGIN_OSXAUDIO
// #define GST_IOS_PLUGINS_CORE
//#define GST_IOS_PLUGINS_CODECS
//#define GST_IOS_PLUGINS_ENCODING
//#define GST_IOS_PLUGINS_NET
//#define GST_IOS_PLUGINS_PLAYBACK
//#define GST_IOS_PLUGINS_VIS
//#define GST_IOS_PLUGINS_SYS
//#define GST_IOS_PLUGINS_EFFECTS
//#define GST_IOS_PLUGINS_CAPTURE
//#define GST_IOS_PLUGINS_CODECS_GPL
//#define GST_IOS_PLUGINS_CODECS_RESTRICTED
//#define GST_IOS_PLUGINS_NET_RESTRICTED
//#define GST_IOS_PLUGINS_GES


//#define GST_IOS_GIO_MODULE_GNUTLS

void gst_ios_init (void);

G_END_DECLS

#endif

To make the IOS definition work, we need to add another line to s.pod_target_xcconfig in native_binding.podspec:

  'GCC_PREPROCESSOR_DEFINITIONS' => '$(inherited) IOS=1'

Now, if we build the app again, we get:

[ +574 ms] Failed to build iOS app
[  +12 ms] Error (Xcode): Undefined symbol: _res_9_dn_expand
           
[   +2 ms] Error (Xcode): Undefined symbol: _res_9_ndestroy
           
[        ] Error (Xcode): Undefined symbol: _res_9_ninit
           
[        ] Error (Xcode): Undefined symbol: _res_9_nquery

This indicates that the plugins we just added have other dependencies. Let’s add those. A hint before you add them: you can try adding them in Xcode. Open app/ios/Runner.xcworkspace, head over to Pods, under TARGETS select native_binding -> General. In Frameworks and Libraries, add the libraries or frameworks you need. For our pipeline, the final result is:

  s.libraries= 'iconv','resolv'
  s.framework= 'AudioToolbox'

After running flutter clean, flutter pub get, and flutter build ios -vv --debug, we can build the app without errors and run it, hearing the sine wave as expected.

Debugging

To debug the native code, we need Xcode. In the Project navigator tab, expand Pods -> Development Pods -> native_binding, and expand that all the way. Open native_binding, hold down the Command key and click "../../src/native_binding.c", then the source file will open. You can now insert breakpoints and debug from there.

GitHub Repository

You can find the repository and the updated files for this tutorial at:

Repository: https://github.com/fengjiongmax/fltgst
Files after changes: https://github.com/fengjiongmax/fltgst/tree/8302ef9a115562c18b0fe4d63c628ababd5317f2