diff --git a/AGENTS.md b/AGENTS.md index d9daf703..3a706c50 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -85,10 +85,17 @@ Be sure to update the directory layout in this file if the directory layout chan |------|-------------| | `include/livekit/` | Public API headers (what SDK consumers include) | | `include/livekit/capture/` | Capture API headers; installed only for capture-enabled SDKs | +| `include/livekit/capture/capture_source.h` | Compatibility umbrella exposing all audio/video capture APIs | +| `include/livekit/capture/audio_source.h` | Audio capture APIs and configurations | +| `include/livekit/capture/video_source.h` | Video capture API and configurations | +| `include/livekit/capture/capture_types.h` | Shared capture results, errors, and exit reasons | | `src/` | Implementation files and internal-only headers (`ffi_client.h`, `lk_log.h`, etc.) | | `src/tests/` | Google Test integration and stress tests | +| `src/tests/manual/` | Interactive hardware testers built with the test targets; not registered with CTest | | `examples/` | In-tree example applications | | `client-sdk-rust/` | Git submodule holding the Rust core of the SDK| +| `client-sdk-rust/livekit-capture/src/audio/` | PCM audio sources and pump | +| `client-sdk-rust/livekit-capture/src/video/` | Pixel and encoded video pumps, sources, and rendering | | `cpp-tools/` | Git submodule holding shared LiveKit C++ engineering guidance, clang-format / clang-tidy configs, scripts, docs, and CI workflow | | `client-sdk-rust/livekit-ffi/protocol/*.proto` | FFI contract (protobuf definitions, read-only reference) | | `cmake/` | Build helpers (`protobuf.cmake`, `spdlog.cmake`, `nlohmann_json.cmake`, `LiveKitConfig.cmake.in`) | diff --git a/CMakeLists.txt b/CMakeLists.txt index 00d4bd32..3cc2e29a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -473,7 +473,10 @@ add_library(livekit SHARED ) # Conditionally compile capture sources if(LIVEKIT_CAPTURE_ENABLED) - target_sources(livekit PRIVATE src/capture_source.cpp) + target_sources(livekit PRIVATE + src/audio_capture_source.cpp + src/capture_source.cpp + ) endif() # Symbol visibility: keep liblivekit's exported ABI restricted to the public diff --git a/client-sdk-rust b/client-sdk-rust index 2811b4be..1387ff32 160000 --- a/client-sdk-rust +++ b/client-sdk-rust @@ -1 +1 @@ -Subproject commit 2811b4be194b091868ed83cca0603842179fed36 +Subproject commit 1387ff328cc9eaf579e6b4aa99c8111f8a3e695a diff --git a/docs/README.md b/docs/README.md index 17158b5b..3dac637e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,7 +4,7 @@ Additional documentation for the SDK. - [Building](building.md) — prerequisites, build scripts, Linux build container, CMake presets, vcpkg, integration into your CMake project, troubleshooting. -- [Capture sources](capture-sources.md) — pattern, clock, and GStreamer sources; +- [Capture sources](capture-sources.md) — video sources plus GStreamer audio capture; build features, release bundles, and publishing. Use capture to stream video through LiveKit without maintaining your own video pipeline. - [Authentication](authentication.md) — generating tokens, token source types diff --git a/docs/building.md b/docs/building.md index fb21e516..07b6b7fc 100644 --- a/docs/building.md +++ b/docs/building.md @@ -534,11 +534,14 @@ sources are available. Check `cargo_features` in `share/livekit/build-info.json` for the selected backends. Creating a source omitted from the SDK build fails through the returned future with `std::runtime_error`. Capture-enabled bundles install the public API under `include/livekit/capture/`; -standard bundles omit that directory. Use `` -for direct inclusion. The source-tree header also rejects use in a standard -build. `share/livekit/build-info.json` in an installed bundle describes the same -build features; neither value tests whether a GPU or a requested GStreamer plugin -is available at runtime. +standard bundles omit that directory. Use `` +for video and `` for GStreamer PCM capture. +`` remains the umbrella that includes both +headers and their shared types from ``. +The source-tree audio and video headers also reject use in a standard build. +`share/livekit/build-info.json` in an installed bundle describes the same build +features; neither value tests whether a GPU or a requested GStreamer plugin is +available at runtime. Release downloads with `-capture` in their names include these features, for example `livekit-sdk-macos-arm64-capture-.tar.gz`. Standard release diff --git a/docs/capture-sources.md b/docs/capture-sources.md index 5f65b032..f3527097 100644 --- a/docs/capture-sources.md +++ b/docs/capture-sources.md @@ -2,15 +2,16 @@ > **NOTE:** LiveKit Capture is not officially supported on Intel-based Macs. -`livekit::capture::CaptureSource` connects a Rust video producer to a regular C++ `VideoSource`. -The capture pump runs in Rust, so C++ does not send one FFI request per frame. -The SDK supports three source configurations: +The capture APIs connect Rust media producers to regular C++ track +types. Media frames stay in Rust, so C++ does not send one FFI request per +frame. The SDK supports video capture and GStreamer PCM audio capture. | Configuration | Output | Runtime requirements | | --- | --- | --- | | `PatternVideoSourceConfig` | GPU-rendered gradient or LiveKit logo | Compatible GPU adapter | | `ClockVideoSourceConfig` | GPU-rendered local time with millisecond precision | Compatible GPU adapter | | `GstreamerVideoSourceConfig` | Pre-encoded video from a GStreamer pipeline | GStreamer runtime and the pipeline's plugins | +| `GstreamerAudioSourceConfig` | PCM S16LE audio into an `AudioSource` | GStreamer runtime and the pipeline's plugins | A custom SDK build can include all sources or a subset selected with `--capture=pattern,clock` or `--capture=gstreamer`. Release `-capture` bundles @@ -29,8 +30,18 @@ requirements, see [Building — Capture sources](building.md#capture-sources). Capture types live in the `livekit::capture` namespace. Capture-enabled bundles install the API in `include/livekit/capture/`: +| Header | Contents | +| --- | --- | +| `capture_source.h` | Umbrella for all capture APIs; existing callers can keep this include | +| `audio_source.h` | Audio capture and its configurations | +| `video_source.h` | Video capture and its configurations | +| `capture_types.h` | Shared capture results, errors, and exit reasons | + +Include the umbrella for the full API, or the audio and video headers directly: + ```cpp -#include +#include +#include ``` Standard SDK bundles omit the `capture/` directory, and their other installed @@ -52,7 +63,7 @@ continues, and the source is released when the callback arrives. ```cpp #include -#include +#include // The SDK is initialized, and room is connected. try { @@ -91,6 +102,43 @@ thread to exit, and reports the exit reason and frame count. For unbounded sources, call `stop()` before `join()`. Destroying the source also stops capture. +## GStreamer audio capture + +`GstreamerAudioSourceConfig` follows the naming of `GstreamerVideoSourceConfig`. +Both source APIs create asynchronously and use +the same `start()`, `stop()`, and `join()` lifecycle, `CaptureResult`, and +`CaptureError` types. + +`GstreamerAudioCaptureSource` asynchronously creates a fixed-format PCM S16LE +pipeline and owns the `AudioSource` fed by its Rust capture pump. Publish that +source through a normal `LocalAudioTrack`, then use the same start, stop, and +join lifecycle as video capture: + +```cpp +livekit::capture::GstreamerAudioSourceConfig config; +config.pipeline = "audiotestsrc ! audioconvert ! " + "audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! " + "appsink name=lk_audio_appsink"; +config.sample_rate = 48000; +config.num_channels = 1; + +auto capture = livekit::capture::GstreamerAudioCaptureSource::create(config).get().value(); +auto track = livekit::LocalAudioTrack::createLocalAudioTrack("gstreamer", capture->audioSource()); +participant->publishTrack(track); +capture->start().value(); + +capture->stop().value(); +auto result = capture->join().get().value(); +// result contains frames_captured (one frame = 10 ms of audio) and exit. +``` + +The negotiated `format()` reports sample rate, channel count, and samples per +channel. Rust validates that pipeline caps match the requested fixed format. +Unlike video resolution and codec discovery, audio requires an explicit fixed +format (48 kHz and mono by default). Audio frames are always 10 ms. The pipeline +must contain the named appsink. Audio capture does not expose video's `publishOptions()` helper; +publish it with normal audio track options. + ## GStreamer encoded pipelines Provide `appsink name=lk_appsink`, or leave exactly one encoded video source pad diff --git a/docs/testing.md b/docs/testing.md index 09525611..e140a587 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -51,6 +51,41 @@ ctest --output-on-failure __Note:__ The integration and stress tests require tokens and a running LiveKit server. See the section below for details. +## Manual microphone capture (macOS) + +`src/tests/manual/capture_audio.cpp` is a standalone GStreamer microphone tester. +It is built with the test binaries on macOS when the GStreamer capture backend +is enabled, and is not run by CTest: + +```bash +./build.sh release-tests --capture=all +./build-release/bin/capture_audio --seconds 10 +``` + +The default pipeline uses `osxaudiosrc` for the default macOS input device and +converts to 48 kHz mono S16LE PCM. Allow microphone access for your terminal in +**System Settings > Privacy & Security > Microphone** if prompted. GStreamer and +the pipeline's plugins must be installed; see [Capture sources](capture-sources.md). +Capture stops at the requested duration (1–600 seconds) or Ctrl+C and prints the frame count +and captured-audio duration. + +To publish the microphone to a LiveKit room, start your server, then run: + +```bash +source scripts/set-test-tokens.sh +./build-release/bin/capture_audio --publish --seconds 30 +``` + +`--publish` reads `LIVEKIT_URL` and `LIVEKIT_TOKEN`, falling back to +`LIVEKIT_TOKEN_A`. Join the same room as another participant to listen. + +Use `--pipeline` to select a different device or test without a microphone: + +```bash +./build-release/bin/capture_audio --seconds 1 --pipeline \ + 'audiotestsrc is-live=true ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! appsink name=lk_audio_appsink' +``` + ## Running a local LiveKit server for tests The integration and stress suites need a running LiveKit server. The easiest @@ -116,8 +151,10 @@ means none. Explicit source selections determine the expected capture flag. The test creates a GStreamer VP8 source when that source is selected. It also requires each omitted backend to report the feature-disabled error. Configuration checks that `include/livekit/capture/` is present only in capture bundles and -that direct inclusion of `livekit/capture/capture_source.h` succeeds only in -those bundles. The obsolete flat capture header must be absent. Both variants must +that each capture header compiles independently only in those bundles, +including `video_source.h` and the compatibility umbrella `capture_source.h`. +The umbrella is also used by the runtime packaging test to exercise both audio +and video APIs. The obsolete flat capture header must be absent. Both variants must still compile with `livekit/livekit.h`. It needs no LiveKit server. GStreamer-enabled runs require the pipeline's plugins. Set `LIVEKIT_VERIFY_PIXEL_SOURCES=ON` to also require successful pattern and clock diff --git a/include/livekit/audio_source.h b/include/livekit/audio_source.h index 249907c5..91ba2d2d 100644 --- a/include/livekit/audio_source.h +++ b/include/livekit/audio_source.h @@ -30,6 +30,7 @@ class FfiResponse; } // namespace proto class FfiClient; +struct AudioSourceAccess; /// Represents a real-time audio source with an internal audio queue. class LIVEKIT_API AudioSource { @@ -131,6 +132,10 @@ class LIVEKIT_API AudioSource { void captureFrame(const AudioFrame& frame, int timeout_ms = 20); private: + friend struct AudioSourceAccess; + + AudioSource(FfiHandle handle, int sample_rate, int num_channels, int queue_size_ms) noexcept; + // Internal helper to reset the local queue tracking (like _release_waiter). void resetQueueTracking() noexcept; diff --git a/include/livekit/capture/audio_source.h b/include/livekit/capture/audio_source.h new file mode 100644 index 00000000..08241b8d --- /dev/null +++ b/include/livekit/capture/audio_source.h @@ -0,0 +1,146 @@ +/* + * Copyright 2026 LiveKit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an “AS IS” BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "livekit/build.h" + +#if !LIVEKIT_CAPTURE_ENABLED && !defined(LIVEKIT_BUILDING_SDK) +#error "Audio capture requires a capture-enabled LiveKit SDK; rebuild with --capture=gstreamer." +#endif + +#include +#include +#include +#include + +#include "livekit/audio_source.h" +#include "livekit/capture/capture_types.h" +#include "livekit/result.h" +#include "livekit/visibility.h" + +namespace livekit { + +class FfiClient; + +namespace proto { +class OwnedAudioCaptureSource; +} // namespace proto + +} // namespace livekit + +namespace livekit::capture { + +/// @brief Fixed PCM format produced in 10 ms frames by an audio capture source. +struct AudioCaptureFormat { + /// @brief Sample rate in Hz. + std::uint32_t sample_rate = 0; + + /// @brief Number of interleaved channels. + std::uint32_t num_channels = 0; + + /// @brief Number of samples per channel in each captured frame. + std::uint32_t samples_per_channel = 0; +}; + +/// @brief Configuration for PCM S16LE audio ingest in 10 ms frames from GStreamer. +struct GstreamerAudioSourceConfig { + /// @brief GStreamer launch description. + /// + /// @note Must contain `appsink name=lk_audio_appsink`. + std::string pipeline; + + /// @brief Output sample rate in Hz. + std::uint32_t sample_rate = 48000; + + /// @brief Number of interleaved output channels. + std::uint32_t num_channels = 1; +}; + +/// @brief GStreamer audio capture source backed by a Rust capture pump. +/// +/// Creation builds the pipeline asynchronously. The object owns both the +/// capture pump and the returned @ref AudioSource. Keep it alive while a local +/// audio track uses that source. This type is not safe for concurrent lifecycle +/// calls. +class LIVEKIT_API GstreamerAudioCaptureSource { +public: + /// @brief Create a GStreamer audio capture source asynchronously. + /// + /// Completes asynchronously after Rust initializes the producer and RTC + /// source. Keep the future to overlap setup with other work, or await it + /// immediately. Destroying the future does not wait for creation; an + /// abandoned result is released when creation completes. + /// + /// @param config Pipeline and fixed PCM output format. + /// @return Future containing the created source on success. + /// @throws std::runtime_error When awaiting the future if setup, transport, + /// protocol validation, or SDK shutdown fails. + /// @throws std::bad_alloc If allocation fails. + [[nodiscard]] static std::future, CaptureError>> create( + const GstreamerAudioSourceConfig& config); + + ~GstreamerAudioCaptureSource(); + + GstreamerAudioCaptureSource(const GstreamerAudioCaptureSource&) = delete; + GstreamerAudioCaptureSource& operator=(const GstreamerAudioCaptureSource&) = delete; + GstreamerAudioCaptureSource(GstreamerAudioCaptureSource&&) = delete; + GstreamerAudioCaptureSource& operator=(GstreamerAudioCaptureSource&&) = delete; + + /// @brief Negotiated fixed PCM format. + /// @return Audio format reported by Rust. + AudioCaptureFormat format() const noexcept; + + /// @brief RTC audio source fed by the capture pump. + /// @return Owned audio source suitable for `LocalAudioTrack::createLocalAudioTrack()`. + std::shared_ptr audioSource() const noexcept; + + /// @brief Start pumping pipeline audio into the RTC source. + /// @return Success when Rust acknowledges the request. + /// @throws std::runtime_error If the request cannot be completed. + [[nodiscard]] Result start(); + + /// @brief Signal capture to stop without waiting for cleanup. + /// + /// Call @ref join() to synchronize and retrieve the terminal result. + /// Stopping an already-finished capture is a no-op. + /// + /// @return Success when Rust acknowledges the request. + /// @throws std::runtime_error If the request cannot be completed. + [[nodiscard]] Result stop(); + + /// @brief Wait asynchronously for capture to finish and return audio statistics. + /// + /// Call @ref stop() first for an unbounded pipeline. The returned + /// @ref CaptureResult includes the number of accepted 10 ms frames and exit reason. + /// Join may be called only once after a successful @ref start(). + /// + /// @return Future containing terminal statistics or a structured capture error. + /// @throws std::runtime_error When awaiting the future if transport, protocol, + /// cancellation, or SDK shutdown fails. + [[nodiscard]] std::future> join(); + +private: + friend class livekit::FfiClient; + + GstreamerAudioCaptureSource(); + static std::shared_ptr fromOwned(const proto::OwnedAudioCaptureSource& owned); + + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace livekit::capture diff --git a/include/livekit/capture/capture_source.h b/include/livekit/capture/capture_source.h index 623e4616..94d9ce11 100644 --- a/include/livekit/capture/capture_source.h +++ b/include/livekit/capture/capture_source.h @@ -16,316 +16,6 @@ #pragma once -#include "livekit/build.h" - -// Internal SDK wrappers are built for both variants; consumers require capture. -#if !LIVEKIT_CAPTURE_ENABLED && !defined(LIVEKIT_BUILDING_SDK) -#error "Capture sources require a capture-enabled LiveKit SDK; rebuild with --capture." -#endif - -#include -#include -#include -#include -#include - -#include "livekit/result.h" -#include "livekit/room_event_types.h" -#include "livekit/video_source.h" -#include "livekit/visibility.h" - -namespace livekit { - -class FfiClient; - -namespace proto { -class OwnedCaptureSource; -} // namespace proto - -} // namespace livekit - -/// @brief Capture sources and their configuration types. -namespace livekit::capture { - -/// @brief Kind of media a capture source produces. -enum class CaptureSourceKind { - /// @brief Pixel frames, published through the WebRTC encoder. - Pixel = 0, - /// @brief Pre-encoded access units, published as passthrough. - Encoded = 1, -}; - -/// @brief Bitrate unit expected by a GStreamer encoder property. -enum class GstreamerBitrateUnit { - /// @brief Bits per second. - Bps = 0, - /// @brief Kilobits per second. - Kbps = 1, -}; - -/// @brief Video resolution in pixels. -struct CaptureResolution { - std::uint32_t width = 0; - std::uint32_t height = 0; -}; - -/// @brief Binding from WebRTC rate-control targets to a GStreamer encoder property. -struct GstreamerRateControl { - /// @brief Name of the encoder element in the pipeline (e.g. `lk_encoder`). - std::string element; - - /// @brief Bitrate property to set on the element (e.g. `bitrate` for x264enc, - /// `target-bitrate` for vp8enc/vp9enc). - std::string property; - - /// @brief Unit the property expects. - GstreamerBitrateUnit unit = GstreamerBitrateUnit::Bps; -}; - -/// @brief Configuration for encoded ingest from a GStreamer pipeline. -struct GstreamerVideoSourceConfig { - /// @brief GStreamer launch description for the encoded producer pipeline. - /// - /// @note Must contain `appsink name=lk_appsink`, or leave exactly one encoded - /// video source pad unlinked for the source to attach one to. - std::string pipeline; - - /// @brief Codec expected from the pipeline; inferred from pipeline caps when - /// omitted. - std::optional codec; - - /// @brief Encoded frame resolution. When omitted, it is discovered from the - /// pipeline's negotiated caps; when set, the pipeline output is verified - /// against it. - std::optional resolution; - - /// @brief Forwards WebRTC rate-control targets to an encoder element's bitrate - /// property. Without this, the pipeline encodes at a fixed bitrate. - std::optional rate_control; -}; - -/// @brief Test pattern rendered by the built-in pattern source. -enum class Pattern { - /// @brief Animated color gradient. - Gradient = 0, - /// @brief Bouncing LiveKit logo. - Logo = 1, -}; - -/// Configuration for the built-in test source rendering a @ref Pattern. -struct PatternVideoSourceConfig { - /// @brief Output resolution; both components must be positive. - CaptureResolution resolution; - - /// @brief Output frame rate in frames per second; must be non-zero. - std::uint32_t framerate_fps = 0; - - /// @brief Pattern to render. - Pattern pattern = Pattern::Gradient; -}; - -/// @brief Wall clock with millisecond precision, rendered on the GPU. -/// Shows the local time of the machine running the FFI server. -struct ClockVideoSourceConfig { - /// @brief Output resolution; both components must be positive. - CaptureResolution resolution; - - /// @brief Output frame rate in frames per second; must be non-zero. - std::uint32_t framerate_fps = 0; -}; - -/// @brief Why a capture ended without error. -enum class CaptureExit { - /// Stopped via @ref CaptureSource::stop (or source destruction). - Stopped = 0, - /// @brief The producer reached the end of its stream. - EndOfStream = 1, -}; - -/// @brief Terminal pump failure category; matches the FFI CaptureErrorKind enum. -enum class CaptureErrorKind { - /// @brief The capture source failed or violated its contract. - Source = 0, - /// @brief The RTC video source rejected a frame. - Rtc = 1, - /// @brief The pump thread panicked. - Panicked = 2, -}; - -/// @brief Structured terminal pump failure reported by the Rust FFI. -struct CaptureError { - /// @brief Human-readable error message. - std::string message; - - /// @brief Error category. - CaptureErrorKind kind = CaptureErrorKind::Source; -}; - -/// @brief Successful terminal statistics of a started capture. -struct CaptureResult { - /// @brief Number of frames or access units captured. - std::uint64_t frames_captured = 0; - - /// @brief Why the capture ended successfully. - CaptureExit exit = CaptureExit::Stopped; -}; - -/// @brief A server-side capture source (livekit-capture) that produces video -/// without per-frame FFI traffic. -/// -/// The source owns its producer (e.g. a GStreamer pipeline) and the pump -/// that feeds an RTC video source. Publish it like any other source: create -/// a track from @ref CaptureSource::videoSource(), merge application options -/// through @ref CaptureSource::publishOptions(), then call @ref CaptureSource::start(). -/// -/// Structured pump failures are returned as Result values containing @ref CaptureError. -/// Failures without a protobuf error kind throw exceptions. Rust validates source -/// configurations and reports failures as std::runtime_error. Invalid enum values -/// cannot be serialized and throw std::invalid_argument. -/// Initialization, transport, protocol, cancellation, and untyped Rust failures -/// throw std::runtime_error. For create() and join(), -/// these exceptions are delivered when awaiting the future. -/// -/// Requires a capture-enabled SDK build (`LIVEKIT_CAPTURE`). Creation fails -/// when the requested source was not compiled into the Rust FFI. -/// -/// @note Keep this object alive while the track is published. Destroying it -/// stops a running capture. -class LIVEKIT_API CaptureSource { -public: - /// @brief Create a capture source from a GStreamer pipeline configuration. - /// - /// Completes asynchronously: construction starts the pipeline and may wait - /// for its first output to discover stream settings. Configuration, producer - /// setup, transport, protocol, and cancellation failures are delivered as - /// exceptions through the future. - /// Destroying the future does not wait for creation. The source is released - /// when creation completes if the future was discarded. - /// - /// @param config GStreamer pipeline configuration. - /// @return A future containing a successful Result with the created source. - /// @throws std::invalid_argument When awaiting the future if enum values cannot be serialized. - /// @throws std::runtime_error When awaiting the future if Rust rejects the configuration - /// or setup, initialization, transport, protocol validation, or cancellation fails. - /// @throws std::bad_alloc If allocation fails, including when awaiting the future. - [[nodiscard]] static std::future, CaptureError>> create( - const GstreamerVideoSourceConfig& config); - - /// @brief Create the built-in test pattern capture source. - /// - /// Completes asynchronously after Rust initializes the GPU renderer. - /// Keep the future to overlap setup with other work, or await it immediately. - /// Destroying the future does not wait for creation; an abandoned result is - /// released when creation completes. - /// - /// @param config Pattern source configuration. - /// @return A future containing a successful Result with the created source. - /// @throws std::invalid_argument When awaiting the future if enum values cannot be serialized. - /// @throws std::runtime_error When awaiting the future if Rust rejects the configuration - /// or setup, initialization, transport, protocol validation, or cancellation fails. - /// @throws std::bad_alloc If allocation fails, including when awaiting the future. - [[nodiscard]] static std::future, CaptureError>> create( - PatternVideoSourceConfig config); - - /// @brief Create the built-in wall clock capture source. - /// - /// Completes asynchronously after Rust initializes the GPU renderer. - /// Keep the future to overlap setup with other work, or await it immediately. - /// Destroying the future does not wait for creation; an abandoned result is - /// released when creation completes. - /// - /// @param config Clock source configuration. - /// @return A future containing a successful Result with the created source. - /// @throws std::runtime_error When awaiting the future if Rust rejects the configuration - /// or setup, initialization, transport, protocol validation, or cancellation fails. - /// @throws std::bad_alloc If allocation fails, including when awaiting the future. - [[nodiscard]] static std::future, CaptureError>> create( - ClockVideoSourceConfig config); - - ~CaptureSource(); - - CaptureSource(const CaptureSource&) = delete; - CaptureSource& operator=(const CaptureSource&) = delete; - CaptureSource(CaptureSource&&) = delete; - CaptureSource& operator=(CaptureSource&&) = delete; - - /// @brief Kind of media this source produces. - /// @return The source kind. - CaptureSourceKind kind() const noexcept; - - /// @brief Declared or discovered stream resolution. - /// @return Width in pixels. - int width() const noexcept; - - /// @brief Declared or discovered stream height. - /// @return Height in pixels. - int height() const noexcept; - - /// @brief Codec produced by the source; encoded sources only. - /// @return The codec, or no value for a pixel source. - std::optional codec() const noexcept; - - /// @brief RTC video source fed by this capture source; use it with - /// LocalVideoTrack::createLocalVideoTrack(). - /// @return The RTC video source. - std::shared_ptr videoSource() const noexcept; - - /// @brief Returns publish options for this track, applying application options. - /// - /// Encoded sources override codec with the source codec, select the - /// PreEncoded encoder backend, and disable simulcast. All other fields are - /// taken from @p options unchanged. Pixel sources preserve every application - /// option; Rust supplies defaults for fields left unset. - /// @param options Application publish options. - /// @return Options with source-required fields applied. - TrackPublishOptions publishOptions(TrackPublishOptions options = {}) const; - - /// @brief Start pumping frames into the RTC video source. - /// - /// @return A successful Result when Rust acknowledges the request. - /// @throws std::runtime_error If the SDK is not initialized, Rust rejects the - /// request, or transport or protocol validation fails. - /// @throws std::bad_alloc If memory allocation fails. - [[nodiscard]] Result start(); - - /// @brief Signal a running capture to stop after the frame in flight. - /// - /// This method does not wait for the pump thread to exit. Call @ref join() - /// to synchronize and retrieve the terminal result. Stopping an - /// already-finished capture is a no-op. - /// - /// @return A successful Result when Rust acknowledges the request. - /// @throws std::runtime_error If the SDK is not initialized, Rust rejects the - /// request, or transport or protocol validation fails. - /// @throws std::bad_alloc If memory allocation fails. - [[nodiscard]] Result stop(); - - /// @brief Consume the running pump and wait asynchronously for it to exit. - /// - /// This method does not signal the pump to stop. Call @ref stop() first when - /// ending an unbounded source. Join may be called only once after a - /// successful @ref start(). - /// - /// @return A future containing statistics on success or a structured pump - /// CaptureError whose kind is supplied by Rust. - /// @throws std::runtime_error When awaiting the future if initialization, - /// transport, protocol validation, or cancellation fails. - /// @throws std::bad_alloc If allocation fails, including when awaiting the future. - [[nodiscard]] std::future> join(); - -private: - friend class livekit::FfiClient; - - CaptureSource(); - - /// @brief Builds the wrapper from the callback payload, adopting its handles. - /// @param owned Rust-owned source handles and metadata. - /// @return The capture source wrapper. - /// @throws std::runtime_error If the source metadata is invalid. - static std::shared_ptr fromOwned(const proto::OwnedCaptureSource& owned); - - struct Impl; - std::unique_ptr impl_; -}; - -} // namespace livekit::capture +// Compatibility umbrella for video and audio capture. +#include "livekit/capture/audio_source.h" +#include "livekit/capture/video_source.h" diff --git a/include/livekit/capture/capture_types.h b/include/livekit/capture/capture_types.h new file mode 100644 index 00000000..895213ab --- /dev/null +++ b/include/livekit/capture/capture_types.h @@ -0,0 +1,64 @@ +/* + * Copyright 2026 LiveKit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an “AS IS” BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include + +/// @brief Capture sources and their configuration types. +namespace livekit::capture { + +/// @brief Why a capture ended without error. +enum class CaptureExit { + /// @brief Stopped by the application or source destruction. + Stopped = 0, + /// @brief The producer reached the end of its stream. + EndOfStream = 1, +}; + +/// @brief Terminal pump failure category; matches the FFI CaptureErrorKind enum. +enum class CaptureErrorKind { + /// @brief The capture source failed or violated its contract. + Source = 0, + /// @brief The RTC audio or video source rejected a frame. + Rtc = 1, + /// @brief The pump thread panicked. + Panicked = 2, +}; + +/// @brief Structured terminal pump failure reported by the Rust FFI. +struct CaptureError { + /// @brief Human-readable error message. + std::string message; + + /// @brief Error category. + CaptureErrorKind kind = CaptureErrorKind::Source; +}; + +/// @brief Successful terminal statistics of a started capture. +struct CaptureResult { + /// @brief Number of frames accepted by the RTC source. + /// + /// Audio frames contain 10 ms of PCM; video frames are pixel frames or + /// encoded access units. + std::uint64_t frames_captured = 0; + + /// @brief Why the capture ended successfully. + CaptureExit exit = CaptureExit::Stopped; +}; + +} // namespace livekit::capture diff --git a/include/livekit/capture/video_source.h b/include/livekit/capture/video_source.h new file mode 100644 index 00000000..d7ccde91 --- /dev/null +++ b/include/livekit/capture/video_source.h @@ -0,0 +1,295 @@ +/* + * Copyright 2026 LiveKit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an “AS IS” BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "livekit/build.h" + +// Internal SDK wrappers are built for both variants; consumers require capture. +#if !LIVEKIT_CAPTURE_ENABLED && !defined(LIVEKIT_BUILDING_SDK) +#error "Capture sources require a capture-enabled LiveKit SDK; rebuild with --capture." +#endif + +#include +#include +#include +#include +#include + +#include "livekit/capture/capture_types.h" +#include "livekit/result.h" +#include "livekit/room_event_types.h" +#include "livekit/video_source.h" +#include "livekit/visibility.h" + +namespace livekit { + +class FfiClient; + +namespace proto { +class OwnedCaptureSource; +} // namespace proto + +} // namespace livekit + +namespace livekit::capture { + +/// @brief Kind of media a capture source produces. +enum class CaptureSourceKind { + /// @brief Pixel frames, published through the WebRTC encoder. + Pixel = 0, + /// @brief Pre-encoded access units, published as passthrough. + Encoded = 1, +}; + +/// @brief Bitrate unit expected by a GStreamer encoder property. +enum class GstreamerBitrateUnit { + /// @brief Bits per second. + Bps = 0, + /// @brief Kilobits per second. + Kbps = 1, +}; + +/// @brief Video resolution in pixels. +struct CaptureResolution { + std::uint32_t width = 0; + std::uint32_t height = 0; +}; + +/// @brief Binding from WebRTC rate-control targets to a GStreamer encoder property. +struct GstreamerRateControl { + /// @brief Name of the encoder element in the pipeline (e.g. `lk_encoder`). + std::string element; + + /// @brief Bitrate property to set on the element (e.g. `bitrate` for x264enc, + /// `target-bitrate` for vp8enc/vp9enc). + std::string property; + + /// @brief Unit the property expects. + GstreamerBitrateUnit unit = GstreamerBitrateUnit::Bps; +}; + +/// @brief Configuration for encoded ingest from a GStreamer pipeline. +struct GstreamerVideoSourceConfig { + /// @brief GStreamer launch description for the encoded producer pipeline. + /// + /// @note Must contain `appsink name=lk_appsink`, or leave exactly one encoded + /// video source pad unlinked for the source to attach one to. + std::string pipeline; + + /// @brief Codec expected from the pipeline; inferred from pipeline caps when + /// omitted. + std::optional codec; + + /// @brief Encoded frame resolution. When omitted, it is discovered from the + /// pipeline's negotiated caps; when set, the pipeline output is verified + /// against it. + std::optional resolution; + + /// @brief Forwards WebRTC rate-control targets to an encoder element's bitrate + /// property. Without this, the pipeline encodes at a fixed bitrate. + std::optional rate_control; +}; + +/// @brief Test pattern rendered by the built-in pattern source. +enum class Pattern { + /// @brief Animated color gradient. + Gradient = 0, + /// @brief Bouncing LiveKit logo. + Logo = 1, +}; + +/// Configuration for the built-in test source rendering a @ref Pattern. +struct PatternVideoSourceConfig { + /// @brief Output resolution; both components must be positive. + CaptureResolution resolution; + + /// @brief Output frame rate in frames per second; must be non-zero. + std::uint32_t framerate_fps = 0; + + /// @brief Pattern to render. + Pattern pattern = Pattern::Gradient; +}; + +/// @brief Wall clock with millisecond precision, rendered on the GPU. +/// Shows the local time of the machine running the FFI server. +struct ClockVideoSourceConfig { + /// @brief Output resolution; both components must be positive. + CaptureResolution resolution; + + /// @brief Output frame rate in frames per second; must be non-zero. + std::uint32_t framerate_fps = 0; +}; + +/// @brief A server-side capture source (livekit-capture) that produces video +/// without per-frame FFI traffic. +/// +/// The source owns its producer (e.g. a GStreamer pipeline) and the pump +/// that feeds an RTC video source. Publish it like any other source: create +/// a track from @ref CaptureSource::videoSource(), merge application options +/// through @ref CaptureSource::publishOptions(), then call @ref CaptureSource::start(). +/// +/// Structured pump failures are returned as Result values containing @ref CaptureError. +/// Failures without a protobuf error kind throw exceptions. Rust validates source +/// configurations and reports failures as std::runtime_error. Invalid enum values +/// cannot be serialized and throw std::invalid_argument. +/// Initialization, transport, protocol, cancellation, and untyped Rust failures +/// throw std::runtime_error. For create() and join(), +/// these exceptions are delivered when awaiting the future. +/// +/// Requires a capture-enabled SDK build (`LIVEKIT_CAPTURE`). Creation fails +/// when the requested source was not compiled into the Rust FFI. +/// +/// @note Keep this object alive while the track is published. Destroying it +/// stops a running capture. +class LIVEKIT_API CaptureSource { +public: + /// @brief Create a capture source from a GStreamer pipeline configuration. + /// + /// Completes asynchronously: construction starts the pipeline and may wait + /// for its first output to discover stream settings. Configuration, producer + /// setup, transport, protocol, and cancellation failures are delivered as + /// exceptions through the future. + /// Destroying the future does not wait for creation. The source is released + /// when creation completes if the future was discarded. + /// + /// @param config GStreamer pipeline configuration. + /// @return A future containing a successful Result with the created source. + /// @throws std::invalid_argument When awaiting the future if enum values cannot be serialized. + /// @throws std::runtime_error When awaiting the future if Rust rejects the configuration + /// or setup, initialization, transport, protocol validation, or cancellation fails. + /// @throws std::bad_alloc If allocation fails, including when awaiting the future. + [[nodiscard]] static std::future, CaptureError>> create( + const GstreamerVideoSourceConfig& config); + + /// @brief Create the built-in test pattern capture source. + /// + /// Completes asynchronously after Rust initializes the GPU renderer. + /// Keep the future to overlap setup with other work, or await it immediately. + /// Destroying the future does not wait for creation; an abandoned result is + /// released when creation completes. + /// + /// @param config Pattern source configuration. + /// @return A future containing a successful Result with the created source. + /// @throws std::invalid_argument When awaiting the future if enum values cannot be serialized. + /// @throws std::runtime_error When awaiting the future if Rust rejects the configuration + /// or setup, initialization, transport, protocol validation, or cancellation fails. + /// @throws std::bad_alloc If allocation fails, including when awaiting the future. + [[nodiscard]] static std::future, CaptureError>> create( + PatternVideoSourceConfig config); + + /// @brief Create the built-in wall clock capture source. + /// + /// Completes asynchronously after Rust initializes the GPU renderer. + /// Keep the future to overlap setup with other work, or await it immediately. + /// Destroying the future does not wait for creation; an abandoned result is + /// released when creation completes. + /// + /// @param config Clock source configuration. + /// @return A future containing a successful Result with the created source. + /// @throws std::runtime_error When awaiting the future if Rust rejects the configuration + /// or setup, initialization, transport, protocol validation, or cancellation fails. + /// @throws std::bad_alloc If allocation fails, including when awaiting the future. + [[nodiscard]] static std::future, CaptureError>> create( + ClockVideoSourceConfig config); + + ~CaptureSource(); + + CaptureSource(const CaptureSource&) = delete; + CaptureSource& operator=(const CaptureSource&) = delete; + CaptureSource(CaptureSource&&) = delete; + CaptureSource& operator=(CaptureSource&&) = delete; + + /// @brief Kind of media this source produces. + /// @return The source kind. + CaptureSourceKind kind() const noexcept; + + /// @brief Declared or discovered stream resolution. + /// @return Width in pixels. + int width() const noexcept; + + /// @brief Declared or discovered stream height. + /// @return Height in pixels. + int height() const noexcept; + + /// @brief Codec produced by the source; encoded sources only. + /// @return The codec, or no value for a pixel source. + std::optional codec() const noexcept; + + /// @brief RTC video source fed by this capture source; use it with + /// LocalVideoTrack::createLocalVideoTrack(). + /// @return The RTC video source. + std::shared_ptr videoSource() const noexcept; + + /// @brief Returns publish options for this track, applying application options. + /// + /// Encoded sources override codec with the source codec, select the + /// PreEncoded encoder backend, and disable simulcast. All other fields are + /// taken from @p options unchanged. Pixel sources preserve every application + /// option; Rust supplies defaults for fields left unset. + /// @param options Application publish options. + /// @return Options with source-required fields applied. + TrackPublishOptions publishOptions(TrackPublishOptions options = {}) const; + + /// @brief Start pumping frames into the RTC video source. + /// + /// @return A successful Result when Rust acknowledges the request. + /// @throws std::runtime_error If the SDK is not initialized, Rust rejects the + /// request, or transport or protocol validation fails. + /// @throws std::bad_alloc If memory allocation fails. + [[nodiscard]] Result start(); + + /// @brief Signal a running capture to stop after the frame in flight. + /// + /// This method does not wait for the pump thread to exit. Call @ref join() + /// to synchronize and retrieve the terminal result. Stopping an + /// already-finished capture is a no-op. + /// + /// @return A successful Result when Rust acknowledges the request. + /// @throws std::runtime_error If the SDK is not initialized, Rust rejects the + /// request, or transport or protocol validation fails. + /// @throws std::bad_alloc If memory allocation fails. + [[nodiscard]] Result stop(); + + /// @brief Consume the running pump and wait asynchronously for it to exit. + /// + /// This method does not signal the pump to stop. Call @ref stop() first when + /// ending an unbounded source. Join may be called only once after a + /// successful @ref start(). + /// + /// @return A future containing statistics on success or a structured pump + /// CaptureError whose kind is supplied by Rust. + /// @throws std::runtime_error When awaiting the future if initialization, + /// transport, protocol validation, or cancellation fails. + /// @throws std::bad_alloc If allocation fails, including when awaiting the future. + [[nodiscard]] std::future> join(); + +private: + friend class livekit::FfiClient; + + CaptureSource(); + + /// @brief Builds the wrapper from the callback payload, adopting its handles. + /// @param owned Rust-owned source handles and metadata. + /// @return The capture source wrapper. + /// @throws std::runtime_error If the source metadata is invalid. + static std::shared_ptr fromOwned(const proto::OwnedCaptureSource& owned); + + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace livekit::capture diff --git a/src/audio_capture_source.cpp b/src/audio_capture_source.cpp new file mode 100644 index 00000000..21ac52fc --- /dev/null +++ b/src/audio_capture_source.cpp @@ -0,0 +1,93 @@ +/* + * Copyright 2026 LiveKit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an “AS IS” BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include + +#include "capture.pb.h" +#include "ffi_client.h" +#include "livekit/capture/audio_source.h" +#include "livekit/ffi_handle.h" + +namespace livekit { + +struct AudioSourceAccess { + static std::shared_ptr adopt(FfiHandle&& handle, int sample_rate, int num_channels) { + return std::shared_ptr(new AudioSource(std::move(handle), sample_rate, num_channels, 0)); + } +}; + +} // namespace livekit + +namespace livekit::capture { + +struct GstreamerAudioCaptureSource::Impl { + FfiHandle handle; + AudioCaptureFormat format; + std::shared_ptr audio_source; +}; + +GstreamerAudioCaptureSource::GstreamerAudioCaptureSource() : impl_(std::make_unique()) {} +GstreamerAudioCaptureSource::~GstreamerAudioCaptureSource() = default; + +std::future, CaptureError>> GstreamerAudioCaptureSource::create( + const GstreamerAudioSourceConfig& config) { + proto::NewAudioCaptureSourceRequest request; + auto* gstreamer = request.mutable_gstreamer(); + gstreamer->set_pipeline(config.pipeline); + gstreamer->set_sample_rate(config.sample_rate); + gstreamer->set_num_channels(config.num_channels); + return FfiClient::instance().newAudioCaptureSourceAsync(std::move(request)); +} + +std::shared_ptr GstreamerAudioCaptureSource::fromOwned( + const proto::OwnedAudioCaptureSource& owned) { + FfiHandle capture_handle(static_cast(owned.handle().id())); + FfiHandle audio_handle(static_cast(owned.info().audio_source().handle().id())); + const auto& info = owned.info(); + constexpr auto kMaxInt = static_cast(std::numeric_limits::max()); + if (!owned.IsInitialized() || info.sample_rate() == 0 || info.num_channels() == 0 || + info.samples_per_channel() == 0 || info.sample_rate() > kMaxInt || info.num_channels() > kMaxInt) { + throw std::runtime_error("GstreamerAudioCaptureSource::create: invalid audio capture source info"); + } + + std::shared_ptr source(new GstreamerAudioCaptureSource()); + source->impl_->handle = std::move(capture_handle); + source->impl_->format = {info.sample_rate(), info.num_channels(), info.samples_per_channel()}; + source->impl_->audio_source = AudioSourceAccess::adopt(std::move(audio_handle), static_cast(info.sample_rate()), + static_cast(info.num_channels())); + return source; +} + +AudioCaptureFormat GstreamerAudioCaptureSource::format() const noexcept { return impl_->format; } + +std::shared_ptr GstreamerAudioCaptureSource::audioSource() const noexcept { return impl_->audio_source; } + +Result GstreamerAudioCaptureSource::start() { + return FfiClient::instance().controlCapture(impl_->handle.get(), true); +} + +Result GstreamerAudioCaptureSource::stop() { + return FfiClient::instance().controlCapture(impl_->handle.get(), false); +} + +std::future> GstreamerAudioCaptureSource::join() { + return FfiClient::instance().joinCaptureAsync(impl_->handle.get()); +} + +} // namespace livekit::capture diff --git a/src/audio_source.cpp b/src/audio_source.cpp index c3416b8b..9b98441f 100644 --- a/src/audio_source.cpp +++ b/src/audio_source.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include "audio_frame.pb.h" #include "ffi.pb.h" @@ -56,6 +57,12 @@ AudioSource::AudioSource(int sample_rate, int num_channels, int queue_size_ms) handle_ = FfiHandle(static_cast(source_info.handle().id())); } +AudioSource::AudioSource(FfiHandle handle, int sample_rate, int num_channels, int queue_size_ms) noexcept + : sample_rate_(sample_rate), + num_channels_(num_channels), + queue_size_ms_(queue_size_ms), + handle_(std::move(handle)) {} + double AudioSource::queuedDuration() const noexcept { if (last_capture_ == 0.0) { return 0.0; diff --git a/src/capture_source.cpp b/src/capture_source.cpp index 8268c461..f39697f9 100644 --- a/src/capture_source.cpp +++ b/src/capture_source.cpp @@ -14,8 +14,6 @@ * limitations under the License. */ -#include "livekit/capture/capture_source.h" - #include #include #include @@ -24,6 +22,7 @@ #include "capture_source_internal.h" #include "ffi.pb.h" #include "ffi_client.h" +#include "livekit/capture/video_source.h" namespace livekit::capture { diff --git a/src/capture_source_internal.h b/src/capture_source_internal.h index 3a9e91ff..a6e57d70 100644 --- a/src/capture_source_internal.h +++ b/src/capture_source_internal.h @@ -20,7 +20,7 @@ #include #include "capture.pb.h" -#include "livekit/capture/capture_source.h" +#include "livekit/capture/video_source.h" #include "livekit/visibility.h" namespace livekit { diff --git a/src/ffi_client.cpp b/src/ffi_client.cpp index 0854def8..df786a21 100644 --- a/src/ffi_client.cpp +++ b/src/ffi_client.cpp @@ -37,7 +37,7 @@ #include "room_proto_converter.h" #if LIVEKIT_CAPTURE_ENABLED -#include "livekit/capture/capture_source.h" +#include "livekit/capture/video_source.h" #endif namespace livekit { @@ -120,6 +120,8 @@ std::optional ExtractAsyncId(const proto::FfiEvent& event) { return event.new_capture_source().async_id(); case E::kJoinCapture: return event.join_capture().async_id(); + case E::kNewAudioCaptureSource: + return event.new_audio_capture_source().async_id(); // low-level data stream callbacks case E::kSendStreamHeader: @@ -470,13 +472,13 @@ bool FfiClient::cancelPendingByAsyncId(AsyncId async_id) { template std::future FfiClient::registerAsync(AsyncId async_id, std::function match, std::function&)> handler, - std::exception_ptr cancellation_error) { + const std::exception_ptr& cancellation_error) { auto pending = std::make_unique>(); pending->async_id = async_id; auto fut = pending->promise.get_future(); pending->match = std::move(match); pending->handler = std::move(handler); - pending->cancellation_error = std::move(cancellation_error); + pending->cancellation_error = cancellation_error; { const std::scoped_lock guard(lock_); pending_by_id_.emplace(async_id, std::move(pending)); @@ -1385,7 +1387,10 @@ JoinOutcome captureJoinResult(const proto::JoinCaptureCallback& callback) { } const auto exit = finished.exit() == proto::CAPTURE_EXIT_END_OF_STREAM ? capture::CaptureExit::EndOfStream : capture::CaptureExit::Stopped; - return JoinOutcome::success(capture::CaptureResult{finished.frames_captured(), exit}); + capture::CaptureResult result; + result.frames_captured = finished.frames_captured(); + result.exit = exit; + return JoinOutcome::success(result); } } // namespace @@ -1501,6 +1506,55 @@ std::future> FfiClient::jo return future; } +std::future, capture::CaptureError>> +FfiClient::newAudioCaptureSourceAsync(proto::NewAudioCaptureSourceRequest request, + const std::function& send) { + using Outcome = Result, capture::CaptureError>; + constexpr const char* kOperation = "GstreamerAudioCaptureSource::create: "; + if (!isInitialized()) { + return captureExceptionFuture(std::string(kOperation) + "LiveKit is not initialized"); + } + const AsyncId async_id = generateAsyncId(); + auto future = registerAsync( + async_id, + [async_id](const proto::FfiEvent& event) { + return event.has_new_audio_capture_source() && event.new_audio_capture_source().async_id() == async_id; + }, + [](const proto::FfiEvent& event, std::promise& promise) { + try { + const auto& callback = event.new_audio_capture_source(); + if (callback.has_error()) { + throw std::runtime_error("GstreamerAudioCaptureSource::create: Rust source setup failed: " + + callback.error()); + } + if (!callback.has_source()) { + throw std::runtime_error("GstreamerAudioCaptureSource::create: callback is missing source or error"); + } + promise.set_value(Outcome::success(capture::GstreamerAudioCaptureSource::fromOwned(callback.source()))); + } catch (...) { + promise.set_exception(std::current_exception()); + } + }, + std::make_exception_ptr(std::runtime_error(std::string(kOperation) + "operation cancelled during SDK shutdown"))); + request.set_request_async_id(async_id); + proto::FfiRequest ffi_request; + ffi_request.mutable_new_audio_capture_source()->CopyFrom(request); + try { + const auto response = send ? send(ffi_request) : sendRequest(ffi_request); + if (!response.has_new_audio_capture_source() || !response.new_audio_capture_source().IsInitialized() || + response.new_audio_capture_source().async_id() != async_id) { + throw std::runtime_error("invalid new_audio_capture_source acknowledgement (missing or mismatched async_id)"); + } + } catch (const std::runtime_error& error) { + cancelPendingByAsyncId(async_id); + return captureExceptionFuture(std::string(kOperation) + error.what()); + } catch (...) { + cancelPendingByAsyncId(async_id); + throw; + } + return future; +} + #endif } // namespace livekit diff --git a/src/ffi_client.h b/src/ffi_client.h index 3183786a..3c6bdc76 100644 --- a/src/ffi_client.h +++ b/src/ffi_client.h @@ -43,7 +43,8 @@ #include "room.pb.h" #if LIVEKIT_CAPTURE_ENABLED -#include "livekit/capture/capture_source.h" +#include "livekit/capture/audio_source.h" +#include "livekit/capture/video_source.h" #endif namespace livekit { @@ -186,6 +187,9 @@ class LIVEKIT_INTERNAL_API FfiClient { // result rather than subscribing to an unsolicited completion event. std::future> joinCaptureAsync( std::uint64_t capture_handle, const std::function& send = {}); + std::future, capture::CaptureError>> + newAudioCaptureSourceAsync(proto::NewAudioCaptureSourceRequest request, + const std::function& send = {}); #endif private: @@ -256,7 +260,7 @@ class LIVEKIT_INTERNAL_API FfiClient { template std::future registerAsync(AsyncId async_id, std::function match, std::function&)> handler, - std::exception_ptr cancellation_error = {}); + const std::exception_ptr& cancellation_error = {}); // Generate a unique client-side async ID for request correlation AsyncId generateAsyncId(); diff --git a/src/tests/CMakeLists.txt b/src/tests/CMakeLists.txt index 75109d87..134b1de7 100644 --- a/src/tests/CMakeLists.txt +++ b/src/tests/CMakeLists.txt @@ -54,6 +54,9 @@ endif() enable_testing() include(GoogleTest) +# Standalone interactive testers; these are not registered with CTest. +add_subdirectory(manual) + # ============================================================================ # Benchmark Utilities (shared by all test types) # ============================================================================ @@ -74,6 +77,7 @@ file(GLOB UNIT_TEST_SOURCES if(NOT LIVEKIT_CAPTURE_ENABLED) list(REMOVE_ITEM UNIT_TEST_SOURCES + "${CMAKE_CURRENT_SOURCE_DIR}/unit/test_audio_capture.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/unit/test_capture_source.cpp") endif() diff --git a/src/tests/integration/test_capture_source.cpp b/src/tests/integration/test_capture_source.cpp index c2135bc5..eefec807 100644 --- a/src/tests/integration/test_capture_source.cpp +++ b/src/tests/integration/test_capture_source.cpp @@ -22,7 +22,7 @@ /// All of them require the Rust FFI built with capture-source support /// (-DLIVEKIT_CAPTURE=all); otherwise they skip. -#include +#include #include #include diff --git a/src/tests/manual/CMakeLists.txt b/src/tests/manual/CMakeLists.txt new file mode 100644 index 00000000..cac4df0b --- /dev/null +++ b/src/tests/manual/CMakeLists.txt @@ -0,0 +1,24 @@ +# Copyright 2026 LiveKit +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Interactive hardware tester, built with the test binaries but never run by CTest. +if(APPLE AND "capture-gstreamer" IN_LIST _lk_capture_features) + add_executable(capture_audio capture_audio.cpp) + target_compile_features(capture_audio PRIVATE cxx_std_17) + target_link_libraries(capture_audio PRIVATE livekit) + add_custom_command(TARGET capture_audio POST_BUILD + COMMAND ${CMAKE_COMMAND} -E copy_if_different + $ $ + COMMAND ${CMAKE_COMMAND} -E copy_if_different + "$/liblivekit_ffi.dylib" $ + COMMENT "Copying dylibs to manual capture tester directory" + ) +endif() diff --git a/src/tests/manual/capture_audio.cpp b/src/tests/manual/capture_audio.cpp new file mode 100644 index 00000000..d4d1d707 --- /dev/null +++ b/src/tests/manual/capture_audio.cpp @@ -0,0 +1,177 @@ +/* + * Copyright 2026 LiveKit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an “AS IS” BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +using namespace std::chrono_literals; + +volatile std::sig_atomic_t interrupted = 0; + +void onSignal(int /*signal*/) { interrupted = 1; } + +struct Options { + int seconds = 10; + bool publish = false; + std::string pipeline = + "osxaudiosrc ! audioconvert ! audioresample ! " + "audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! " + "appsink name=lk_audio_appsink"; +}; + +void printUsage() { + std::cout << "Usage: capture_audio [--seconds 1..600] [--publish] [--pipeline DESCRIPTION]\n" + "Captures the default macOS microphone at 48 kHz, mono, S16LE.\n" + "Stops after 10 seconds by default, or on Ctrl+C.\n" + "--publish uses LIVEKIT_URL and LIVEKIT_TOKEN (or LIVEKIT_TOKEN_A).\n" + "--pipeline overrides the microphone pipeline, e.g. for audiotestsrc.\n"; +} + +Options parseOptions(int argc, char* argv[]) { + Options options; + for (int index = 1; index < argc; ++index) { + const std::string_view argument(argv[index]); + if (argument == "--publish") { + options.publish = true; + } else if ((argument == "--seconds" || argument == "--pipeline") && index + 1 < argc) { + const std::string_view value(argv[++index]); + if (argument == "--pipeline") { + if (value.empty()) { + throw std::invalid_argument("--pipeline must not be empty"); + } + options.pipeline = value; + } else { + const auto parsed = std::from_chars(value.data(), value.data() + value.size(), options.seconds); + if (parsed.ec != std::errc{} || parsed.ptr != value.data() + value.size() || options.seconds < 1 || + options.seconds > 600) { + throw std::invalid_argument("--seconds must be an integer from 1 to 600"); + } + } + } else { + throw std::invalid_argument("Unknown option or missing value: " + std::string(argument)); + } + } + return options; +} + +void publish(livekit::Room& room, const std::shared_ptr& track) { + const char* url = std::getenv("LIVEKIT_URL"); + const char* token = std::getenv("LIVEKIT_TOKEN"); + if (token == nullptr || *token == '\0') { + token = std::getenv("LIVEKIT_TOKEN_A"); + } + if (url == nullptr || *url == '\0' || token == nullptr || *token == '\0') { + throw std::runtime_error("--publish requires LIVEKIT_URL and LIVEKIT_TOKEN (or LIVEKIT_TOKEN_A)"); + } + if (!room.connect(url, token, {})) { + throw std::runtime_error("Could not connect to the LiveKit room"); + } + const auto participant = room.localParticipant().lock(); + if (!participant) { + throw std::runtime_error("Connected room has no local participant"); + } + livekit::TrackPublishOptions options; + options.source = livekit::TrackSource::SOURCE_MICROPHONE; + participant->publishTrack(track, options); + std::cout << "Published capture-audio; join the same room with another participant to listen.\n"; +} + +int capture(const Options& options) { + livekit::capture::GstreamerAudioSourceConfig config; + config.pipeline = options.pipeline; + std::cout << "Opening audio pipeline...\n"; + auto pending = livekit::capture::GstreamerAudioCaptureSource::create(config); + if (pending.wait_for(10s) != std::future_status::ready) { + throw std::runtime_error("Audio source creation timed out"); + } + const auto created = pending.get(); + if (!created) { + throw std::runtime_error(created.error().message); + } + const auto& source = created.value(); + const auto track = livekit::LocalAudioTrack::createLocalAudioTrack("capture-audio", source->audioSource()); + livekit::Room room; + if (options.publish) { + publish(room, track); + } + const auto format = source->format(); + std::cout << "Capturing " << format.sample_rate << " Hz, " << format.num_channels << " channel(s), " + << format.samples_per_channel << " samples/frame for up to " << options.seconds + << " seconds. Press Ctrl+C to stop.\n"; + source->start().value(); + auto finished = source->join(); + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(options.seconds); + while (finished.wait_for(100ms) != std::future_status::ready) { + if (interrupted != 0 || std::chrono::steady_clock::now() >= deadline) { + source->stop().value(); + break; + } + } + if (finished.wait_for(5s) != std::future_status::ready) { + throw std::runtime_error("Audio capture shutdown timed out"); + } + const auto result = finished.get(); + if (!result) { + throw std::runtime_error(result.error().message); + } + const auto& stats = result.value(); + std::cout << "Captured " << stats.frames_captured << " frames (" << static_cast(stats.frames_captured) / 100.0 + << " seconds of audio).\n"; + if (stats.frames_captured == 0) { + throw std::runtime_error("No audio frames captured; check microphone permission and the selected input device"); + } + return 0; +} + +} // namespace + +int main(int argc, char* argv[]) { + if (argc == 2 && (std::string_view(argv[1]) == "--help" || std::string_view(argv[1]) == "-h")) { + printUsage(); + return 0; + } + int exit_code = 1; + bool initialized = false; + try { + const auto options = parseOptions(argc, argv); + if (std::signal(SIGINT, onSignal) == SIG_ERR || std::signal(SIGTERM, onSignal) == SIG_ERR) { + throw std::runtime_error("Could not install signal handlers"); + } + initialized = livekit::initialize(livekit::LogLevel::Warn); + if (!initialized) { + throw std::runtime_error("Could not initialize LiveKit"); + } + exit_code = capture(options); + } catch (const std::exception& error) { + std::cerr << "capture_audio: " << error.what() << '\n'; + } + if (initialized) { + livekit::shutdown(); + } + return exit_code; +} diff --git a/src/tests/packaging/CMakeLists.txt b/src/tests/packaging/CMakeLists.txt index c76f9768..174c2a90 100644 --- a/src/tests/packaging/CMakeLists.txt +++ b/src/tests/packaging/CMakeLists.txt @@ -103,7 +103,10 @@ if(EXISTS "${_sdk_include_dir}/livekit/capture_source.h") message(FATAL_ERROR "Installed SDK contains the obsolete flat capture header") endif() if(_expected_capture) - if(NOT EXISTS "${_sdk_include_dir}/livekit/capture/capture_source.h") + if(NOT EXISTS "${_sdk_include_dir}/livekit/capture/capture_source.h" OR + NOT EXISTS "${_sdk_include_dir}/livekit/capture/video_source.h" OR + NOT EXISTS "${_sdk_include_dir}/livekit/capture/audio_source.h" OR + NOT EXISTS "${_sdk_include_dir}/livekit/capture/capture_types.h") message(FATAL_ERROR "Capture SDK is missing the capture public API directory") endif() elseif(EXISTS "${_sdk_include_dir}/livekit/capture") @@ -114,17 +117,19 @@ endif() include(CheckCXXSourceCompiles) set(CMAKE_REQUIRED_INCLUDES "${_sdk_include_dir}") set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) -unset(_capture_header_compiles CACHE) -check_cxx_source_compiles("#include -int main() { return 0; }" - _capture_header_compiles) +foreach(_header IN ITEMS capture_source video_source audio_source capture_types) + set(_check "_capture_${_header}_compiles") + unset(${_check} CACHE) + check_cxx_source_compiles("#include +int main() { return 0; }" ${_check}) + if(_expected_capture AND NOT ${_check}) + message(FATAL_ERROR "Capture SDK must allow including livekit/capture/${_header}.h") + elseif(NOT _expected_capture AND ${_check}) + message(FATAL_ERROR "Standard SDK must reject including livekit/capture/${_header}.h") + endif() +endforeach() unset(CMAKE_REQUIRED_INCLUDES) unset(CMAKE_TRY_COMPILE_TARGET_TYPE) -if(_expected_capture AND NOT _capture_header_compiles) - message(FATAL_ERROR "Capture SDK must allow including livekit/capture/capture_source.h") -elseif(NOT _expected_capture AND _capture_header_compiles) - message(FATAL_ERROR "Standard SDK must reject including livekit/capture/capture_source.h") -endif() add_executable(livekit_capture_package_test test_capture_bundle.cpp) target_compile_features(livekit_capture_package_test PRIVATE cxx_std_17) diff --git a/src/tests/packaging/test_capture_bundle.cpp b/src/tests/packaging/test_capture_bundle.cpp index a4a469e3..3e130930 100644 --- a/src/tests/packaging/test_capture_bundle.cpp +++ b/src/tests/packaging/test_capture_bundle.cpp @@ -87,6 +87,31 @@ bool checkSources() { clock.framerate_fps = 30; success = checkSource("clock", clock, LIVEKIT_EXPECT_CLOCK != 0, livekit::capture::CaptureSourceKind::Pixel) && success; + + if (LIVEKIT_EXPECT_GSTREAMER) { + try { + livekit::capture::GstreamerAudioSourceConfig audio; + audio.pipeline = + "audiotestsrc samplesperbuffer=480 num-buffers=2 ! " + "audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! " + "appsink name=lk_audio_appsink"; + auto created = livekit::capture::GstreamerAudioCaptureSource::create(audio).get(); + if (!created || !created.value()->audioSource() || created.value()->format().samples_per_channel != 480 || + !created.value()->start()) { + std::cerr << "gstreamer audio capture: unexpected source metadata\n"; + success = false; + } else { + const auto finished = created.value()->join().get(); + if (!finished || finished.value().frames_captured == 0) { + std::cerr << "gstreamer audio capture: missing terminal statistics\n"; + success = false; + } + } + } catch (const std::exception& error) { + std::cerr << "gstreamer audio: " << error.what() << '\n'; + success = false; + } + } return success; } diff --git a/src/tests/unit/test_audio_capture.cpp b/src/tests/unit/test_audio_capture.cpp new file mode 100644 index 00000000..bfbee4d5 --- /dev/null +++ b/src/tests/unit/test_audio_capture.cpp @@ -0,0 +1,88 @@ +/* + * Copyright 2026 LiveKit + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an “AS IS” BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include + +#include + +namespace livekit::test { +namespace { + +using namespace std::chrono_literals; + +class GstreamerAudioTest : public ::testing::Test { +protected: + void SetUp() override { ASSERT_TRUE(initialize(LogLevel::Warn)); } + void TearDown() override { shutdown(); } +}; + +TEST_F(GstreamerAudioTest, CapturesSyntheticPcmAndReportsFormatAndStats) { + capture::GstreamerAudioSourceConfig config; + config.pipeline = + "audiotestsrc wave=sine samplesperbuffer=480 num-buffers=8 ! " + "audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! " + "appsink name=lk_audio_appsink"; + + auto pending = capture::GstreamerAudioCaptureSource::create(config); + ASSERT_EQ(pending.wait_for(10s), std::future_status::ready); + const auto created = pending.get(); + ASSERT_TRUE(created.ok()) << created.error().message; + const auto source = created.value(); + ASSERT_NE(source, nullptr); + EXPECT_EQ(source->format().sample_rate, 48000u); + EXPECT_EQ(source->format().num_channels, 1u); + EXPECT_EQ(source->format().samples_per_channel, 480u); + ASSERT_NE(source->audioSource(), nullptr); + EXPECT_EQ(source->audioSource()->sampleRate(), 48000); + EXPECT_EQ(source->audioSource()->numChannels(), 1); + + ASSERT_TRUE(source->start().ok()); + auto joined = source->join(); + ASSERT_EQ(joined.wait_for(10s), std::future_status::ready); + const auto finished = joined.get(); + ASSERT_TRUE(finished.ok()) << finished.error().message; + EXPECT_EQ(finished.value().exit, capture::CaptureExit::EndOfStream); + EXPECT_EQ(finished.value().frames_captured, 8u); +} + +TEST_F(GstreamerAudioTest, RechunksStereoBuffersIntoTenMsFrames) { + capture::GstreamerAudioSourceConfig config; + config.num_channels = 2; + config.pipeline = + "audiotestsrc samplesperbuffer=960 num-buffers=4 ! " + "audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2 ! " + "appsink name=lk_audio_appsink"; + + auto pending = capture::GstreamerAudioCaptureSource::create(config); + ASSERT_EQ(pending.wait_for(10s), std::future_status::ready); + const auto created = pending.get(); + ASSERT_TRUE(created.ok()) << created.error().message; + const auto source = created.value(); + EXPECT_EQ(source->format().num_channels, 2u); + EXPECT_EQ(source->format().samples_per_channel, 480u); + ASSERT_TRUE(source->start().ok()); + auto joined = source->join(); + ASSERT_EQ(joined.wait_for(10s), std::future_status::ready); + const auto finished = joined.get(); + ASSERT_TRUE(finished.ok()) << finished.error().message; + EXPECT_EQ(finished.value().exit, capture::CaptureExit::EndOfStream); + EXPECT_EQ(finished.value().frames_captured, 8u); +} + +} // namespace +} // namespace livekit::test diff --git a/src/tests/unit/test_capture_source.cpp b/src/tests/unit/test_capture_source.cpp index 2f4d531c..c304dcd3 100644 --- a/src/tests/unit/test_capture_source.cpp +++ b/src/tests/unit/test_capture_source.cpp @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include