Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,17 @@ Be sure to update the directory layout in this file if the directory layout chan
|------|-------------|
| `include/livekit/` | Public API headers (what SDK consumers include) |
| `include/livekit/capture/` | Capture API headers; installed only for capture-enabled SDKs |
| `include/livekit/capture/capture_source.h` | Compatibility umbrella exposing all audio/video capture APIs |
| `include/livekit/capture/audio_source.h` | Audio capture APIs and configurations |
| `include/livekit/capture/video_source.h` | Video capture API and configurations |
| `include/livekit/capture/capture_types.h` | Shared capture results, errors, and exit reasons |
| `src/` | Implementation files and internal-only headers (`ffi_client.h`, `lk_log.h`, etc.) |
| `src/tests/` | Google Test integration and stress tests |
| `src/tests/manual/` | Interactive hardware testers built with the test targets; not registered with CTest |
| `examples/` | In-tree example applications |
| `client-sdk-rust/` | Git submodule holding the Rust core of the SDK|
| `client-sdk-rust/livekit-capture/src/audio/` | PCM audio sources and pump |
| `client-sdk-rust/livekit-capture/src/video/` | Pixel and encoded video pumps, sources, and rendering |
| `cpp-tools/` | Git submodule holding shared LiveKit C++ engineering guidance, clang-format / clang-tidy configs, scripts, docs, and CI workflow |
| `client-sdk-rust/livekit-ffi/protocol/*.proto` | FFI contract (protobuf definitions, read-only reference) |
| `cmake/` | Build helpers (`protobuf.cmake`, `spdlog.cmake`, `nlohmann_json.cmake`, `LiveKitConfig.cmake.in`) |
Expand Down
5 changes: 4 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,10 @@ add_library(livekit SHARED
)
# Conditionally compile capture sources
if(LIVEKIT_CAPTURE_ENABLED)
target_sources(livekit PRIVATE src/capture_source.cpp)
target_sources(livekit PRIVATE
src/audio_capture_source.cpp
src/capture_source.cpp
)
endif()

# Symbol visibility: keep liblivekit's exported ABI restricted to the public
Expand Down
2 changes: 1 addition & 1 deletion client-sdk-rust
Submodule client-sdk-rust updated 167 files
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Additional documentation for the SDK.

- [Building](building.md) — prerequisites, build scripts, Linux build
container, CMake presets, vcpkg, integration into your CMake project, troubleshooting.
- [Capture sources](capture-sources.md) — pattern, clock, and GStreamer sources;
- [Capture sources](capture-sources.md) — video sources plus GStreamer audio capture;
build features, release bundles, and publishing. Use capture to stream video through LiveKit
without maintaining your own video pipeline.
- [Authentication](authentication.md) — generating tokens, token source types
Expand Down
13 changes: 8 additions & 5 deletions docs/building.md
Original file line number Diff line number Diff line change
Expand Up @@ -534,11 +534,14 @@ sources are available. Check `cargo_features` in `share/livekit/build-info.json`
for the selected backends. Creating a source omitted from the SDK build fails
through the returned future with `std::runtime_error`.
Capture-enabled bundles install the public API under `include/livekit/capture/`;
standard bundles omit that directory. Use `<livekit/capture/capture_source.h>`
for direct inclusion. The source-tree header also rejects use in a standard
build. `share/livekit/build-info.json` in an installed bundle describes the same
build features; neither value tests whether a GPU or a requested GStreamer plugin
is available at runtime.
standard bundles omit that directory. Use `<livekit/capture/video_source.h>`
for video and `<livekit/capture/audio_source.h>` for GStreamer PCM capture.
`<livekit/capture/capture_source.h>` remains the umbrella that includes both
headers and their shared types from `<livekit/capture/capture_types.h>`.
The source-tree audio and video headers also reject use in a standard build.
`share/livekit/build-info.json` in an installed bundle describes the same build
features; neither value tests whether a GPU or a requested GStreamer plugin is
available at runtime.

Release downloads with `-capture` in their names include these features, for
example `livekit-sdk-macos-arm64-capture-<version>.tar.gz`. Standard release
Expand Down
58 changes: 53 additions & 5 deletions docs/capture-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@

> **NOTE:** LiveKit Capture is not officially supported on Intel-based Macs.

`livekit::capture::CaptureSource` connects a Rust video producer to a regular C++ `VideoSource`.
The capture pump runs in Rust, so C++ does not send one FFI request per frame.
The SDK supports three source configurations:
The capture APIs connect Rust media producers to regular C++ track
types. Media frames stay in Rust, so C++ does not send one FFI request per
frame. The SDK supports video capture and GStreamer PCM audio capture.

| Configuration | Output | Runtime requirements |
| --- | --- | --- |
| `PatternVideoSourceConfig` | GPU-rendered gradient or LiveKit logo | Compatible GPU adapter |
| `ClockVideoSourceConfig` | GPU-rendered local time with millisecond precision | Compatible GPU adapter |
| `GstreamerVideoSourceConfig` | Pre-encoded video from a GStreamer pipeline | GStreamer runtime and the pipeline's plugins |
| `GstreamerAudioSourceConfig` | PCM S16LE audio into an `AudioSource` | GStreamer runtime and the pipeline's plugins |

A custom SDK build can include all sources or a subset selected with
`--capture=pattern,clock` or `--capture=gstreamer`. Release `-capture` bundles
Expand All @@ -29,8 +30,18 @@ requirements, see [Building — Capture sources](building.md#capture-sources).
Capture types live in the `livekit::capture` namespace. Capture-enabled bundles
install the API in `include/livekit/capture/`:

| Header | Contents |
| --- | --- |
| `capture_source.h` | Umbrella for all capture APIs; existing callers can keep this include |
| `audio_source.h` | Audio capture and its configurations |
| `video_source.h` | Video capture and its configurations |
| `capture_types.h` | Shared capture results, errors, and exit reasons |

Include the umbrella for the full API, or the audio and video headers directly:

```cpp
#include <livekit/capture/capture_source.h>
#include <livekit/capture/video_source.h>
#include <livekit/capture/audio_source.h>
```

Standard SDK bundles omit the `capture/` directory, and their other installed
Expand All @@ -52,7 +63,7 @@ continues, and the source is released when the callback arrives.

```cpp
#include <livekit/livekit.h>
#include <livekit/capture/capture_source.h>
#include <livekit/capture/video_source.h>

// The SDK is initialized, and room is connected.
try {
Expand Down Expand Up @@ -91,6 +102,43 @@ thread to exit, and reports the exit reason and frame count. For unbounded
sources, call `stop()` before `join()`. Destroying the source also stops
capture.

## GStreamer audio capture

`GstreamerAudioSourceConfig` follows the naming of `GstreamerVideoSourceConfig`.
Both source APIs create asynchronously and use
the same `start()`, `stop()`, and `join()` lifecycle, `CaptureResult`, and
`CaptureError` types.

`GstreamerAudioCaptureSource` asynchronously creates a fixed-format PCM S16LE
pipeline and owns the `AudioSource` fed by its Rust capture pump. Publish that
source through a normal `LocalAudioTrack`, then use the same start, stop, and
join lifecycle as video capture:

```cpp
livekit::capture::GstreamerAudioSourceConfig config;
config.pipeline = "audiotestsrc ! audioconvert ! "
"audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! "
"appsink name=lk_audio_appsink";
config.sample_rate = 48000;
config.num_channels = 1;

auto capture = livekit::capture::GstreamerAudioCaptureSource::create(config).get().value();
auto track = livekit::LocalAudioTrack::createLocalAudioTrack("gstreamer", capture->audioSource());
participant->publishTrack(track);
capture->start().value();

capture->stop().value();
auto result = capture->join().get().value();
// result contains frames_captured (one frame = 10 ms of audio) and exit.
```

The negotiated `format()` reports sample rate, channel count, and samples per
channel. Rust validates that pipeline caps match the requested fixed format.
Unlike video resolution and codec discovery, audio requires an explicit fixed
format (48 kHz and mono by default). Audio frames are always 10 ms. The pipeline
must contain the named appsink. Audio capture does not expose video's `publishOptions()` helper;
publish it with normal audio track options.

## GStreamer encoded pipelines

Provide `appsink name=lk_appsink`, or leave exactly one encoded video source pad
Expand Down
41 changes: 39 additions & 2 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,41 @@ ctest --output-on-failure

__Note:__ The integration and stress tests require tokens and a running LiveKit server. See the section below for details.

## Manual microphone capture (macOS)

`src/tests/manual/capture_audio.cpp` is a standalone GStreamer microphone tester.
It is built with the test binaries on macOS when the GStreamer capture backend
is enabled, and is not run by CTest:

```bash
./build.sh release-tests --capture=all
./build-release/bin/capture_audio --seconds 10
```

The default pipeline uses `osxaudiosrc` for the default macOS input device and
converts to 48 kHz mono S16LE PCM. Allow microphone access for your terminal in
**System Settings > Privacy & Security > Microphone** if prompted. GStreamer and
the pipeline's plugins must be installed; see [Capture sources](capture-sources.md).
Capture stops at the requested duration (1–600 seconds) or Ctrl+C and prints the frame count
and captured-audio duration.

To publish the microphone to a LiveKit room, start your server, then run:

```bash
source scripts/set-test-tokens.sh
./build-release/bin/capture_audio --publish --seconds 30
```

`--publish` reads `LIVEKIT_URL` and `LIVEKIT_TOKEN`, falling back to
`LIVEKIT_TOKEN_A`. Join the same room as another participant to listen.

Use `--pipeline` to select a different device or test without a microphone:

```bash
./build-release/bin/capture_audio --seconds 1 --pipeline \
'audiotestsrc is-live=true ! audioconvert ! audioresample ! audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=1 ! appsink name=lk_audio_appsink'
```

## Running a local LiveKit server for tests

The integration and stress suites need a running LiveKit server. The easiest
Expand Down Expand Up @@ -116,8 +151,10 @@ means none. Explicit source selections determine the expected capture flag.
The test creates a GStreamer VP8 source when that source is selected. It also
requires each omitted backend to report the feature-disabled error. Configuration
checks that `include/livekit/capture/` is present only in capture bundles and
that direct inclusion of `livekit/capture/capture_source.h` succeeds only in
those bundles. The obsolete flat capture header must be absent. Both variants must
that each capture header compiles independently only in those bundles,
including `video_source.h` and the compatibility umbrella `capture_source.h`.
The umbrella is also used by the runtime packaging test to exercise both audio
and video APIs. The obsolete flat capture header must be absent. Both variants must
still compile with `livekit/livekit.h`. It needs no LiveKit server.
GStreamer-enabled runs require the pipeline's plugins. Set
`LIVEKIT_VERIFY_PIXEL_SOURCES=ON` to also require successful pattern and clock
Expand Down
5 changes: 5 additions & 0 deletions include/livekit/audio_source.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ class FfiResponse;
} // namespace proto

class FfiClient;
struct AudioSourceAccess;

/// Represents a real-time audio source with an internal audio queue.
class LIVEKIT_API AudioSource {
Expand Down Expand Up @@ -131,6 +132,10 @@ class LIVEKIT_API AudioSource {
void captureFrame(const AudioFrame& frame, int timeout_ms = 20);

private:
friend struct AudioSourceAccess;

AudioSource(FfiHandle handle, int sample_rate, int num_channels, int queue_size_ms) noexcept;

// Internal helper to reset the local queue tracking (like _release_waiter).
void resetQueueTracking() noexcept;

Expand Down
146 changes: 146 additions & 0 deletions include/livekit/capture/audio_source.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
/*
* Copyright 2026 LiveKit
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an “AS IS” BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#pragma once

#include "livekit/build.h"

#if !LIVEKIT_CAPTURE_ENABLED && !defined(LIVEKIT_BUILDING_SDK)
#error "Audio capture requires a capture-enabled LiveKit SDK; rebuild with --capture=gstreamer."
#endif

#include <cstdint>
#include <future>
#include <memory>
#include <string>

#include "livekit/audio_source.h"
#include "livekit/capture/capture_types.h"
#include "livekit/result.h"
#include "livekit/visibility.h"

namespace livekit {

class FfiClient;

namespace proto {
class OwnedAudioCaptureSource;
} // namespace proto

} // namespace livekit

namespace livekit::capture {

/// @brief Fixed PCM format produced in 10 ms frames by an audio capture source.
struct AudioCaptureFormat {
/// @brief Sample rate in Hz.
std::uint32_t sample_rate = 0;

/// @brief Number of interleaved channels.
std::uint32_t num_channels = 0;

/// @brief Number of samples per channel in each captured frame.
std::uint32_t samples_per_channel = 0;
};

/// @brief Configuration for PCM S16LE audio ingest in 10 ms frames from GStreamer.
struct GstreamerAudioSourceConfig {
/// @brief GStreamer launch description.
///
/// @note Must contain `appsink name=lk_audio_appsink`.
std::string pipeline;

/// @brief Output sample rate in Hz.
std::uint32_t sample_rate = 48000;

/// @brief Number of interleaved output channels.
std::uint32_t num_channels = 1;
};

/// @brief GStreamer audio capture source backed by a Rust capture pump.
///
/// Creation builds the pipeline asynchronously. The object owns both the
/// capture pump and the returned @ref AudioSource. Keep it alive while a local
/// audio track uses that source. This type is not safe for concurrent lifecycle
/// calls.
class LIVEKIT_API GstreamerAudioCaptureSource {
public:
/// @brief Create a GStreamer audio capture source asynchronously.
///
/// Completes asynchronously after Rust initializes the producer and RTC
/// source. Keep the future to overlap setup with other work, or await it
/// immediately. Destroying the future does not wait for creation; an
/// abandoned result is released when creation completes.
///
/// @param config Pipeline and fixed PCM output format.
/// @return Future containing the created source on success.
/// @throws std::runtime_error When awaiting the future if setup, transport,
/// protocol validation, or SDK shutdown fails.
/// @throws std::bad_alloc If allocation fails.
[[nodiscard]] static std::future<Result<std::shared_ptr<GstreamerAudioCaptureSource>, CaptureError>> create(
const GstreamerAudioSourceConfig& config);

~GstreamerAudioCaptureSource();

GstreamerAudioCaptureSource(const GstreamerAudioCaptureSource&) = delete;
GstreamerAudioCaptureSource& operator=(const GstreamerAudioCaptureSource&) = delete;
GstreamerAudioCaptureSource(GstreamerAudioCaptureSource&&) = delete;
GstreamerAudioCaptureSource& operator=(GstreamerAudioCaptureSource&&) = delete;

/// @brief Negotiated fixed PCM format.
/// @return Audio format reported by Rust.
AudioCaptureFormat format() const noexcept;

/// @brief RTC audio source fed by the capture pump.
/// @return Owned audio source suitable for `LocalAudioTrack::createLocalAudioTrack()`.
std::shared_ptr<AudioSource> audioSource() const noexcept;

/// @brief Start pumping pipeline audio into the RTC source.
/// @return Success when Rust acknowledges the request.
/// @throws std::runtime_error If the request cannot be completed.
[[nodiscard]] Result<void, CaptureError> start();

/// @brief Signal capture to stop without waiting for cleanup.
///
/// Call @ref join() to synchronize and retrieve the terminal result.
/// Stopping an already-finished capture is a no-op.
///
/// @return Success when Rust acknowledges the request.
/// @throws std::runtime_error If the request cannot be completed.
[[nodiscard]] Result<void, CaptureError> stop();

/// @brief Wait asynchronously for capture to finish and return audio statistics.
///
/// Call @ref stop() first for an unbounded pipeline. The returned
/// @ref CaptureResult includes the number of accepted 10 ms frames and exit reason.
/// Join may be called only once after a successful @ref start().
///
/// @return Future containing terminal statistics or a structured capture error.
/// @throws std::runtime_error When awaiting the future if transport, protocol,
/// cancellation, or SDK shutdown fails.
[[nodiscard]] std::future<Result<CaptureResult, CaptureError>> join();

private:
friend class livekit::FfiClient;

GstreamerAudioCaptureSource();
static std::shared_ptr<GstreamerAudioCaptureSource> fromOwned(const proto::OwnedAudioCaptureSource& owned);

struct Impl;
std::unique_ptr<Impl> impl_;

Check warning on line 143 in include/livekit/capture/audio_source.h

View workflow job for this annotation

GitHub Actions / Builds / Build (windows-x64)

'livekit::capture::GstreamerAudioCaptureSource::impl_': 'std::unique_ptr<livekit::capture::GstreamerAudioCaptureSource::Impl,std::default_delete<livekit::capture::GstreamerAudioCaptureSource::Impl>>' needs to have dll-interface to be used by clients of 'livekit::capture::GstreamerAudioCaptureSource' [D:\a\client-sdk-cpp\client-sdk-cpp\build-release\livekit.vcxproj]

Check warning on line 143 in include/livekit/capture/audio_source.h

View workflow job for this annotation

GitHub Actions / Builds / Build (windows-x64)

'livekit::capture::GstreamerAudioCaptureSource::impl_': 'std::unique_ptr<livekit::capture::GstreamerAudioCaptureSource::Impl,std::default_delete<livekit::capture::GstreamerAudioCaptureSource::Impl>>' needs to have dll-interface to be used by clients of 'livekit::capture::GstreamerAudioCaptureSource' [D:\a\client-sdk-cpp\client-sdk-cpp\build-release\livekit.vcxproj]

Check warning on line 143 in include/livekit/capture/audio_source.h

View workflow job for this annotation

GitHub Actions / Tests / Test (windows-x64)

'livekit::capture::GstreamerAudioCaptureSource::impl_': 'std::unique_ptr<livekit::capture::GstreamerAudioCaptureSource::Impl,std::default_delete<livekit::capture::GstreamerAudioCaptureSource::Impl>>' needs to have dll-interface to be used by clients of 'livekit::capture::GstreamerAudioCaptureSource' [D:\a\client-sdk-cpp\client-sdk-cpp\build-release\livekit.vcxproj]

Check warning on line 143 in include/livekit/capture/audio_source.h

View workflow job for this annotation

GitHub Actions / Tests / Test (windows-x64)

'livekit::capture::GstreamerAudioCaptureSource::impl_': 'std::unique_ptr<livekit::capture::GstreamerAudioCaptureSource::Impl,std::default_delete<livekit::capture::GstreamerAudioCaptureSource::Impl>>' needs to have dll-interface to be used by clients of 'livekit::capture::GstreamerAudioCaptureSource' [D:\a\client-sdk-cpp\client-sdk-cpp\build-release\livekit.vcxproj]
};

} // namespace livekit::capture
Loading
Loading