Insertable Streams for MediaStreamTrack API

When processing real-time video, you sometimes want to insert visual elements or otherwise process the stream of video frames. For example, an application might include two tracks that need to be combined, such as a weather map and video of a presenter explaining the map. Or, you may want to do processing on a track to blur backgrounds, or introduce other elements (such as adding funny hats to people, and so on). The APIs described here provide direct access to the video stream, allowing you to manipulate it in real time.

To ensure optimal performance, the APIs are only available in dedicated workers (unless otherwise stated).

MediaStreamTrackProcessor Experimental

Consumes a MediaStreamTrack object's source and produces a stream of video frames.

VideoTrackGenerator Experimental

Creates a WritableStream that acts as a MediaStreamTrack video source.

MediaStreamTrackGenerator Experimental Non-standard

Creates a WritableStream that acts as a MediaStreamTrack source for either video or audio. Only available on the main thread.

The following example is from the article Unbundling MediaStreamTrackProcessor and VideoTrackGenerator. It transfers a camera MediaStreamTrack to a worker for processing. The worker creates a pipeline that applies a sepia tone filter to the video frames and mirrors them. The pipeline culminates in a VideoTrackGenerator whose MediaStreamTrack is transferred back and played. The media now flows in real time through the transform off the main thread.

const stream = await navigator.mediaDevices.getUserMedia({ video: true });
const [track] = stream.getVideoTracks();
const worker = new Worker("worker.js");
worker.postMessage({ track }, [track]);
const { data } = await new Promise((r) => (worker.onmessage = r));
video.srcObject = new MediaStream([data.track]);

worker.js:

onmessage = async ({ data: { track } }) => {
  const vtg = new VideoTrackGenerator();
  self.postMessage({ track: vtg.track }, [vtg.track]);
  const { readable } = new MediaStreamTrackProcessor({ track });
  await readable
    .pipeThrough(new TransformStream({ transform }))
    .pipeTo(vtg.writable);
};