This post is the second in a series about building custom video chat applications that support large meetings using the Daily call object. The features we’ll add specifically target use cases where dozens (or hundreds!) of participants in an up to 1000 person call might turn on their cameras.
In browser-based WebRTC video calls, like Daily's, each participant has separate tracks for their video and audio for other participants to see and hear.
Daily deals with the complexity of routing all of those distinct sets of audio, video, and screen media tracks to every other participant on a call by default. Like all video calls, Daily calls work on a publish-subscribe model: call participants publish audio, video, and screen data tracks, and are subscribed to other participants’ tracks.
Subscribing every participant to every track works well for most applications. However, end users can start to see performance issues as the number of call participants increases. This is especially true for users on mobile or older devices. We learned a lot about this when we added performance improvements to Daily Prebuilt.
If you’re building larger calls, manually handling tracks can help you support larger call sizes, deliver better call quality, and preserve participants’ CPU.
In this tutorial, we’ll add manual track subscriptions as a feature to a paginated video chat app. We’ll use it to subscribe to the tracks of participants on a current page, queue the tracks on the previous and next pages, and unsubscribe from the rest.
To add track subscriptions, we will:
- Get to know and control tracks and their Daily subscription states
- Keep track 🙊 of staged and subscribed participant ids
- Update the Daily call object with the latest staged and subscribed data
- Play and pause tracks based on participant events
- Render subscribed tracks
While we’ll build on top of the /daily-demos/track-subscriptions repository introduced in the previous post in this series, you can use the same methods and events covered in this tutorial to add track subscriptions to any app.
🔥 Tip: Check out our "Notes" section in the demo repo to see how to make it production ready.
Whether you’re using the template repository or your own, to add track subscriptions you definitely need a Daily account if you don’t have one already, and to use daily-js 0.17.0 or higher.
Get to know and control tracks and their Daily subscription states
A track refers to an individual media stream track object, sent from a participant’s camera, microphone, screen video or screen audio.
We can access each participant’s tracks and each track’s subscribed status from the Daily participants object. Here’s an abridged example for one participant:
{
"e20b7ead-54c3-459e-800a-ca4f21882f2f": {
"user_id": "e20b7ead-54c3-459e-800a-ca4f21882f2f",
"audio": true,
"video": false,
"screen": false,
"joined_at": "Date(2019-04-30T00:06:32.485Z)",
"local": false,
"owner": false,
"session_id": "e20b7ead-54c3-459e-800a-ca4f21882f2f",
"user_name": "",
"tracks": {
"audio": {
"subscribed": true,
"state": "blocked",
// "blocked" | "off" | "sendable" | "loading" | "interrupted" | "playable"
"blocked": {
"byDeviceMissing": true,
"byPermissions": true
},
"off": {
"byUser": true,
"byBandwidth": true
},
"track": <MediaStreamTrack>,
"persistentTrack": <MediaStreamTrack>
},
"video": {
// same as above
},
"screenAudio": {
// same as above
},
"screenVideo": {
// same as above
}
}
}
}
Each track type within tracks (audio, video, screenAudio, and screenVideo) includes a raw media stream track object available on both track and persistentTrack (We explain the difference between these in our docs, but in short, persistentTrack solves a Safari-specific bug that’s beyond the scope of this post). Each track type also includes the state of the track, and its subscribed value.
state tells us if the track can be played (see full possible states in our docs). subscribed tells us whether or not the local participant is receiving the track. subscribed’s value is true if the local participant is receiving, false if they are not, or "staged".
A subscribed status of "staged" keeps the connection for that track open, but stops any bytes from flowing across. Compared to tearing down the connection with a complete unsubscribe, staging a track speeds up the process of showing or hiding participants' video and audio. Staging also cuts out the processing and bandwidth required for that track.
In calls with over 50 participants, it’s best to take advantage of both staging and unsubscribing to maximize both quickly showing videos and minimizing the load on connections, processing and CPU.
In our demo app, we’ll set the subscribed status of participant tracks on the page before and on the page after the one currently being viewed to "staged". We’ll unsubscribe from the rest, limiting subscriptions to only participants on the current page. This will minimize the total number of subscriptions while maintaining an easily accessible queue for page turns.
To turn on the ability to set "subscribed" track states manually, we set the Daily subscribeToTracksAutomatically property to false. We’ll pass subscribeToTracksAutomatically: false on join():
const join = useCallback(
async (callObject) => {
await callObject.join({
url: roomUrl,
subscribeToTracksAutomatically: false
});
setLocalVideo(callObject.localVideo());
},
[roomUrl]
);
There are a few other ways to set up manual track subscriptions: passing
subscribeToTracksAutomatically: falseas a parameter when callingcreateCallObject(), or via with thesetSubscribeToTracksAutomatically()method. The method is a good option if you want to switch over to track subscriptions only after a certain number of participants have joined the call.
Keep track 🙊 of subscribed and staged participant ids
Once we've set up direct track control over tracks, we can start managing subscriptions by keeping lists of subscribedIds and stagedIds. We do that in a useMemo hook in our <PaginatedGrid /> component:
const [subscribedIds, stagedIds] = useMemo(() => {
const maxSubs = 3 * pageSize;
let renderedOrBufferedIds = [];
switch (page) {
case 1:
renderedOrBufferedIds = participants
.slice(0, Math.min(maxSubs, 2 * pageSize))
.map((p) => p.id);
break;
case Math.ceil(participants.length / pageSize):
renderedOrBufferedIds = participants
.slice(-Math.min(maxSubs, 2 * pageSize))
.map((p) => p.id);
break;
default:
{
const buffer = (maxSubs - pageSize) / 2;
const min = (page - 1) * pageSize - buffer;
const max = page * pageSize + buffer;
renderedOrBufferedIds = participants.slice(min, max).map((p) => p.id);
}
break;
}
const subscribedIds = [];
const stagedIds = [];
renderedOrBufferedIds.forEach((id) => {
if (id !== "local") {
if (visibleParticipants.some((vp) => vp.id === id)) {
subscribedIds.push(id);
} else {
stagedIds.push(id);
}
}
});
return [subscribedIds, stagedIds];
}, [page, pageSize, participants, visibleParticipants]);
There’s a lot going on here, so let’s look at each piece.
First piece: dependencies. Our hook listens for changes to the current displayed page, pageSize, participants on the call, and visibleParticipants. Have a look at the first post in this series for a refresher on those values. When any of those values changes, the hook recalculates the subscribedIds and stagedIds.
First, it creates a temporary empty array for storing values that will need to be subscribed or staged.
let renderedOrBufferedIds = [];
Then, it populates the array depending on the current page using the participants array from the ParticipantProvider.
If the current page is the first, then we need to subscribe to the tracks of the participants on the first page, and stage the ones on the second page. We pass those participant id’s to renderedOrBufferedIds:
// First page
case 1:
renderedOrBufferedIds = participants
.slice(0, Math.min(maxSubs, 2 * pageSize))
.map((p) => p.id);
break;
If the current page is between the first and the last page, then we need to subscribe to the tracks of participants on the current page, and stage the ones on both the previous and next pages. We send the participant id’s starting on the page before the current page to the id’s starting on the page after the current page to renderedOrBufferedIds.
default:
{
const buffer = (maxSubs - pageSize) / 2;
const min = (page - 1) * pageSize - buffer;
const max = page * pageSize + buffer;
renderedOrBufferedIds = participants.slice(min, max).map((p) => p.id);
}
break;
Finally, if we’re on the last page, we need to subscribe to the tracks of participants on the last page, and stage the ones on the next-to-last page. We send ids counting backwards from the end of the participants array to renderedOrBufferedIds:
case Math.ceil(participants.length / pageSize):
renderedOrBufferedIds = participants
.slice(-Math.min(maxSubs, 2 * pageSize))
.map((p) => p.id);
break;
Once we have the renderedOrBufferedIds relevant to the page being viewed, we can calculate subscribedIds and stagedIds:
renderedOrBufferedIds.forEach((id) => {
if (id !== "local") {
if (visibleParticipants.some((vp) => vp.id === id)) {
subscribedIds.push(id);
} else {
stagedIds.push(id);
}
}
});
We compare the renderedOrBufferedIds, filtering out the "local" participant, against currently visibleParticipants (for a refresher on that value, see our previous post).
If a participant is visible on the page, we push their id to subscribedIds. If they’re not visible, we push their id to stagedIds.
Update the Daily call object with the latest staged and subscribed data
With an accurate list of subscribedIds and stagedIds, we can listen for changes to those lists and call an updateCamSubscriptions() handler in response:
useDeepCompareEffect(() => {
if (!subscribedIds || !stagedIds) return;
const timeout = setTimeout(() => {
updateCamSubscriptions(subscribedIds, stagedIds);
}, 50);
return () => clearTimeout(timeout);
}, [subscribedIds, stagedIds, updateCamSubscriptions]);
We import updateCamSubscriptions() from TracksProvider. Let’s look at its definition:
const updateCamSubscriptions = useCallback(
(subscribedIds, stagedIds = []) => {
if (!callObject) return;
const stagedIdsFiltered = [
...stagedIds,
...recentSpeakerIds.filter((id) => !subscribedIds.includes(id)),
];
const updates = remoteParticipantIds.reduce((u, id) => {
let desiredSubscription;
const currentSubscription =
callObject.participants()?.[id]?.tracks?.video?.subscribed;
if (!id || id === "local") return u;
if (subscribedIds.includes(id)) {
desiredSubscription = true;
} else if (stagedIdsFiltered.includes(id)) {
desiredSubscription = "staged";
} else {
desiredSubscription = false;
}
if (desiredSubscription === currentSubscription) return u;
return {
...u,
[id]: {
setSubscribedTracks: {
video: desiredSubscription,
},
},
};
}, {});
callObject.updateParticipants(updates);
},
[callObject, remoteParticipantIds, recentSpeakerIds]
);
Once again, let’s take this snippet in pieces.
If a call exists, we first add recentSpeakerIds to the stagedIds list, because we assume they might be in the middle of a conversation and likely to speak again soon:
const stagedIdsFiltered = [
...stagedIds,
...recentSpeakerIds.filter((id) => !subscribedIds.includes(id)),
];
Then, we calculate the tracks updates. We need updates to be in the form of an object with keys as participant ids, and values as setSubscribedTracks objects indicating the new desiredSubscription for each video track (This format is required for the Daily updateParticipants() method we’ll use in a minute).
We calculate updates by calling .reduce() on remoteParticipantIds, an array of all participants except the "local" participant. We check if the remoteParticipantId is also in our list of stagedIds or subscribedIds, and set its desiredSubscription status accordingly. If the id can’t be found in either list, we set video’s desiredSubscription to false.
const updates = remoteParticipantIds.reduce((u, id) => {
let desiredSubscription;
const currentSubscription =
callObject.participants()?.[id]?.tracks?.video?.subscribed;
if (!id || id === "local") return u;
if (subscribedIds.includes(id)) {
desiredSubscription = true;
} else if (stagedIdsFiltered.includes(id)) {
desiredSubscription = "staged";
} else {
desiredSubscription = false;
}
if (desiredSubscription === currentSubscription) return u;
return {
...u,
[id]: {
setSubscribedTracks: {
video: desiredSubscription,
},
},
};
}, {});
Finally, we send the consolidated updates to the updateParticipants() method to apply them to the Daily call object.
callObject.updateParticipants(updates);
The call object now has all the up-to-date information about which tracks are subscribed, staged, or unsubscribed. Now, we need to reflect that information in the UI.
Play and pause tracks based on participant events
A useEffect in TracksProvider listens for changes to the Daily call object. Here’s the full hook, but don’t worry, we’ll walk through each part:
useEffect(() => {
if (!callObject) return false;
const handleTrackUpdate = ({ action, participant, track }) => {
if (!participant) return;
const id = participant.local ? "local" : participant.user_id;
switch (action) {
case "track-started":
case "track-stopped":
if (track.kind !== "video") break;
setVideoTracks((prevState) => ({
...prevState,
[id]: participant.tracks.video,
}));
break;
case "participant-updated":
setVideoTracks((prevState) => ({
...prevState,
[id]: participant.tracks.video,
}));
break;
case "participant-left":
setVideoTracks((prevState) => {
delete prevState[id];
return prevState;
});
break;
}
};
callObject.on("track-started", handleTrackUpdate);
callObject.on("track-stopped", handleTrackUpdate);
callObject.on("participant-updated", handleTrackUpdate);
callObject.on("participant-left", handleTrackUpdate);
return () => {
callObject.off("track-started", handleTrackUpdate);
callObject.off("track-stopped", handleTrackUpdate);
callObject.off("participant-updated", handleTrackUpdate);
callObject.off("participant-left", handleTrackUpdate);
};
}, [callObject]);
Let’s start at the bottom with the event listeners. The updateParticipants() method fires a "participant-updated" Daily event for every participant included in the updates. We need to update our app state not only on those events, but also for other events like a participant muting ("track-stopped") or unmuting ("track-started"), or leaving the call ("participant-left").
callObject.on("track-started", handleTrackUpdate);
callObject.on("track-stopped", handleTrackUpdate);
callObject.on("participant-updated", handleTrackUpdate);
callObject.on("participant-left", handleTrackUpdate);
handleTrackUpdate covers all of those cases. It updates the videoTracks in local state to include the relevant update:
const handleTrackUpdate = ({ action, participant, track }) => {
if (!participant) return;
const id = participant.local ? "local" : participant.user_id;
switch (action) {
case "track-started":
case "track-stopped":
if (track.kind !== "video") break;
setVideoTracks((prevState) => ({
...prevState,
[id]: participant.tracks.video,
}));
break;
case "participant-updated":
setVideoTracks((prevState) => ({
...prevState,
[id]: participant.tracks.video,
}));
break;
case "participant-left":
setVideoTracks((prevState) => {
delete prevState[id];
return prevState;
});
break;
}
};
With an accurate list of videoTracks, we’re ready to display them.
Render subscribed tracks
To display all the participant video tiles, <PaginatedGrid /> maps over visibleParticipants, passing information about each participant on a page to the <Tile /> component.
export const PaginatedGrid = ({ autoLayers }) => {
// Other functionality here
const tiles = useDeepCompareMemo(
() =>
visibleParticipants.map((p) => (
<Tile
participant={p}
key={p.id}
autoLayers={autoLayers}
style={{
maxHeight: tileHeight,
maxWidth: tileWidth,
}}
/>
)),
[tileWidth, tileHeight, autoLayers, visibleParticipants]
);
// Other functionality here
return (
<!-- Other components here -->
<div ref={gridRef} className="grid">
<div className="tiles">{tiles}</div>
</div>
<!-- Other components and styling here -->
);
};
One of the first things <Tile /> does is pass the participant’s id to a useVideoTrack hook:
const Tile = ({ participant, autoLayers, ...props }) => {
const videoTrack = useVideoTrack(participant.id);
// Other things here
};
useVideoTrack finds the participant’s videoTrack in the videoTracks from the TracksProvider. It then checks for the track’s state and subscribed status. If the track is subscribed to and available, useVideoTrack returns the persistentTrack:
export const useVideoTrack = (id) => {
const { videoTracks } = useTracks();
const videoTrack = useDeepCompareMemo(
() => videoTracks?.[id],
[id, videoTracks]
);
return useDeepCompareMemo(() => {
const videoTrack = videoTracks?.[id];
if (
videoTrack?.state === "off" ||
videoTrack?.state === "blocked" ||
(!videoTrack?.subscribed && id !== "local")
)
return null;
return videoTrack?.persistentTrack;
}, [id, videoTrack, videoTrack?.persistentTrack?.id]);
};
<Tile /> takes that track, listens for changes to it, and uses a ref to set the <video /> element src to play it:
const Tile = ({ participant, autoLayers, ...props }) => {
const videoEl = useRef();
const videoTrack = useVideoTrack(participant.id);
// Other things here
useEffect(() => {
const video = videoEl.current;
if (!video || !videoTrack) return;
video.srcObject = new MediaStream([videoTrack]);
}, [videoTrack]);
return (
<!-- Other components here -->
<video
autoPlay
muted
playsInline
ref={videoEl}
className={videoTrack ? "play" : "pause"}
/>
<!-- Other components and styling here -->
);
};
We’re on track!
With that, our app only subscribes to the video tracks on a current page, keeps adjacent tracks "staged", and unsubscribes from the rest. We’re ready to add the final feature in the series: selective simulcast encoding display. For a head start, you can explore all the source code in the repository. Stay tuned!
Never miss a story
Get the latest direct to your inbox.
