This post is part five of a series on how to build an app with spatialization features using Daily's real time video and audio APIs.
Introduction
In the previous posts for our spatialization series, we reviewed how to set up our 2D world and the users within it, and how to manage users' video and audio tracks based on participants' proximity to each other in a traversal space.
Now, let's see how we can create different kinds of interactive elements for users to join different zones or broadcast to everyone else in the world.
As a reminder, we are creating a 2D world with daily-js and TypeScript, in which video call participants' video and audio tracks are manipulated based on their proximity to other users. We're using the PixiJS rendering framework to assist us in this quest.
What are "zones"?
In our demo, the world consists of multiple "zones". Each zone is distinguished by a numeric ID. Different zones can have different types of behavior. We have three main zone types implemented in our demo:
- A global traversal zone: This is where the user starts out, and it's the zone they navigate around as they walk through the world.
- A broadcast focus zone: This is the zone where, when entered, the occupant is seen and heard by everyone else in the world, regardless of their current zone or distance from the broadcasting user.
- An isolated focus zone: When entering this zone, the user can see and hear everyone else who is in the same zone. In our world these are implemented as "desks" at which a user can sit, but we could really spawn these as any type of furniture, alcove, or other area in our world.
In our previous posts, we already covered how users navigate within the global traversal zone, how their proximity to other users in the traversal zone is calculated, and how that impacts their video and audio tracks.
Now, let's go through our focus zones. These zones contain one or more "spots" for users to occupy. For example, our desk zone consists of four spots (which in the context of a desk we call “seats” in the UI). It also contains a table, a border, and a text label:
We won't dig into our Spot sprite too much here, because its main purpose is very barebones: a Spot is there to show a visual representation of each occupiable position in a focus zone, and to keep track of who is currently in it. It does nothing beyond these two things. Feel free to check out the code and reach out if you have any questions!
Let's see how we implement the unique behaviors of the BroadcastZone and DeskZone.
BroadcastZone : "You will listen to me and you will like it"
When a user enters the BroadcastZone , their video is put into focus on top of the world canvas for all other users in the world. The broadcast zone is a class that extends PIXI.Container. A container is a display object which is intended to hold other display objects within it. In the case of our broadcast zone, the container holds two children:
- a
Spot - a text display object
It looks like this:
The BroadcastZone class implements our IZone TypeScript interface. This means that it will have to implement the following methods and properties:
export interface IZone extends ICollider {
tryInteract: (user: User) => void;
tryPlace: (user: User, spotID: number) => void;
tryUnplace: (userID: string, spotID: number) => void;
getID: () => number;
}
- The
tryInteract()method above attempts to perform whatever interaction logic the zone implements (which we will go through shortly). - The
tryPlace()andtryUnplace()methods are used to attempt to position a user within the zone or remove them from the zone. getID()returns the unique ID of the zone.
Because IZone extends ICollider, the zone will also need to implement the following:
export interface ICollider {
physics: boolean;
hits: (other: ICollider) => boolean;
}
- The
physicsboolean above lets us know whether the class implementing theIColliderinterface should allow users to pass through it or not. - The
hits()method tells us if the other collider (in our case the user) is touching the broadcast zone.
You might remember from part three of our series that the broadcast zone is instantiated when we start the world:
const zoneBroadcast = new BroadcastZone(
broadcastZoneID,
0,
defaultWorldSize / 2
);
zoneBroadcast.moveTo({
x: defaultWorldSize / 2 - zoneBroadcast.width / 2,
y: zoneBroadcast.y,
});
this.focusZonesContainer.addChild(zoneBroadcast);
this.focusZones.push(zoneBroadcast);
Above, we position our new broadcast zone in the middle of our world. We then add the zone to our focus zones container. Like the broadcast zone, focusZonesContainer is a PixiJS container which holds a collection of PIXI.DisplayObject.
Because we'll be frequently referring to our custom interface methods on this type, we also append the object to our world's own focusZones collection, which is an array of IZone.
Instantiating the BroadcastZone
When instantiating BroadcastZone, the constructor of the zone creates a Spot:
constructor(id: number, x: number, y: number) {
super();
this.id = id;
this.name = id.toString();
this.x = x;
this.y = y;
// The position is in relation to the container, not global
// which is why we set it to 0,0
this.spot = new Spot(
0,
{ x: 0, y: 0 },
{ width: spotSize, height: spotSize },
"📣"
);
this.addChild(this.spot);
this.createLabel();
this.sortableChildren = true;
}
It then adds the spot as a child of a container, creates our aforementioned text label via this.createLabel(), and sorts the children.
Interacting with the BroadcastZone
We check the local user's interaction with all of our focus zones as part of our update loop, which we covered in part three of our series. As a refresher, each tick we loop through all focus zones and see if the local user is interacting with any of them by calling tryInteract() on the zone object. Here's the tryInteract() method for our BroadcastZone:
public tryInteract(other: User) {
if (this.hits(other) && !this.spot.occupantID) {
this.spot.occupantID = other.id;
other.updateZone(this.id);
return;
}
if (other.id === this.spot.occupantID && !this.hits(other)) {
this.spot.occupantID = null;
other.updateZone(globalZoneID);
}
}
Above, we first check if the user is colliding with the broadcast zone. The broadcast zone container is the size of all of its children, in this case our single spot. So if the user touches our spot, it'll also be colliding with the broadcast zone container.
If the user hits the broadcast zone and the zone's single spot is unoccupied, the user occupies the spot. We update the spot's occupantID and call user.updateZone(). We'll go through this code path in detail below.
If it turns out the spot is already occupied by the interacting user and the user is not colliding with the zone, we know they must have left! We unset the occupant ID and update the user's zone to the ID of our global traversal zone.
And that covers the meat of our broadcast zone implementation!
DeskZone: Small groups for isolated chats
When a user enters a DeskZone, their video is placed over the world canvas much like with a broadcast zone. The difference is that several users can be in the desk zone at the same time. All of their video tracks will be shown in identical larger DOM elements for a more targeted chat in their small group, and they will only be able to see and hear each other while they're in the zone.
Each DeskZone consists of a desk which a user is prevented from walking through, and however many spots we choose. For our demo, we've chosen to have four seating spots per desk.
We create two desk zones when we start the world, right after instantiating the broadcast zone:
// Create two desk zones
const yPos = defaultWorldSize / 2 + 325;
const zone1 = new DeskZone(1, "Koala", 4, { x: 0, y: yPos });
zone1.moveTo({
x: defaultWorldSize / 2 - zone1.width - zoneBroadcast.width,
y: zone1.y,
});
this.focusZonesContainer.addChild(zone1);
this.focusZones.push(zone1);
const zone2 = new DeskZone(2, "Kangaroo", 4, { x: 0, y: yPos });
zone2.moveTo({ x: defaultWorldSize / 2 + zoneBroadcast.width, y: zone2.y });
this.focusZonesContainer.addChild(zone2);
this.focusZones.push(zone2);
Same as the broadcast zone, we add each desk zone to the focus zone container as well as our separate focus zones array. Let's take a look at the most important data the desk zone holds:
// The desk sprite in the middle of the desk zone, which
// a user cannot pass through
private desk: Desk;
// Array of Spots that users can occupy to join the
// desk zone
private spots: Array<Spot> = [];
// How many free seats are available in the desk zone
private freeSeats: number;
// A blue visual marker surrounding the zone
private zoneMarker: PIXI.Graphics;
// Text indicating the name and occupancy of the zone
private labelGraphics: PIXI.Text;
All of the above elements are instantiated in the DeskZone constructor. We won't go into it in detail here, but please feel free to check out the code if you're curious!
Now, let's look at how interaction works for a desk zone.
Interacting with the DeskZone
The desk zone's tryInteract() method is a little more involved than that of the broadcast zone, because the desk zone has multiple spots instead of just one:
public async tryInteract(user: User) {
let hadPriorSpot: boolean;
let hasNewSpot: boolean;
const oldFreeSeats = this.freeSeats;
for (let spot of this.spots) {
// If the user is already registered in this spot...
if (spot.occupantID === user.id) {
// ...and is still in the spot, do nothing
if (spot.hits(user) && !hasNewSpot) return;
// User is no longer in the spot - clear the spot
spot.occupantID = null;
hadPriorSpot = true;
continue;
}
// If this spot has no occupant but the user
// hits it, occupy it.
if (!spot.occupantID && spot.hits(user)) {
spot.occupantID = user.id;
user.updateZone(this.id, spot.id);
hasNewSpot = true;
if (user.isLocal) this.hideZoneMarker();
continue;
}
}
// If the user has just left a spot and has not
// joined a new one, go back to the global zone.
if (hadPriorSpot && !hasNewSpot) {
this.freeSeats++;
user.updateZone(globalZoneID);
if (user.isLocal) this.showZoneMarker();
} else if (!hadPriorSpot && hasNewSpot) {
this.freeSeats--;
}
// Update the label text if needed.
if (oldFreeSeats !== this.freeSeats) {
this.updateLabel();
}
}
Above, we first loop through all spots present in the zone. If the spot is currently being occupied by the given user and the user is still in that spot, we early out.
If the user is no longer in the spot they were occupying, we clear that spot's occupant ID and set our hadPriorSpot boolean to true.
If the given spot is not occupied by anyone, but the user is colliding with it, the user occupies that spot and we set our hasNewSpot boolean to true. We also hide/fade out the blue border marker of the zone, to indicate that the user is inside the given zone.
Finally, if the user had a previous spot which they are no longer occupying and doesn't have a new spot in the same zone, we know they've left the zone and move them to the global traversal zone.
Finally, if the count of free seats has changed, we update the zone label to show the new count.
How is the user's zone actually updated?
The call to user.updateZone() is where all the zone updating magic happens. Let's go through it now. We've left comments inline, and will go through some of the calls you see in this function right afterwards.
// updateZone updates the zone ID of the user
updateZone(zoneID: number, spotID: number = -1) {
const oldZoneID = this.zoneData.zoneID;
const oldSpotID = this.zoneData.spotID;
// If the old zone is identical to this zone, there's
// nothing more to do - early out.
if (zoneID === oldZoneID && spotID === oldSpotID) return;
// Update the user's zone data
this.zoneData.zoneID = zoneID;
this.zoneData.spotID = spotID;
// If the old zone was a broadcast zone, leave
// the broadcast. Otherwise if the new zone is
// a broadcast zone, enter the broadcast.
if (oldZoneID === broadcastZoneID) {
this.media.leaveBroadcast();
} else if (zoneID === broadcastZoneID) {
// If the user is already in another focus zone,
// leave that zone.
if (this.media.currentAction === Action.InZone) {
this.media.leaveZone();
}
this.alpha = maxAlpha;
this.media.enterBroadcast();
this.isInVicinity = false;
}
// If the new zone is not the global zone or broadcast zone,
// set the default sprite texture. We'll be showing video in
// separate, larger DOM elements.
if (zoneID !== globalZoneID && zoneID !== broadcastZoneID) {
this.setDefaultTexture();
}
// The rest of the function is only relevant to the local user.
if (!this.isLocal) return;
// Reset the user's zonemates if they're not coming
// from the global zone.
if (oldZoneID !== globalZoneID) {
this.localZoneMates = {};
}
if (this.onJoinZone) {
this.onJoinZone({ zoneID: zoneID, spotID: spotID });
}
if (zoneID === globalZoneID) {
// If the local user's camera isn't disabled, set
// the video texture.
if (!this.media.cameraDisabled) {
this.setVideoTexture();
}
// Since their previous zone is not a global traversal zone,
// it must be a focus zone - leave it.
this.media.leaveZone();
return;
}
// If the user is not in the global zone or the broadcast zone,
// they must be in an interactive focus zone - enter it.
if (zoneID !== globalZoneID && zoneID !== broadcastZoneID) {
this.media.enterZone();
}
}
We hope the inline comments gave you a good idea of what the function is doing. Now, let's take a look at the media calls we see above.
UserMedia focus tile management
As we mentioned in our previous post, UserMedia handles the management of tracks and focus tiles. This includes showing and hiding focus tiles. For example, if the new zone the user is joining is the broadcast zone, we call this.media.enterBroadcast():
enterBroadcast() {
if (this.audioTag) this.muteAudio();
this.currentAction = Action.Broadcasting;
this.showOrUpdateBroadcast();
}
Above, the UserMedia instance mutes the user's audio tag, updates the current action, and shows the broadcast focus tile:
showOrUpdateBroadcast() {
let videoTrack = null;
if (this.videoTrack && !this.cameraDisabled) {
videoTrack = this.videoTrack;
}
showBroadcast(this.userName, videoTrack, this.audioTrack);
}
showBroadcast() is an exported function in our util/tile.ts file:
export function showBroadcast(
name: string,
videoTrack?: MediaStreamTrack,
audioTrack?: MediaStreamTrack
) {
const tracks: Array<MediaStreamTrack> = [];
if (videoTrack) tracks.push(videoTrack);
if (audioTrack) tracks.push(audioTrack);
if (tracks.length > 0) {
broadcastVideo.srcObject = new MediaStream(tracks);
}
// Update name and show broadcast div
broadcastName.innerText = name;
broadcastDiv.style.visibility = "visible";
broadcastDiv.draggable = true;
}
As we can see, showBroadcast() creates a MediaStream from the provided tracks and displays the broadcast div.
Likewise, the user's UserMedia instance also handles hiding the broadcast tile as well as showing and hiding focus tiles when entering a desk zone.
Among the most important things we do when updating a user's zone is the call to user.onJoinZone(). Let's go through it now.
Broadcasting our zone update
onJoinZone() is a function defined in our Room class, which is passed to the World when a user joins the meeting:
// The function World will call when the local user changes zone.
// This will update their bandwidth and broadcast their new zone
// to other participants.
const onJoinZone = (zoneData: ZoneData, recipient: string = "*") => {
if (zoneData.zoneID === globalZoneID) {
this.setBandwidth(BandwidthLevel.Tile);
} else {
this.setBandwidth(BandwidthLevel.Focus);
}
const data = {
action: "zoneChange",
zoneData: zoneData,
};
this.broadcast(data, recipient);
};
As we can see, this makes a call to setBandwidth(), which we covered in post two of the series. It then broadcasts the new zone data to the other participants in the call.
So what happens when a user receives someone else's "zoneChange" message?
Handling others' zone updates
When a user receives a zone change message, our room calls the world's updateParticipantZone() method:
updateParticipantZone(
sessionID: string,
zoneID: number,
spotID: number = -1
) {
let user = this.getUser(sessionID);
if (!user) {
user = this.createUser(sessionID, -100, -100);
}
const priorZone = user.getZoneData();
const oldZoneID = priorZone.zoneID;
const oldSpotID = priorZone.spotID;
user.updateZone(zoneID, spotID);
this.localUser.updateStoredZonemates(user);
if (user.isZonemate(this.localUser)) {
// Send data back to make sure the newly joined participant knows exactly
// where we are
this.sendPosDataToParticipant(sessionID);
}
// Iterate through all focus zones and try to place/unplace user in
// communicated zone and spot as needed. "Placement" does not impact
// user behavior itself (that is done via `user.updateZone()` above).
// Placement affects zone spot occupation status and remote positioning.
for (let zone of this.focusZones) {
if (oldZoneID === item.getID()) {
zone.tryUnplace(user.id, oldSpotID);
// If the new zone is the global zone, don't bother
// checking for any further placement.
if (zoneID === globalZoneID) return;
}
if (zoneID === zone.getID()) {
zone.tryPlace(user, spotID);
}
}
}
Above, we first retrieve the user being updated. We create one if it doesn't yet exist.
Then, we retrieve the user's previous zone data and call the updateZone() method we covered above with the new zone data obtained from the "zoneChange" "app-message" event.
We then make sure the local user's zonemates are updated via a call to this.localUser.updateStoredZonemates().
If the user who just entered a new zone is now a zonemate of the local user, we send our local position data to this zonemate (so that they can see exactly where we are within the zone).
Finally, we iterate through all of our focus zones. We try to unplace the user from their old zone and place them into their new zone.
For example, the BroadcastZone zone placement/un-placement attempt looks like this:
public tryPlace(user: User, _: number) {
if (!this.spot.occupantID) {
this.spot.occupantID = user.id;
const np = {
x: this.x + this.spot.x,
y: this.y + this.spot.y,
};
user.moveTo(np);
}
}
public tryUnplace(userID: string, _: number = -1) {
if (this.spot.occupantID === userID) {
this.spot.occupantID = null;
}
}
Above, when trying to place the user, if the zone's single spot is not already occupied, we set the occupantID to that of the user and move the user to the exact location of the spot.
When removing the user from a spot, we set the spot's occupantID back to null.
For a DeskZone, the placement and removal are very similar, but the implementation involves looping through all of the spots in the zone. You can check out the implementation of that here.
One last thing: ticks (the good kind)
You might recall from our last post that we call a processUsers() function on our local user with each tick of the world. In the last post, we focused on going through the proximity-based behavior that happens between users in the global traversal zone.
The processing of a user in relation to our local participant involves some more steps for users which are moving between zones. For example, if during this check we see that a remote user is now in the same focus zone as the local user when they weren't before, we update the remote user sprite's transparency and subscribe to their tracks:
// If the users are in the same zone that is not the default zone,
// enter vicinity and display them as zonemates in focused-mode.
if (ozID > 0 && ozID === tzID) {
// Store this in the localZoneMates array for more efficient
// broadcasting later.
this.doSaveZonemate(o.id);
if (o.media.currentAction !== Action.InZone) {
if (!o.isInVicinity) {
o.alpha = 1;
o.isInVicinity = true;
if (this.onEnterVicinity) this.onEnterVicinity(o.id);
}
o.media.enterZone();
}
// Mute the other user's default audio, since we'll
// be streaming via a zone.
o.media.muteAudio();
return;
}
We've left comments throughout the remainder of this function that we hope will help you follow how this zone-related logic is structured for every tick. You can check them out here and let us know if you have any questions.
Conclusion
In this post, we went over how focus zones are defined and interacted with in our spatialization demo, how users broadcast their zone updates to others in the world, and how others' zone updates are handled in relation to the local user.
In the next post of our spatialization series, we'll take a look at how to extend our existing demo with a new feature: screen sharing.
Never miss a story
Get the latest direct to your inbox.
