If, by the tribunes' leave, and yours, good people,
I may be heard, I would crave a word or two;
The which shall turn you to no further harm
Than so much loss of time.

So. It's been a while! This was a busy year for MeXanimatoR. I'll need to write a wrap-up post for the 2023-24 academic year. But first, I'd like to go over some of the details in the voice chat system's ongoing development process. Be forewarned, this will be more documentative than instructional. The journey is the destination, or something like that.

As a reminder, MeXanimatoR started as an asynchronous experience: recording a player's body movement and audio as a digital performance allows you to play it back and perform as another player in the scene, casting and layering the performances in a layered fashion. Microphone access and recording as an AudioClip, then saving to storage has been functional for a while. By playing the AudioClip from the recorded player's head position and using 3D sound, the spatialization of audio makes you feel like you're really sharing a space with another person.

Incorporating live multiplayer between separate devices necessitates a system for transmitting voice. Body movement is synchronized using NetworkBehaviour transforms as part of the player objects, but there was no system in place for broadcasting voice. This was fine, at first. Initial tests and demos all made use of the same physical space for multiple players, close enough in proximity that they can simply hear one another. This presents other problems related to synchronization between play spaces, physical bodies, and avatar placement, but that's another post on my todo list related to co-location in VR.

Back to voice chat over the network, with the assumption that players are in a separate space: how do we do it? My first approach to this was to try Dissonance, a plugin in the Unity Asset Store. In the development of MeXanimatoR, I quite often looked at existing solutions to solve portions of the system whenever applicable. This choice in itself is a fulcrum of thinking in programming: when should we approach a problem as a configuration of existing tools, and when should we really dive into the details and develop our own system? This Twitter post has an interesting perspective on it:

That said, knowing how and when to apply both techniques is important. But Dissonance comes highly regarded in just about every Unity community I've encountered, and it solves a ton of problems. Our networking system comes from Fish-Net, another plugin, and there's a community-provided integration between the two. Huzzah! I installed and configured the plugin, and after an hour or so of tinkering, I had voice chat coming through on two local instances! On that note: testing network functionality can be tricky. I've been using ParrelSync to clone the project for running separate Unity Editor instances with one as host and the other as client, and this has been the fastest and easiest system.

So here's where the issues began. Despite being in a LAN environment with tiny frames (10ms), voice chat with Dissonance creates a latency of something like 450ms. It's wild! Where's all this coming from? Especially when the estimated latency should be much better: this writeup claims a latency of roughly 75ms while also addressing that Unity's going to add around 100ms. While 150-175ms is better than what I was encountering, it's still rather bad for timing-critical performances. And as it happens, Unity's default Microphone API has some fairly significant lag, too. Putting something like this in your Start() function:

var audio = GetComponentInChildren<AudioSource>();
audio.clip = Microphone.Start("Microphone", true, 10, 48000);
audio.loop = true;
while (!(Microphone.GetPosition(null) > 0)) { }
audio.Play();

lets the player hear their own voice, with a delay of almost 1 second. I came across a few other posts talking about this issue, many of whom were concerned with Quest 2 development, so it's a nice indicator of standalone VR development. But I've been on my Windows machines up to this point and dealing with the same problem!

By default, Dissonance uses the Unity Microphone calls to handle voice access. They offer an FMOD plugin that is supposed to help with this kind of delay. I've never worked with FMOD before, but the Dissonance devs state that it can provide faster access to the recording device and avoid the overhead from Unity's Microphone call. In trying it out, the delay did improve, but we were still around something like 200ms. Something else was going on. Turning on debug mode in Dissonance, I started getting warnings that echoed closed issues on the Dissonance GitHub: "encoded audio heap is getting very large (46 items)" and the like. Into the past issues and Dissonance Discord I dove. A Discord thread referred to a similar problem that just so happened to involve the Fish-Net plugin and the community-provided integration layer. There was also an issue with player position tracking for correct audio spatialization related to network player Ids not being available at the right moment. I managed to fix this by implementing my own component for the IDissonancePlayer interface, but it did speak of other issues that might appear.

As a test, I made a new project that used Dissonance and Mirror for the network system, which has an integration layer provided by Dissonance. Combined with FMOD, this did perform much better: I think it was around 60-70ms. Mirror was actually my first choice for MeXanimatoR's networking system back when I started toying around with it. There were some issues as summarized in this amusing commit history:

Well, crud. I really didn't want to go back to Mirror from Fish-Net after having developed a considerable number of networking components for it. So began the old, dark thoughts once more: what if we didn't use Dissonance at all? We'd lose all of its (very nice and helpful) features at the cost of having to make some ourselves, but could be handle this in the data-processing manner and not depend on the laggy black box?

That brings us up to speed. Over the last week, I've been toying with FMOD for Unity. At startup, I create a Sound for recording and a Sound for playback. The latency here is much, much better - around 20-30ms between speaking and hearing yourself. Not so much of a speech jammer, but I'll also look into NAudio as another option. So far it's showing improvements that I was hoping for with the Dissonance FMOD plugin.

At the moment, here's what the system can do:

• Start recording and playback for latency testing.
• Access and copy samples from the sound recording on a per-frame basis.
• Save the total recording to storage as an .ogg or .wav file.
• Encode the per-frame samples with Vorbis.

What's left to try:

• Transmitting encoded samples via the network.
• Decoding Vorbis samples from the network back to PCM
• Playing decoded samples live.
• Measuring the latency in this system.

There's still quite a ways to go, but I'm hoping to have most of these things testable before taking a vacation next week. Oh, ambition, you love to take such huge bites.

Direct access has its own cost - we lose some of the conveniences provided by Dissonance, such as audio quality configuration, noise suppression, background sound removal, and adding these back in takes time (both for development and as little increments that add to the voice system's latency). For measuring latency, I've just been using OBS to record the microphone and the desktop audio to separate tracks. Then I extract the tracks with ffmpeg and compare the waveforms in Audacity. It's a bit of a manual process (the coder's anathema), but it works for the moment.

That's all I've got for now! I'm hoping to write more over the summer, if only to try and keep track of all the configurations as I go through them.

One of my favorite parts about calculus was learning about integrals. The mental leap I had when I realized you could compute the exact area under the curve if only you could use an infinitely small $dx$ in a Riemann sum is a particularly fond memory. That marks a major shift in my understanding of mathematics and, consequently, how I view the world around me (if you were wondering, probability theory, theory of computation, and JRPGs account for most of my other shifts). The following is a byproduct of post-calculus Nick.

If you're anything like me, occasionally you need to compute the area of a polygon. Sometimes it's for finding the barycentric coordinates of a point in a triangle; other times it's used to construct a function that selects uniformly random points from an irregularly-shaped polygon surface. The right area computation method depends on the circumstances. If it is known that a polygon will never deform, you can pre-compute the area before doing anything exciting with it. Otherwise, the area will have to be re-computed any time the polygon reports a significant deformation. If the polygon is convex (whether by construction, or verified to be so during runtime, or whatever), you can treat it like a triangle fan and find the sum of all the triangles in the fan. For n-sided regular polygons, this method should also work. Pick a vertex $v_0$ and find the area of the triangle between $v_i$ and $v_{i+1}$ for $i$ between 1 and $n - 2$. The triangle fans would look something like this:

Note the variety in triangle size. You have to compute all n triangle areas to get the proper sum. If you recognize that most triangles have a mirror on the other side of the fan, you can cut this down to about $n / 2$ area calculations. However, if you picked $v_0$ be the polygon's center and iterated over $i$ from 0 to $n$ (where $v_o =  v_n$), the fan would look like this:

Those triangles look much more uniform. If you know the area of a single triangle as $A$, the polygon's total area is $nA$, since there are $n$ such triangles in the regular polygon. One float multiplication can be much nicer than multiple float additions. You know, because these ideal polygon cases pop up all the time and you want your code to be efficient ;p

To find $A$, note that the triangles in the regular polygon's fan are isosceles. The two equal sides are of length $r$, the regular polygon's radius. The angle between these two sides is $2π / n  = θ$. A single triangle is shown below with labels:

The area for this triangle, and the total area of the regular polygon, are computed as such:

So in the gifs posted above, you'll notice that a regular polygon starts to look mighty circular when $n$ grows large. For a radius $r$ and center $c$, a circle is the set of points that are distance $r$ from $c$. This is an infinite set, so a circle has an infinite number of points. Since a regular polygon's points are equidistant from its center, it seems reasonable to think of circles as regular polygons with infinitely many sides. For an infinity-gon, the area formula above should evaluate to  $πr^2$. Except, you can't really substitute infinity for $n$, or you get infinity * 0, which is undefined. Like most cases when dealing with infinity, it's better to look at the limit as $n$ approaches infinity!

It's been a few years since I spoke fluent L'Hôpital's rule, so I turned to WolframAlpha for my limit-solving needs. You'll see that the limit of the area function as n approaches infinity evaluates to  $πr^2$, which is a pretty cool result to get. I was really excited, so I signed up for the free WolframAlpha Pro trial to download the step-by-step results to share here. Free users can see up to 3 step-by-step solutions per day, so you can check it out on the WolframAlpha page, too.

That's some damn fine limit substitution! So there it is. Considering circles as infinity-gons is cooler, and the math checks out, too. The rearrangement proof and triangle proof for finding the area of a circle, however, are still way cooler.

The easiest way for me to resolve this was to rollback to previous versions of both VS Code and SSH FS. So far on Windows 10 and 11, VS Code version 1.76.2 and SSH FS version 1.25.0 have been fast and stable. You'll most likely want to use the User download link for Windows rather than the System download link. The main difference is User installs to your account's AppData folder and does not need admin rights, while System installs to your Program Files directory and may need admin rights.

To roll back to a previous version of SSH FS, find it in the Extensions tab on VS Code and click the arrow next to Uninstall, then choose Install Another Version and choose 1.25.0:

You may also want to turn off automatic updates for VS Code. In File > Preferences > Settings, type update in the Search bar and change the method to none or manual.

Maybe once the semester's over, I'll go back to the latest versions of these and see if the issue has been fixed. But in the last few weeks of busyness, waiting > 20 seconds to view a folder's content, open a file, or save changes is just too much to bear.

Before jumping in, I spun my wheels for a bit researching current solutions. Netcode for GameObjects doesn't seem quite ready, but I had heard good things about Mirror as a spiritual successor to UNet. But, I hit some issues that couldn't be resolved after a couple days of debugging, and I wasn't that deep into integration, so I jumped ship and am trying out Fish Networking. Special thanks to this Google Sheet for helping me navigate the options.

So far, it's been promising! I've been able to get two editors running and connected to each other with player movement being properly broadcast. At the moment, I'm testing how well it works with a VR headset. Good news - the Quest 2 connects to an Editor session and we can see each other's movement! Bad and funny news - the Quest player can locally control both their avatar and the other player's. There's also a bit of a strobing issue on the Editor side; the VR player's movement is being shown on the networked player, but the position data is intermittently popping in and out of place. I've not dug too deeply into the configuration to find the cause, but it's nice to make rapid progress. Soon, I'll need to start experimenting with dedicated servers and headless builds to have a server instance running on Solaire. I've also got my eye on the plugin Dissonance (a namely homage to Discord?) for voice chat, which supports Fish Networking as a transport.

Prototypes and tests

Prior to any of this in Unity, I spent some time over winter break experimenting with peer-based networking in the browser. Different ball game - as is demonstrated when you look at Unity networking plugins and every single one has caveats and workarounds listed for WebGL builds - but nothing too crazy. These tests were basically "how do I get 2D player data shared between multiple instances using divs and some client-side JS?" And I built a few different versions, each with client-side JS:

• PHP middleware, MariaDB storage (lol)
• PHP middleware, PHP session storage (forcing all clients to access the same session with a server-side predetermined ID)
• Node middleware using socket.io
• Node middleware using PeerJS
• Node middleware using geckos.io

I treated these as a warmup for (and distraction from) Unity-side networking. There's something very satisfying about VS Code, a terminal, and a browser for your dev env. And let us never forget the incredibly powerful browser DevTools 🙏🏼. It did bring up some interesting questions:

• For small groups ( < 10 players) in collaboratory/social VR, do we want peer-to-peer networking or a client-server arrangement?

Peer networking is appealing - a server is still needed to match players together, but then something (e.g. WebRTC) handles communication between clients afterward. But it gets worse with increased players, and there's no central authority in cases of logging or abuse.

• If we use client-server, do we want to allow players to be hosts, or should we use a dedicated server?

The default for Mirror and Fish is client-server, and out of the box it allows a player to be both the host and a client. There's also support for headless builds (Unity dedicated servers), which will be helpful for persistence.

• With networked play, how should we approach performance recording?

Initially, the project is set up to record one player's movement and voice to disk storage during a performance. This works fairly well on Quest 2, which is currently the most critical platform (I'm not as worried about disk performance on desktop VR). But I don't know how well it will scale with multiple players at a time. Granted these recordings have stayed fairly small, and there's room for some optimization in audio compression and delta transform tracking. Additionally, the recording system for movement data depends on tagged components with unique ids - these ids will also need to be synced across the network before recording begins to avoid crossover issues. So player 1's set of IDs is the same on all ends of the network.

Those are all reasonable problems with solutions I can more or less imagine before committing to code. The trickier one is below:

Timing time

• Given that the network introduces lag, and synchronization is important for performances, how do we rectify the lag with the performance?

By default, each client (besides the host-client) has a bit of lag. Here's a scenario:

• Players P1 and P2, where P1 is the host,
• P1's lag: 0 ms and P2's lag: 25 ms (a generous measure on a good day for Spectrum)
• P1 picks a 2-player scene and clicks start. A 3-second countdown begins before the lines start playing.

The "Start scene" action can be decorated as a remote procedure call so that all connected clients run the same code with the same data. But whether they should run the command at the same actual time is less clear.

1. There's always a 3-second countdown before a scene begins.
2. P1 clicks start.
3. Each connected player knows their latency.

What about offsetting a player's start time by their latency? After the start scene command is received, P2 could begin the scene after 2.975 seconds instead of 3. From P1's perspective, P2's lines and movement come in right on time. But for P2, P1's data would be twice as late, having started the scene early and still having to account for their lag. If P2 starts after 3 seconds and P1 starts after 2.975 seconds, then it's the opposite problem. If both players start the scene "on time", then P1 will feel P2's 25 ms lag, but P2 would seemingly get P1's data right on time!

I'm amazed whenever I play social VR games that handle this problem well, VRChat being at the top of the list. I need to revisit my Gaffer on Games for further reading.

In Record Time

We can record performance data, but how should that work in the networked version?

1. Each client records their own performance and just that.
2. Each client records performances of all players.
3. The server records all performances.

Option 1 is quite appealing - locally the data is all on-time, and consolidating the files after the scene ends should give a reproduction that is fairly faithful to the actual timing. You'll bear witness to lag still - P1 will physically react 25 ms too late to P2's gesture - but it's a close match to how things "should be" for a finished recording.

Option 2 can help reveal differences between experiences for debugging, if it's not too much strain on the disk I/O for the Quest 2.

Option 3 is a nice choice as well, if the server-that-is-also-a-client can keep up with the quests. But if #2 works, then this should too? It's also a good option for a dedicated server and keeping a log of player behavior.

Knowing me, I will most likely ignore the recording challenges until networking is working as I like it, then return to recording and break many things in the process – the fixes of which will in turn break some things on the networking side, and so it oscillates until a steady state is reached. Ah, agile software development. It's too ambitious to expect all of these issues to be worked out before the semester starts again in a couple of weeks, but hopefully we'll have enough in-motion to build on and flesh out by then.

However, if you move away from OpenXR and switch to developing with the Oculus or SteamVR plugins, everything gets a little more complicated. The whole point of using OpenXR is to avoid some of the pain points found with developing for multiple runtimes. Because of a push to get things ready for a demo at Meaningful Play (a post for another day), the entire focus had been getting this to run standalone on a Meta Quest 2. Now that the conference has passed, I've wanted to spend some time on checking how it runs elsewhere. Not to mention, having the ability to rapidly test iterations without a device deployment is invaluable. So the motivation is there just to aid in debugging.

So with the Unity project configured to use OpenXR, I've been able to run the scene under Play mode in the Unity Editor with both the Valve Index (via SteamVR on Windows, Linux TBD) as well as the Meta Quest 2 (via Oculus Link on Windows). The Unity project's Android configuration is also set to use OpenXR and utilize the OculusXR and Meta Quest Features. Deploying this and running it on a Quest 2 has still worked, which is very encouraging.

Woe de plate-forme*

That about summarizes the good parts of using OpenXR. Now here's some of the bad:

Good things come to those who wait, and evidently compositor layer support is coming to OpenXR at some point. Until then, I would need to switch to using OVR for the provider, which means giving up the use of OpenXR conveniences. The same goes for using anything specific to SteamVR. For instance, the HTC Vive Tracker 3.0 devices work well with the Valve Index base stations – hello low-cost mocap! But these trackers aren't available by default in OpenXR. Valiant efforts have been made to consolidate tracking features from OpenXR and the SteamVR plugin for Unity, but the unofficial solutions given in the thread did not work for me after a night spent trying, which is discouraging. I've come across a couple of other threads of people dealing with similar issues and frustrations at the lack of support. So for the moment, further attempts to supplement lower body IK with additional trackers are on pause.

Speaking of lower body movement, Meta's Movement SDK is another thing we currently miss out on by sticking with OpenXR. We gave this a shot soon after release, but came up short in terms of seeing anything usable from the Body Tracking SDK:

We followed the release notes as best as we could, but no luck. After that, I figured it was worth waiting until there were more examples floating around, which thankfully didn't take too long to appear. Now that these are out, I'll give it another try soon to see how things stand. The real win would be getting reasonable lower body tracking on a Quest 2.

Of course, these things utilize the specific features from the Oculus Utilities plugin, which means supporting them comes at a project management cost. So here's the short list of the things that have been interesting and/but troublesome to support in the same context:

• Body and hand tracking from the Meta Movement SDK (Quest platforms)
• Vive trackers (SteamVR and eventually OpenXR)
• Compositor layers for text readability (Oculus and eventually OpenXR)

What's next?

I'm aspiring to have this game supported across a range of devices. This won't solve the problems that came out of limited Kinect longevity, but it will mitigate it as much as possible and ideally make it easier to support newer, better devices as they emerge. OpenXR goes a long way in making this possible, but the tradeoff involves waiting for certain things. When it comes to delivering the best experience on each specific headset, the advice from Unity is to handle it with separate prefabs and/or scenes that cater to that platform's needs. Which... works, but feels a little bad?

I'm not very far off from following this pattern, but it doesn't solve everything. For instance, if I wanted to use Vive trackers and specify SteamVR/OpenVR as the XR plugin over OpenXR, that means changing a project setting, not a scene setting. To avoid issues when making a build for one platform vs. another, I might end up having the same repository cloned into adjacent folders with descriptive names - one for the Quest build, another for the SteamVR build, etc. This is more applicable when dealing with the choice of build platform, as the "Switch platforms" option in the Unity build system can take a staggering amount of time to complete due to asset reimports and such.

Ultimately, things are looking up for supporting multiple devices. Some of the really cool features may have to wait, but that also helps draw boundaries around core features that deserve attention first before extending further support. As fast as things move, there's always a risk of creating something that works out of madness, only for it to be made obsolete by official packages and specification updates. Good problems to have, but bad for burnout and sanity.

Still, who knows what the next semester will bring? A great deal of effort was made in the fall - general quality-of-life improvements across the board - and the wishlist is still quite long:

• live multiplayer support
• downloadable builds for desktop (easy)
• downloadable apk for Quest (easy to give, harder to install since it requires dev mode)
• possible pursuit of AppLab support to ease Quest 2 installation (requires a fairly involved review and approval process that may be complicated to satisfy given our features involve deliberately recording voice and movement data)
• more avatars and play spaces
• web-based performance viewer
• possibly even a web-based version? Hello WebXR!
• internal development for pedagogy and experiment purposes

So, going into 2023, I think we'll have our hands full :)

_{*Platform woes, but spoken in the same manner as eue de toilette}

That thread is worth a look-through. I've always been a sucker for game games. You know what I mean: games that are really a platform for players and creators to make insane stuff: LittleBigPlanet, Second Life, Roblox, Garry's Mod, etc.. VRChat is no exception.

A few years ago, late one night, I hosted a public room called "late nite computering" in Bigscreen Beta. A friend and I were just screen sharing dumb gifs and webms when a guy joined and chatted about his work in VR. That guy turned out to be Ron Millar, whose name I recognized from my old Blizzard game manuals. I gushed about reading the WarCraft II manuals over and over and poring over the art work, and got to hear anecdotes about his time there and Chris Metzen, so that was a childhood meet-your-hero moment come-to-life that I'm still not completely over.

Gushing aside, he told us about the work he had been doing on VRChat, and talked about how exciting the platform development was at that time (and still is). We spoke about social VR experiences we had thus far: Bigscreen was and truly is a comfortable feeling. It can feel immensely cozy having spatialized audio with head and hand gestures from the folks sharing a space with you. IMO, Bigscreen's advantage was having fixed seating with teleportation, so the distance between players is a feature. Ron talked about about what VRChat had to offer: all of the above, plus highly-customized avatars, player-created games and content, live events, concerts, and more.

Temptations

As a platform, VRChat solves quite a lot of problems that I'm anticipating with my own VR project:

• Network connectivity.
• So many avatar choices.
• Interactive props!
• Cross-platform support: desktop mode (KBM and gamepad), PC VR, Quest, and more. The more platform support, the better.
• Voice chat between players. With spatialized audio!
• Hosted rooms, sharing contexts and instances.
• Social features: friends, invites, avatar/voice muting.
• Content creation and sharing.
• Plugin support for lots of nice Unity things: IK solvers, rich text, video playback (even with subtitle files!)
• Honestly decent documentation!

So rather than try to reinvent enough wheels to build a semitruck, it's very tempting to consider VRChat as the platform for shared virtual performances. A self-contained solo-dev project simply won't be able to beat it in terms of audience, never mind the features listed above.

There's always a but...

However, there are always terms and conditions. For starters, the use of Unity components in building a VRChat world is restricted to a whitelist. Makes sense - arbitrary component support from Unity would probably be a nightmare. So, the nice karaoke text player component I made recently? The one that can make super smooth text like this?

That's probably not an option. Which is sad, because it's made using the TextMeshPro component, which is supported, but it's also using its own shader and script to control the shader's uniforms, which doesn't look like it's supported. I might be wrong, but that seems to be the case.

There's also the future plans for this project. I've been working on making performance recordings, which essentially contain layered movement and audio data from each player (or from the same player multiple times) synchronized for playback. I've done this before with Kinect data, but our synchronization setup was painful and prone to drift - even for a single performance's video and audio!

For the movement data in VR, this would actually be fewer points than needed to record Kinect data (20 or 25 joint positions and rotations per player) since it primarily comes down to the avatar's root transform, plus the head and hand transforms. Saving these values and replaying them on a rig set up with FinalIK is – OK, stuff like that is always a bit of a pain, especially when it comes to setting up a generalizable prefab that can basically be spawned in and told to replay the movement – within reach. Even for desktop players, having the player's root position and rotation saved will allow the IK system to approximate the other bits: locomotion, orientation, head gaze. Granted the VR players will steal the show with more natural-looking movement, it's still exciting to think about! And it just doesn't seem possible with VRChat.

Capturing captivating performances

Like FinalIK, AVPro is another external tool supported by VRChat. In their capacity, I believe it's mainly used for audio/video playback. But it also has components for capturing audio and video. I've used these before to record the screen performance for Play the Knave. From the look of it, AV capture isn't on the supported list for VRChat. So if players want to record a performance in-game, they have to do it themselves. Granted OBS, Radeon Record, and ShadowPlay have all been excellent when I've used them. But there's a few reasons why we want to go with a game-based replay file (i.e., record the players' movements and voice), as opposed to a screen-based capture approach:

1. Screen recordings from first-person VR perspectives are not especially compelling. It feels great to play a game in VR, but it feels less than great to watch someone else's first-person VR experience, especially as a recording. This is why some VR streamers make the extra effort to setup external cameras, use depth sensing or green screens to filter out the background, and synchronize an in-game camera's position with the external camera to make more compelling screen recordings. Beat Saber is great for this, and I believe Serious Sam VR: The Last Hope supports this as well, to name a couple.
2. To develop the previous point: a screen recording's perspective is quite limited. Although AI+ML can do insanely cool things with video, it sure would be nice to have a customizable camera position for a scene replay.
3. To develop the point even further: a screen recording's audio is also finite. Yes, the game may have spatialized audio – players at your side sound like they're at your side – but if we want to move the camera around for a replay, we have to make the audio respond in turn.
4. Here's a new point: file size! A screen recording produces a video, which is subject to encoding choices (resolution, audio/video bitrate, framerate, etc.) for optimal file size. We can't get around paying the cost of recording audio, but certainly a performance's lossless^[1] movement data will be considerably smaller than a high-resolution screen recording. Remember: we just have to save a few positions and rotations per frame per player, rather than w ✕ h pixels per frame.
5. Another benefit based on file size: performance! Much kinder to the player's machine (or how about the server instead?) to task them with recording the movement data rather than record and encode a video.

Points 5 and 6 actually lend themselves well to the idea of running a server for multiplayer. You've gotta send movement and voice data between players anyway, so why not have the server just write that down as it happens? Ethics of data collection aside for a moment, this practice can be in the best interest of social VR server admins. You can review these performances to verify reports of abusive players and hold them accountable. I'm trying and failing to find a rant about this issue from someone I follow (I wanna say Doc Ok?). But the issue is an old one that repeatedly appears across multiple platforms:

• https://www.theverge.com/2018/10/24/18019376/oculus-go-samsung-gear-vr-new-record-report-abuse-feature
• https://www.technologyreview.com/2021/12/16/1042516/the-metaverse-has-a-groping-problem/

I'll acknowledge that moderation is a tricky thing. But in the shared spaces that VR can provide, it's probably in your best interest to assume that everything you say and do can be subject to playback at a later time. As a kid, the advice frequently given in school was along the lines of "don't write anything you wouldn't want your mom to see on the front page of tomorrow's newspaper." So that's how old I am. Incidentally, VRChat does have a reporting process in place and they highly recommend including - you guessed it! - video recordings of the behavior being reported.

This post is getting longer than I expected, and I have more to say on this part in particular, so I'll wrap this up and shuffle the follow-up on game recordings to a new post.

In short

Overall, I'm keeping my eye on VRChat as a choice of platform when it comes to <secret VR collaborative performance project>. The component restrictions most likely make it a non-option for my specific needs, but it's hard to argue with a successful platform and its community. Still, it's nice to see so much that does work in one system. Inspiration, at the very least.

By manually turning off the Virtual Mouse component when I'm using the system mouse, the UI interactions work as intended. Happy about that, not so happy about the two of them not being able to stay synced. I attempted to use the scripts referenced above, but had no luck. I actually ended up disabling the GameObject with the Virtual Mouse Input component entirely. Instead, I'm using the code I had before to warp the mouse position.

OK, WEIRD. I don't know what I changed or how it started working, but directional navigation in the UIs (at least for scene selection) are now working. The Cursor + Virtual Mouse stuff is all disabled, and it only starts working once I use the mouse to click on a scene. But once it has focus, it's just... working?! And the same thing does not work for the main controls to begin a scene/calibrate, etc. Possibly because the scene selection canvas has a grid layout group component, so it's got built-in support for that.

Things like this build up over time, and before you know it, you're > 50% convinced that making your own engine would just be better, and somehow faster.

Whelp, I got frustrated surveying the landscape of folks with similar issues, so I stepped away from the KBM/gamepad inputs and worked on getting the Index to be recognized. Involved the following:

• Switching SteamVR to be the default system provider for OpenXR
• Adding Valve Index controller schemes in Unity XR Plugin Management
• Rebinding the input actions for locomotion, pointing, etc.

I also wanted a way to delay the launch of VR mode, so that's being done with InitializeLoader and StartSubsystems calls. In doing so, I found a very nice way to make a coroutine wait for another coroutine to complete: simply call:

yield return StartCoroutine(OtherCoroutine());

Overall this worked, but the player height became drastically high in doing so. This wasn't the case when the XR provider is configured to initialize at startup for your platform. You'd think the fix would involve having manual checks for player height vs. the floor, etc., but my gut told me to just disable the XROrigin component by default and enable it after the XR subsystems are started. Voila! As far as I can tell, doing so didn't cause any issues in desktop mode, which is nice.

Unity does give a warning about the PlayerInput not matching the actions that the UI Input Module supports, and even offers a button to "fix" it. I've yet to press that button but I think it boils down to providing action event handlers for the mousey things: pointer move, left button, scroll, etc. It may not be that, though. I hope it's not, anyway.

Not much more to say than that for now. The semester's end is looming and I need to catch up on grading before spending too much more time on this. On the bright side, the Valve Index arrive today :)

Update: the VirtualMouseInput component is doing pretty well. Not quite at perfect harmony between actual mouse and gamepad-driven virtual mouse, but it's getting there. It appears to be a bigger hassle to use both 1) a software cursor and 2) a canvas in world space. Evidently there will be a fix for this in 1.4, but the latest is 1.3. I'm honestly a little relieved to find other folks with a similar issue.

I gave a passing try and running my scene with OpenXR configured as the XR plugin, but it didn't make it very far. Even with OpenXR as the configuration before, the Oculus provider seemed to just work. The best I got today was the right hand controller moving and aiming the camera?? So, try another night.

• requiring the player to move/look to view the entire menu,
• managing looking vs. selecting state,
• forcing raycast/"laser pointer"-only selections via mouse/hand pointer,
• moving the mouse cursor with a gamepad (shudder)

And yet, I'm doing all of these – for now. Part of the reason is due to using Unity's "new" Input System and their XR Interaction Toolkit. The purpose of such choices is to save time and effort in the long run. For instance, the same "move" action and player logic can be bound to multiple inputs simultaneously, such as: WSAD on a keyboard, the left analog stick on a gamepad, and a left-hand XR controller's  analog stick. Some of the work is deferred to configuration files outside of script, but the jist isn't so bad to work with:

1. Give an action a name
2. Bind it to input(s) and configure
3. Define an "On(ActionName)" method in your player class

Still, tedium creeps in. You know how games will often seamlessly switch between input prompts? "Press enter to start" becomes "Press Ⓐ to start" as soon as you touch a controller's analog stick. Suddenly, the benefits of handling input actions like an interface is restrictive – you aren't able to see which device the input came from in the handler, so you aren't sure if and when to switch displays. Additionally, allowing users to rebind becomes quite involved, too. The InputSystem allegedly has some things built-in to help, but nothing is free. In turn, there's a plugin for the InputSystem called PlayerInput that's supposed to help with both of these issues, and so the perpetual struggle to work with Unity input and not against it continues.

With all that in mind, I'm happy I can test out KBM and gamepad interactions in the same scene. Longer-term goals for the UI options include having directional navigation in the menus for all devices - arrows on keyboard, sticks/dpads on controllers - to avoid the need for raycasting. That could be a single raycast to select the menu using any collision on its canvas, then transition to an interaction state so the directional controls navigate between menu options. And which is more evil?

Let's see how well that test poll does...

Side note: I took a major detour in writing this post that went something like this:

1. Can Ghost embed polls? Well, it can embed Google Forms...
2. Can Ghost embed JS and HTML? Yes! Now we're talking...
3. Can I whip together a poll database? Yes! (You see where this is going)
4. Can I write some PHP to interface with the database and invoke with with cURL?
5. How about with fetch from JS?
6. Can I generate the form in HTML and the submit event handler in JS from the PHP?
7. Can I add the returned JS to the page so it can be executed on the form submission?
8. Can I show results for a poll after it's been submitted?
9. Can I allow users to change their vote?

Conclusion: cascading "Yes!"es are dangerous.

LBS (left) vs spring-based skinning (right)

With normals visible

Armadillo

Spring forces with LBS

Spring forces with rigid skinning

Splits wireframe

Left: all forces but length. Loss of detail around abdominal muscles
Right: all forces. Better preservation, minor loss

Same, but with normal map

LBS ROM

A little difficult to compare due to video size. Will crop these so more of the mesh is visible on each side without so much filler space in between.
Left: without forces
Right: with forces. A couple of frame-to-frame artifacts (slight bulging on the arms), but also better volume preservation in the bends.

Force combinations

Length force only. In this LBS pose, the mesh has 14% less volume than when it's in bind. After the length forces are applied, the mesh has 3.74% less volume than bind.

Length and surface forces. Almost identical to using length-only forces

Length, surface, attachment torque, and bone torque

Using position-based updater - no acceleration, only velocity.

Using Physics-based updater - acceleration, velocity

Today I added some controls to my spring force solver. My goal is to have the spring solver "always-on" (updating the mesh once per frame based on the spring forces), but whenever I'd try letting the solver run continuously, the spring forces would eventually skew too much to be considered usable or stable. So a mesh skinned with LBS may look like this:

And using the results from LBS as the initial state, this is how it looks after a few rounds of the solver:

The middle regions look OKish, but vertices near the root and the leaf have distorted considerably. Eventually the spring forces stop altering the mesh and it looks like this:

So it definitely needs some help. To help stabilize the solver, I considered 2 choices:

1. Modify the solver so vertices along the bone can have varying flexibility. For example, vertices near the center of the bone should essentially retain their shape from LBS, while vertices near joints should be allowed to flex as much as possible.
2. Try to add an additional force that counteracts the tendency for the mesh to skew.

Flex weights

I went with the first approach and implemented a few simple functions as flex weights. Each bone gets essentially a 1D, 1-channel float texture that the solver can use to accordingly weigh the computed force vectors. If the flex weight for certain vectors is 0, then this effectively adds static constraints to the solver and should prevent the skewing collapse. Here's what some of the flex functions look like:

Flex shape: U
Function: (float t) { return glm::pow(glm::abs<float>(t - 0.5f) * 2.0f, 2.0f); }

This represents a smooth change of flexibility across the bone, allowing the ends to vary but keeping the center rigid. "U" can see the effect this has on the mesh - since LBS introduces scale vector length changes, this makes the inner bends appear pinched as vertices in between the rigid regions deform to reduce the spring system's energy.

Flex shape: /
Function: (float t) { return t; }

The linear flex function makes vertices at the start of a bone rigid, and more flexible as the mesh deforms. This looks a little more acceptable than the U function, but it makes for some strange deformations on the leaf joint's vertices.

Bone torque

With this approach, I'm setting the variable flexibility concept aside for a moment and focusing on what causes the skew during long runs. The spring force solver works by computing spring forces between adjacent vertices. One of those spring forces is a torsional force between adjacent scale vectors. So if the solver is left to run on its own, it will continue trying to minimize the scale vectors angles - there is no incentive to stop running if the angles can still be minimzed. This is how the skewing result happens.

I was counting on convergence checks to prevent this from happening, but that's less reliable for an always-on solver. Instead, I added a new torsion force. This one measures the angle between a scale vector and its connecting bone at bind, then when the solver is running, it computes a force to restore this angle.

Here's a look at the mesh with bone torque and stiffness coefficient=1

Not bad, maybe a little too rigid. The inner bends still look like they're overlapping, and there's some noticeable gaps on the outer bends. Here's the result using coefficient=0.1

The outer bends look a little more pointy, but the inner bends have converged to nice pinches. The leaf joint's vertices are staying put, too!

Combined

Here are some figures with both flex weights and bone torque.

Bone coeff=0.1, flex U:

The pinching artifact returns, but looks a little more consistent and stylesque.

Bone coeff=0.1, flex /:

Bone coeff=0.1, flex \:

Subtle differences in the squish direction between / and \...

This is a post comparing two techniques for mesh deformation: spring forces, and ARAP. I've written about as-rigid-as-possible deformation before, but I haven't written much about springs before. The spring forces solver run on an OpenGL Compute shader, while the ARAP solver runs on the CPU using Eigen/LibIGL.

In this post on attachment and rigidity, I brought up some artifacts that can occur when deforming a mesh using a skeleton. Here's an example:

For the record: The mesh I'm using has 5634 vertices, 11264 faces, and 17 bones. The plus-sign shape makes it very useful for deformation studies.

Similar artifacts happen with LBS and DQS:

The skinning method's rigidity is causing that drastic stretch artifact. In the last post, I also mentioned a few things I might try to remove the artifact. Let's start with ARAP. I'm using LibIGL's implementation with default parameters: Young's modulus is set to 1, and the maximum iteration count is 100.

ARAP has a preprocess phase for the selected boundary vertices (highlighted yellow). Any time the selected set changes, the preprocess is reprocessed. The solver runs separately. For the deformation above, compute time was ~400 ms, and solver time was 1100 ms.

Now let's look at the same kind of deformation, but used with spring forces. The techniques are rather different, but I'll use the same cap of 100 iterations. Note: ARAP runs and solves everything in one pass. The spring forces can run like this too, or I can set it to run one iteration per frame to show the change over time. I'll show both below.

There's no preprocess phase for the spring force solver. If I'm allowing the solver to run until complete, then the total compute time is under 5 ms. When I restrict the solver to running once per frame, the total compute time is closer to 12 ms. The additional time comes from frame-to-frame overhead - mostly updating and binding shader storage buffers. Even so, it's 12 ms compared to ARAP's total of 1500 ms: around 125x speedup.

So, the performance improvement is nice, and the runtime makes it almost suitable for real-time use. However, the techniques are quite different. ARAP is an energy minimization problem solved with a sparse linear system. It depends on the mesh's adjacency matrix and edge lengths at bind.

I've started by creating springs out of every mesh edge. Let's bust out Hooke's Law:

$$
\begin{equation}
F = -kx
\end{equation}
\label{eq:hookeslaw}
$$

where $F$ is the spring's force, $k$ is the spring constant that represents the spring's stiffness, and $x$ is the spring's displacement from rest position. The displacement's sign determines the force direction, and at 0 displacement, the spring exerts no force.

We can also compute the potential energy stored in a spring:

$$
\begin{equation}
U = \frac{1}{2}kx^2
\end{equation}
\label{eq:springpe}
$$

How does this relate to our mesh? If the mesh is a set of vertices and edges $M=(V,E)$, then we can compute the energy of any edge given a deformation $M'=(V',E)$:

$$
\begin{equation}
U(i,j) = \frac{1}{2}k (||V'(i)-V'(j)||-||(V(i)-V(j)||)^2
\end{equation}
\label{eq:edgepe}
$$

And the total potential energy in the mesh deformation:

$$
\begin{equation}
E(M') = \sum_{i,j \in E} U(i,j)
\end{equation}
\label{eq:meshenergy}
$$

Now things can get interesting. The edge's energy is a quadratic function of the displacement. The mesh's energy is a sum over the edge energies. A good way to reduce the mesh's energy is to redistribute large displacements across many adjacent edges. Because of that square power, it ends up being better to have many edges with a little displacement than to have only a few edges with large displacements.

Glossary:

• ABCD: Attachment-based character deformation. Performs rigid skinning first, then deforms mesh vertices relative to a point of attachment on the skeleton.
• ARAP: As-rigid-as-possible surface modeling technique. Uses ROIs to define mesh deformation area. Switches between optimizing vertex positions and rotations to minimize cell energy until convergence.
• LSE: Laplacian surface editing. Uses ROIs to define mesh deformation area. Attempts to minimize changes to differential geometry.
• ROI: Region of interest on a mesh. Geometry is separated into solved (fixed in place) and unsolved (allowed to vary) groups. Solved groups include region boundaries to control the deformation size, and handles that can be deformed by the user to drive deformation. Unsolved groups include whatever geometry is contained within the boundaries.

This post concerns the abilities and limitations of ABCD. Namely, the attachment system produces the most visually appealing results when mesh skeletons are chains: single parent, single child.

That's basically the poster child mesh of ABCD. Uniform thickness, a chain skeleton, and reasonable bone lengths. If the mesh and skeleton are bent, the default attachment scheme (direct projection onto nearest bone) fails considerably:

Why exactly is that a failure? Visually, it seems clear: the inner bend attachs to the skeleton strangely, leaving these large gaps in between adjacent attachment points. Intuitively, those attachment points should probably be close together. Why?

To deform a mesh vertex around a bending joint, ABCD has to find answers to a few questions:

• How close is the vertex to the bone? Is it within the bending joint's ROI?
• Is the vertex on the inner side of the bend, or the outer side?
• Is the bending joint on the parent side of the bone, or the child?
• How much is that joint bending, and along what axis?

Based on those answers, the vertex receieves a percentage of the joint's bend (including 0% if there's no influence). For this to smoothly deform the mesh, it requires some preservation of distances between attachment point, relative to the mesh. That is, if two mesh vertices are adjacent, then their attachment points should be, too. Or at least, they should have roughly the same distance between them. Watch what happens when ABCD skins the bent cylinder, both before and after smoothing the attachment points:

Pretty gross! Granted that the mesh doesn't seem well-suited for large outer bends, I was hopeful that ABCD would do a decent job uniformly straightening out the surface. For comparison, let's look at an inner bend:

Not a big surprise, but the mesh is better suited for inner bends. Still, it'd be nice to have more consistent behavior around the outer bend.

ABCD accommodates changes in bone length fairly well, but it's still problematic. Because of the linear mapping between attachment points and bone length, large bone length changes introduce their own discontinuities. The vertices along the bone are consistently-spaced with each other, and so are the vertices outside the bone's influence. But the boundary between them is too strong, and results in poor spacing:

The same artifact could occur with large bone twists. Ultimately, the issue is this: ABCD behaves as though every vertex is mapped to exactly one bone in the skeleton. After rigid skinning with this data, it does what it can to smooth out the regions in between rigidly-skinned meshes. The issue gets even worse with more attached skeletons (joints with multiple children). Let's start with the LBS version of a pose:

Not pretty, but compared to this (ABCD, direct projection):

There are two bad artifacts: one in the upper left, and one in the lower left. They are both the result of the single bone vertex mapping. One way to try and "fix" them would be to edit the skeleton: uniformly extend the bone lengths around the root so the joints extend past the saddle-shaped regions, making the central region more rigid in the process. Better results, but only because the mesh is more rigid. If we want that region to deform smoothly, then we need another approach. What happens if we just try to smooth the attachment points a bit?

That's actually quite a bit better! Attachment quality definitely makes a pretty big difference in the skinning results. How does it handle a few more bends? Here's a comparison between ABCD and LBS. The biggest issue seems to be the skin weight quality I got from Maya (geodesic voxel @ 512 resolution, classic linear weights. I thought that would be more than sufficient!).

And here's another clip with alternating up-down bends on the "limbs":

It's promising, but we still need to address the rigid boundaries between bone regions. The surface editing approach to shape control is to freely deform a small set of handle geometry (vertices, triangles), then optimize a linear least squares problem to minimize energy changes for each vertex. These energy values are defined using the vertex's neighbors, so mesh adjacency and spacing are critical factors. What if we could do the same for skinning?

• Some mesh vertices are more "cleanly" mapped to the skeleton than others. Think about the middle of an arm bone compared to the center of the chest.
• Since we have a strong understanding of how those vertices deform, we should have some sense of how rigidly attached to the skeleton they are. Let's call this vertex rigidity.
• Based on that rigidity and an initial rigid skinning phase, we should be able to inform a solver (similar to that used for surface editing) on how much "wiggle room" each vertex is granted.
• With the right optimization, this should result in smoother deformations along bending joints (low rigidity) while preserving the desired shape for singular chains (high rigidity).

In application, this will probably be a 2-stage pass. Stage 1 is ABCD skinning with the GPU, stage 2 is optimization using a solver. Hopefully, since most vertices will already be near their ideal position, the solve time will be short. This setup is similar in structure to the elastic implicit skinning work, which does rigid isosurface deformation, then vertex marching to restore offset, then interleaved tangential relaxation/ARAP passes to achieve global deformation.

Questions of which solver to use are harder to answer immediately. The first guesses would be:

• ARAP, but with using scale vectors (vertex position - attachment point) to determine rigidity rather than position alone.
• LSE, but this would probably look bad with large rotations, as even the optimized LSE has limitations for large transformations.
• Spring force solver. Possibly a more natural approach to a problem involving rigidity, and we can likely take advantage of the attachment and scale vectors as constraints to preserve thickness, etc. May have stability issues, but could also be very easy to implement and test. This may be where I go first with experiments, but first, more research!

More to come later...

Institute of Visual Computing of ETH Zürich
Prof. Olga Sorkine-Hornung

Research topics

• interactive shape modeling and animation,
• digital geometry processing,
• digital fabrication,
• image and video processing

Publications

And many more...

libigl - A C++ Geometry Processing Library

A simple C++ geometry processing library with wide functionality, including construction of sparse discrete differential geometry operators and finite-elements matrices such as the contangent Laplacian and diagonalized mass matrix, simple facet and edge-based topology data structures, mesh-viewing utilities for OpenGL and GLSL, and many core functions for matrix manipulation which make Eigen feel a lot more like MATLAB

Github / Tutorial

What I like about it:

• header-only library, but supports external high-performance libraries
• Uses Eigen for matrices and linear algebra (another header-only library)
• Has a ton of useful, self-contained, and best of all, working tutorials!

In my experience, 99% of the effort in setting up the tutorials was performed by git and cmake. But compilation time takes a while (sometimes a drawback of header-only libraries), so I'll go through the setup process, but then I'll run demos from a precompiled version.

Tutorials to try:

• 106_ViewerMenu
• 201_Normals
• 202_GaussianCurvature
• 205_Laplacian
• 405_AsRigidAsPossible
• 605_Tetgen
• 707_SweptVolume
• Taking requests!

What you need (Windows):

• Git
• CMake
• Visual Studio 2017 Community (2015 should also work, but 64-bit is a must either way!)

The scene files are changing once again. The progression so far has only been from XML files to json, with changes to object behavior to better align the two. Essentially I want everything serializable for saving state between files.

Right now, the json file just contains a bunch of fields to specify how to load a mesh. Here's an example:

{
	"name": "cylinder scene",
	"meshes": [
		{
			"name": "cylinder",
			"description": "5 bone cylinder",
			"path": "../../data/models/cylinder",
			"skeletonFile": "cylinderskeleton.dae",
			"meshFile": "cylinder.obj",
			"tfType": "mat4",
			"skinning": {
				"load": [ "heatmap-0.45.weights" ],
				"methods": [ "ir" ]
			},
			"attachment": {
				"methods": [ "DirectProjection" ]
			}
		}
	]
}

I'm used to tweaking this file by hand, then reloading the program. But the last few overhauls made me outgrow this approach (read: I have a better UI that will be faster than this), so instead I'm interested in doing it the "right" way. I'm interested in a file storage system that:

• maintains a directory for data storage. Textures, skin weights, UV sets, and animations would all be located here. Currently an input mesh knows its root folder ("path" in the above json) so it can load files relative to it. I want to do the same thing for the entire scene's data.
• is binary-formatted. I'm working with meshes, which means lots of sequential data of a known size. Loses human readability, but I'm OK with that. The documentation will likely include a schema for parsing the scene file if necessary. Why not just stick with json text and list out the relevant binary files to load? Well, that makes file history harder to control. I think it's still OK to have the mesh and skeleton data (skin weights, animations, UVs, etc.) exportable and importable to their own binary format for reuse, but the general scene file should provide this information on its own.
• preserves object state. Deform objects are responsible for changing behavior on meshes and skeletons. They're applied in order of creation and generally support undo/redo operations. Serializing these would allow me to save out command history, but I'm not convinced I want to copy Maya that much just yet. To have the current state preserved would be enough for iterative animation development. Reminder that the software's goals include supporting fast and easy exploration of ideas
• I use a lot of smooth splines created by input device position (list of 2D points on desktop, or 3D points in VR, possibly filtered down with RDP, then chained smoothly together as a series of Bézier curves), and I want them available in between sessions. Or at least, I want the abstraction that they ultimately provide available in between sessions.
• Flexible parsing. I want it to handle gradual changes to the format gracefully. An old scene file should still load if it doesn't have something a new parser expects.

Let's warm up with some of the types we care most about storing

• Mesh
• numVertices (size_t), followed by that many MeshVertex values representing the mesh vertices at bind
  • MeshVertex: position (vec3), normal(vec3), and uv(vec2). Ordered by index
• numFaces (size_t), followed by that many uivec3 representing the mesh faces
• SkinWeights
• name (string), numVertices (size_t), followed by that many SkinWeight (indices (ivec4), weights (vec4))
• SkeletonAttachment
• name (string), mesh(Mesh), and binding (SkeletonBinding)
• SkeletonBinding:
• numVertices (size_t), followed by that many VertexAttachment (boneIndices (ivec2), t (vec2))
• Skeleton
• numJoints (size_t), followed by output of the root Transform. Each one processes itself, followed by its children.
  • Transform:
    • name (string)
    • color (vec4)
    • parent index (int) -1 if no parent
    • translation (vec3): bind translation
    • rotation (quaternion): bind rotation
    • scale (vec3): bind scale
    • numChildren (size_t), followed by that many child indices (int)