AR Superpower: Trajectory Prediction

Continuing my AR prototyping in VR attempts, I prototyped an idea for a AR ‘superhuman ability’: showing the trajectory of objects to the user for easier catching:

This is done by running a physics simulation of the thrown object for 2 seconds once it’s thrown. Since the controller is tracked, with accurate position and velocity, I created a rigidbody in Unity then ran the physics simulation to run a few seconds ahead of time, showing the result in a ghosted version of the controller.
The trajectory calculation happens when I let go of the controller (for the Vive wand I tracked this by when the grip buttons are released). And since it’s an actual physics simulation that is ran (rather than a ballistic trajectory function) it will accurately bounce off the walls and ground. The walls have been previously scanned with ZedMini’s spatial mapping so it has a Unity mesh collider. Physics run ahead of time by calling Unity’s Physics.Simulate method. For better visuals I increased the bounce restitution of the physics body, otherwise it would have just slided off the wall.

You can see it’s accuracy with a longer throw in slow motion:

I had a lot of feedback from the video that the trajectory display wouldn’t actually be helpful – that since the mind does this anyway the trajectory display would just be in the way. This is definitely true for the short throws shown in the video – but I do think it’s usefulness would increase with farther trajectories.

The problem with this approach is that it only calculates the trajectory once (after letting go), and that it shows you the complete path rather than just where you should put your hand to catch. The trajectory path is usually accurate – but other factors like air resistance or inaccuracy with initial tracking could mean the object will land off of it’s predicted spot.

So because of that I had the idea of not showing the full trajectory path, but just the important position of where you should catch it. And update this position constantly:

The simulation is ran until the physics object hits a specific height. When throwing to the ground that is just the ground – but when throwing back and forth between hands/people that height is it’s height it was at when thrown.
By updating the prediction every frame it now gets more accurate as it gets closer to it’s landing.

From experimenting with this briefly I absolutely think it’s a improvement for easier catching. I can now accurately catch a controller before having even seen it be thrown.

Ideally both my hands would be tracked, and the position shown would be the best possible spot to catch the controller (made based off of the closest distance the simulated object gets to either hand). This wouldn’t show the path of the flight, but that information isn’t needed when just trying to catch the object at the right position. Also ideally this wouldn’t require a perfectly tracked VR controller, but any object the headset sensors could see.

Here is the Update code for the constantly updating prediction (for the old way this method would only be called on letting go of the controller)

void Update()
{
	for (int i = 0; i < GhostModels.Count; i++)
		GhostModels[i].gameObject.SetActive(false);
	
	Body.MovePosition(Tracked.transform.position);
	Body.MoveRotation(Tracked.transform.rotation);
	Body.velocity = Tracked.GetVelocity();
	Body.angularVelocity = Tracked.GetAngularVelocity();

	for (int i = 0; i < GhostModels.Count; i++)
	{
		if (thrownGoalHeight != -1)
		{
			// show trajectory just when at 'goal height'
			if (Body.position.y < thrownGoalHeight)
			{
				GhostModels[i].gameObject.SetActive(true);
				GhostModels[i].transform.rotation = Body.rotation;
				GhostModels[i].transform.position = Body.position;
				break;
			}
		}
		else
		{
			// show full trajectory
			GhostModels[i].gameObject.SetActive(true);
			GhostModels[i].transform.position = Body.transform.position;
			GhostModels[i].transform.rotation = Body.transform.rotation;
		}
		for (int j = 0; j < SimulateAmountPerGhostModel; j++)
		{
			Physics.Simulate(Time.fixedDeltaTime);
		}
	}
}