Did you miss Transform 2022? Check out all the summit sessions in our on-demand library now! See here.
While you do push-ups, squats or ab work, heft dumbbells, jump or stretch, a device on your TV will follow you throughout your workout.
Tracks your form, your completion of an exercise (or lack thereof); you will receive recommendations on what to do next in cardio, bodyweight, strength training or yoga workouts; and you can work towards achievement badges.
This is the next-level home fitness experience powered by the Peloton Guide, a camera-based, TV-mounted training device and system powered by computer vision, AI, advanced algorithms and synthetic data.
Sanjay Nichani, head of Peloton’s computer vision team, discussed the technology’s progress — and ongoing improvements — in a livestream this week at Transform 2022.
Peloton Guide’s computer vision capability tracks members and identifies their activity, gives them credit for completed moves, provides recommendations and real-time feedback. The “self mode” mechanism also allows users to pan and zoom their device to look at themselves on the screen and make sure they are showing the right shape.
Nichani emphasized the power of metric-driven accountability when it comes to fitness, saying “the insight and progress is very encouraging.”
Getting to the final Peloton Guide commercial product was an “iterative process,” he said. The initial goal of AI was to “bootstrap quickly” by finding a small amount of custom data and combining it with open-source data.
Once a model is developed and deployed, detailed analysis, evaluation and telemetry are applied to continue improving the system and making “focused improvements,” Nichani said.
The machine learning (ML) flywheel “all starts with data,” he says. Peloton’s developers use real data supplemented with “a heavy dose of synthetic data,” creating datasets using nomenclatures specific to exercises and poses combined with appropriate materials in reference.
The development teams also applied pose estimation and matching, accuracy recognition models and optical flow, which Nichani called a “classic computer vision technique.”
One of the challenges of computer vision, said Nichani, is the “wide variety of attributes that must be considered.”
- Characteristics of nature: background (walls, floor, furniture, windows); lights, shadows, reflections; other people or animals in the field of view; equipment used.
- Member attributes: gender, skin color, body type, health level and clothing.
- Geometric attributes: Setting the camera-user; height and tilt of camera mounting; member orientation and distance from camera.
Peloton’s developers have conducted extensive field testing to validate content cases and have included a capability to “nudge” users if the camera is unable to do so due to any number of issues. cause, said Nichani.
The challenge of bias
Fairness and inclusivity are both important in the process of developing AI models, Nichani said.
The first step to mitigating bias in models is to make sure the data is diverse and has a sufficient amount of different characteristics for training and testing, he said.
However, he said, “a variety of data alone cannot ensure unbiased systems. Bias tends to creep in, in deep learning models, even if the data is unbiased.
Through Peloton’s process, all source data is tagged with attributes. This allows the models to measure the performance of “different slices of attributes,” ensuring that no bias is observed in the models before they are released into production, Nichani explained.
When bias is found, it is addressed – and ideally corrected – through the flywheel process and deep dive analysis. Nichani said Peloton’s developers are looking at an “equality of the odds” measure of fairness.
That is, “for any particular label and attribute, a classifier predicts that the label is the same for all values of that attribute.”
For example, to predict whether a member does a crossbody curl, a squat, or a dumbbell swing, models are built to assign body type characteristics (“underweight,” “average ,” “overweight”) and skin tone based on the Fitzpatrick classification – which, although widely accepted for classifying skin tone, still has some limitations
However, any challenges are outweighed by significant opportunities, Nichani said. AI has many implications in the home fitness realm — from personalization, to accountability, to convenience (e.g. voice-activated commands), to guidance, to overall engagement.
Providing insights and metrics can help improve a user’s performance “and really push them to do more,” Nichani said. Peloton aims to provide personalized gaming experiences “so you don’t have to watch the clock when you’re working out.”
See the full length conversation from Transform 2022.