Bouldering and Computer Vision

Bouldering and Computer Vision

As per my previous post, I have begun delving into the topic of combining computer vision and deep learning techniques with my recent hobby, indoor climbing, specifically bouldering. I intend to use this page as a comprehensive note to collect what I have discovered regarding this topic and to potentially further develop real-world algorithms and applications.

Disclaimer: As I have no experience in outdoor climbing, all terms related to climbing mentioned here are in reference to bouldering.

Thoughts

There are two general goals, which I think climbers maybe benefits from enhancing their bouldering experience in meaninful ways; not just fancy pose estimation.

  1. Tutoring on bouldering moves for beginner climbers.
  2. Recording bouldering moves in a quantified-self manner.

But of course, these are only my own opinions so far. I plan to gather more feedback from the climbers around me to discuss how they would feel about incorporating technologies in climbing, whether it's good or bad. As for the second point, I've noticed that current apps for climbing journaling have a lot of room for improvement in terms of UI; and incorporating computer vision techniques can better enhance the user experience. Well, some may also ask, why not just use a notebook? I don't have an answer for that.

To achieve these two goals, there are some detailed (and interesting) problems to be explored or further improved based on existing projects, products, or literature.

Bouldering Holds Segmentation

This idea involves attempting to extract bouldering holds with different colors and shapes from a specific photo in 2D image space. The output would identify the type of a certain hold and understand which holds belong to which route. One naive solution might involve using k-means clustering in HSV or Lab color space, but some prefer using machine learning models trained on ground truth labeled data to perform dense prediction afterward.

A naive solution that applies k-means clustering after an color-quantized image.
Each bouldering gym and manufacturer has different colors and shapes for the holds, so the algorithm should adaptively extract segments of the holds with more flexibility while maintaining detection precision. Above are some samples of holds that appeared in my local gym.

I've also considered whether automatically generating bouldering routes is a good idea. However, at the moment, I believe that without creativity and human interaction, purely randomly generated routes have no meaningful value, and you lose the fun in climbing. This topic will be discussed further in the future.

Some relevant resource I've collected so far regarding this subsection:

3D Reconstructed Bouldering Routes

As you may have seen in many IFSC videos, when the competition climber takes the stage, there is typically an animation beforehand showing what the route looks like. The idea here is to reconstruct the entire 3D scene from a user's captured photos, for later use in simulation or route analysis. One possible solution is to use Gaussian splatting for precise reconstruction, and an existing application that all users can try is Luma Studio.

Center-of-mass (CoM) Analysis

The goal is to analyze the CoM based on 3D pose estimation, to determine how climbers might benefit from adopting different postures to improve their balance and reach the next hold.

$$C=\frac{1}{M'}\sum_{i=1}^{M'}w_i\mathbf{j}_i$$

where $\mathbf{j}_i$ represents the 3D positions of the mid-points between two connected joints (e.g., we define the CoM of a left lower arm as a mid-point between a hand joint and an elbow joint), and $M'$ denotes the number of the mid-points.

💡
Application 1: Adjustment on virtualized posture through simulation

had fun climbing a V4-5 slab today at Mosaic Berkeley. It was a bit of a bummer because I almost finished it using my own approach. However, I only managed to touch the second-to-last hold before losing my balance.

0:00
/0:24

While I didn't have any energy left to physically validate my next idea for that movement before falling off, I wonder if applying some CoM analyses in a virtual 3D scene would be useful.

What if I drag the red dot shown above to the left and slowly move my shoulder in that direction (I will also have to twist my right palm slowly)? Then, if I catch the second-to-last crimp with my left hand, would it be more comfortable without losing my balance?

What I imagine is that after the pose estimation is applied to the video, we will have 3D coordinates for the body (and also the center of mass). Wouldn't it be nice to let the user drag a certain joint or part of the pose to another location to see if it falls off? In other words, this would allow us to validate the idea for that movement virtually without having to climb the wall. This is also relevant to the "Virtual Agent for Learning Progress Simulation" section. We could set a CoM goal for the virtual agent to learn to reach using reinforcement learning, allowing it to learn how to avoid falling off the wall through countless simulations.

Bouldering Routes Evaluation

This idea revolves around exploring possibilities of "solving" bouldering routes or evaluating the difficulties of certain routes. While grading can sometimes be difficult for humans to evaluate, this challenge could lead to making grading more quantifiable.

Some existing works:

Virtual Agent for Learning Progress Simulation

This may seem absurd to some, but the essence of this idea is to apply deep reinforcement learning to potentially predict how a climber may progress and become better, based on existing quantifiable metrics such as finger/arm/core strength or flexibility. Imagine duplicating a climber's biometric data (finger strength, height, arm length, etc.) in a simulated world and allowing the agent to simulate the beginner climber's movements through hundreds of thousands of iterations, generating an evaluation report or even a training plan. This idea is still in its early stages, and I have not yet delved into any specific details. Updates will follow in the future.

Real-to-Sim Skill Learning

Motivated by the paper from X. B. Peng et al. titled "SFV: Reinforcement Learning of Physical Skills from Videos," this idea involves utilizing deep reinforcement learning to study the skills demonstrated by world-renowned climbers such as Adam Ondra and Janja Garnbret. The goal is to compile a set of valuable information and instructions based on these climbers' movements into a knowledge base for learning purposes.

Other Relevant Information

Last update: Apr 16, 2024
  • Red-Point "is the first ever Augmented Reality Climbing Guide that works with no internet connection."
  • Climbalyzer "is an AI powered 3D body position and movement analysis app for coaches and self-coached climbers."
  • Belay.ai: "AI that understands climbing." Here's a video where the Norwegian climber Magnus reviews their technologies .
  • Valo Motion: "Gamify your [ordinary] climbing walls with our augmented climbing wall game platform."
  • Babylon.js is an open web rendering engine that uses WebGL. The latest Gaussian splatting feature may be helpful for quickly building a proof-of-concept 3D demo related to any of the ideas mentioned above.