Bouldering and Computer Vision

As per my previous posts, I have begun delving into the topic of combining computer vision and deep learning techniques with my recent hobby, indoor climbing, specifically bouldering. I intend to use this page as a comprehensive note to collect what I have discovered regarding this topic and to potentially further develop real-world algorithms and applications.

Disclaimer: As I have no experience in outdoor climbing, all terms related to climbing mentioned here are in reference to bouldering.

Ideas

There are two general goals, which I think climbers maybe benefits from enhancing their bouldering experience in meaninful ways; not just fancy pose estimation.

Tutoring on bouldering moves for beginner climbers.
Recording bouldering moves in a quantified-self manner.

But of course, these are only my own opinions so far. I plan to gather more feedback from the climbers around me to discuss how they would feel about incorporating technologies in climbing, whether it's good or bad. As for the second point, I've noticed that current apps for climbing journaling have a lot of room for improvement in terms of UI; and incorporating computer vision techniques can better enhance the user experience. Well, some may also ask, why not just use a notebook? I don't have an answer for that.

To achieve these two goals, there are some detailed (and interesting) problems to be explored or further improved based on existing projects, products, or literature.

Bouldering holds segmentation

This idea involves attempting to extract bouldering holds with different colors and shapes from a specific photo in 2D image space. The output would identify the type of a certain hold and understand which holds belong to which route. One naive solution might involve using k-means clustering in HSV or Lab color space, but some prefer using machine learning models trained on ground truth labeled data to perform dense prediction afterward.

A naive solution that applies k-means clustering after an color-quantized image.

Each bouldering gym and manufacturer has different colors and shapes for the holds, so the algorithm should adaptively extract segments of the holds with more flexibility while maintaining detection precision. Above are some samples of holds that appeared in my local gym.

I've also considered whether automatically generating bouldering routes is a good idea. However, at the moment, I believe that without creativity and human interaction, purely randomly generated routes have no meaningful value, and you lose the fun in climbing. This topic will be discussed further in the future.

Some relevant resource I've collected so far regarding this subsection:

3D Reconstructed Bouldering Routes

As you may have seen in many IFSC videos, when the competition climber takes the stage, there is typically an animation beforehand showing what the route looks like. The idea here is to reconstruct the entire 3D scene from a user's captured photos, for later use in simulation or route analysis. One possible solution is to use Gaussian splatting for precise reconstruction, and an existing application that all users can try is Luma Studio.

Center-of-mass (CoM) Analysis

The goal is to analyze the CoM based on 3D pose estimation, to determine how climbers might benefit from adopting different postures to improve their balance and reach the next hold.

Bouldering Routes Evaluation

This idea revolves around exploring possibilities of "solving" bouldering routes or evaluating the difficulties of certain routes. While grading can sometimes be difficult for humans to evaluate, this challenge could lead to making grading more quantifiable.

Some existing works:

Graph Neural Networks in Classifying Rock Climbing Difficulties

Virtual Agent for Learning Progress Simulation

This may seem absurd to some, but the essence of this idea is to apply deep reinforcement learning to potentially predict how a climber may progress and become better, based on existing quantifiable metrics such as finger/arm/core strength or flexibility. Imagine duplicating a climber's biometric data (finger strength, height, arm length, etc.) in a simulated world and allowing the agent to simulate the beginner climber's movements through hundreds of thousands of iterations, generating an evaluation report or even a training plan. This idea is still in its early stages, and I have not yet delved into any specific details. Updates will follow in the future.

Real-to-Sim Skill Learning

Motivated by the paper from X. B. Peng et al. titled "SFV: Reinforcement Learning of Physical Skills from Videos," this idea involves utilizing deep reinforcement learning to study the skills demonstrated by world-renowned climbers such as Adam Ondra and Janja Garnbret. The goal is to compile a set of valuable information and instructions based on these climbers' movements into a knowledge base for learning purposes.

SFV: Reinforcement Learning of Physical Skills from Videos

Other Relevant Information

Last update: Apr 16, 2024

Red-Point "is the first ever Augmented Reality Climbing Guide that works with no internet connection."
Climbalyzer "is an AI powered 3D body position and movement analysis app for coaches and self-coached climbers."
Belay.ai: "AI that understands climbing." Here's a video where the Norwegian climber Magnus reviews their technologies .
Valo Motion: "Gamify your [ordinary] climbing walls with our augmented climbing wall game platform."
Babylon.js is an open web rendering engine that uses WebGL. The latest Gaussian splatting feature may be helpful for quickly building a proof-of-concept 3D demo related to any of the ideas mentioned above.