Build an Olympic Skating Sports Analytics System using AI

Manual technical calling in figure skating is incredibly difficult. This guide explores building an automated Olympic-grade AI Jump Classifier using Python, YOLO11, and MediaPipe. Learn to track rapid rotations in real-time, analyze temporal patterns with LSTM networks, and build a digital technical panel that identifies every Axel, Lutz, and Salchow with mathematical precision.

The Challenge: Why Human Eyes Often Miss the Mark

Figure skating is a sport defined by milliseconds and millimeters. We have all watched a competition where a skater performs a triple jump at high speed, and the commentators debate the takeoff edge. Was it a Lutz with an outside edge or a Flip with an inside edge? Even for experts, the speed of rotation and the spray of ice make manual tracking flawed.

Technical panels rely on slow-motion replays, but even then, human bias and fatigue can lead to inconsistent scoring. Most current apps rely on basic accelerometers which cannot distinguish the specific geometry of a takeoff. We decided we needed a better solution, so we built an objective, digital technical specialist.

In this guide, we explore how to build a robust AI-Powered Jump Classifier. This system does not just track movement; it identifies specific technical maneuvers. We utilize Computer Vision and Pose Estimation to track skeletal joints in real-time. By the end of this project, we established a system that classifies jumps by analyzing the entry, airtime, and landing. We achieved this by combining Python, YOLO11, MediaPipe, and data annotation tools from Labellerr.

The Solution: A Specialized AI Analytics Pipeline

Our solution follows a clear, four-step pipeline that moves from raw broadcast footage to a real-time classification application. We started with data preparation, utilizing Labellerr to annotate skating frames with specific keypoints like the skates, knees, and hips. Once the data was ready, we used YOLO11 to locate the skater and MediaPipe to extract high-fidelity skeletal data. We developed an LSTM (Long Short-Term Memory) model to process these sequences over time. Finally, we wrapped everything in an OpenCV application that draws the skeleton and updates the jump type live.

Step 1: Data Collection and Annotation

High-quality data is the bedrock of Olympic-level AI. For this project, a generic person detector was insufficient. We needed the model to understand the subtle edge changes of a skater's blade, so we recorded high-frame-rate footage of various jumps from multiple angles. The next challenge was labeling the specific body positions and blade angles so the model could learn the nuances of each jump.

Annotating figure skating poses is demanding due to the high velocity and motion blur. To speed this up, we used Labellerr's automated platform. We uploaded our raw clips and defined a Schema consisting of critical points: the nose, shoulders, hips, knees, and ankles. The platform allowed us to quickly mark these joints across complex rotations. We then exported the data in a JSON format, which contained all the coordinate information needed to train our custom detection layers.

Step 2: Pose Estimation with MediaPipe and YOLO11

With our dataset ready, we moved to the extraction phase. We chose YOLO11 for initial athlete detection because of its incredible speed and accuracy in crowded environments. Once the skater is located, we pass the cropped region to MediaPipe Pose. MediaPipe provides 33 3D landmarks, which is essential for capturing the rotational geometry of a triple or quadruple jump.

  MediaPipe

By utilizing both frameworks, we achieved a stable "stick-figure" representation of the athlete even during mid-air spins. This approach allows the system to maintain tracking even when the skater is moving at peak velocity across the ice. After processing our dataset, the system could reliably track the coordinates of every joint throughout the entire jump duration.

Step 3: LSTM Sequence Classification

This phase served as the intelligence of the project. A single frame of a skater in the air looks identical for many jumps. We needed to look at the "history" of the movement. We implemented an LSTM (Long Short-Term Memory) network, which is a type of neural network designed for sequence data. The LSTM looks at the 30 frames leading up to the jump and the 30 frames during the airtime.

  LSTM

To ensure accuracy, our logic analyzes the path of the ankle keypoints. For a Lutz, the system detects a specific back-outside edge entry. For an Axel, it looks for the forward-facing takeoff. The LSTM processes these coordinate sequences and outputs a probability for each jump type. This creates a data-driven classification loop that identifies the jump based on the athlete's actual movement patterns over time.

Step 4: Real-Time Inference and HUD

The final step was to bring these components into a visual application. We used OpenCV to process the video feed and overlay a futuristic "Heads-Up Display" (HUD). The workflow operates in a continuous loop: the system reads the frame, extracts the MediaPipe skeleton, feeds the sequence into the LSTM, and displays the result.

We added a technical overlay for better feedback. As the skater jumps, the system draws a "motion trail" indicating the number of rotations completed. It also calculates the vertical jump height and airtime in real-time. This visual interface provides instant confirmation, highlighting the skeleton in different colors once a jump is successfully identified and classified.

Conclusion

We successfully built a system that replaces subjective calling with machine precision. By combining MediaPipe’s pose tracking with the sequential intelligence of LSTM models, we created a tool that truly understands the art of figure skating. This project demonstrates that AI can master even the most complex human movements. Whether you are a coach looking for objective feedback or a developer interested in sports analytics, this project sets a new standard for Olympic-grade technology.

Also If you need help with annotation services or dataset creation for similar sports or vision/robotics use cases, feel free to reach out and book a call with us

Frequently Asked Questions