2608.09400-Sign Language Recognition Using Original and Synthetic Depth Image Based Point Cloud Data Models

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Sign Language Recognition Using Original and Synthetic Depth Image Based Point Cloud Data Models".

Jane: The paper was written by Rüstem Özakar and Eyüp Gedikli from Erzurum Technical University and Trabzon University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: The biggest twist for me is that the synthetic depth data actually beat the original in one of the datasets. That's not what you expect going in. On the KArSL dataset, the point gesture map model with synthetic depth hit 86 point 81 percent accuracy, while the original depth version only reached 32 point 11. And the synthetic LSTM model scored 97 point 56 percent, compared to 95 point 19 for the original. So we have to ask why fake depth sometimes outperforms real depth.

Jane: Wait, how can fake depth be better than the real thing? That sounds backwards.

Tom: That's exactly what I thought. Lu, you've dealt with synthetic data before — what's your reading of this?

Lu: It's surprising, but it's not unprecedented. Depth Anything V2 generates depth from RGB, and that process can smooth over sensor noise and inconsistencies that exist in real Kinect captures. Plus, KArSL was recorded against a green screen, so the original depth may have artifacts from that setup. The synthetic PGM result was so much better that it suggests something about the point distribution helped the network generalize, maybe acting as a form of regularization.

Meng: But that boost didn't appear in the other two datasets, right?

Lu: Right. On Dataset-A and Dataset-C, the original depth was clearly stronger. So this isn't a universal advantage — it's dataset-specific. The authors themselves say the synthetic data might create a distinctive positive effect that the original data doesn't have. That's worth digging into, but it definitely complicates the story.

Jane: So what exactly did they do with the point clouds? I remember three different models: frame-based, point gesture map, and LSTM.

Tom: Right. The frame-based model feeds individual point clouds into PointNet. The point gesture map merges frames from a gesture — around 49 frames in Dataset-A, or all frames in the other datasets — into one large point cloud, then samples 6400 points. And the LSTM approach extracts features from a pretrained frame-based PointNet and feeds those features into an LSTM across time.

Meng: The LSTM did best overall on both KArSL and AUTSL. On AUTSL, the original depth LSTM reached 68 point 43 percent accuracy. That's much lower than the other datasets, but AUTSL has 226 gestures across 43 signers with varied backgrounds, so it's genuinely harder.

Lalam: That trade-off matters for the bigger picture. Depth cameras like the Kinect aren't everywhere, but ordinary RGB cameras are. If you can create decent synthetic depth from RGB, you could apply point cloud methods to the huge amount of existing sign language video. That could make recognition systems far more scalable and accessible.

Jane: But they also showed synthetic depth isn't always good enough. So are we ready to rely on it?

Lu: Not yet. The authors trained on raw point clouds without separating hands or arms, and they list that as future work. This is really a feasibility study — it maps out where synthetic depth works and where it doesn't, and it gives us a solid baseline to improve on.

Tom: I also noticed the total training time was around a hundred days, spread over five months. That's a massive computational effort for three datasets.

Meng: And the AUTSL training curves oscillated a lot, especially for validation loss. The models clearly struggled more. That tells me there's still room for better preprocessing, better architectures, and maybe data augmentation.

Jane: So what's the main thing you'd want a listener to remember? Aside from the numbers.

Tom: I'd say it's that synthetic depth is a promising substitute, but not a perfect one. And the dramatic win on KArSL means we really need to understand why synthetic sometimes helps. That could reshape how we build sign language recognition systems.

Lalam: It's also part of a wider pattern in computer vision: generating missing data modalities instead of always collecting them. Depth, radar, lidar — if the generator is reliable enough, it unlocks whole libraries of existing data for new techniques. This paper gives us a clear example, quirks and all.

Jane: That's a good place to wrap up. Thanks to everyone for the insights.

Page 1 of the paper: Tom: Page one really sets up the central question. The authors want to know if you can generate depth images from regular RGB video using a neural network, then turn those into point clouds, and still get reliable sign language recognition without a depth camera at all.

Jane: That's a bold idea. Depth cameras like Kinect aren't everywhere, and they have their own limitations.

Tom: Exactly. They pick Depth Anything V2 to do the depth estimation, which is a recent model that produces depth maps from single RGB frames. Then they convert each depth map into a three dee point cloud. That point cloud becomes the input to a network called PointNet, which handles unordered points directly.

Jane: So the comparison is original depth data versus synthetic depth data across three different sign language datasets.

Tom: Right, and the abstract already teases that it's not a one-sided win. Sometimes synthetic data actually outperforms the original, which is a surprising thing to watch for as we go through the results.

Jane: I also appreciate that they don't gloss over the difficulty. They list challenges like different sign languages, varied backgrounds, and continuous signing, so this isn't a toy problem.

Tom: That realism drives the whole motivation. Page one makes the point that RGB images are sensitive to lighting and color changes, while depth images can be far more stable. That's exactly why they want to synthesize depth from RGB in the first place.

Jane: So the big idea is to get the robustness of depth data without needing the specialized hardware.

Tom: Precisely. And the way they describe testing frame-based models, gesture maps, and LSTM sequences tells you they're going to evaluate this across multiple architectures, not just one lucky setup.

Jane: That sets up a lot of results to dig through.

Page 2 of the paper: Jane: So page three is where we actually meet the three datasets they used, and I like that they picked ones that have both RGB and depth from the start. That lets them compare real depth against the synthetic depth they generate from RGB with Depth Anything V2.

Tom: Right, and all three were recorded with Microsoft Kinect cameras, which matters because the depth images have known camera parameters. The authors list the exact intrinsic values they plugged into Open3d to turn those depth frames into point clouds.

Jane: What impressed me is the range of difficulty in these datasets. Dataset-A is the Real-time ASL Fingerspelling set, just 24 letters with 65,000 frames, while Dataset-B is KArSL with 502 Arabic sign gestures performed by only three signers.

Tom: Three signers for 502 gestures, that's a lot of vocabulary per person. And Dataset-C is AUTSL, the Turkish set, with 226 gestures but 43 different signers and over 38,000 videos, so much more variety in who's signing.

Jane: Exactly, and that spread is a feature, not an accident. Dataset-A is static fingerspelling, mostly letters held in one pose, while the other two are dynamic, real sign language with movement over time.

Tom: That dynamic aspect sets up a big choice in the methodology. Since Dataset-A has no temporal dimension, they'll only use frame-based point cloud models and Point Gesture Maps. But for the video datasets they can also train LSTMs on sequences of point clouds.

Jane: And you can see them preparing for that here on this page. They mention PointNet for extracting features from the unordered points, and they introduce Point Gesture Maps as a way to merge frames from a gesture into one point cloud, basically compressing time into space.

Tom: The synthetic depth part is the real twist though. They generate depth images from the RGB frames using Depth Anything V2, which is a recent model trained on a huge amount of real-world data, and then they treat those generated depth maps exactly like the original ones.

Jane: So the whole page is really about setting up a fair comparison. Same datasets, same point cloud creation, same networks for both types of depth, with only the origin of the depth image being different.

Tom: And the numbers in Table 1 give you the scale of what they're attempting. Nearly two million depth frames for KArSL alone, and over two million for AUTSL. That's not a small experiment.

Jane: No kidding. But it also makes the results they report later more convincing, because you're seeing these models trained on genuinely large datasets, not just a few hundred examples.

Page 3 of the paper: Tom: So they’re walking through the KArSL dataset now, and the first thing they have to nail down is how to turn those depth frames into point clouds using the Kinect V2’s camera parameters.

Jane: Right, the focal length and the principal point coordinates — if you get those wrong, every point in the cloud lands in the wrong place, and the whole model learns from distorted geometry.

Tom: Exactly, and they use the same settings later for the Turkish dataset because it was recorded with the same camera, so that’s a smart way to keep things consistent across experiments.

Jane: Then they shrink each point cloud down to 512 samples for the frame-based models — that’s a practical move so PointNet doesn’t choke on millions of raw points.

Tom: But the real meat of this page is how they prep the temporal data for the LSTM. KArSL gestures are videos, not stills, so each gesture has a variable number of frames.

Jane: So they count the average frame length and decide on 25 frames per gesture as the fixed input size.

Tom: And if a video has more than 25 frames, they just pick 25 of them in order, but if it has fewer — between 14 and 25 — they interpolate. They literally blend the previous and next frames proportionally to create the missing ones.

Jane: That’s the clever bit. Instead of padding with zeros or repeating the same frame, they synthesize intermediate frames so the motion stays smooth and the LSTM sees a natural sequence.

Tom: And then they don’t feed raw point clouds into the LSTM at all. They use a pretrained frame-based PointNet to pull out a feature vector from each frame — specifically from the GlobalMaxPooling1D layer — and that sequence of features becomes the input.

Jane: So the LSTM is learning temporal patterns on top of features that are already meaningful in three dee space. That’s a really common pattern in gesture recognition, but it’s good to see it spelled out with the exact layer they used.

Tom: One detail that stood out to me is the cross-validation choice. For the PGM models, they had to switch from five-fold to ten-fold just because the point gesture maps are so large they couldn’t fit in memory otherwise.

Jane: That’s the kind of practical constraint you only hit when you’re dealing with 56,000 PGM samples, each with 6,400 points — the numbers get big fast.

Tom: And they mention the depth-scale is set to 500 and depth-trunc to 1000 for this dataset, which is different from the fingerspelling dataset, so the depth conversion clearly isn’t one-size-fits-all.

Jane: Right, those parameters depend on the sensor and the scene, and getting them right is what separates a clean point cloud from a noisy mess.

Page 4 of the paper: Tom: This page is really where you see the scale of what they actually built. Table 2 lists every single data model across all three datasets, with train and test counts for both the original and synthetic depth point clouds, and the network input shape for each one.

Jane: That's a lot of numbers. Dataset-B alone has over 1 point 9 million frame point clouds, and Dataset-C tops two million.

Tom: Exactly. And that's why the training section says the whole process took roughly a hundred days of compute, stretched over five months with breaks. When you see those numbers, that timeline makes sense.

Jane: They also get specific about the network tweaks. For the PGM models, they took the standard PointNet and swapped out the last two dense layers for four bigger ones — 4096, 2048, 1024, 512 — each with dropout at 0 point 3.

Tom: So they're making the classifier deeper to handle those 6400-point gesture maps. And for the LSTM models, they put a 256-unit LSTM layer first, then two dense layers with dropout at 0 point 2. They say these layers were determined by observing various training experiments, which tells you it was empirical.

Jane: I noticed the optimizer and learning rate are fixed too — Adam at 0 point 0001. That's a pretty standard choice, but it's good they state it plainly so people can reproduce the work.

Tom: The input shapes in the table are the part I keep coming back to. Frame models take 512 points, PGM models take 6400, and the LSTM models take a sequence of 25 or 30 frames, each with 512 points sampled from the point cloud.

Jane: So the temporal dimension is handled by feeding the PointNet's extracted features frame by frame into the LSTM. And they mention all data was shuffled before splitting, which is a small but important detail.

Tom: Right, it avoids any ordering bias when you're separating training from validation. It's a page full of practical decisions, not just theory — the kind of specifics you need if you want to compare your own results.

Page 5 of the paper: Tom: So page nine is where they show the actual training schedules for the first two datasets. We get exact epoch counts and how long each epoch took, which is the kind of practical detail that normally gets buried.

Jane: And those numbers tell a story. The frame-based PointNet for Dataset-A runs about fifty seconds per epoch, while the point gesture maps finish in eight seconds.

Tom: Right, because the gesture maps compress a whole sequence into one point cloud, so there are far fewer samples to iterate over. But each one is much denser.

Jane: Then for Dataset-B, the frame models jump to twenty-five minutes per epoch, and they ran the original depth network for two hundred forty epochs. That's roughly a hundred hours on a GPU for one model.

Tom: They didn't run the synthetic version as long, though. It stopped at fifty epochs, likely because the validation accuracy plateaued earlier.

Jane: They also admit they removed a few extreme error spikes from some training plots to make them readable. That's a transparency worth appreciating.

Tom: And they close the page by pointing out that Dataset-C had frequent oscillations in validation loss. Given what we saw about that dataset having 43 signers and varied backgrounds, that's not surprising.

Jane: Exactly. That's why the Turkish dataset results are lower, and it sets up the comparison on the next page.

Page 6 of the paper: Tom: Page 11 is where all those AUTSL training curves are plotted, and they look a lot messier than the KArSL ones we saw before. The validation loss lines keep bouncing up and down, and the authors actually noted those oscillations in the text.

Jane: So the Turkish dataset really is the hardest of the three.

Tom: That's what the figures show. You can also spot the difference in training length: the synthetic LSTM model runs for two hundred epochs, while the original only needs a hundred. That's double the time just to reach a comparable point.

Jane: And the PGM curves spike hard at the start, too.

Tom: They do, though the authors say they trimmed some of the worst spikes to keep the plots readable. What's interesting is that even with those rough curves, the synthetic depth models eventually level out to something close to the original ones, just later.

Jane: So the page really gives you the visual proof that the synthetic data is workable, but it costs more training effort.

Tom: Exactly. And it sets up the numbers we're about to see in the results tables, where the synthetic LSTM actually ends up slightly ahead for KArSL.

Page 7 of the paper: Tom: So after seeing those accuracy numbers for Dataset-B, page thirteen lays out the confusion matrices, and they really show where the synthetic depth data pulled ahead. Look at Figure 31, the original depth PGM, versus Figure 32, the synthetic one. The original matrix has a lot of scattered off-diagonal dots, which explains that 32 percent accuracy, while the synthetic matrix is much cleaner along the diagonal.

Jane: And that cleaner diagonal is what matches the 86 point 8 percent we saw in the results table earlier. The synthetic data isn't just improving one or two gestures, it's helping across the whole set.

Tom: Exactly. Then the LSTM matrices in Figures 33 and 34 tell a similar story. The original version is already pretty solid, but the synthetic one shows an even tighter diagonal, which lines up with the 97 point 56 percent against 95 point 19 percent.

Jane: I did notice the axes only say "True Label" and "Predicted Label," so we can't see which specific Arabic sign letters are getting confused. But even without the class names, you can see whether the mistakes are concentrated or spread out randomly.

Tom: Right. And that's important because earlier the frame-based models actually favored the original depth data. On this page, the synthetic advantage only shows up in the spatio-temporal models, the PGM and the LSTM.

Jane: Which makes sense, since those models use the sequence of frames rather than a single snapshot. So page thirteen gives us the visual proof that synthetic depth preserves enough temporal structure for those point cloud networks to work with.

Page 8 of the paper: Tom: So on this page we finally get to see the confusion matrices for the Turkish dataset, AUTSL. The numbers in the table earlier told us the overall accuracy, but these grids show us exactly which signs are getting mixed up with which.

Jane: Right, and the most striking thing to me is that the synthetic PGM model, the Point Gesture Map one, performed so poorly they didn't even include its confusion matrix. In the results table it had that tiny 14 percent accuracy, and here it's just absent.

Tom: Exactly. The authors called it an insignificant accuracy, which is a polite way of saying the model was basically guessing. But the other five matrices are there, and they're useful for spotting patterns.

Jane: Looking at the original depth frame confusion matrix, you can see the diagonal is pretty bright, meaning most signs are being classified correctly. But there are some off-diagonal blocks where the model consistently confuses certain hand shapes.

Tom: And for the synthetic depth frames, the matrix looks a lot messier overall. That matches the 29 percent accuracy we saw, but interestingly the confusion isn't random. It's concentrated among similar-looking gestures.

Jane: That's actually a good sign, isn't it? Even when the synthetic data fails, it fails in a structured way. That suggests Depth Anything V2 is capturing something meaningful about the hand shape, just not as reliably as the real depth sensor.

Tom: The original LSTM confusion matrix is cleaner than the frame-based one, which makes sense because it has temporal information. The synthetic LSTM matrix is noisier, but it's still clearly picking up the right structure.

Jane: And there's something curious about that synthetic LSTM. It got 61 percent accuracy, which is lower than the original's 68 percent, but looking at the matrix, the errors seem more spread out. The model isn't fixating on one particular sign.

Tom: Right, that's the kind of detail a single accuracy number hides. You'd want to know whether the mistakes are dangerous confusions, like mixing up signs that mean opposite things, or just minor confusions between similar hand positions.

Jane: One thing I appreciate is that they show all the matrices with the same color scale, so you can compare them directly. The original depth matrices are visibly brighter on the diagonal, which gives you an intuitive sense of the performance gap.

Tom: And the fact that they included both original and synthetic for the LSTM on the same page makes it easy to see the temporal model handles synthetic depth much better than the frame model does. That's a nice insight for future work.

Jane: So if I were building on this paper, I'd take that synthetic LSTM matrix as evidence that temporal fusion can compensate for noisy depth estimation. The frame-based synthetic model collapses, but the LSTM still manages to pull out useful structure.

Conclusion: Tom: So the big takeaway for us is that synthetic depth data can handle most of the job that real depth cameras do, with a few interesting exceptions.

Jane: Right, and the real surprise was on the Arabic dataset, where the generated point clouds actually outperformed the originals in two of the models.

Tom: Exactly. That LSTM model hitting nearly 98 percent accuracy shows synthetic depth can be genuinely useful, not just a fallback.

Jane: That has practical implications too, because most sign language videos available online are plain RGB, so you could synthesize depth and then run point cloud recognition on top of it.

Tom: That could make sign language tools more accessible, especially for languages that don't have expensive depth camera datasets.

Jane: They also kept the raw data without hand segmentation, which leaves clear room for improvement in future work.

Tom: And they plan to test more datasets and newer depth generation networks to see if that KArSL result holds up.

Jane: Solid paper with honest limitations. I'll be curious to see what comes next.

Tom: Then let's move on to the next one.