BibTeX
@article{hong2026mogloss,
title = {{MoGloss}: Joint Motion and Pseudo-Gloss Pretraining for American Sign Language Understanding},
author = {Hong, Rui and Ko{\v{s}}eck{\'a}, Jana},
journal = {arXiv preprint arXiv:2611.25333},
year = {2026}
}
Highlights
Method
A Transformer encoder reads RTMPose keypoints, one frame per token, and is pretrained in two stages before translation.
A curated ASL corpus
One documented processing of five public datasets, with the same keypoints and cleaned captions for every clip. For setting B, no training clip comes from the same video as a test clip.
| Source | Videos | Clips | Hours |
|---|---|---|---|
| YouTube-ASL | 8,790 | 228,399 | 440.4 |
| ASL STEM Wiki | 229 | 59,109 | 291.7 |
| OpenASL | 1,476 | 70,763 | 169.0 |
| YouTube-SL-25, ASL | 4,069 | 83,181 | 152.1 |
| How2Sign | 2,192 | 30,794 | 58.8 |
| Total | 15,687 unique | 472,246 | 1,112.1 |
Results
Setting A: YouTube-ASL pretraining and per-dataset fine-tuning (BLEU-4)
Every method is trained for translation on the same 785 hours of YouTube-ASL and then fine-tuned per dataset. OpenASL is also split by whether the source video is in YouTube-ASL. All baselines are our reproductions under the same protocol. Uni-Sign is not re-implemented. We run its official code unmodified and only match our data to it, using our own RTMPose keypoints rather than any keypoints or weights released by Uni-Sign. SHuBERT (A), (C1) and (C2) are three reproductions that differ in preprocessing and weight decay.
| Method | Input | How2Sign | STEM | OpenASL all | unseen | seen |
|---|---|---|---|---|---|---|
| YouTube-ASL baseline | pose | 3.32 | 3.19 | 13.23 | 6.37 | 16.10 |
| SHuBERT (A) | pose + RGB | 0.04 | 0.13 | 1.30 | 0.97 | 1.44 |
| SHuBERT (C1) | pose + RGB | 0.01 | 0.05 | 0.21 | 0.12 | 0.25 |
| SHuBERT (C2) | pose + RGB | 0.35 | 0.39 | 0.46 | 0.25 | 0.52 |
| Uni-Sign | pose | 9.28 | 19.85 | 21.56 | 15.42 | 24.19 |
| Uni-Sign | pose + RGB | 9.55 | 20.51 | 22.36 | 16.48 | 24.71 |
| Ours, no pretraining | pose | 6.96 | 15.34 | 20.41 | 14.12 | 23.06 |
| Ours, MMM only | pose | 9.27 | 19.75 | 24.69 | 17.40 | 27.77 |
| Ours, joint only | pose | 8.43 | 17.95 | 24.28 | 15.19 | 27.99 |
| Ours, MMM + joint | pose | 9.79 | 22.00 | 27.84 | 18.92 | 31.66 |
Setting B: one model on all data, no fine-tuning (BLEU-4)
Data is the share of translation sentences. For 50% and 25% only the translation stage sees less data, and the pretrained encoders are the same. The gap between the two encoders grows as the data shrinks.
| Data | Method | STEM | OpenASL | How2Sign | YT-ASL |
|---|---|---|---|---|---|
| 100% | Uni-Sign | 3.43 | 9.57 | 3.35 | 6.33 |
| Ours, no pretraining | 0.76 | 4.76 | 1.77 | 2.88 | |
| Ours, joint only | 18.74 | 20.29 | 8.53 | 13.57 | |
| Ours, MMM only | 21.20 | 21.11 | 9.98 | 15.19 | |
| Ours, MMM + joint | 21.99 | 21.64 | 10.38 | 15.81 | |
| 50% | Ours, MMM only | 9.59 | 13.22 | 5.12 | 9.74 |
| Ours, MMM + joint | 16.80 | 19.21 | 8.37 | 13.06 | |
| 25% | Ours, MMM only | 0.58 | 3.95 | 1.49 | 2.07 |
| Ours, MMM + joint | 10.55 | 13.75 | 5.78 | 10.04 |
Isolated sign recognition
What joint pretraining changes in the encoder
Code, data and weights
GitHub: the model, both pretraining stages, translation, recognition and evaluation scripts, the preprocessing for new data (RTMPose extraction, pseudo-gloss prompt and parser, sign lexicon), and our reproductions of Uni-Sign, the YouTube-ASL baseline and SHuBERT. For Uni-Sign we use only its official code, unmodified. Our scripts read our keypoints, captions and splits into the inputs that code expects and run it on one GPU with gradient accumulation instead of deepspeed.
Hugging Face: RTMPose keypoints of every clip, MediaPipe landmarks used by the SHuBERT reproduction, captions and pseudo-glosses, every split used in the paper, and the weights of our models and of the Uni-Sign and YouTube-ASL baseline reproductions (CC BY-NC 4.0).
BibTeX
@article{hong2026mogloss,
title = {{MoGloss}: Joint Motion and Pseudo-Gloss Pretraining for American Sign Language Understanding},
author = {Hong, Rui and Ko{\v{s}}eck{\'a}, Jana},
journal = {arXiv preprint arXiv:2611.25333},
year = {2026}
}