MoGloss: Joint Motion and Pseudo-Gloss Pretraining for American Sign Language Understanding

Rui Hong    Jana Košecká

George Mason University

Paper (arXiv) Code Data & Weights

Radar chart comparing MoGloss with Uni-Sign, SHuBERT, the YouTube-ASL baseline and PoseFormer on How2Sign, ASL STEM Wiki and OpenASL translation and on ASL Citizen, WLASL100 and WLASL2000 recognition
Translation (BLEU-4, setting A) and isolated sign recognition (top-1). Every method except the published SHuBERT and PoseFormer recognition results was trained and tested by us under the same protocol.

Abstract

Sign language translation and recognition have been shown to benefit from various pretraining approaches. We propose a novel two-stage pretraining approach for the motion encoder, which first learns to reconstruct masked motion, followed by joint masked prediction of pseudo-gloss units and motion. Pseudo-glosses serve as an intermediate structured representation of natural language text and are obtained from the captions by a large language model. We curate a corpus of about 1,100 hours of American Sign Language from five public datasets, estimate 2D pose for all clips and define an evaluation protocol. We then reproduce state-of-the-art translation methods and evaluate them using the same train and test splits. We propose a translation model built on our pretraining strategy, and this model outperforms the previous state-of-the-art methods Uni-Sign and SHuBERT, as well as the YouTube-ASL baseline, on the How2Sign, ASL STEM Wiki and OpenASL datasets. Furthermore, the gain from the joint pretraining using motion and pseudo-gloss grows as translation data shrinks. A classifier built on the same encoder also achieves state-of-the-art accuracy on ASL Citizen isolated sign recognition and outperforms our reproduction of Uni-Sign on WLASL2000.

BibTeX

@article{hong2026mogloss,
  title   = {{MoGloss}: Joint Motion and Pseudo-Gloss Pretraining for American Sign Language Understanding},
  author  = {Hong, Rui and Ko{\v{s}}eck{\'a}, Jana},
  journal = {arXiv preprint arXiv:2611.25333},
  year    = {2026}
}

Highlights

1,112 h
curated ASL corpus from five public datasets, 472K clips, one set of RTMPose keypoints for every clip
27.84
BLEU-4 on OpenASL, against 22.36 for our reproduction of Uni-Sign with RGB, while reading keypoints only
78.0%
top-1 on ASL Citizen (2,731 signs, official signer-disjoint split), 61.1% without pretraining

Method

A Transformer encoder reads RTMPose keypoints, one frame per token, and is pretrained in two stages before translation.

Two-stage pretraining. Stage 1 masks motion frames and recovers them. Stage 2 feeds pseudo-gloss units and motion into the same encoder and masks either units or frames.
Two-stage pretraining. Stage 1 (masked motion modeling) masks 30% of the frames and recovers them. Stage 2 feeds the pseudo-gloss units and the motion of the same clip into the encoder from stage 1. Each example masks either units or frames, and the model recovers what was masked. To recover a masked unit it has to find that sign in the motion.

Pseudo-glosses

ASL corpora have English captions but no glosses. An open-weight LLM (Qwen3.5-27B) rewrites each caption into a short sequence of gloss-like units, guided by a sign lexicon of 3,339 entries built from ASL Signbank and ASL Citizen.

CaptionPseudo-gloss
Police shot and killed the knifeman.police shoot kill knifeman
Hi, my name is Katie Moore.hi my name katie moore
The Japanese government has approved this process.japanese government approve process

Translation

The pretrained encoder is followed by a linear projector and T5-base, and all parts are trained end to end.

Translation model: motion encoder, linear projector, T5-base encoder and decoder
Translation model.

A curated ASL corpus

One documented processing of five public datasets, with the same keypoints and cleaned captions for every clip. For setting B, no training clip comes from the same video as a test clip.

SourceVideosClipsHours
YouTube-ASL8,790228,399440.4
ASL STEM Wiki22959,109291.7
OpenASL1,47670,763169.0
YouTube-SL-25, ASL4,06983,181152.1
How2Sign2,19230,79458.8
Total15,687 unique472,2461,112.1

Results

Setting A: YouTube-ASL pretraining and per-dataset fine-tuning (BLEU-4)

Every method is trained for translation on the same 785 hours of YouTube-ASL and then fine-tuned per dataset. OpenASL is also split by whether the source video is in YouTube-ASL. All baselines are our reproductions under the same protocol. Uni-Sign is not re-implemented. We run its official code unmodified and only match our data to it, using our own RTMPose keypoints rather than any keypoints or weights released by Uni-Sign. SHuBERT (A), (C1) and (C2) are three reproductions that differ in preprocessing and weight decay.

MethodInputHow2SignSTEMOpenASL allunseenseen
YouTube-ASL baselinepose3.323.1913.236.3716.10
SHuBERT (A)pose + RGB0.040.131.300.971.44
SHuBERT (C1)pose + RGB0.010.050.210.120.25
SHuBERT (C2)pose + RGB0.350.390.460.250.52
Uni-Signpose9.2819.8521.5615.4224.19
Uni-Signpose + RGB9.5520.5122.3616.4824.71
Ours, no pretrainingpose6.9615.3420.4114.1223.06
Ours, MMM onlypose9.2719.7524.6917.4027.77
Ours, joint onlypose8.4317.9524.2815.1927.99
Ours, MMM + jointpose9.7922.0027.8418.9231.66

Setting B: one model on all data, no fine-tuning (BLEU-4)

Data is the share of translation sentences. For 50% and 25% only the translation stage sees less data, and the pretrained encoders are the same. The gap between the two encoders grows as the data shrinks.

DataMethodSTEMOpenASLHow2SignYT-ASL
100%Uni-Sign3.439.573.356.33
Ours, no pretraining0.764.761.772.88
Ours, joint only18.7420.298.5313.57
Ours, MMM only21.2021.119.9815.19
Ours, MMM + joint21.9921.6410.3815.81
50%Ours, MMM only9.5913.225.129.74
Ours, MMM + joint16.8019.218.3713.06
25%Ours, MMM only0.583.951.492.07
Ours, MMM + joint10.5513.755.7810.04

Isolated sign recognition

ASL Citizen, 2,731 signs, official signer-disjoint split (top-1 / top-5, %). Published results are taken from the cited papers. †The paper does not state which split it evaluates on.

MethodInputTop-1Top-5
ST-GCNpose6083
SignCLIPpose6084
PoseFormer†pose75.193.2
I3DRGB6386
SHuBERTpose + RGB6587
SignDEGRGB70.489
Ours, no pretrainingpose61.183.6
Ours, MMM onlypose74.391.6
Ours, MMM + jointpose78.094.0

WLASL, official split, per-instance top-1 (%). Both methods start from the same 1,112 hours and read the same RTMPose keypoints. Uni-Sign is our reproduction with its released WLASL recipe.

MethodWLASL100WLASL2000
Uni-Sign84.547.3
Ours, MMM + joint83.752.2

What joint pretraining changes in the encoder

t-SNE of encoder features of ASL Citizen test clips for six signs, for the encoder without pretraining, after MMM, and after MMM and joint pretraining
Encoders trained on YouTube-ASL only, applied to ASL Citizen test clips from unseen signers without any fine-tuning. For each clip we take the closest clip of another signer. It shows the same sign for 1.9% of the clips without pretraining, 5.5% after MMM and 42.6% after MMM and joint pretraining (random guessing 0.04%).

Code, data and weights

GitHub: the model, both pretraining stages, translation, recognition and evaluation scripts, the preprocessing for new data (RTMPose extraction, pseudo-gloss prompt and parser, sign lexicon), and our reproductions of Uni-Sign, the YouTube-ASL baseline and SHuBERT. For Uni-Sign we use only its official code, unmodified. Our scripts read our keypoints, captions and splits into the inputs that code expects and run it on one GPU with gradient accumulation instead of deepspeed.

Hugging Face: RTMPose keypoints of every clip, MediaPipe landmarks used by the SHuBERT reproduction, captions and pseudo-glosses, every split used in the paper, and the weights of our models and of the Uni-Sign and YouTube-ASL baseline reproductions (CC BY-NC 4.0).

BibTeX

@article{hong2026mogloss,
  title   = {{MoGloss}: Joint Motion and Pseudo-Gloss Pretraining for American Sign Language Understanding},
  author  = {Hong, Rui and Ko{\v{s}}eck{\'a}, Jana},
  journal = {arXiv preprint arXiv:2611.25333},
  year    = {2026}
}