Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation

• Taehwan Kim, Jonathan Keane, Weiran Wang, Hao Tang, Jason Riggle, Gregory Shakhnarovich, Diane Brentari, Karen Livescu.
• Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation.
• Computer Speech \& Language, vol. 46, pp. 209––232, 2017.

Download: pre-print | from publisher | via doi

We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: it involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92% letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83% letter accuracies in this setting.

@Article{Kim2017ab,
  author = {Taehwan Kim and Jonathan Keane and Weiran Wang and Hao Tang and Jason Riggle and Gregory Shakhnarovich and Diane Brentari and Karen Livescu},
  doi = {10.1016/j.csl.2017.05.009},
  journal = {Computer Speech \& Language},
  month = {November},
  pages = {209––232},
  title = {Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation},
  url = {https://arxiv.org/abs/1609.07876},
  volume = {46},
  year = {2017},
  bdsk-file-1 = {YnBsaXN0MDDSAQIDBFxyZWxhdGl2ZVBhdGhZYWxpYXNEYXRhXxB1UGFwZXJzL0tpbUtlYW5lV2FuZy1MZXhpY29uLUZyZWUgRmluZ2Vyc3BlbGxpbmcgUmVjb2duaXRpb24gZnJvbSBWaWRlbyBEYXRhLCBNb2RlbHMsIGFuZCBTaWduZXIgQWRhcHRhdGlvbiAoMjAxNykucGRmTxEC6gAAAAAC6gACAAAMTWFjaW50b3NoIEhEAAAAAAAAAAAAAAAAAAAAAAAAAEJEAAH/////H0tpbUtlYW5lV2FuZy1MZXhpYyNGRkZGRkZGRi5wZGYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAQADAAAKIGN1AAAAAAAAAAAAAAAAAAZQYXBlcnMAAgCaLzpVc2Vyczpqa2VhbmU6RHJvcGJveDp3ZWI6dml0YUtlYW5lOlBhcGVyczpLaW1LZWFuZVdhbmctTGV4aWNvbi1GcmVlIEZpbmdlcnNwZWxsaW5nIFJlY29nbml0aW9uIGZyb20gVmlkZW8gRGF0YSwgTW9kZWxzLCBhbmQgU2lnbmVyIEFkYXB0YXRpb24gKDIwMTcpLnBkZgAOAN4AbgBLAGkAbQBLAGUAYQBuAGUAVwBhAG4AZwAtAEwAZQB4AGkAYwBvAG4ALQBGAHIAZQBlACAARgBpAG4AZwBlAHIAcwBwAGUAbABsAGkAbgBnACAAUgBlAGMAbwBnAG4AaQB0AGkAbwBuACAAZgByAG8AbQAgAFYAaQBkAGUAbwAgAEQAYQB0AGEALAAgAE0AbwBkAGUAbABzACwAIABhAG4AZAAgAFMAaQBnAG4AZQByACAAQQBkAGEAcAB0AGEAdABpAG8AbgAgACgAMgAwADEANwApAC4AcABkAGYADwAaAAwATQBhAGMAaQBuAHQAbwBzAGgAIABIAEQAEgCYVXNlcnMvamtlYW5lL0Ryb3Bib3gvd2ViL3ZpdGFLZWFuZS9QYXBlcnMvS2ltS2VhbmVXYW5nLUxleGljb24tRnJlZSBGaW5nZXJzcGVsbGluZyBSZWNvZ25pdGlvbiBmcm9tIFZpZGVvIERhdGEsIE1vZGVscywgYW5kIFNpZ25lciBBZGFwdGF0aW9uICgyMDE3KS5wZGYAEwABLwAAFQACAA3//wAAAAgADQAaACQAnAAAAAAAAAIBAAAAAAAAAAUAAAAAAAAAAAAAAAAAAAOK},
  bdsk-url-1 = {https://arxiv.org/abs/1609.07876},
}