Despite the United Nations Convention on the Rights of Persons with Disabilities emphasising the importance of sign language for individuals with hearing impairment, support in Hong Kong is limited, with only 50 registered sign language interpreters to assist over 200,000 hearing-impaired individuals. This can lead to hindrance to access to important information and services, as well as feelings of isolation and exclusion. Efforts to increase the awareness of their needs have been made, but more action is needed.
We propose to improve communication between hearing and hard-of-hearing individuals in Hong Kong by translating between spoken language and Hong Kong Sign Language (HKSL). The solution combines two modules: the first module recognises HKSL and translates it into spoken languages using Convolutional Neural Networks (CNNs), while the second module generates sign language videos sequence from spoken language sentences using Neural Machine Translation (NMT) and Generative Adversarial Network (GAN) with motion generation.
The CNNs module can effectively recognise HKSL, enabling real-time communication in various settings, such as medical emergencies or law enforcement situations. The NMT and GAN modules can translate textual information into HKSL in realistic video sequences, providing the deaf community with access to any textual information, including live news or emergency announcements.
The proposed solution has the potential to revolutionise communication for the deaf and hard-of-hearing community in Hong Kong, promoting equal opportunities for all to access information and services, regardless of their abilities. By combining the two modules, the solution can facilitate communication in various settings, creating a more inclusive society and fostering a barrier-free city.
