Overview / Usage

Even with the second-most population in the world, India's languages are underrepresented on the web. Due to a lack of good models and data, NLU models on the Indian language perform worse than their other counterparts leading to a sub-par web experience for the approximately 1.4 billion Indian users. I try to solve this by using Intel NLPArchitect to improve upon baseline performance on QA tasks in Hindi, by using the dataset from Google Research's recently released dataset, chaii-1.

​

​

Methodology / Approach

For the dataset, I'm using the recently released chaii-1 dataset by Google Research, which includes training and test sets. The train set includes the context, questions, and answers, and a starting character for the answer. The test set contains only context and questions.

The next steps are to convert the data into SQuAD format and then preprocessing it. The data contains some noise, which is ideal as our aim should be to make models that are robust to noise as we encounter in real life.

Finally, the open-source Intel NLP Architect would be used for modeling and inference. Right now, I'm working on the Match LSTM and Answer Pointer Network. It forms a question-aware representation of the passage and then the representation is used by the answer pointer network to identify the indices between which the answer could be present. I'm also using the GloVe non-contextual monolingual Hindi language embeddings, which were released by IIT B. I've also planned to implement and test out transformer models, going further.

A model with a respectable performance could then be deployed in applications such as online customer service, and personalized chatting, and for even educational purposes in Indian schools.

Technologies Used

Intel MKL, Intel Optimized TensorFlow, Intel NLP Architect, Intel CPU

Other links

• Dataset Link