Automated Testing Linguistic Capabilities of NLP Models

Jaeseong Lee, Simin Chen, Austin Mordahl, Cong Liu, Wei Yang, Shiyi Wei

Summary

ALiCT tests NLP applications against user-specified linguistic capabilities. It generates diverse test suites with test oracles and evaluates them on sentiment analysis and hate speech detection. The experiments demonstrate greater semantic and syntactic diversity than existing techniques and identify more model failures across most evaluated capabilities.