Language proficiency diagnosis is essential to extract fine-grained information about the linguistic knowledge states and skill mastery levels of test takers based on their performance on language tests. Different from comprehensive standardized tests, many language learning apps often revolve around word-level questions. Therefore, knowledge concepts and linguistic skills are hard to define, and diagnosis must be well-designed. Traditional approaches are widely applied for modeling knowledge in science or mathematics, where skills or knowledge concepts are easy to associate with each item. However, only a few works focus on defining knowledge concepts and skills using linguistic characteristics for language knowledge proficiency diagnosis. In addressing this, we propose a framework for language proficiency diagnosis based on neural networks. Specifically, we propose a series of methods based on our framework that uses different linguistic features to define skills and knowledge concepts in the context of the language learning task. Experimental results on a real-world second-language learning dataset demonstrate the effectiveness and interpretability of our framework. We also provide empirical evidence with comprehensive experiments and analysis to prove that our knowledge concept and skill definitions are reasonable and critical to the performance of our model.