The project addresses the problem of high-level semantic understanding in mult-mode data of on-line learning, such as mouse, keyboard, video, in order to recognize and track e-Learner's behviors, emotions, intentions, while overcome the difficulties caused by dynamic, uncertainty and open-world. The characteristics of the techs adopted in this project include incremental learning with long-tailed dataset, contrastive learning, etc.
Demo(be shown in nearly future)
The project addresses the problem of Facial Expressions Recognition (FER) on multi-scale low resolution images, which is widely existed in applications with crowd scenes (station, classroom etc.). Real world crowd scene-oriented FER faces several challenging issues: Firstly, in a crowd scene, facial images of different person have different size, which makes it challenging to achieve high performance with a single FER model; Moreover, with the decrease of resolution, the distribution of features for FER is changing; at the same time, lower resolution image leads to the larger loss of feature information extracted by traditional methods. The above reason results in the degradation of discrimination. In this work, inspired by feature super-resolution methods for object detection, we proposed a novel generative adversary network-based feature level super-resolution method for robust facial expression recognition, which takes the Feature Super Resolution’s advantages on increasing resolution and restore details in images, as well as overcomes the problems of Image super-resolution(ISR), such as difficult to recover the details at very low resolution, to guarantee the recovery of features that are discriminative enough for FER task. Three significant outcomes are (a)We propose a facial expression recognition method based on feature super-resolution that can be used for low-resolution inputs, it can reduce the chance of privacy leaking without restoring high resolution facial images. (b)We use Generative Adversarial Network(GAN) to transform features of low-resolution images to more discriminative ones by making them closer to the ones of corresponding high-resolution images, in order to solve the FER problem of multi-scale facial images in crowd scenes. (c)We introduced a new class-aware loss re-weighting strategy for GAN training, helping model focus on samples not only with large loss but hard to be classified into corresponding category.
Code:
By the way, our source code is available on https://github.com/nanfangAlan/FSRFER.
Proposed a multi-person multi-object-oriented three-phase HOI recogniztion method, which achieve precision above 86%--97% over a self-built classroom image dataset with tyical 12 learning behaviors.
Propose a low-resolution-oriented facial expression recognization method to detect various resolution facial images in multi-person multi-object scenarios, such as classroom.
When Applicats want to fill the application on-line, this system compares their photos with your image in their Certificate. This system has two important parts: load balancing and Deep Learning Model for face segementation, identification and comparison. It had been deployed in China Higher Education Student Information and carried out in a range of few cities in Shandong and Jiangsu Province seperately, covering over 70K applicants. The precision of this system tested currently is above 96~99% depending on the image quality and sizes.
· Research on modelling e-learner's cognition performance and methods of adaptive cognition regulation（NSFC, 2015-2018）
Aiming at the problem of lack of 'self-supervision service' in current e-Learning systems, firstly, based on thinking map, this project is to build an e-learner cognition performance model combined with the non-intelligence factors, such as e-learner's personality, interest and emotion. This model solves the problem of the quantifiable and computable representation of cognition components and their relationships among them. Secondly, to recognize e-learner’s cognition states from the heterogeneous multi-sources in multimodal interaction, a two-layer information fusion framework is proposed. At the same time, an utility based information fusion algorithm is introduced into the second layer fusion center, which considers the robustness, accuracy, performance and cost of each recognition method in various interaction mode, an utility based multi-objective function is constructed, and an optimization method is used to acquire the optimal parameters of fusion algorithm. Thirdly, learner behavior instruction following with the law of metacognition and learning content recommendation based on Wundt curve are to be exploited further so as to achieve the goal of adaptive recognition regulation of e-learners. Finally, to verify and validate the proposed methods, these methods will be carried out in internet education School of our university. Obviously, this work will improve the learning efficiency and quality, and encourage to a learning ecological environment with characteristics of virtuous cycle and incentive interactions among e-learner’s interest, cognition, behavior and emotion. Through this research, strive to publish 8-10 articles, four of which is in high level international journals.
Multi-channel Data Fusion Framework
Spatial and Temporal Features Extraction of facial expression
【1】Annual Report of NSFC (grant No. 61472315) in 2015，download
-
Platform of Education quality monitoring based on BIG DATA （2011-Current）
Aiming at solve four major problems (how to collect status of classroom, evaluate teaching, improve education quality, and serve students's learning) in classroom quality monitoring, this project first adopted advanced technologies, such as Internet of Things, Cloud computing, Big data, to collect the data and status related to teaching and learning in every classroom of Xian Jiaotong University; then, applied AI techniques, especially deep learning (CNNRNN, etc), NLP and Decision Science to evaluate the quality of teaching in classroom and mining out the issues and problems of teaching content and students' reviews; moreover, students are characterized via six dimensions (social relationship, consuming behavior, day routine, learning achievment, etc.) and guide their learning plan for future.
The platform has won the second prize of the National Teaching Achievement Award, China, and the outstanding prize of Shaanxi Province Teaching Achievement in 2017. It has been highly acknowledged by two vice ministers of Ministry of Education, China and by the former chairman of the China Association of Higher Education. It has been covered in such media as CCTV (China Central Television) and Sina.com. So far, 26 other Chinese universities and six provincial offices of education have conducted on-site survey of the platform.
· Knowledge graph/forrest-based personalized learning path recommendation
1) achievement 1 ：
A multi-constraint learning path recommendation algorithm based on knowledge map （KBS, Volume 143, 1 March 2018, Pages 102-114）
It is difficult for e-learners to make decisions on how to learn when they are facing with a large amount of learning resources, especially when they have to balance available limited learning time and multiple learning objectives in various learning scenarios. This research presented in this paper addresses this challenge by proposing a new multi-constraint learning path recommendation algorithm based on knowledge map. The main contributions of the paper are as follows. Firstly, two hypotheses on e-learners’ different learning path preferences for four different learning scenarios (initial learning, usual review, pre-exam learning and pre-exam review) are verified through questionnaire-based statistical analysis. Secondly, according to learning behavior characteristics of four types of the learning scenarios, a multi-constraint learning path recommendation model is proposed, in which the variables and their weighted coefficients considers different learning path preferences of the learners in different learning scenarios as well as learning resource organization and fragmented time. Thirdly, based on the proposed model and knowledge map, the design and implementation of a multi-constraint learning path recommendation algorithm is described. Finally, it is shown that the questionnaire results from over 110 e-learners verify the effectiveness of the proposed algorithm and show the similarity between the learners’ self-organized learning paths and the recommended learning paths.
2) Achievment 2：
A group-oriented recommendation algorithm based on similarities of personal learning generative networks, IEEE Access, Vol. 6, 2018, pp:42729-42739. (on-line published, PDF, DOI:10.1109/ACCESS.2018.2856753)
To solve the lack of consideration of the learning time sequence and knowledge dependencies in group-based recommendation, we proposed a novel group-oriented recommendation algorithm which is characterized by mapping the user’s learning log to a personal learning generative network (PLGN) based on a knowledge map. In this paper, we first provide calculation methods of similarity and temporal correlation between knowledge points, where we provide the construction method of the PLGN. Second, a method for measuring the similarities between any two PLGNs is proposed. According to the similarities, we perform the CURE clustering algorithm to obtain learning groups. Third, based on the group clustering, the group learning generative network using a graph overlay method is generated. We calculate the importance of the vertices on the different learning needs and propose a group-oriented recommendation algorithm. Finally, we compare the effect of the proposed recommendation to that of a group-based collaborative filtering recommendation for the aspects of precision rate, recall rate, normalized discounted cumulative gain, and the average accuracy of parameters (MAP). The experimental results show that the group-oriented learning recommendation based on the learning generated network outperforms the group recommendation-based collaborative filtering when the amount of data is large enough.
3) Achiement 3
A Cross-Curriculum Video Recommendation Algorithm Based on a Video-Associated Knowledge Map,(IEEE Access, Vol. 6, 2018, pp:42729-42739.)
Learning resource recommendation, such as curriculum video recommendation, is an effective way to reduce cognitive overload in online learning. The existing curriculum video recommendation systems are generally limited to one course, ignoring the knowledge correlation between courses. In this work, we propose a two-stage cross-curriculum video recommendation algorithm that considers both the learners’ implicit feedback and the knowledge association between course videos. First, we use collaborative filtering to generate a video seed set, which is based on the learner’s implicit video feedback, such as video learning frequencies, video learning duration, and video pausing and dragging frequencies. Second, we construct a cross-curriculum video-associated knowledge map and use a random walk algorithm to measure the relevance of the course videos. The relevance is based on each video seed as a starting node and is extended to a video subgraph. Then, several cross-curricular video-oriented subgraphs are recommended for the learners. The experimental results indicate that our cross-curriculum video recommendation algorithm performs better than the traditional collaborative filtering-based recommendation algorithms in terms of accuracy, recall rate and knowledge relevance.
