Jingling Li

I am a Ph.D. student in Computer Science at the University of Maryland, College Park, where I am fortunate to be advised by Prof. John Dickerson. Before that, I obtained my Bachelor’s degree in Computer Science and Mathematics from Bryn Mawr College. My research focuses on understanding and enriching the reasoning capabilities of current deep learning models. I believe having the ability to reason is an important and necessary step to achieving general intelligence.

Across my Ph.D., I have interned at Microsoft Research under the guidance of Adith Swaminathan on designing better hindsight learning schemes for combinatorial optimization problems. I have also interned at DeepMind under the supervision of Dr. Petar Veličković, working on how to re-use the learned knowledge and skills in reinforcement learning. In addition, I have worked closely with Prof. Jimmy Ba and Prof. Taiji Suziki when doing research internships at Vector Institute and RIKEN AIP. During my research, I also received great guidance from Prof. Don Perlis, Prof. Furong Huang, and Prof. Justin Brody.

news

Aug 26, 2022	I finished my summer internship at Microsoft Research, and I am continuing working on designing better RL algorithms for combintorial optimization problems.
Jan 21, 2022	I passed my preliminary exam, and I am now a Ph.D. candidate!
Sep 28, 2021	Our work on How Does a Neural Network’s Architecture Impact its Robustness to Noisy Labels has been accepted to NeurIPS 2021.
Sep 28, 2021	Our work on VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization has been accepted to NeurIPS 2021.
May 24, 2021	Started internship at DeepMind.

selected publications

Neurips

How Does a Neural Network’s Architecture Impact Its Robustness to Noisy Labels?

Li, Jingling, Zhang, Mozhi, Xu, Keyulu, Dickerson, John, and Ba, Jimmy

In Advances in Neural Information Processing Systems, 2021

Abs HTML Code

Noisy labels are inevitable in large real-world datasets. In this work, we explore an area understudied by previous works — how the network’s architecture impacts its robustness to noisy labels. We provide a formal framework connecting the robustness of a network to the alignments between its architecture and target/noise functions. Our framework measures a network’s robustness via the predictive power in its representations — the test performance of a linear model trained on the learned representations using a small set of clean labels. We hypothesize that a network is more robust to noisy labels if its architecture is more aligned with the target function than the noise. To support our hypothesis, we provide both theoretical and empirical evidence across various neural network architectures and different domains. We also find that when the network is well-aligned with the target function, its predictive power in representations could improve upon state-of-the-art (SOTA) noisy-label-training methods in terms of test accuracy and even outperform sophisticated methods that use clean labels.
ICLR

How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks

Xu, Keyulu, Zhang, Mozhi, Li, Jingling, Du, Simon S., Kawarabayashi, Ken-ichi, and Jegelka, Stefanie

In International Conference on Learning Representations (Oral) 2021

Abs HTML Code

We study how neural networks trained by gradient descent extrapolate, i.e., what they learn outside the support of the training distribution. Previous works report mixed empirical results when extrapolating with neural networks: while feedforward neural networks, a.k.a. multilayer perceptrons (MLPs), do not extrapolate well in certain simple tasks, Graph Neural Networks (GNNs) – structured networks with MLP modules – have shown some success in more complex tasks. Working towards a theoretical explanation, we identify conditions under which MLPs and GNNs extrapolate well. First, we quantify the observation that ReLU MLPs quickly converge to linear functions along any direction from the origin, which implies that ReLU MLPs do not extrapolate most nonlinear functions. But, they can provably learn a linear target function when the training distribution is sufficiently "diverse". Second, in connection to analyzing the successes and limitations of GNNs, these results suggest a hypothesis for which we provide theoretical and empirical evidence: the success of GNNs in extrapolating algorithmic tasks to new data (e.g., larger graphs or edge weights) relies on encoding task-specific non-linearities in the architecture or features. Our theoretical analysis builds on a connection of over-parameterized networks to the neural tangent kernel. Empirically, our theory holds across different training settings.
ICLR

What Can Neural Networks Reason About?

Xu, Keyulu, Li, Jingling, Zhang, Mozhi, Du, Simon S., Kawarabayashi, Ken-ichi, and Jegelka, Stefanie

In International Conference on Learning Representations (Spotlight) 2020

Abs HTML Code

Neural networks have succeeded in many reasoning tasks. Empirically, these tasks require specialized network structures, e.g., Graph Neural Networks (GNNs) perform well on many such tasks, but less structured networks fail. Theoretically, there is limited understanding of why and when a network structure generalizes better than others, although they have equal expressive power. In this paper, we develop a framework to characterize which reasoning tasks a network can learn well, by studying how well its computation structure aligns with the algorithmic structure of the relevant reasoning process. We formally define this algorithmic alignment and derive a sample complexity bound that decreases with better alignment. This framework offers an explanation for the empirical success of popular reasoning models, and suggests their limitations. As an example, we unify seemingly different reasoning tasks, such as intuitive physics, visual question answering, and shortest paths, via the lens of a powerful algorithmic paradigm, dynamic programming (DP). We show that GNNs align with DP and thus are expected to solve these tasks. On several reasoning tasks, our theory is supported by empirical results.
AISTATS

Understanding Generalization in Deep Learning via Tensor Methods

Li, Jingling, Sun, Yanchao, Su, Jiahao, Suzuki, Taiji, and Huang, Furong

In International Conference on Artificial Intelligence and Statistics 2020

Abs HTML

Deep neural networks generalize well on unseen data though the number of parameters often far exceeds the number of training examples. Recently proposed complexity measures have provided insights to understanding the generalizability in neural networks from perspectives of PAC-Bayes, robustness, overparametrization, compression and so on. In this work, we advance the understanding of the relations between the network’s architecture and its generalizability from the compression perspective. Using tensor analysis, we propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks; thus, in practice, our generalization bound outperforms the previous compression-based ones, especially for neural networks using tensors as their weight kernels (e.g. CNNs). Moreover, these intuitive measurements provide further insights into designing neural network architectures with properties favorable for better/guaranteed generalizability. Our experimental results demonstrate that through the proposed measurable properties, our generalization error bound matches the trend of the test error well. Our theoretical analysis further provides justifications for the empirical success and limitations of some widely-used tensor-based compression approaches. We also discover the improvements to the compressibility and robustness of current neural networks when incorporating tensor operations via our proposed layer-wise structure.