Jie Fu (付杰) is a happy and funny visiting scholar at Hong Kong University of Science and Technology (HKUST), chasing his human-friendly big AI dream.

He was a postdoctoral fellow (funded by Microsoft Research Montreal) supervised by Yoshua Bengio at University of Montreal, Quebec AI Institute (Mila). He was an IVADO postdoctoral fellow supervised by Chris Pal at Polytechnique Montreal, Quebec AI Institute (Mila). He worked as a researcher (PI) at BAAI (智源人工智能研究院). He obtained his PhD from National University of Singapore under the supervision of Tat-Seng Chua. He received ICLR 2021 Outstanding Paper Award.

His research agenda is centered around establishing an LLM within a unified, safe, and scalable world model framework, thus enabling tight integration of System-1 (e.g., intuitive, fast, unconscious) and System-2 (e.g., slow, logical, conscious, algorithmic, planning) functionalities. More concretely:

  • Safe and Scalable System-2 Language Models: Human-AI Alignment, Reinforcement learning, Meta learning, Lifelong learning, Reasoning, Modular neural architectures, etc
  • Applications: Multi-modal learning (e.g., NLP, CV), Multi-modal embodied intelligence, AI for science, etc

Selected Publications

I’m (still…) training myself (slowly…)

My full publications can be found in Google Scholar.
Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction
arXiv
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
arXiv
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
arXiv
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
ICLR 2024
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
arXiv
TACO: Topics in Algorithmic COde generation dataset
arXiv
AI Alignment: A Comprehensive Survey
arXiv
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
arXiv
When Do Graph Neural Networks Help with Node Classification: Investigating the Homophily Principle on Node Distinguishability
NeurIPS 2023
MARBLE: Music Audio Representation Benchmark for Universal Evaluation
NeurIPS 2023, Datasets and Benchmarks Track
Running Ahead of Evolution - AI based Simulation for Predicting Future High-risk SARS-CoV-2 Variants
IJHPCA, ACM Gordon Bell COVID Finalist 2022
Think Before You Act: Decision Transformers with Internal Working Memory
arXiv
Interactive Natural Language Processing
arXiv
MUDiff: Unified Diffusion for Complete Molecule Generation
LoG 2023
Chinese Open Instruction Generalist: A Preliminary Release
arXiv
Learning Multi-Objective Curricula for Robotic Policy Learning
CoRL 2022
Biological Sequence Design with GFlowNets
ICML 2022