Zhejiang University · DCD Lab · Hangzhou
Undergraduate Researcher · Multimodal Speech Intelligence
I study how machines can perceive, generate, and interact through sound. My recent work spans spoken-dialogue reward modeling, multimodal spatial audio, and full-duplex spoken interaction.
Background
Building academically grounded audio systems from data and modeling to evaluation.
I am an undergraduate student in Computer Science and Technology at Zhejiang University, also pursuing the Advanced Honors Class of Engineering Education at Chu Kochen Honors College.
Since September 2024, I have worked as a research assistant in the DCD Lab under the supervision of Prof. Zhou Zhao. My current work connects spoken-language intelligence with generative audio: learning better reward signals for natural dialogue, building large-scale multimodal audio data, and developing controllable spatial and full-duplex audio systems.
Research output
Peer-reviewed work in spoken dialogue, spatial audio, and singing voice synthesis, with links to official records and open research resources.
An end-to-end reward model, preference dataset, and evaluation benchmark for assessing modality awareness and colloquialness in multi-turn spoken dialogue.
I contributed to preference-data construction, reward-model design and training, ablation studies, and benchmark validation.
A survey and empirical audit that separates where duplex decisions are made, which interaction is occurring, and how system behavior evolves moment by moment.
I developed the L0–L3 architectural hierarchy, the T×I×R interaction ontology, and a five-state decision machine, and audited representative models, training datasets, and evaluation benchmarks.
A 484-hour multimodal spatial-audio dataset spanning speech, singing, music, and everyday scenes, with synchronized audio, video, trajectories, and refined annotations.
I contributed to data collection and processing, audio-video and 3D-position synchronization, preprocessing, and speech-text alignment.
Research agenda
One coherent direction: making audio agents more perceptive, generative, and natural in real-time interaction.
Reward modeling and preference data for modality-aware, colloquial, multi-turn dialogue.
Multimodal generation, refined annotation, and evaluation for multi-source spatial scenes.
Low-latency spoken systems that can listen, respond, and coordinate naturally in real time.
Studying evaluation and learning methods for spoken systems that must listen, respond, and manage turn-taking continuously rather than alternate between fixed user and assistant turns.
Contributed to the ISDrama end-to-end spatial-audio architecture and the 484-hour MRSAudio dataset, spanning data collection, cross-modal synchronization, preprocessing, and refined annotation.
Academic & industry
Research, industry, open-source, and academic experience in speech and machine learning.
Industry research
Alibaba Group
Research and evaluation of end-to-end real-time spoken dialogue models, with emphasis on concurrent listening and speaking, interruption handling, turn transitions, and multi-turn interaction quality.
Open source
Reinforcement Learning Infrastructure for Embodied and Agentic AI
Implemented a complete D4RL offline-IQL training and checkpoint-resume workflow in PyTorch, with dataset, Runner, Worker, and multi-task configuration support for MuJoCo, AntMaze, Kitchen, and Adroit tasks.
Research
Zhejiang University · Supervisor: Prof. Zhou Zhao
Research on spatial audio generation, audio and speech processing, spoken-dialogue modeling, and full-duplex audio interaction.
Education
Zhejiang University
Minor · Advanced Honors Class of Engineering Education, Chu Kochen Honors College
Recognition
A concise selection of scholarships and academic awards.
National cohort · 30 recipients
Awarded in two consecutive academic years · Top 3%
College of Computer Science and Technology · 10 / 1,305
One of five recipients in the college
Engineering category
Theoretical category
