Research Approach
I care not only about which problems to study, but also about the shape of a good research bet. I am most excited by two modes of exploration: opening a new direction with practical value, and building complete systems where many pieces have to work together.
Opening New Directions with Practical Value
I like research that identifies a useful new formulation early, then makes it practical enough for others to build on. Representative works include:
- MarDini, among the first hybrid autoregressive-diffusion models enabling scalable autoregressive video generation. Homepage arXiv
- T-GATE, among the first works to successfully apply KV-caching-style reuse to Diffusion Transformers for both image and video generation. arXiv GitHub ★ 418
- NLSOM, scaling the number of agents to study group intelligence in natural-language societies of mind; Best Paper Award at the NeurIPS 2023 R0-FoMo Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models. arXiv GitHub ★ 80 Best Paper
- BoxDiff, an early diffusion-based approach to controllable generation, developed contemporaneously with ControlNet. arXiv GitHub ★ 274
- Neural Computers, among the first end-to-end efforts to use diffusion dynamics to simulate general-purpose computation. arXiv GitHub ★ 202 Blogpost Karpathy mention
Building Complete Systems
I also enjoy systematic, result-driven research: building a complete system rather than relying on a single algorithmic breakthrough. In this mode, data, architecture, training, inference, evaluation, and product constraints have to co-evolve. Representative systems include:
- SANA-Video, an efficient video generation system in the Sana family. Homepage arXiv GitHub ★ 8.3k
- SANA-WM, a minute-scale world-modeling system built around efficient hybrid linear Diffusion Transformers. Homepage arXiv GitHub ★ 8.3k
- TUNA, a native unified multimodal model system for taming unified visual representations. Homepage arXiv GitHub ★ 727
- Meta in-house foundation model work, large-scale model building where product impact, data, evaluation, and system design have to co-evolve.