Sasha Rush+

srush starred muscriptor/muscriptor

srush opened a pull request in chalx

srush opened a pull request in chalx

srush opened a pull request in colm-org.github.io

srush pushed rush-nlp

srush pushed rush-nlp

srush pushed rush-nlp

AI World Models (Keyon Vafa)

What do we know about how AI systems understand the world? Guest talk by Keyon Vafa (https://keyonvafa.com/) Main papers discussed: - Evaluating the World Model Implicit in a Generative Model: https://arxiv.org/abs/2406.03689 - What has a Foundation Model Found? Using Inductive Bias to Probe for World Models: https://arxiv.org/abs/2507.06952

Reasoning in the Wild - Wenting Zhao

Wenting Zhao. An overview of challenges in modern reasoning. (https://wenting-zhao.github.io/ ). * Part 1: WildChat: What do users ask LLMs * Part 2: Learning to reason with structural supervision * Part 3: Commit-0: How to reason in code environments * Future Challenges

Linear Attention and Beyond (Interactive Tutorial with Songlin Yang)

Songlin Yang, the author of the influential Flash Linear Attention https://github.com/fla-org/flash-linear-attention library, joined me to give an interactive tutorial on the current state of linear attention. This is a more technical coverage of much of the latest work on this topic.

Compute-Constrained Data Selection (Junjie Oscar Yin)

Guest talk on post-training data selection in a scaling laws framework. Intern project from Junjie Oscar Yin https://oseyincs.io/about/ Paper (ICLR 2025): https://arxiv.org/abs/2410.16208

How DeepSeek Changes the LLM Story

Quick turnaround survey of DeepSeek v3 and DeepSeek R1 the two technical papers behind the recent open-source LLM news. Presented at Simons Institute Feb 3, 2024. Slides: https://docs.google.com/presentation/d/e/2PACX-1vRTikiWLPInajDs8GIfTf8hEicxbApz7sWZVx7TVY2r9uJmyFOKek_37U5OWt51Q1f3-RGIG8o85Xe0/pub?start=false&loop=false&delayms=60000

Flash LLMs: Block Quantization

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: Data Parallel + ZeRO-1

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: Tensor Cores

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: MFU

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: BF16

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: Computation / Communication Overlap

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: Mixture of Experts

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: FP8

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: Expert Parallel

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).

Flash LLMs: Pipeline Parallel

Primer for understanding modern LLMs such as Llama 3 (https://arxiv.org/abs/2407.21783) and DeepSeek v3 (https://arxiv.org/abs/2412.19437).