Yaroslav Bulatov

The point of "bitter lesson" is that simple things often work at scale. You can "muscle your way" past bad algorithms by scaling up compute. But if you fix your compute, the only way to scale up is through algorithms. Epoch AI found that in the last 12 years of LLMs, our

In the last 5 years, we mostly figured out how compress the knowledge of 100B people who ever lived into some matmuls, "System 1 thinking". What's next? 1) Collect more knowledge. But that takes robots which we don't have. 2) Come up with arch for "System 2 thinking". But that's

@HazanPrinceton (sanity checks on the formula collected in https://t.co/WqlV93ISr5)

@HazanPrinceton For that form, loss after t/2 steps of GD with step size 1 has a nice form in terms of the complementary error function. https://t.co/ClDuRK3im6