Links for September

General

• “every company has the same hiring criteria” by Ethan Ding: agency, intelligence, and character.
• Twyman’s law: “Any figure that looks interesting or different is usually wrong”. Very important to remember for research!
• Conventional Commits: I didn’t know that a convention for writing git commits existed, and had picked up (pieces of) convention mostly from reading actual commits.
• “What Is Emotional Truth?” is an essay by Bryan Caplan that resonates with me. Like him, I am a nerd who never cared much for science fiction, because they lack “emotional truth”. As Caplan writes, “Why can’t hard sci-fi or true stories fulfill this ideal?  In principle, they could.  But when creators spend a lot of mental energy on the accuracy of their physics or the historical sequence of events, they tend to lose sight of their characters’ inner lives.”
• “Everything I know about good API design” by Sean Goedecke has lots of good advice. Examples: don’t break userspace, and if you must make a breaking change, use API versioning (with something like \v1 in the API URL).

AI

• Hugging Face’s “Ultra-Scale Playbook”
• The “How to Think About GPUs” chapter in DeepMind’s JAX scaling book
• A guide to AI research interviews by Bas van Opheusden
• BeyondWeb: Datology’s paper on scaling synthetic pretraining data
• Stanford’s Systems for Machine Learning Fall 2023 class (with slides)
• Tencent’s Yan, a video foundation model
• “How to ML Paper” by Jakob Foerster has a nice checklist of writing pitfalls.
• Neel Nanda’s highly opinionated advice on how to write ML papers.
• “The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget”: “Source code is usually formatted with elements like indentation and newlines to improve readability for human developers. However, these visual aids do not seem to be beneficial for large language models (LLMs) in the same way since the code is processed as a linear sequence of tokens. … LLMs can maintain performance across formatted code and unformatted code, achieving an average input token reduction of 24.5% with negligible output token reductions.”