Post #871

GraphML News (Nov 2nd) - The Debate on Equivariance, MoML and Stanford Graph workshop 🎃 Writing ICLR reviews and LOG rebuttals might have delivered you enough of the Halloween spirit with spooky papers and (semi)undead reviewers - it’s almost over though! 🥊 The debate on equivariance, namely, is it worth to bake symmetries right into the model or learn from data, remains to be a hot topic in the community with new evidence appearing every week supporting both sides. Is torch.nn.TransformerEncoder all you need? In the blue corner, the work Does equivariance matter at scale? by Johann Brehmer et al compares a vanilla transformer with the E(3)-equivariant Geometric Algebra Transformer (GATr) on the rigid-body modelling task with a wide range of sizes to derive scaling laws (akin to Kaplan and Chinchilla laws) and finds that the equivariant transformer scales better overall. In the red corner, we have The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains by Eric Qu et al who modify a vanilla transformer and outperform hefty Equiformer, GemNet, and MACE on ML potential benchmarks on molecules and crystals. Another wrestler in the red corner is the tech report on ORB v2 by Mark Neumann and Orbital Materials - ORB v2 is a vanilla MPNN potential trained with a denoising objective and delivers SOTA (or close to) performance while trained on only 8 A100s (compared to 64+ GPUs needed for Equiformer V2 but subject to different training datasets). 🏆 Overall, “no equivariance” wins this week 2-1 (2.5 - 1 if including a recent work on relaxed equivariance). 🎤 Next Tuesday, Nov 5th, is not just the election day in the US, but also the day of two graph learning events: MoML 2024 at MIT and the Graph Learning Workshop 2024 at Stanford. Programs of both events are now visible and there might be livestreams as well, keep an eye on the announcements. Weekend reading: Generator Matching: Generative modeling with arbitrary Markov processes by Peter Holderrieth feat. Ricky Chen and Yaron Lipman - a generalization of diffusion, flow matching (both continuous and discrete), and jump processes (outstanding paper award at ICLR’24). Expect a new generation of generative models for images / proteins / molecules / SBDD / RNAs / crystals to adopt this next year. Long-context Protein Language Model by Yingheng Wang and (surprisingly) Amazon team - introduces a Mamba-based bidirectional protein LM that outperforms ESM-2 on a variety of tasks while being much smaller and faster. Iambic announced NeuralPLexer 3 competitive with AlphaFold 3. While we are waiting for the tech report and more experiments, it seems that NP3 features Triton kernels for efficient triangular attention akin to FlashAttention but on triples of nodes.