Graph Machine Learning

​​GraphML News (Jan 26th) - Graph learning overtakes ⚽, GPT-4b ⚽TacticAI, the system for analyzing football games using MPNNs and geometric learning developed by Google DeepMind for Liverpool FC (one of the coolest applications highlighted in the post about 2023 graph learning advancements), is making rounds in the UK sports industry: now Arsenal FC is looking for Research Engineers to work on “spatiotemporal transformer with built-in geometric equivariances”. This hints about the success of the original TacticAI (as well as about the small world of English Premier League), and makes us wonder: - Would graph learning get a surprising vanity boost from a rather unexpected place? Transfers of best GraphML researchers from one club to another along with the head coach team, eg, Christopher Morris to Borussia Dortmund or Stephan Günnemann to Bayern Munich? - Would 💰 from sheikhs and rich owners of football clubs investing into GPUs result in more expected scaling laws (instead of ridiculously inflated players’ contracts)? - When will Mikel Arteta (head coach of Arsenal) tell English newspapers “Equivariance rules!” like Geoff Hinton? We will keep you posted about those important matters. Meanwhile, post your fantasy teams of graph researchers and FCs in the comments. 🧬 OpenAI reportedly finished training of GPT-4b, the protein LLM, together with Retro Biosciences (where Sam Altman conveniently invested $180M) that focuses on longevity studies. GPT-4b aims at re-engineering a specific set of proteins, Yamanaka factors, that can turn human skin cells into young stem cells. We don’t know more details, the model is likely to stay closed - one could hypothesize it might look like the ESM family of protein models with the knowledge of protein function and trained on a massive dataset of proprietary data (the key to a successful biotech startup in 2025). 🎙️ The Graph Signal Processing Workshop 2025 will take place on May 14-16 at Mila in Montreal supported by Centre de recherches mathématiques (CRM) and Valence Labs. The workshop invites theoretical works in signal processing on graphs and will showcase examples of applications in gene expression patterns defined on top of gene networks, the spread of epidemics over a social network, the congestion level at the nodes of a telecommunication network, and patterns of brain activity defined on top of a brain network. Submission deadline is Feb 1st. Weekend reading: ICLR 2025 announced accepted papers but the full list is not yet available. Moreover, expect a flurry of the announcements next week after the ICML submission deadline.

GraphML News (Jan 18th) - MatterGen release, Aviary, Metagene-1 We are getting back to the regular schedule after the winter break! ⚛️ MSR AI 4 Science has finally released code and weights of MatterGen, the generative model for inorganic materials, together with its publication on Nature (free and no paywall). The final version includes new evaluation pipeline that accounts for compositional disorder (which unrealistically increases performance metrics of recent generative models) and experimental validation of the first generated material TaCr2O6. MatterSim, an ML potential model, was also used during the filtering stages along with standard DFT calculations. Great result for MSR, materials science community, and diffusion model experts who can apply a bag of tricks to a new domain 👏 📜 Together with MatterGen, MaSIF-neosurf from EPFL got published on Nature as well. MaSIF-neosurf is a geometric model for studying surfaces of protein-ligand complexes which was experimentally evaluated on binders agains three real-world protein complexes. To conclude the celebration of massive scientific publications, ESM 3 - the foundation model for proteins - got accepted at Science. 🕊️ FutureHouse released Aviary, the agentic framework for scientific tasks like molecular cloning, protein stability, and scientific QA. Aviary extends the framework of PaperQA (the best open source scientific RAG tool) with learnable RL environments and demonstrated that even small LLMs like LLaMa 3.1 7B excel at these tasks with enough inference compute. Get ready to hear about inference time scaling all over 2025 😉 ✏️ The AI 4 Science consortium published a blog post AI 4 Science 2024 highlighting AF3 and its replications, the success of non-equivariant models, scientific foundation models, new progress in small molecules and quantum chemistry. A short but insightful read. 🪣Metagene-1 (by USC and Prime Intellect) is a foundation model trained on metagenomic sequences (”human wastewater samples” we all know what it means 💩) that might help in pandemic monitoring and pathogen detection. It’s a standard LLaMa 2-7B architecture but, interestingly, outperforms some state space models like HyenaDNA on genome understanding. Weekend reading: GenMol: A Drug Discovery Generalist with Discrete Diffusion by Seul Lee, NVIDIA and KAIST - a generalist generative model for a suite of drug discovery tasks like de-novo generation, fragment-conditioned generation, and lead optimization. The Jungle of Generative Drug Discovery: Traps, Treasures, and Ways Out by Riza Özçelik and Francesca Grisoni - on metrics and benchmarking for generative models for molecules. Explaining k-Nearest Neighbors: Abductive and Counterfactual Explanations by Pablo Barceló and CENIA team from Chile - a theoretical work tackling classical (but still important) kNN classifiers and how their predictions can be explained. Experiments on MNIST and can run on a laptop

Let's use this post for comments!

GraphML News (Dec 20th) - Thoughts after NeurIPS, ICLR workshops, new blogposts In the last op-ed of this year and returning back from NeurIPS, it’s about time to reflect on the state of graph learning research. Comments to this post should be open (hopefully?) 🤔 At the age when o3 solves some of incredibly hard FrontierMath problems, when NotebookLM allows to call into a podcast generated about a paper of your choice, and Veo2 generates 4K videos with increasingly correct physics, it is somewhat frustrating to see that the graph learning (meaning vanilla 2D graph learning here, because geometric DL is flourishing in the AI 4 Science areas) community is still obsessed with WL tests, node classification on OGB, and other toy tasks that increasingly lose relevance in the modern deep learning world. Is it the issue of too toyish benchmarks, the lack of cool applications, or something else? Is Graph ML to be confined in the recsys and retail predictions domain or it could get its own “RLHF revival moment”? 🏗️ ICLR 2025 started announcing the accepted workshops, you might find some of those interesting: - Frontiers in Probabilistic Inference: Sampling Meets Learning - Generative and Experimental Perspectives for Biomolecular Design - Weight Space Learning - Learning Meaningful Representations of Life - AI for Accelerated Materials Design will be back, too 📝 New blogposts! Understanding Transformer reasoning capabilities via graph algorithms by Google Research elaborating on the NeurIPS 2024 paper on when and where transformers can outperform GNNs on graph tasks. A massive 3-part study of pooling in GNNs by Filippo Maria Bianchi (Arctic University of Norway) introduces the common pooling framework (select-reduce-connect), studies a variety of pooling methods and their evaluation protocols.

GraphML News (Dec 8th) - NeurIPS’24, ESM Cambrian, Antiviral Competition, The Well 🍻 NeurIPS 2024 starts next week, the full schedule is available, the main conference is scheduled for Wed-Fri with two days of workshops (Sat-Sun) and unknown amount of private parties and gatherings throughout the week. See you in Vancouver! 🧬 EvolutionaryScale announced ESM Cambrian (ESM C), a new family of embedding models replacing ESM-2 with better performance across all sizes (300M, 600M, and 6B), dramatically smaller memory requirements and faster inference (think of Triton kernels here). ESM C was trained on UniRef, MGnify, and JGI data, smaller models are already available on GitHub, the 6B is available through the API service. 💊 Polaris Hub launches the Antiviral Competition together with ASAP discovery and OpenADMET. The competition includes three tracks: - Predicting ligand poses of MERS-CoV based on SARS-CoV2 structures (metric: RMSD) - Predicting ligand fluorescence potencies based on SARS and MERS data (metrics: MAE of pIC50 and ranking) - Predicting ligands’ ADMET properties (MAE and ranking) The competition starts on Jan 13th and ends on March 25th, prepare your big GNNs ⚔️ ⚛️ After the announcement in May, MSR released the code and weights of MatterSim, a universal ML potential akin to MACE-MP-0 and Orb models. MatterSim is based on the M3GNet message passing GNN and is available in 1M and 5M params versions. 🪣 Polymathic AI, Flatiron Institute, and a collab of universities and national labs released The Well, a 15 TB dataset of physical simulations (think PDEs and Neural Operators) covering 16 different areas from fluid dynamics to supernova explosions. In the accompanying preprint, the authors compared several variants of Fourier Neural Operators (FNO) and U-Nets. A great resource for scientific and industrial applications where expensive simulations eat up a huge bulk of supercomputers time.

GraphML News (Nov 30th) - LOG recordings, The Chip Has Sailed, Illustrated Flow Matching 📺 LOG 2024 has just wrapped up, and all recordings are now available on the official YouTube channel, featuring: - Keynotes from Yusu Wang (UCSD), Zach Ulissi (Meta), Xavier Bresson (NUS), Alden Hung (Isomorphic Labs) - Tutorials on geometric generative models, neural algorithmic reasoning, GNNs for time series, heterophilic graph learning, and KGs + LLMs - All oral presentations A lot of stuff to digest over the weekend if you missed it! ⛵️ The GNNs for chip design saga continues: - the seminal 2021 Nature paper (now known as AlphaChip) spun off some controversy from the chip design community about reproducibility (however, other people say it’s rather a skill issue and the CD community just doesn’t have proper deep learning chops), - several rounds of message exchange led to the official Addendum on the Nature paper clarifying the concerns and adding that pre-training is a must. - In Oct 2024, Igor Markov (Synopsys) published an article at ACM Communications with a further criticism of the approach. - Recently, in Nov 2024, the lead authors of AlphaChip (Anna Goldie, Azalia Mirhoseini, and Jeff Dean himself) recently put an arxiv paper The Chip Has Sailed with the full timeline and emphasizing that the approach is actually already working within Google and AlphaChip has been used in several generations of chips used in production. Perhaps one of the most important messages from this paper is that while academics and industry debate whether the approach could work, it is actually already working. We’ll keep you posted! 🖌️A Visual Dive into Conditional Flow Matching by Anne Gagneux, Ségolène Martin, Rémi Emonet, Quentin Bertrand, and Mathurin Massias to flow matching is what the Illustrated Transformer to transformers - a detailed visual guide on the inner workings of flow matching and math behind it, a highly recommended reading. 🪠PLUMBER from Bioptic is the new protein-ligand benchmark of 1.8M data points based on PLINDER and enriched with more data from BindingDB, ChEMBL, and BioLip 2 to probe robustness of protein-ligand binding models in more diverse compositional generalization tests.

GraphML News (Nov 23rd) - LOG 2024, Boltz-1 and Chai-1r, OCx24, cuEquivariance 🎙️LOG 2024 starts next Tuesday! The schedule of orals and posters is already available, registration is free, we’ll be happy to see you there online or at one of many local meetups! 🧬 The “Stable Diffusion moment” in generative models for structural biology is spinning up: two models released during the past week starting with fully open-source Boltz-1 from MIT (tech report, code) that achieves AF3-like quality and outperforms a recent Chai-1 (open inference code). Meanwhile, Chai folks released Chai-1r that supports complexes with user-specified restraints - and already compared with Boltz-1. Open source and competition really drive the field forward 👏 The next step for AF3-like models seems to be integrating the recent GPU-enabled MSA tool MMSeqs2-GPU that might shave off another order of magnitude of inference time of protein structure prediction models. 🧊 FAIR Chemistry at Meta, University of Toronto, and VSParticle presented Open Catalyst experiments (OCx24) - a new dataset of 600 mixed metal catalysts synthesized and probed physically in the lab (a huge step beyond DFT-only simulations) along with analytical data for 19k catalyst materials with 685M relaxations. It’s already the 3rd huge dataset openly published by Meta in addition to OpenDAC and OMAT - Meta is a firm leader in this area of AI 4 Science. Fun fact: models from the Open Catalyst ecosystem are directly used in Meta’s products like recent Orion AR glasses. 🔱 Faster spherical harmonics and tensor products for geometric GNNs: following EquiTriton, an open-source collection of Triton kernels for fast and memory-efficient computation of spherical harmonics developed by Intel Labs (enabling harmonics of up to the 10th order), NVIDIA released cuEquivariance - CUDA kernels (closed-source kernels with public bindings for PyTorch, JAX, and numpy) for spherical harmonics and tensor products which speed up DiffDock, MACE and other models by 2-20x, this is especially useful in tasks where a model is called multiple times like a generative model or MD calculations. cuEquivariance is a part of the new BioNeMo suite for drug discovery. Weekend reading: 📚 Check out accepted orals and posters of LOG 2024 on OpenReview

GLOW Reading Group on Nov 20th 🌟Graph Learning on Wednesdays (GLOW) is a new reading group about foundations and latest developments in Graph Machine Learning - its inaugural meeting was held on Oct 9th, and the next one is happening on Wednesday, Nov 20th at 5pm CET (11am Eastern) featuring two papers presented by the first authors: - On the Expressivity and Sample Complexity of Node-Individualized Graph Neural Networks by Paolo Pelizzoni - Graph Neural Networks Use Graphs When They Shouldn’t by Maya Bechler-Speicher The format of the RG is rather interactive: short presentations by the authors are followed by discussions and brainstorms, and sometimes expert panels study the paper together. Consider joining on Wednesday!

GraphML News (Nov 16th) - ICLR 2025 Stats, Official AlphaFold 3 and RFam, Faster MD 📊 PaperCopilot comprised the basic statistics of ICLR 2025 initial reviews and best scored papers - a few GNN papers are in top-100 and we’ll keep an eye on them! Meanwhile, Azmine Toushik Wasi compiled a list of accepted graph papers at NeurIPS 2024 grouped by categories - from GNN theory to generative models to transformers to OOD generalization and many more. 🧬 Google DeepMind finally released the official code for AlphaFold 3 featuring a monstrous 3k lines of code Structure class and some kernels implemented in Triton (supporting the fact that Triton is the future and you should implement your most expensive neural net ops as efficient kernelized ops). Somewhat orthogonally to AF3, the Baker Lab presented RFam, the improved version of RFdiffusion, in a paper about metallohydrolases. RFam now uses flow matching (welcome on board), allows for scaffolding arbitrary atom-level motifs and sequence-position-agnostic scaffolding. Waiting for the code soon! 🏎️ Microsoft Research announced AI2BMD - a freshly accepted to Nature method for accelerating ab-initio molecular dynamics of proteins with equivariant GNNs (based on VisNet, already in PyG) scaling it up to impressive 10k atoms in a structure (far beyond the capabilities of standard MD tools). Besides, the authors collected a new dataset of 20M DFT-computed snapshots which would be of great help to the MD community. 🌊 Continuing the simulation note, NXAI and JKU Linz presented NeuralDEM, a neural approach to replace Discrete Element Method (DEM) in complex physical simulations (like fluid dynamics) with transformers and neural operators. NeuralDEM is as accurate and stable as vanilla DEM while being much faster and allowing for longer sim times. Weekend reading: Generalization, Expressivity, and Universality of Graph Neural Networks on Attributed Graphs by Levi Rauchwerger, Stefanie Jegelka, and Ron Levie - it is known that the vanilla WL test assumes no node features. This is one of the first works to study GNN properties on featurized graphs. GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts by Shirley Wu et al feat. Bruno Ribeiro and Jure Leskovec Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning by Haitz Borde et al feat. Michael Bronstein - a transformer-like block where multi-head attention is replaced with a GCN (keeping layer norms, residual stream and MLPs intact) is suprisingly competitive on very large graphs

GraphML News (Nov 9th) - ELLIS PhD Applications, Protenix AF3, New papers The next week is going to be busy with ICLR rebuttals, so we still have a bit of time to check the news and read new papers. 🎓 The call for PhD applications within ELLIS, the European network of ML and DL labs, is ending soon (November 15th) - this is a great opportunity to start (or continue) your academic journey in a top machine learning lab! 🧬 ByteDance released Protenix, a trainable PyTorch reproduction of AlphaFold 3, with model checkpoints (so you can run it locally) and with the inference server. The tech report is coming, would be interesting to compare with Chai-1 and other open source reproductions. Weekend reading: A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing by Julia Balla et al feat Tess Smidt - introduces cool new graph datasets - given a point cloud of 5000 galaxies, predict their cosmological properties on the graph level and node level (eg, galaxy velocity). FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions by Anuroop Sriram and FAIR (NeurIPS 24): extension of FlowMM (ICML 2024), a flow matching model for crystal structure generation, but now instead of sampling from a Gaussian, the authors fine-tuned LLaMa 2 on the Materials Project to sample 10000 candidates (using just a small academic budget of 300 A100 gpus) - this prior yields 3x more stable structures. Flow Matching for Accelerated Simulation of Atomic Transport in Materials by Juno Nam and MIT team feat. Rafael Gómez-Bombarelli - introduces LiFlow, a flow matching model for MD simulations of crystalline materials: where ab-initio methods would take 340 days to simulate 1 ns of a 200-atoms structure, LiFlow takes only 48 seconds 🏎️

GraphML News (Nov 2nd) - The Debate on Equivariance, MoML and Stanford Graph workshop 🎃 Writing ICLR reviews and LOG rebuttals might have delivered you enough of the Halloween spirit with spooky papers and (semi)undead reviewers - it’s almost over though! 🥊 The debate on equivariance, namely, is it worth to bake symmetries right into the model or learn from data, remains to be a hot topic in the community with new evidence appearing every week supporting both sides. Is torch.nn.TransformerEncoder all you need? In the blue corner, the work Does equivariance matter at scale? by Johann Brehmer et al compares a vanilla transformer with the E(3)-equivariant Geometric Algebra Transformer (GATr) on the rigid-body modelling task with a wide range of sizes to derive scaling laws (akin to Kaplan and Chinchilla laws) and finds that the equivariant transformer scales better overall. In the red corner, we have The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains by Eric Qu et al who modify a vanilla transformer and outperform hefty Equiformer, GemNet, and MACE on ML potential benchmarks on molecules and crystals. Another wrestler in the red corner is the tech report on ORB v2 by Mark Neumann and Orbital Materials - ORB v2 is a vanilla MPNN potential trained with a denoising objective and delivers SOTA (or close to) performance while trained on only 8 A100s (compared to 64+ GPUs needed for Equiformer V2 but subject to different training datasets). 🏆 Overall, “no equivariance” wins this week 2-1 (2.5 - 1 if including a recent work on relaxed equivariance). 🎤 Next Tuesday, Nov 5th, is not just the election day in the US, but also the day of two graph learning events: MoML 2024 at MIT and the Graph Learning Workshop 2024 at Stanford. Programs of both events are now visible and there might be livestreams as well, keep an eye on the announcements. Weekend reading: Generator Matching: Generative modeling with arbitrary Markov processes by Peter Holderrieth feat. Ricky Chen and Yaron Lipman - a generalization of diffusion, flow matching (both continuous and discrete), and jump processes (outstanding paper award at ICLR’24). Expect a new generation of generative models for images / proteins / molecules / SBDD / RNAs / crystals to adopt this next year. Long-context Protein Language Model by Yingheng Wang and (surprisingly) Amazon team - introduces a Mamba-based bidirectional protein LM that outperforms ESM-2 on a variety of tasks while being much smaller and faster. Iambic announced NeuralPLexer 3 competitive with AlphaFold 3. While we are waiting for the tech report and more experiments, it seems that NP3 features Triton kernels for efficient triangular attention akin to FlashAttention but on triples of nodes.

GraphML News (Oct 26th) - LOG meetups, Orbital round, ESANN 2025 🍻 The Learning of Graphs conference continues to update the list of local meetups - the networks already includes 13 places from well-known graph learning places like Stanford, NYC, Paris, Oxford, Aachen, Amsterdam, Tel Aviv down to Tromsø, Uppsala, Siena, New Delhi, Suzhou, and Vancouver (Late November in Tromsø, talking graphs with a cup of glühwein and snow outside must be a quite a cozy venue). The call for meetups is still open! On this note, the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2025) will host 3 special sessions on graph learning: Foundation and Generative Models for Graphs, Graph Generation for Life Sciences, and Network Science Meets AI. Submission deadline is November 20, 6 pages tops. Thanks to Manuel Dileo for the pointer! ESANN 2025 will take place on April 23-25 (2025) in Bruges (jokes about the movie and Tottenham are welcome). 💸 Orbital Materials secured a new funding round led by NVIDIA Ventures (financial details undisclosed) – timed nicely coinciding with the recent release of the ML potential GNN Orb-v2. A new unicorn from AI 4 Science is coming? 🤔 Weekend reading: Learning Graph Quantized Tokenizers for Transformers by Limei Wang, Kaveh Hassani et al and Meta - an unorthodox approach for graph tokenization via vector quantization and codebook learning, conceptually similar to VQ-GNNs (NeurIPS 2021), strange to not see this older paper cited Relaxed Equivariance via Multitask Learning by Ahmed Elhag et al feat Michael Bronstein - instead of baking equivariances right into models, let’s add it as a loss component and allow a model to learn and use as much equivariance as necessary, brings 10x inference speedups. Homomorphism Counts as Structural Encodings for Graph Learning by Linus Bao, Emily Jin, et al - introduces motif structural encoding (MoSE) for graph transformers. Paired with GraphGPS, brings MAE on ZINC from 0.07 down to 0.062 and to 0.056 with GRIT.