Graph Machine Learning

GraphML News (Oct 19th) - Orb-v2, OMat24, Stanford Graph Learning Seminar, new PhD positions 🔥 The competition in materials science heats up: ML potentials (models that estimate the potential energy of an atomistic system and often predict energy, forces, and stresses) are one of the main drivers in the field as they can significantly speed up expensive molecular dynamics (MD) calculations. Matbench Discovery is one of the main benchmarks for ML potentials. 🔮 During the week, Orbital Materials released the code and weights of Orb-v2, the next version of the non-equivariant MPNN (Orbital folks explicitly bet against equivariant GNNs) that outperforms mighty MatterSim from MSR with just 25M parameters. Besides, Orb-v2 offers increased stability during MD calculations. 📈 A few days later, FAIR chemistry released OMat24, a new large dataset with 100M+ structures for training ML potentials (much larger than existing datasets) that required 400M+ core hours to complete DFT calculations for (preprint). Together with OMat24, FAIR released EquiformerV2, equivariant transformer, pre-trained on this dataset and fine-tuned on MatBench discovery (using just 64 A100s - 🌚 an entry-level 🌚 of compute those days) and claimed SOTA on Matbench Discovery. Interestingly, Equiformer got a significant performance boost when trained with the denoising objective - similar to what Orb models are trained on. It is likely that the benchmark will be fully saturated next year. Meanwhile, Google DeepMind together with Japanese institutes released a paper on applying GNoME (the flagship tool for materials discovery introduced last year) to synthesizing cesium chlorides. 🎙️ The Stanford Graph Learning Workshop will take place on November 5th physically at Stanford with the online stream, expect some new announcements and releases! 🎓 Finally, the application season for PhD positions and internships is open: we’d highlight the call for fully-funded PhD positions from Viacheslav Borovitskiy at the University of Edinburgh on Geometric Learning and Uncertainty Quantification (Geometric Kernels is one of the most recent works). Application deadline: Dec 15th, start date: September 2025. Let us know if your lab is hiring this season and we’ll compile a larger list of open geometric learning positions! Weekend reading: PDFs of ICLR 2025 submissions are now visible - you can open and read everything from the list we prepared a few weeks ago.

GraphML News (Oct 12th) - Nobel Prizes, Mediterranean ML Summer School 🏅If you lived under the rock this week, Deep Learning got two Nobel Prizes this year: Geoff Hinton and John Hopfield got the physics prize (less expected), and David Baker, John Jumper, and Demis Hassabis got the chemistry prize (more than expected after AF 2 received almost all other scientific awards). The acknowledgement of deep learning advancements was not rushed as it might seem - it took already 10+ years since the ImageNet revolution and the entire new industry has grown on top of it. It roughly took the same time for CRISPR (another chemistry Nobel Prize in 2020) to get acknowledged. What does the prizes mean for the field and industry (other than DL researchers could claim to be a bit of physicists and chemists themselves)? It is likely that AI 4 Science as a field in general would receive a significant attention with more researchers entering the area and more funding for commercializing some of the tech behind it. The potential of using DL methods in accelerating scientific discovery is still largely untapped (yes, Geometric DL did enable the recent successes in protein design and pharma but, for example, we can’t say that protein generative models truly learn underlying physics phenomena for now), so it is as exciting time as ever to start your research journey in this area. There is a plenty of space to do impactful research and we’ll probably see more labs and companies pivoting there. (Fun fact - brace yourselves as every 2nd talk at NeurIPS 2024 would probably start with the same Nobel Prize slides). 📺 The recordings of the Mediterranean ML Summer School are finally available! The school took place in September in Milan packing a week of talks on transformers, reasoning, diffusion models and flow matching, GNNs, RL, RLHF, optimization, and many more. Weekend reading (while waiting for ICLR papers to go public) is featuring a fresh lineup of works by Google DeepMind on studying the guts of transformers: softmax is not enough (for sharp out-of-distribution) by Petar Veličković et al arguing that softmax necessarily looses sharpness on longer OOD inputs Positional Attention: Out-of-Distribution Generalization and Expressivity for Neural Algorithmic Reasoning by Artur Back de Luca, George Giapitzakis, Shenghao Yang et al Round and Round We Go! What makes Rotary Positional Encodings useful? by Federico Barbero et al

Generative modeling with proteins (hundreds of them either): EquiJump: Protein Dynamics Simulation via SO(3)-Equivariant Stochastic Interpolants Design of Ligand-Binding Proteins with Atomic Flow Matching RapidDock: Unlocking Proteome-scale Molecular Docking Deep Learning for Protein-Ligand Docking: Are We There Yet? ProteinBench: A Holistic Evaluation of Protein Foundation Models Fast and Accurate Blind Flexible Docking Solving Inverse Problems in Protein Space Using Diffusion-Based Priors Crystals and Materials: Flow Matching for Accelerated Simulation of Atomic Transport in Materials MOFFlow: Flow Matching for Structure Prediction of Metal-Organic Frameworks Learning the Hamiltonian of Disordered Materials with Equivariant Graph Networks Designing Mechanical Meta-Materials by Learning Equivariant Flows SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models Rethinking the role of frames for SE(3)-invariant crystal structure modeling A Periodic Bayesian Flow for Material Generation ECD: A Machine Learning Benchmark for Predicting Enhanced-Precision Electronic Charge Density in Crystalline Inorganic Materials Wyckoff Transformer: Generation of Symmetric Crystals PDDFormer: Pairwise Distance Distribution Graph Transformer for Crystal Material Property Prediction

GraphML News (Oct 5th) - ICLR 2025 Graph and Geometric DL Submissions 📚 Brace yourselves, for your browser is about to endure 50+ new tabs. All accepted NeurIPS 2024 papers are now visible (titles and abstracts), and a new batch of goodies from ICLR’25 has just arrived. Tried to select the papers that haven't yet appeared during the ICML/NeurIPS cycles. PDFs will be available on the respective OpenReview pages shortly: Towards Graph Foundation Models: GraphProp: Training the Graph Foundation Models using Graph Properties GFSE: A Foundational Model For Graph Structural Encoding Towards Neural Scaling Laws for Foundation Models on Temporal Graphs Graph Generative Models: Quality Measures for Dynamic Graph Generative Models Improving Graph Generation with Flow Matching and Optimal Transport Equivariant Denoisers Cannot Copy Graphs: Align Your Graph Diffusion Models Topology-aware Graph Diffusion Model with Persistent Homology Hierarchical Equivariant Graph Generation Smooth Probabilistic Interpolation Benefits Generative Modeling for Discrete Graphs GNN Theory: Towards a Complete Logical Framework for GNN Expressiveness Rethinking the Expressiveness of GNNs: A Computational Model Perspective Learning Efficient Positional Encodings with Graph Neural Networks Equivariant GNNs: Improving Equivariant Networks with Probabilistic Symmetry Breaking Does equivariance matter at scale? Beyond Canonicalization: How Tensorial Messages Improve Equivariant Message Passing Spacetime E(n) Transformer: Equivariant Attention for Spatio-temporal Graphs Rethinking Efficient 3D Equivariant Graph Neural Networks Generative modeling with molecules (hundreds of them actually): AssembleFlow: Rigid Flow Matching with Inertial Frames for Molecular Assembly RoFt-Mol: Benchmarking Robust Fine-tuning with Molecular Graph Foundation Models Multi-Modal Foundation Models Induce Interpretable Molecular Graph Languages MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra Reaction Graph: Toward Modeling Chemical Reactions with 3D Molecular Structures Accelerating 3D Molecule Generation via Jointly Geometric Optimal Transport

GraphML News (September 28th) - AlphaChip, Generate + Novartis deal, MolPhenix NeurIPS results for both tracks have arrived - congrats to those who made it, the datasets track this year was particularly egregious with hard score cutting below average 6.3. Good luck with the final ICLR push and see you in Vancouver! 💻 Google DeepMind presented AlphaChip - the improved version of the famous 2021 Nature paper that introduced the RL agent that uses edge-level GNNs for chip placement - that is, placing dozens of smaller blocks (often implementing certain logical function) on a canvas to optimize common design metrics like HPWL or PPA. The addendum highlights that pre-training with large compute is rather crucial and reports that AlphaChip has been successfully used for several generations of TPUs (25 RL-designed blocks in the latest TPU) as well as for external customers like MediaTek. The paper got some controversial reputation in the chip design community and some professors even argued for retracting the work from Nature for lack of clarity and reproducibility. Over time, however, it seems more like a skill issue of those who tried to replicate it - generally, the level of ML expertise in the chip design community is pretty low (some accepted papers at top venues like DAC are just 🫣) and most university teams are stuck somewhere between MLPs and convnets. Professors gonna hate, Google gonna continue making impactful real-world products, and we will have new pre-trained checkpoints of AlphaChip with some Colab tutorials 🍿. 💸 Generate:Biomedicines (the authors of Chroma, a generative model for protein design) announced collaboration with Novartis resulting in $65M upfront payments and $1B in biobucks (royalties and other performance-based milestones typically split across many years). 🐦 Valence Labs announced MolPhenix, a CLIP-like model to study phenomics (how cells respond to perturbations). Practically, it is trained on pairs of microscopy images and molecules using ViT as image encoder and MolGPS for molecules. Experiments report massive 10x improvements in Top-1% recall of active molecules over previous SOTA 👏. Weekend reading: TabGraphs: A Benchmark and Strong Baselines for Learning on Graphs with Tabular Node Features by Gleb Bazhenov et al - a fresh collection of new graph datasets where features are interpretable (numerical, categorical) - a stark contrast to boring text-attributed graphs or Planetoid datasets with bag-of-words as features. Design of Ligand-Binding Proteins with Atomic Flow Matching by Junqi Liu et al feat. Jian Tang - generate a docked protein-ligand 3D structure conditioned just on 2D ligand graph and protein sequence with flow matching. Outperforms RFDiffusionAA on several metrics.

Discrete Neural Algorithmic Reasoning Guest post by Gleb Rodionov Paper: https://www.arxiv.org/abs/2402.11628 Blog: https://research.yandex.com/blog/discrete-neural-algorithmic-reasoning Code: https://github.com/yandex-research/dnar In this paper, we focus on generalizable and interpretable neural algorithmic reasoners. Starting with attention-based GNN, we inspect the reasons for generalization errors and propose several architectural modifications: feature discretization, hard attention and separating discrete and continuous data flows. All of these blocks are important for generalization: ⁃ State discretization prevents the model to use complex and redundant dependencies in the data; ⁃ Hard attention ensures that attention weights are not annealed for larger graphs. Also, hard attention limits the set of possible messages that each node can receive; ⁃ Separating discrete and continuous flows is needed to ensure that state discretization does not lose information about continuous data. As a result, we achieve a model that provably imitates the execution of several algorithms for any test data when trained with hints. Practically, on SALSA-CLRS, trained on problem sizes of 16 nodes, the model demonstrates perfect graph- and node-level scores generalizing to problems of up to 1600 nodes. For future work, it would be interesting to enhance the expressivity of the proposed model to a broader set of algorithms and investigate whether it is possible to train these models without hints.

GraphML News (September 21st) - AITHYRA, Fragrance 2o, LOG meetups 🧬 The Austrian Academy of Sciences together with Boehringer Ingelheim Foundation launched AITHYRA - the Institute for Biomedical AI - with a generous €150M funding over the next 12 years as a part of the Vienna BioCenter with Michael Bronstein as the first scientific director! AITHYRA plans to host 10-15 research groups supporting them with compute resources and robotic lab. Chances are AITHYRA might become the European version of the Institute for Protein Design (behold, David Baker) and the hub for Geometric Deep Learning research. Big win for Vienna 👏 👃Osmo, a generative fragrance startup founded by ex-Google researchers who worked on the Principal Odor Map, uncovered a bit more details on the Fragrance 2o platform - essentially, this is a molecule search / generation for potential fragrance molecules with further conditional generation capabilities. It would certainly be exciting to discover a personalized scent like “of a sweaty researcher submitting an ICLR paper while camping in Yosemite forests”. We will keep you up to date whether GNNs conquer the perfume world and beauty industry and when Fragrantica starts to list LLM prompts as ingredients. 🍻 One of the unique ideas of the Learning on Graphs conference are local meetups about graph learning research. To date, seven meetups spanning October-December have been announced: Tel Aviv, New Jersey, Aachen, Amsterdam, Paris, Kunshan, and Siena - feel free to attend or organize one at your place! Weekend reading: Accelerating Training with Neuron Interaction and Nowcasting Networks by Boris Knyazev et al and collab between Samsung and Mila - pretty amazing work where every k-th optimization step model weights are predicted by a graph transformer conditioned on the neural net architecture (supports convnets, GPT2, BERT, Llama, and ViTs), brings up to 50% speed ups in optimization. The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof by Derek Lim, Moe Putterman feat. Haggai Maron - another interesting work on neural parameter symmetries. Turns out that fixing weights in MLPs via freezing or non-linearities breaks parameter symmetries and enables better model merging (you can interpolate between pre-trained models to get even better performance). Can Graph Reordering Speed Up Graph Neural Network Training? An Experimental Study by Nikolai Merkel et al (VLDB 2025) - The answer is yes, avg speedup is 25%. The idea of partitioning the graph into several components to optimize memory reads is similar to the findings of Graph Segment Pre-training (by Google) and Sequential Aggregation and Rematerialization (Intel). Deep learning-based predictions of gene perturbation effects do not yet outperform simple linear methods by Constantin Ahlmann-Eltze et al 🫳🎤

GraphML News (September 17th) - Chai-1, GenMS 🍓 This week offered a significant portion of strawberries that might result in major improvements in scientific applications. For now, let’s try to check what’s there beyond the berries. 🧬 Chai Discovery emerged from stealth and released Chai-1 - a reproduction of AlphaFold 3 with trained weights (thanks to a month on 128 A100 which saved you roughly $500k), a tech report, open inference server, and inference code (interestingly, no model code). Initial experiments report numbers close to AF 3. Chai is backed by OpenAI and many famous VCs, so it might appear as a new strong player in the industry, we’ll keep an eye. 🔮 Google DeepMind announced GenMS: Generative Hierarchical Materials Search by Sherry Yang, Simon Batzner, and the team that brought us UniMat last year. GenMS employs three components: (1) Gemini 1.5 to sample candidate formulae after a natural language query, eg, “give me the formula for a stable, chalcogenide with atom ratio 1:1:2 that's not in the ICSD database”. Samples are filtered through some rule-based heuristics and re-reranked by an LLM; (2) best candidates are sent to a diffusion model (non-equivariant, attention-based 3D Unet) to generate 3D structures; (3) the structures are scored by a pre-trained ML potential (NequIP) - if they are stable and exhibit target characteristics, we add them as a tree branch for the new iteration by LLMs. GenMS excels at perovskites, pyrochlore, and spinel crystals with structures confirmed by DFT formation energy calculations. Almost no geometric DL whatsoever 🙀 Weekend reading: Recurrent Aggregators in Neural Algorithmic Reasoning by Kaijia Xu and Petar Veličković - the first model capable to solve quickselect from the CLRS benchmark happened to be a Triplet MPNN with a non permutation-invariant LSTM aggregator (GraphSAGE vibes). Back in January in our annual review post quickselect was the most unlikely candidate for traction, and looks like it is almost solved now! On the design space between molecular mechanics and machine learning force fields by Yuanqing Wang and a huge collab of physicists and chemists led by NYU (feat. Kyunghyung Cho) - a nice intro to molecular mechanics, force fields, and potentials approachable by folks without a degree in physics. The survey includes a discussion on foundational ML potential models and “a nihilstic epilogue” worth checking out.

GraphML News (September 7th) - AF 3 reproductions, AlphaProteo, ORB, Entalpic round Just the first week of September, but already so much news in the protein design and materials science! 🧬 Two AlphaFold 3 reproductions are now available: HelixFold 3 from Baidu (tech report) and AF3 from Ligo Bioscience (no tech report yet). Training HelixFold 3 on PDB and custom data yields results roughly similar to the OG AlphaFold 3 on PoseBusters and CASP 15 - good news for science and reproducibility (and for Nature editors, hehe). Getting more data will be the key to the full reproduction - probably no other lab has as large and diverse dataset as DM and Iso. Meanwhile, Google DeepMind announced AlphaProteo - a generative model for binders conditioned on the target protein and possible binding sites. The preprint has no information about the generative model itself (an educated guess would be either autoregressive transformer or discrete diffusion as a backbone) but the training dataset is similar to that of the full AlphaFold 3. Experimentally, AlphaProteo generates plausible binders in several use-cases like Epstein-Barr virus protein, COVID-19 spike protein, and proteins involved in cancer. 🔮 In the computational materials science, Orbital Materials announced ORB - a family of forcefield models to compute energy, forces, and stresses of atomistic systems (like bulk materials or semiconductors). ORB trained on Alexandria and Materials Project trajectories with the denoising objective (improved Noisy Nodes) yields SOTA on MatBench Discovery outperforming big boys MatterSim from MSR and GNoME from DeepMind. The authors highlight that ORB are non-equivariant GNNs - in fact, the backbone is very similar to the Graph Network Simulator from 2020 with an optional attention interaction. It will be fun to watch equivariant vs non-equivariant folks beating each others SOTA in the next few months 🍿 💸Entalpic, a French materials discovery startup with founders graduated from Mila, announced €8.5m seed round co-lead by Breega, Cathay Innovation and Felicis - congrats to Mathieu, Victor, and Alexandre! Entalpic joins CuspAI and Orbital Materials in the emerging market of DL-based materials discovery companies - we’ll be keeping an eye on their advances. Weekend reading: Two papers from Shuiwang Ji’s lab on SE(3)-invariant 1D tokenization of 3D molecules for autoregressive generation: Geometry Informed Tokenization of Molecules for Language Model Generation - for small molecules on QM9 and Geom-Drugs. Fragment and Geometry Aware Tokenization of Molecules for Structure-Based Drug Design Using Language Models - for generating ligands for protein pockets. Talking about autoregressive molecule generation, Any-Property-Conditional Molecule Generation with Self-Criticism using Spanning Trees is another strong baseline improving spanning tree-based graph generation.

GraphML News (August 31st) - When GNNs help, randomized transformers, and new papers August is a dry month in terms of news, but soon we’ll start to see upcoming ICLR submissions! 🔨 Meanwhile, have a look at the Measuring and Exploiting Network Usable Information blogpost by Meng-Chieh Lee (based off the spotlight ICLR 2024 paper) that touches upon the question asked every day in industrial labs - will GNNs outperform MLPs on my data? Are there any hints or data characteristics (well, apart from the homophily ratio) that could indicate which model would be better without training one? The authors introduce the notion of Network Usable Information (NUI) as a function of structural embeddings, node features, and neighbors’ features and find some correlations between the new score and performance on node classification and link prediction. We submitted a position paper to ICML’24 studying a similar question but it didn’t get through because reviewers demanded more experiments (in the positions track, yeah). 🎰Learning Randomized Algorithms with Transformers by Google and ETH Zurich - a intriguing blend of theoretical CS, math, and randomized algorithms with expressiveness of transformers. Experiments shows that randomized transformers can solve graph coloring problems on small sizes and explore grid worlds. More weekend reading: 💊Graph Artificial Intelligence in Medicine by Ruth Johnson, Michelle Li, feat Marinka Zitnik - a massive survey on GNNs in clinical applications. Do Graph Neural Networks Work for High Entropy Alloys? by Zhang et al - the answer is yes, but with proper modeling. High-entropy alloys are unordered at the atomic scale but can be represented as sets of graphs (each graph is a local env for an alloy). Practically, adding a set pooling function like DeepSet(GNN(set of graphs)) is what we are looking for. Expressive Power of Temporal Message Passing by Przemysław Wałega and Michael Rawson - Weisfeiler and Leman Go Temporal! Another fun fact about temporal GNNs: two models named DyG-Mamba (one, two, both add Mamba on top of GNN encoders) were submitted on arxiv with a few days gap.

GraphML News (August 24th) - Psiformer, ML potentials arena, Single-cell foundation models ⚛️ DeepMind announced the updated version of Psiformer (together with the paper in Science, twitter thread, and source code in Jax) - a transformer for quantum physics tasks. The new model can approximate excited states of molecules on par or better than existing gold standard models. Excited energy states are responsible for lasers, semiconductors, solar panels, fluorescence, and many other phenomena - a huge potential for Psiformer in industrial applications. 🏆 Continuing with energy states - you probably know that the ultimate LLM benchmark those days is the ELO rating on the Chatbot Arena. Yuan Chiang started a similar effort for ML potential models (MLIP Arena) featuring 3 tasks: two atoms of the same type (the only LB for now) and two molecular dynamics tasks (loading time is slow). The supported models for now are Equiformer V2, CHGNet, MACE MP, M3GNet, SevenNet, and the GPAW DFT baseline from the DFT world. 🎻 Single-cell foundation models are getting more attention. The new scCello by Mila is a transformer trained on the masked LM task together with the alignment loss using the Cell Ontology. scCello in the zero-shot inference regime outperforms end-to-end trained models on tasks like cell type classification, marker gene prediction, and batch integration. If you are interested to learn more, have a look at the fresh survey on transformers in SC omics. Weekend reading: more foundation models and materials science: A foundation model for clinician-centered drug repurposing by Kexin Huang et al feat. Jure Leskovec and Marinka Zitnik - introduces TxGNN, a graph foundation model for drug repurposing trained on a medical KG of 17k diseases and 8k drugs, strong zero-shot performance included. The model and example weights are already on Github. Microsoft published the source code of Aurora - FM for atmospheric forecasting, consists of Perceiver encoder/decoder and SwinTransformer as the backbone. Crystalline Material Discovery in the Era of Artificial Intelligence by Zhenzhong Wang et al (thanks to Wanyu Lin for highlighting the work) - a survey on predictive and generative models for crystals, with the github repo of relevant papers From Text to Insight: Large Language Models for Materials Science Data Extraction + tutorial online book by Mara Schilling-Wilhelmi, Martiño Ríos-García et al. LLMs are surprisingly strong in generating 3D structures of solid-state materials (ICLR 2024) on par with fancy equivariant diffusion models, this survey studies how much MatSci data LLMs could possibly feed.

GraphML News (August 17th) - Spanner Graph, some new papers 🔧 Google announced Spanner Graph - the infinitely scalable graph database (as the vanilla Spanner) with all the bells and whistles GDBMS have in 2024: support both Graph Query Language (GQL, finally standardized by ISO in April after 8 years of work) and SQL, vector search and full-text search, basic graph algorithms at query time. Otherwise, it’s mid-August and vacation time, so probably no major news for the next few weeks. Weekend reading: Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability from the large DeepMind team - turns out reducing hallucinations when training LLMs on KGs (ie, recalling training triples) requires an order of magnitude more compute than Chinchilla scaling laws. Lots of qualitative results - have a look! Besides, it is one of the accepted papers at COLM - a new conference specifically tailored for LLM research (rip, ACL/EMNLP). Topological Blind Spots: Understanding and Extending Topological Deep Learning Through the Lens of Expressivity by Yam Eitan et al. feat Haggai Maron - one of the first studies of expressive power of topological (higher-order) MPNNs. Turns out standard models based on simplicial complexes or cellular networks cannot distinguish many common topological patterns like a Möbius strip vs cylinder. The authors then derive provably more powerful scalable multi-cell networks. Tokenized and Continuous Embedding Compressions of Protein Sequence and Structure by Amy X. Lu et al feat. Pieter Abbeel and Kyunghyun Cho - a deep dive into the latent space of ESMFold which happens to be quite sparse, it can reduced by 128x without losing in prediction performance.