Graph Machine Learning

GraphML News (August 10th) - Summer School recordings, DD merger 🖥️Recordings from the ML for Drug Discovery Summer School are now available covering 5 days of talks with 28 videos - from basics of GNNs for chemistry and equivariance to protein folding, ML potentials, simulations, protein-protein (-ligand) binding, to generative modeling and causal discovery. 🖥️ The Eastern European ML Summer School’24 also published their recordings - 25 videos covering a more general area of deep learning including LLMs, reasoning, VLMs, RL, generative models, Bayesian DL, and many more. Notebooks from the practical sessions are available on GitHub. Both schools feature the most up-to-date material from the top experts in the field, quite the gems to watch during the summer break 💎. ⚛️ Continuing with the quality content, Sophia Tang published a massive, 2.5h-read guide to spherical equivariant graph transformers deriving them from the first principles and spherical harmonics to TensorField nets to the SE(3)-Transformer. Lots of illustrations with the code going along. The best tutorial so far. 💸 News from the Geometric Wall Street Journal: a huge merger between Recursion and Exscientia (focusing on precision oncology) - actually, Recursion bought Exscientia for $688M in stocks continuing its acquisition spree (besides the BioHive-2 with 500 H100’s). (Not a stonks advice) Weekend reading: The Heterophilic Graph Learning Handbook: Benchmarks, Models, Theoretical Analysis, Applications and Challenges by Sitao Luan feat. Rex Ying and Stefanie Jegelka - everything you wanted to know about heterophilic graphs in 2024 When Heterophily Meets Heterogeneity: New Graph Benchmarks and Effective Methods by Junhong Lin et al - introduces H2DB, a collection of known and new heterophilic and heterogeneous graphs, much larger than existing datasets.

GraphML News (August 3rd) - NeurIPS workshops, MoML @ MIT, RUM and GraM ⛷️ NeurIPS’24 announced 56 accepted workshops (brace yourself, Vancouver convention center). In addition to a good bunch of LLM, VLM, and foundation model-focused events, graph and geometric learning folks might be interested in: - AI for New Drug Modalities - Machine Learning in Structural Biology - Symmetry and Geometry in Neural Representations - Multimodal Algorithmic Reasoning - Machine Learning and the Physical Sciences - AI for Accelerated Materials Design 🧬 The second part of MoML 2024 (Molecular ML) will be happening at MIT on November 5, you can submit short papers until October 10th. The authors of accepted papers get free admission! 💎 The GraM workshop of ICML’24 published accepted blogposts with some hidden gems like JAX implementation of EGNN, intro to equivariant neural fields, and the study of how consistency models don’t work for 3D molecule generation. Check out others as well - most of them require only entry-level background. 📈Non-convolutional Graph Neural Networks by Yuanqing Wang and Kyunghyun Cho (the OG of GRUs) introduce RUM (random walk with unified memory) nets free of convolutions. Practically, the recipe of RUM included sampling random walks with anonymous node ID sequences (tracking the first occurrence of a node ID in the sequence), encodes both sequences via RNNs (sure, you can drop-in your fav Mamba here), concats both vectors with an MLP on top. The authors show RUMs are more expressive than 1-WL GNNs while not suffering from oversmoothing and oversquashing (and beating the baselines on a bunch of benchmarks). Interestingly, RUMs look like DeepWalk on steroids with several improvements. Is Bryan Perozzi the Noam Shazeer of graph learning? 🤔 More weekend reading: Spatio-Spectral Graph Neural Networks by Simon Geisler et al feat. Stephan Günnemann - spectral GNNs can be strong performers, too - just to contrast with RUMs Learning production functions for supply chains with graph neural networks by Serina Chang et al feat Jure Leskovec - a cool work that frames supply chains as temporal graphs, shows significant gains in prediction accuracy, and releases the data simulator What Are Good Positional Encodings for Directed Graphs? by Yinan Huang, Haoyu Wang, and Pan Li. The answer is the Magnetic Laplacian with multiple potential factors (multi-q) - your best choice for DAGs.

​​Seminar on Graph-based Causal Discovery in Computational Biology 🎓 Topic: "Causal discovery from multivariate information in biological and biomedical data" 👨‍🔬 Who: Hervé Isambert, The Isambert Lab, CNRS, Institut Curie, Paris ⌚ When: Monday, July 29th, 5pm CEST Abstract: In this webinar, I will present the principles and limitations of graph-based causal discovery methods and their improvement using multivariate information decomposition, recently developed in my lab. Applications will range from gene expression data in single cells to nationwide medical databases of cancer patients. I will then discuss the theoretical link between graph-based causality and temporal (Granger-Schreiber) causality, which can both be expressed in terms of conditional multivariate information. While temporal causality is shown to imply graph-based causality, the converse may not be true (see Figure). An application to time series data concerns the analysis of video images of reconstituted tumor ecosystems, which uncovered a novel antagonistic effect of cell-cell interactions under therapeutically relevant conditions. The Zoom link will appear in this channel shortly before 5pm

GraphML News (July 27th) - LLMs in Chemistry, Discrete Flow Matching ICML kept most of the community busy (Saturday is the last day of workshops) while in the other news Llama 3.1, SearchGPT, AlphaProof, and AlphaGeometry 2 took the headlines of approaching AGI singularity. Anyways, August would likely be a quieter month. Some fresh works for the weekend reading: A Review of Large Language Models and Autonomous Agents in Chemistry by Mayk Ramos, Christopher Collison, and Andrew White - a massive survey on what current gen of LLMs can do in chemistry - from property prediction and synthesis prediction to tool-augmented and multi-modal frontier models for orchestrating automated discovery labs. (paying respects to the LLM week) Discrete Flow Matching by Itai Gat and Meta FAIR including Ricky Chen and Yaron Lipman - the OG authors of (Riemannian) Flow Matching. Discrete FM is now competitive to Llama 2/3 on coding tasks - so we should expect that module to be in all generative models for molecules, proteins, and crystals around ICLR’25 submissions and later. Generative Modeling of Molecular Dynamics Trajectories by Bowen Jing and Hannes Stärk - MD via stochastic interpolants, supports accurate forward simulation, upsampling, interpolation between two states in the trajectory, and even inpainting of the simulated structure.

GraphML News (July 20th) - Pinder and Plinder, LAB bench, ICML 2024 🎙️ ICML 2024 starts next week - enjoy the conference and Vienna if you are participating this year! Beside the main program, Monday will feature the Graph learning tutorial, Thursday and Friday have a handful of graph-related workshops. 🧬 VantAI together with MIT, NVIDIA, UniBasel, and SIB introduce two novel large-scale benchmarks: Pinder (Protein INteraction Dataset and Evaluation Resource) and Plinder (Protein-Ligand Interaction Dataset and Evaluation Resource). Pinder includes 500x more data than PPIRef, and Plinder is roughly 10x larger than DockGen, previous largest datasets in the area susceptible to test set leakages. Re-training SOTA diffusion models on Pinder and Plinder shows much lower results indicating that saturation is far away (at least for the coming year). Besides, it is great to see the industrial company (from a highly competitive CompBio area) contributing to the field with open datasets. Pinder and Plinder will be the main datasets for the upcoming ML for Structural Bio challenge at NeurIPS 2024, so prepare your GPUs and diffusion models. 🔬 FutureHouse released the LAB bench for studying LLMs in Biology and Chemistry. The benchmark includes 8 categories where LLMs have to deal with figures, images, scientific literature, databases, and designing protocols. Recent LLMs and VLMs (GPT-4o, Claude, and LLama-3) all show rather underwhelming results on those tasks - it is finally a new unsaturated benchmark for the LLM crowd! The authors saved some data to check training contamination of future models (eg, when training data for the next gen of such models would include validation and test splits of the datasets). Weekend reading: Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures by Sophia Sanborn, Johan Mathe, Mathilde Papillon, et al - a massive survey with amazing illustrations PINDER: The protein interaction dataset and evaluation resource by Daniel Kovtun, Mehmet Akdel, and VantAI folks feat. Michael Bronstein PLINDER: The protein-ligand interactions dataset and evaluation resource by Janani Durairaj, Yusuf Adeshina, and VantAI folks LAB-Bench: Measuring Capabilities of Language Models for Biology Research by Jon M. Laurent, Joseph D. Janizek, et al feat. Andrew White

GraphML News (July 13th) - Recursion goes brrr, Acquisition of Graphcore, Illustrated AF3 💸 Recursion and NVIDIA launched BioHive-2, a GPU cluster made of 504 H100’s which is roughly equivalent to 1 petaflops in FP16 / BF16 and perhaps sub-$50M in the costs. Some napkin math indicates it could train and fine-tune a full AlphaFold 3-like model in about 4 days. Except for ESM-3, we haven’t yet seen drug discovery models trained on such compute - congrats to Recursion, Valence, and researchers with engineers who can now really go brrr. 💸 Graphcore, a UK hardware startup offering their hardware platform (BOW IPUs), was acquired by SoftBank for rumored $500M (back in 2020 valuation was about $2.8B). Former employees likely lost their vested options ($500M is still less than $600M originally invested into the company) but let’s hope that now the future would be more stable for Graphcore and we will see more successful products. 🧬The Illustrated AlphaFold by Elana Simon and Jake Silberg from Stanford (inspired by the Illustrated Transformer) explains visually the main building blocks of the model - starting from the input data down to PairFormer, triangular attention to the diffusion module to the training losses. Things get much simpler indeed when you know which shapes are involved at each particular step. Weekend reading: Link Prediction with Untrained Message Passing Layers by Lisi Qarkaxhija, Anatol E. Wegner, and Ingo Scholtes - the unreasonable effectiveness of untrained MPNNs strikes back SE(3)-Hyena Operator for Scalable Equivariant Learning by Artem Moskalev et al - FFT with Clifford MLPs enable equivariant Hyena on long sequences up to 3.5M tokens on a single GPU On the Expressive Power of Sparse Geometric MPNNs by Yonatan Sverdlov, Nadav Dym - enabling equivariant GNNs on sparse graphs (usually EGNNs work on fully-connected graphs)

This year's ICLM will finally have a tutorial on graphs! Adrian Arnaiz-Rodriguez and Ameya Velingker will present a tutorial on on Graph Learning: Principles, Challenges, and Open Directions. 🗓️ Date: Monday, July 22 🕒 Time: 15:30 CEST - 17:30 CEST 📍 ICML In-person Event: Hall A8, ICML Venue 📍 Virtual attendance: https://icml.cc/virtual/2024/tutorial/35233 What to expect? - Intro to Graph Learning and GNNs: Introduction to Traditional graph representation, Graph Neural Networks (GNNs), Message Passing Networks (MPNNs), Graph Transformers (GTs) and spectral quantities. - Expressiveness and Generalizability: GNN expressivity linked with the WL test, generalizability of MPNNs, and their performance implications. - Challenges in GNNs: Understanding and addressing under-reaching, over-smoothing, over-squashing, and graph rewiring techniques. - Panel Discussion on Future Directions: Panel discussion with Michael Bronstein, Bryan Perozzi, Christopher Morris and more panelist TBC. We will discuss about GNN limitations, graph foundation models, and integrating GNNs with large language models (LLMs). This tutorial balances introductory content and advanced insights, aimed to both general audiences and experts. Don’t miss this opportunity to deepen your understanding of GNNs!

GraphML News (July 7th) - ICML Workshops, AI4Science Lectures, GraphRAG release 📚 ICML workshops started publishing their accepted papers on OpenReview. Remember that workshop papers send a good signal of future full papers at next big conferences so you might find something interesting! Among others, check out: - GRaM workshop (Geometry-grounded Representation Learning and Generative Modeling), - AI4Science, - TF2M (Theoretical Foundations of Foundation Models) - SPIGM (Structured Probabilistic Inference & Generative Modeling) 📺 Simons Institute at Berkeley recently organized a workshop AI≡Science: Strengthening the Bond Between the Sciences and Artificial Intelligence with a stellar lineup including Tess Smidt, Mohammed AlQuraishi, Rafael Gomez-Bombarelli, and many others. All lectures recordings are now available. 🚒 Microsoft Research released GraphRAG, their take on graph-enriched RAG, on GitHub along with the accompanying blogpost. The repo received 6k stars just in 5 days 📈. Weekend reading: Foundations and Frontiers of Graph Learning Theory by Yu Huang et al. feat Muhan Zhang - a survey on the GNN theory, can accompany the recent ICML position paper by Morris et al. Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization by Siyi Gu, Minkai Xu et al feat. Jure Leskovec - perhaps the first diffusion model for ligand generation (conditioned on the pocket) with the DPO alignment (RLHF without H).

GraphML News (June 29th) - ESM 3, TDC 2, AI 4 Genomics Conference 🧬 Evolutionary Scale (formerly a team in Meta, now a standalone startup) released ESM 3 - the next version of the SOTA protein LM, pretty much GPT-4 of pLMs. Now it’s not only a sequence model, but also a structure and function model. Following best LLM practices, ESM 3 even employs RLHF for aligning with human feedback! Besides, the model features SE(3)-invariant geometric attention based on distances between frames (equivariance not dead!) and VQ-VAE to tokenize structures and functions. The ESM 3 family is available in three sizes: 1.4B is open weights, 8B and 98B are available in the API (it’s time to embrace that). The preprint is quite informative about training data, pre-/post-training details, and RLHF details - kudos for not sweeping it under the rug. The model code is also available, so you only need 10,000 H100’s to train it on your own 🙂 💊 The team of Harvard, MIT, and Stanford researchers led by Marinka Zitnik released Therapeutic Data Commons 2 adding even more datasets and modalities: over 1000 single-cell datasets over 85M cells, the first protein-peptide binding dataset, drug-target interaction data, clinical trials data, and much more covering 10+ modalities. TDC-2 packages several pre-trained embeddings and can be used for evaluating a variety of models - from LLMs to GNNs. TDC-1 received somecritic from drug discovery people back in the days, let’s see if TDC-2 closes those gaps. The AI for Genomics and Health conference will be held in Boston, Oct 17-18th, with a stellar lineup of speakers including Shekoofeh Azizi (DeepMind, the author of Med-PaLM), Mo Lotfollahi (Sanger Institute), Sergey Ovchinnikov (MIT), Marinka Zitnik (Harvard), and James Zou (Stanford). 🔮 A small update: MatterSim by MSR AI4 Science became SOTA on MatBench Discovery beating recent GNoME from DeepMind - competition makes wonders even in such advanced scientific topic as materials discovery, ML potentials, and molecular dynamics. Weekend reading: Multimodal Graph Benchmark by Zhu et al feat. Danai Koutra - three datasets combining graphs, texts, and images for node classification and link prediction tasks. Transformers meet Neural Algorithmic Reasoners by Bounsi et al feat. Petar Veličković - Transformer cross-attenting to the pre-trained Triplet-GMPNN solves algorithmic reasoning tasks (CLRS-text) better than the vanilla Transformer (but still struggles with OOD generalization though) Clifford-Steerable Convolutional Neural Networks by Maksim Zhdanov et al - ConvNets go spacetime — equivariant to the Lorentz group and useful for electrodynamics. The thread by Maurice Weiler explains the work much more in details. Someday (after another PhD in math and physics) I will be able to understand the math behind this paper

GraphML News (June 22nd) - $30M seed for CuspAI, Graph Foundation Models, MoML 2024 💸 A new startup CuspAI by Max Welling and Chad Edwards focusing on materials discovery and materials design for clean energy and sustainability raised $30M in the seed round (led by Hoxton, Basis Set, and Lightspeed). The support from the godfathers is significant - Geoff Hinton is a board advisor and Yann LeCun commented on the collaboration with FAIR and OpenCatalyst teams on OpenDAC. The materials design area gets hotter - not as hot as drug discovery and protein design though - but is steadily growing. In addition to Radical AI, Orbital Materials, new CuspAI, a fresh Entalpic by ex-Mila founders raised $5M+. 🔖 Together with Michael Bronstein, we released a new blog post on Graph Foundation Models. First, we define what GFMs are and what are the key design challenges covering heterogeneous model expressivity, scaling laws, and data scarcity. Then, we describe several successful examples of recent generalist models that can be considered GFMs in a particular area, eg, GraphAny for node classification, ULTRA for KG reasoning, and MACE MP-0 as universal potentials. We made sure to include all the recent references including position papers to appear at ICML’24! 🧬 The Molecular ML 2024 conference took place in Montreal this week (concluding the ML for Drug Discovery summer school) and featured talks on drug discovery and drug design. The recording is already available - check out talks by Jian Tang (BioGeometry) on geometric DL for proteins and by Max Jaderberg (Chief AI Officer at Isomorphic Labs) on AlphaFold 3. Might be one of the first public talks on AF3! Weekend reading: More benchmarks (brought to you by the NeurIPS Datasets & Benchmarking track deadline). Temporal Graph Benchmark 2.0 by Gastinger, Huang et al - the first large-scale benchmark for temporal KGs and heterogeneous graphs Text-space Graph Foundation Models by Chen et al feat. Anton Tsitsulin and Bryan Perozzi - a collection of text-attributed graphs for node classification, link prediction, and graph-level tasks Towards Neural Scaling Laws for Foundation Models on Temporal Graphs by Shirzadkhani, Ngo, Shamsi et al - perhaps the first evidence that one temporal GNN can generalize to different temporal graphs (here those are token transactions in Ethereum) RNA-FrameFlow: Flow Matching for de novo 3D RNA Backbone Design by Rishabh Anand, our own Chaitanya K. Joshi, et al - equivariant flow matching for generating 3D RNA structures.

GraphML News (June 15th) - ICML’24 graph papers, musings on AF3, more Flow Matching 🎉 ICML 2024 papers (including orals and spotlights) are now visible on OpenReview (however, without search). If you don’t want to scroll through 100 pages of accepted papers manually or write a custom parser, Azmine Toushik Wasi compiled a collection of accepted Graph ML papers with a nice categorization. 👨‍🔬 More blogs on AlphaFold 3 and reflexions about the future of TechBio: Charlie Harris focuses more on the technical side whereas Carlos Outeiral presents the CompBio perspective highlighting some cases where AF3 still underperforms. 🔀 Flow Matching continues to reach new heights with recently released papers: Variational Flow Matching (you didn’t forget ELBO and KL divergence, right?) by the UvA team of Floor Eijkelboom, Grigory Bartosh, et al (feat. Max Welling) derives a generalized flow matching formulation that naturally allows for categorical data (😼 CatFlow) and graph generation - the model outperform DiGress and other diffusion baselines. At the same time, the NYU team of Boffi et al propose Flow Map Matching - pretty much the Consistency Models for FMs that enable generation in one step instead of 20-100. Finally, Ross Irwin et al from AstraZeneca come up with MolFlow - flow matching for generating 3D conformations of molecules showing compelling results on QM9 and Geom-Drugs. 📚Weekend reading (no flow matching): GraphStorm: all-in-one graph machine learning framework for industry applications by Da Zheng and AWS - we wrote about a new GNN framework for enterprises back in 2023, here is the full paper with details. CRAG -- Comprehensive RAG Benchmark from Meta (and a Kaggle competition for $30k) - the factual QA benchmark that simulates queries to knowledge graphs and APIs. Vanilla RAG yields only 44% accuracy and fancy industrial models barely reach 63% - so a plenty of room for improvements. Explainable Graph Neural Networks Under Fire - by Zhong Li feat Stephan Günnemann. Turns out most GNN explainers utterly fail and cannot be trusted in the presence of simple adversarial perturbations. Let us know if you ever found a successful working case for GNN explainers 🤭

GraphML News (June 8th) - LOG’24, FoldFlow 2, more new papers 🎙️The biggest announcement of the week is that the virtual LOG’24 actually happens before going physical at UCLA in 2025. The dates are Nov 26-29th 2024, and submission deadline is September 11th. LOG is known for a much higher review quality - a considerable part of the whole budget is dedicated to monetary rewards for reviewers (one of the few events that ever appreciate good reviews). 🧬 The Dreamfold team announced FoldFlow 2 - an improved version of the protein structure generative model that made Riemannian flow matching a mainstream topic. FoldFlow 2 adds an ESM2 encoder for protein sequences and is trained on a much bigger dataset (featuring filtered synthetic structures from SwissProt and AlphaFold 2 DB). Experimentally, FoldFlow 2 substantially improves over previous SOTA big guys, RFDiffusion and Chroma, on unconditional and conditional (motif scaffolding) generation tasks. Besides, it’s never too late to remind that Federico Errica is hiring interns and visiting researchers at NEC Labs in Heidelberg. 📚 The weeks after the NeurIPS deadline continue to bring cool submissions and accepted ICML papers! - Topological GNNs went equivariant all the way: Topological Neural Networks go Persistent, Equivariant, and Continuous (ICML’24) by Yogesh Verma et al E(n) Equivariant Topological Neural Networks by Claudio Battirolo et al E(n) Equivariant Message Passing Cellular Networks by Veliko Kovač et al feat Erik Bekkers - Theory on graph transformers and spectral GNNs (all will be at ICML’24) What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding by Hongkang Li et al Aligning Transformers with Weisfeiler–Leman by Luis Müller and Chris Morris On the Expressive Power of Spectral Invariant Graph Neural Networks by Bohang Zhang et al feat. Haggai Maron - Transformers through the graph lens (both featuring Petar Veličković) Transformers need glasses! Information over-squashing in language tasks by Federico Barbero et al - the old friend over-squashing is confirmed to be present in transformers The CLRS-Text Algorithmic Reasoning Language Benchmark by Markeeva, McLeish, Ibarz et al - the text version of CLRS for all you LLM folks, a fresh unsaturated benchmark - Combinatorial optimization with GNNs Towards a General GNN Framework for Combinatorial Optimization by Frederik Wenkel, Semih Cantürk, et al A Diffusion Model Framework for Unsupervised Neural Combinatorial Optimization by Sebastian Sanokowski et al