Graph Machine Learning

Towards Geometric Deep Learning IV: Chemical Precursors of GNNs In the final post of the series, Michael Bronstein covers the role of chemistry and computational chemistry in developing mathematical concept that were further used in creating GNNs. For instance, the problem of patent offices when registering a new drug required a way to compare a new molecule with those in the existing database - starting from strings continuing with molecular fingerprints and finally arriving to the WL-test and its modern variants.

Graph Machine Learning @ ICML 2022 In case you missed all the ICML’22 fun, we prepared a comprehensive overview of graph papers published at the conference: 35+ papers in 10 categories: - Generation: Denoising Diffusion Is All You Need - Graph Transformers - Theory and Expressive GNNs - Spectral GNNs - Explainable GNNs - Graph Augmentation: Beyond Edge Dropout - Algorithmic Reasoning and Graph Algorithms - Knowledge Graph Reasoning - Computational Biology: Molecular Linking, Protein Binding, Property Prediction - Cool Graph Applications

Upcoming Graph Workshops If you are finishing a project and would like to probe your work and get the first round of reviews, consider submitting to recently announced workshops: - Federated Learning with Graph Data (FedGraph) @ CIKM 2022 - deadline August 15 - Trustworthy Learning on Graphs (TrustLOG) @ CIKM 2022 - deadline September 2 - New Frontiers in Graph Learning (GLFrontiers) @ NeurIPS 2022 - deadline September 15 - Symmetry and Geometry in Neural Representations (NeurReps) @ NeurIPS 2022 - deadline September 22

​​ESMFold: Protein Language Models Solve Folding, Too Today, Meta AI Protein Team announced ESMFold - a protein folding model that uses representations right from a protein LM. Meta AI has been working on BERT-style protein language models for a while, e.g., they created a family of ESM models that are currently SOTA in masked protein sequence prediction tasks. “A key difference between ESMFold and AlphaFold2 is the use of language model representations to remove the need for explicit homologous sequences (in the form of an MSA) as input.” To this end, the authors design a new family of protein LMs ESM-2. ESM-2 are much more parameter efficient compared to ESM-1b, e.g., 150M ESM-2 is on par with 650M ESM-1b, and 15B ESM-2 leaves all ESM-1 models far behind. Having pre-trained an LM, ESMFold applies Folding Trunk blocks (simplified EvoFormer blocks from AlphaFold 2) and yields 3D predictions. ESMFold outperforms AlphaFold and RoseTTAFold when only given a single-sequence input w/o MSAs and also much faster! Check out the attached illustration with architecture and charts. “On a single NVIDIA V100 GPU, ESMFold makes a prediction on a protein with 384 residues in 14.2 seconds, 6X faster than a single AlphaFold2 model. On shorter sequences we see a ~60X improvement. … ESMFold can be run reasonably quickly on CPU, and an Apple M1 Macbook Pro makes the same prediction in just over 5 minutes.” Finally, ESMFold shows remarkable scaling properties: “We see non-linear improvements in protein structure predictions as a function of model scale, and observe a strong link between how well the language model understands a sequence (as measured by perplexity) and the structure prediction that emerges.” Are you already converted to the church of Scale Is All You Need - AGI Is Coming? 😉

Origins of Geometric Deep Learning - Part 2 and 3 A while ago we referenced the first article of the series on the Origins of Geometric DL by Michael Bronstein. Recently, the series got new episodes - Part 2 focuses on the high hopes about the perceptron, the curse of dimensionality, and first AI winters. Part 3 introduces first architectures with baked geometrical priors - the neocognitron (precursor of convnets) and convolutional neural networks. As always, Michael did a great and meticulous job of finding original references and adding some comments to them - often the references section is as interesting and informative as the main text! 🍿

ICML 2022 - Graph Workshops ICML starts today with the full week of tutorials, main talks, and workshops. While we are preparing a blog post about interesting graph papers, you can already check the contents of graph- and related workshops to be held on Friday and Saturday. - Topology, Algebra, and Geometry in Machine Learning (TAG in ML) - Knowledge Retrieval and Language Models (KRLM) - Beyond Bayes: Paths Towards Universal Reasoning Systems - Machine Learning in Computational Design

TensorFlow GNN TensorFlow GNN (TF-GNN) is a new scalable library for Graph Neural Networks in TensorFlow. It is designed from the bottom up to support the kinds of rich heterogeneous graph data that occurs in real-life use-cases. Many production models at Google use TF-GNN and it has been recently released as an open source project. Google has released a paper that describe the TF-GNN data model, its Keras modeling API, and relevant capabilities such as graph sampling, distributed training, and accelerator support. A new version was just pushed to GitHub.

Graph ML Workshops and Summer Schools🇬🇧🇨🇦🇨🇭🇮🇹 This week is surprisingly well-packed with physical meetings of the GraphML community with top speakers and lecturers. We would expect all the materials to be recorded and available online. - London Geometry and ML Summer School (🇬🇧) - Deep Exploration of non-Euclidean Data with Geometric and Topological Representation Learning (🇨🇦) - Swiss Equivariant Machine Learning Workshop (🇨🇭) Also, in 2 weeks there is going to be Italian Summer School on Geometric DL (🇮🇹).

🔥New Course:An Introduction to Group Equivariant Deep Learning Erik Bekkers from University of Amsterdam created a fantastic new course covering the most up-to-date flavor of GNNs, namely, equivariant and group-equivariant GNNs. The course consists of 3 lectures, starts from the introduction to the group theory, gradually comes to equivariance and steerable kernels, covers tensor products and irreducible representations (hello Wigner matrices). After the course, you won’t be afraid of cryptic abbreviation like SO(3) or E(n)! The course includes a YouTube playlist, slides, lecture notes, and Colab notebooks to play around with the real code. If you got inspired by this topic, we highly recommend the upcoming course by Joey Bose (Mila and McGill) on Geometry and Generative Models with even deeper study of manifolds (hyperbolic, spherical, product) to normalizing flows, ODEs, and denoising diffusion models.

​​Recap: Fields Medal & Graph Theory, Origins of Geometric Deep Learning 1. Fields Medal is often considered “the Nobel Prize in mathematics”. This year, International Mathematical Union (IMU) announced 4 awardees: brilliant mathematicians Hugo Duminil-Copin (Université de Genève and IHÉS), June Huh (Princeton), James Maynard (Oxford), and Maryna Viazovska (EPFL). It is heartwarming for channel’s editors that the research of June Huh has direct connections to the graph theory - first, he proved the 40-years-unsolved Read’s conjecture on counting ways to color the graph using chromatic polynomials, studied those polynomials even deeper, and generalized the framework to matroids. Check this wonderful Quanta Magazine’s article dedicated to June and his research. 2. Just in case you had all your browser tabs closed and looked for something new to read - Michael Bronstein comes to help and publishes a new blog on the origins of Geometric Deep Learning. This is going to be a series of articles tracing the history of geometry from Greeks to GNNs.

EURO Meets NeurIPS 2022 Vehicle Routing Competition ”The EURO Meets NeurIPS 2022 Vehicle Routing Competition aims to bring together researchers from operations research (OR) and machine learning (ML) to address the vehicle routing problem with time windows (VRPTW) as well as a dynamic VRPTW.” Recently, we have been observing a surge in applying GNNs for Combinatorial Optimization problems (like Traveling Salesman Problem) - here is the top challenge combining combinatorial optimization and dynamic graphs. Data and problem description are already available.

OpenFold & Open Molecular Software Foundation News from the sister adjacent where Geometric Deep Learning is the main workhorse: OpenFold is a new, non-profit AI research consortium to foster free and open-source tools for biology and drug discovery. OpenFold is founded by the Lab of Mohammed AlQuraishi at Columbia University, Arzeda, Cyrus Biotechnology, Prescient Design, and Outspace Bio. The first big release of OpenFold is OpenFold, (citing the authors) “a trainable reproduction” of AlphaFold 2 in PyTorch with the aim to open all the training data and model weights. The OpenFold consortium is designed to be “OpenAI in drug discovery”, let’s hope they will be a bit more open than OpenAI itself about their models and code 😉