Post #801

​​ULTRA: Towards Foundation Models for Knowledge Graph Reasoning by Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu arxiv: https://arxiv.org/abs/2310.04562 code: https://github.com/DeepGraphLearning/ULTRA It’s time to announce our latest work - ULTRA - a pre-trained foundation model for knowledge graph reasoning that works on any graph and outperforms supervised SOTA models on 50+ graphs. For years, ML on knowledge graphs implied training one model per dataset and those were fixed to a particular set of entities and relations, hence not transferable at all. The key problem: different, often non-overlapping sets of entities and relations (eg, Freebase and Wikidata). Since graph learning is a question of symmetries and invariances, we pose a question: what is the transferable invariance in seemingly different graphs even with different relations? We find the invariance in relation interactions in the graph of relations! Even though relations are different, their interactions remain the same - and we model 4 such interactions (edge types) in the graph of relations. Learning those means we can have a single trained model working on any multi-relational graph. Practically, ULTRA consists of 2 GNNs with the labeling trick (Neural Bellman-Ford nets in our case): given a query (h,r,?), one produces relational features conditioned on the query relation and interaction graph, the 2nd one uses those for inductive reasoning on the main graph We pre-train ULTRA on 3 standard KGs (from Freebase, Wordnet, and Wikidata) and evaluate on 50+ other graphs of various sizes. A single ULTRA outperforms supervised SOTA even in the 0-shot regime never seeing those graphs before. Fine-tuning bumps the performance by additional 10%! Cool fact: it can be shown that ULTRA is a distributionally double-equivariant (to nodes and relations permutations) model - thanks to Bruno Ribeiro for noticing that! Double equivariance is a theoretical framework for inductive reasoning and (probably) is a necessary condition for designing inductive neural nets (arxiv). We publish the code in the latest PyG 2.4 and PyTorch 2.1 and also release the pre-trained checkpoints so you can run them on your own graphs right away!