×

Inclusion of domain-knowledge into GNNs using mode-directed inverse entailment. (English) Zbl 07510322

Summary: We present a general technique for constructing Graph Neural Networks (GNNs) capable of using multi-relational domain knowledge. The technique is based on mode-directed inverse entailment (MDIE) developed in Inductive Logic Programming (ILP). Given a data instance \(e\) and background knowledge \(B\), MDIE identifies a most-specific logical formula \(\bot_B(e)\) that contains all the relational information in \(B\) that is related to \(e\). We represent \(\bot_B(e)\) by a “bottom-graph” that can be converted into a form suitable for GNN implementations. This transformation allows a principled way of incorporating generic background knowledge into GNNs: we use the term ‘BotGNN’ for this form of graph neural networks. For several GNN variants, using real-world datasets with substantial background knowledge, we show that BotGNNs perform significantly better than both GNNs without background knowledge and a recently proposed simplified technique for including domain knowledge into GNNs. We also provide experimental evidence comparing BotGNNs favourably to multi-layer perceptrons that use features representing a “propositionalised” form of the background knowledge; and BotGNNs to a standard ILP based on the use of most-specific clauses. Taken together, these results point to BotGNNs as capable of combining the computational efficacy of GNNs with the representational versatility of ILP.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Ando, HY; Dehaspe, L.; Luyten, W.; Van Craenenbroeck, E.; Vandecasteele, H.; Van Meervelt, L., Discovering h-bonding rules in crystals with inductive logic programming, Molecular Pharmaceutics, 3, 6, 665-674 (2006) · doi:10.1021/mp060034z
[2] Bai, S.; Zhang, F.; Torr, PH, Hypergraph convolution and hypergraph attention, Pattern Recognition, 110, 107637 (2021) · doi:10.1016/j.patcog.2020.107637
[3] Besold, T. R., Garcez, A. D., Bader, S., Bowman, H., Domingos, P., Hitzler, P., Kühnberger, K. U., Lamb, L. C., Lowd, D., Lima, P. M. V., et al. (2017). Neural-symbolic learning and reasoning: A survey and interpretation. arXiv:abs/1711.03902
[4] Bianchi, F. M., Grattarola, D., Livi, L., & Alippi, C. (2021). Graph neural networks with convolutional arma filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1. doi:10.1109/TPAMI.2021.3054830
[5] Bravo, H. C., Page, D., Ramakrishnan, R., Shavlik, J., & Costa, V. S. (2005). A framework for set-oriented computation in inductive logic programming and its application in generalizing inverse entailment. In International conference on inductive logic programming, Springer, pp. 69-86. · Zbl 1134.68467
[6] Cangea, C., Veličković, P., Jovanović, N., Kipf, T., & Liò, P. (2018). Towards sparse hierarchical graph classifiers. arXiv:abs/1811.01287
[7] Chollet, F., et al. (2015). Keras. https://keras.io.
[8] Dash, T., Srinivasan, A., Vig, L., Orhobor, O. I., & King, R. D. (2018). Large-scale assessment of deep relational machines. In International conference on inductive logic programming, Springer, pp. 22-37.
[9] Dash, T., Srinivasan, A., Joshi, R. S., & Baskar, A. (2019). Discrete stochastic search and its application to feature-selection for deep relational machines. In International conference on artificial neural networks, Springer, pp. 29-45.
[10] Dash, T., Chitlangia, S., Ahuja, A., & Srinivasan, A. (2021a). How to tell deep neural networks whatwe know. arXiv:abs/2107.10295
[11] Dash, T., Chitlangia, S., Ahuja, A., & Srinivasan, A. (2021b). Incorporating domain knowledge into deep neural networks. arXiv:abs/2103.00180
[12] Dash, T., Srinivasan, A., & Vig, L. (2021c). Incorporating symbolic domain knowledge into graph neural networks. Machine Learning, 1-28. · Zbl 07465651
[13] Du, SS; Hou, K.; Salakhutdinov, RR; Poczos, B.; Wang, R.; Xu, K., Graph neural tangent kernel: Fusing graph neural networks with graph kernels, Advances in Neural Information Processing Systems, 32, 5723-5733 (2019)
[14] Feng, Y.; You, H.; Zhang, Z.; Ji, R.; Gao, Y., Hypergraph neural networks, Proceedings of the AAAI Conference on Artificial Intelligence, 33, 3558-3565 (2019) · doi:10.1609/aaai.v33i01.33013558
[15] Fey, M., & Lenssen, J. E. (2019). Fast graph representation learning with PyTorch Geometric. In ICLR workshop on representation learning on graphs and manifolds.
[16] Fischer, M., Balunovic, M., Drachsler-Cohen, D., Gehr, T., Zhang, C., & Vechev, M. (2019). Dl2: Training and querying neural networks with logic. In International conference on machine learning, PMLR, pp. 1931-1941.
[17] França, MV; Zaverucha, G.; Garcez, ASd, Fast relational learning using bottom clause propositionalization with artificial neural networks, Machine Learning, 94, 1, 81-104 (2014) · doi:10.1007/s10994-013-5392-1
[18] Frasconi, P.; Costa, F.; De Raedt, L.; De Grave, K., klog: A language for logical and relational learning with kernels, Artificial Intelligence, 217, 117-143 (2014) · Zbl 1405.68288 · doi:10.1016/j.artint.2014.08.003
[19] Garcez, AD; Gori, M.; Lamb, LC; Serafini, L.; Spranger, M.; Tran, SN, Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning, FLAP, 6, 4, 611-632 (2019) · Zbl 07594179
[20] Garcez, ASA; Zaverucha, G., The connectionist inductive learning and logic programming system, Applied Intelligence, 11, 1, 59-77 (1999) · doi:10.1023/A:1008328630915
[21] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. In International conference on machine learning, PMLR, pp. 1263-1272.
[22] Gori, M., Monfardini, G., & Scarselli, F. (2005). A new model for learning in graph domains. In Proceedings. 2005 IEEE international joint conference on neural networks, 2005, IEEE, vol. 2, pp. 729-734.
[23] Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034.
[24] Hamilton, WL, Graph representation learning, Synthesis Lectures on Artifical Intelligence and Machine Learning, 14, 3, 1-159 (2020) · Zbl 1457.68005 · doi:10.2200/S01045ED1V01Y202009AIM046
[25] Heckerman, D., Meek, C., & Koller, D. (2007). Probabilistic entity-relationship models, prms, and plate models. Introduction to statistical relational learning, pp. 201-238.
[26] Jankovics, V. (2020). vakker/cilp. https://github.com/vakker/CILP
[27] Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., & Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910.
[28] Kersting, K., Kriege, N. M., Morris, C., Mutzel, P., & Neumann, M. (2016). Benchmark data sets for graph kernels. http://graphkernels.cs.tu-dortmund.de
[29] King, RD; Rowland, J.; Oliver, SG; Young, M.; Aubrey, W.; Byrne, E.; Liakata, M.; Markham, M.; Pir, P.; Soldatova, LN, The automation of science, Science, 324, 5923, 85-89 (2009) · doi:10.1126/science.1165620
[30] Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In ICLR (Poster), arXiv:1412.6980
[31] Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In 5th International conference on learning representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
[32] Kitano, H., Artificial intelligence to win the nobel prize and beyond: Creating the engine for scientific discovery, AI Magazine, 37, 1, 39-49 (2016) · doi:10.1609/aimag.v37i1.2642
[33] Krizhevsky, A.; Sutskever, I.; Hinton, GE, Imagenet classification with deep convolutional neural networks, Communications of the ACM, 60, 6, 84-90 (2017) · doi:10.1145/3065386
[34] Kursuncu, U., Gaur, M., & Sheth, A. (2020). Knowledge infused learning (k-il): Towards deep incorporation of knowledge in deep learning. arXiv:abs/1912.00512
[35] Lamb, L. C., Garcez, A. D., Gori, M., Prates, M. O., Avelar, P. H., & Vardi, M. Y. (2020). Graph neural networks meet neural-symbolic computing: A survey and perspective. In C. Bessiere (Ed.) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI-20, international joint conferences on artificial intelligence organization, pp. 4877-4884. doi:10.24963/ijcai.2020/679, survey track.
[36] Lavrač, N., Džeroski, S., & Grobelnik, M. (1991). Learning nonrecursive definitions of relations with linus. In European working session on learning, Springer, pp. 265-281.
[37] Lavrač, N.; Podpečan, V.; Robnik-Šikonja, M., Propositionalization of relational data, 83-105 (2021), Cham: Springer, Cham · Zbl 1466.68004 · doi:10.1007/978-3-030-68817-2_4
[38] Lee, J., Lee, I., & Kang, J. (2019). Self-attention graph pooling. In International conference on machine learning, pp. 3734-3743.
[39] Lodhi, H. (2013). Deep relational machines. In International conference on neural information processing, Springer, Berlin, pp. 212-219.
[40] Marx, K. A., O’Neil, P., Hoffman, P., & Ujwal, M. (2003). Data mining the nci cancer cell line compound gi50 values: identifying quinone subtypes effective against melanoma and leukemia cell classes. Journal of Chemical Information and Computer Sciences, 43(5), 1652-1667.
[41] Morris, C.; Ritzert, M.; Fey, M.; Hamilton, WL; Lenssen, JE; Rattan, G.; Grohe, M., Weisfeiler and leman go neural: Higher-order graph neural networks, Proceedings of the AAAI Conference on Artificial Intelligence, 33, 4602-4609 (2019) · doi:10.1609/aaai.v33i01.33014602
[42] Muggleton, S., Inverse entailment and progol, New Generation Computing, 13, 3-4, 245-286 (1995) · doi:10.1007/BF03037227
[43] Muggleton, S.; De Raedt, L., Inductive logic programming: Theory and methods, The Journal of Logic Programming, 19, 629-679 (1994) · Zbl 0816.68043 · doi:10.1016/0743-1066(94)90035-3
[44] Muralidhar, N., Islam, M. R., Marwah, M., Karpatne, A., & Ramakrishnan, N. (2018). Incorporating prior domain knowledge into deep neural networks. In 2018 IEEE International conference on big data (Big Data), IEEE, pp. 36-45.
[45] Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. arXiv:abs/1609.03499
[46] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8024-8035.
[47] Plotkin, G. (1972). Automatic methods of inductive inference. The University of Edinburgh. Ph.D. dissertation.
[48] Plotkin, GD, A note on inductive generalization, Machine intelligence, 5, 1, 153-163 (1970) · Zbl 0219.68045
[49] Prechelt, L. (1998). Early stopping-but when? In Neural Networks: Tricks of the trade, Springer, Berlin, pp. 55-69.
[50] Saha, A., Srinivasan, A., & Ramakrishnan, G. (2012). What kinds of relational features are useful for statistical learning? In International conference on inductive logic programming, Springer, pp. 209-224. · Zbl 1382.68203
[51] Schlichtkrull, M., Kipf, T.N., Bloem, P., Van Den Berg, R., Titov, I., & Welling, M. (2018). Modeling relational data with graph convolutional networks. In European semantic web conference, Springer, pp. 593-607.
[52] Sheth, A.; Gaur, M.; Kursuncu, U.; Wickramarachchi, R., Shades of knowledge-infused learning for enhancing deep learning, IEEE Internet Computing, 23, 6, 54-63 (2019) · doi:10.1109/MIC.2019.2960071
[53] Sourek, G.; Aschenbrenner, V.; Zelezny, F.; Schockaert, S.; Kuzelka, O., Lifted relational neural networks: Efficient learning of latent relational structures, Journal of Artificial Intelligence Research, 62, 69-100 (2018) · Zbl 1444.68163 · doi:10.1613/jair.1.11203
[54] Šourek, G., Železnỳ, F., & Kuželka, O. (2021). Beyond graph neural networks with lifted relational neural networks. Machine Learning, pp. 1-44. · Zbl 1515.68279
[55] Srinivasan, A. (2001). The aleph manual. https://www.cs.ox.ac.uk/activities/programinduction/Aleph/aleph.html
[56] Srinivasan, A., & Ramakrishnan, G. (2011). Parameter screening and optimisation for ilp using designed experiments. Journal of Machine Learning Research, 12(2). · Zbl 1280.68197
[57] Srinivasan, A., King, R. D., & Bain, M. E. (2003). An empirical study of the use of relevance information in inductive logic programming. Journal of Machine Learning Research, 4(Jul):369-383. · Zbl 1102.68597
[58] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R., Dropout: A simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 15, 1, 1929-1958 (2014) · Zbl 1318.68153
[59] Stevens, R., Taylor, V., Nichols, J., Maccabe, A. B., Yelick, K., & Brown, D. (2020). Ai for science. Tech. rep., Argonne National Lab.(ANL), Argonne, IL (USA).
[60] Van Craenenbroeck, E., Vandecasteele, H., & Dehaspe, L. (2002). Dmax’s functional group and ring library. https://dtai.cs.kuleuven.be/software/dmax/
[61] Velic̆ković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks. In International conference on learning representations, https://openreview.net/forum?id=rJXMpikCZ
[62] Vishwanathan, SVN; Schraudolph, NN; Kondor, R.; Borgwardt, KM, Graph kernels, Journal of Machine Learning Research, 11, 1201-1242 (2010) · Zbl 1242.05112
[63] Wang, H., Zhao, M., Xie, X., Li, W., & Guo, M. (2019). Knowledge graph convolutional networks for recommender systems. In The world wide web conference, pp. 3307-3313.
[64] Williams, K.; Bilsland, E.; Sparkes, A.; Aubrey, W.; Young, M.; Soldatova, LN; De Grave, K.; Ramon, J.; De Clare, M.; Sirawaraporn, W., Cheaper faster drug development validated by the repositioning of drugs against neglected tropical diseases, Journal of the Royal society Interface, 12, 104, 20141289 (2015) · doi:10.1098/rsif.2014.1289
[65] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:abs/1609.08144
[66] Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Philip, S. Y. (2020). A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems.
[67] Xie, Y., Xu, Z., Kankanhalli, M. S., Meel, K. S., & Soh, H. (2019). Embedding symbolic knowledge into deep networks. In Advances in neural information processing systems, pp. 4233-4243.
[68] Xu, J., Zhang, Z., Friedman, T., Liang, Y., & Broeck, G. (2018). A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, PMLR, pp. 5502-5511.
[69] Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2019). How powerful are graph neural networks? In International conference on learning representations, https://openreview.net/forum?id=ryGs6iA5Km
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.