×

Neural probabilistic logic programming in DeepProbLog. (English) Zbl 1520.68022

Summary: We introduce DeepProbLog, a neural probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques of the underlying probabilistic logic programming language ProbLog can be adapted for the new language. We theoretically and experimentally demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.

MSC:

68N17 Logic programming
68T07 Artificial neural networks and deep learning
PDFBibTeX XMLCite
Full Text: DOI arXiv Link

References:

[1] Manhaeve, R.; Dumancic, S.; Kimmig, A.; Demeester, T.; De Raedt, L., Deepproblog: neural probabilistic logic programming, (Advances in Neural Information Processing Systems (2018)), 3749-3759
[2] Kahneman, D., Thinking, Fast and Slow (2011), Farrar: Farrar Straus and Giroux New York
[3] Santoro, A.; Raposo, D.; Barrett, D. G.; Malinowski, M.; Pascanu, R.; Battaglia, P.; Lillicrap, T., A simple neural network module for relational reasoning, (Advances in Neural Information Processing Systems, vol. 30 (2017)), 4974-4983
[4] De Raedt, L.; Kersting, K.; Natarajan, S.; Poole, D., Statistical relational artificial intelligence: logic, probability, and computation, Synth. Lect. Artif. Intell. Mach. Learn., 10, 1-189 (2016) · Zbl 1352.68005
[5] Getoor, L.; Taskar, B., Introduction to Statistical Relational Learning (2007), MIT Press · Zbl 1141.68054
[6] De Raedt, L.; Kimmig, A., Probabilistic (logic) programming concepts, Mach. Learn., 100, 5-47 (2015) · Zbl 1346.68050
[7] Garcez, A. S.d.; Broda, K. B.; Gabbay, D. M., Neural-Symbolic Learning Systems: Foundations and Applications (2012), Springer Science & Business Media
[8] Bošnjak, M.; Rocktäschel, T.; Riedel, S., Programming with a differentiable forth interpreter, (Proceedings of the 34th International Conference on Machine Learning, vol. 70 (2017)), 547-556
[9] Rocktäschel, T.; Riedel, S., End-to-end differentiable proving, (Advances in Neural Information Processing Systems, vol. 30 (2017)), 3788-3800
[10] Cohen, W. W.; Yang, F.; Mazaitis, K. R., Tensorlog: deep learning meets probabilistic databases, J. Artif. Intell. Res., 1, 1-15 (2018)
[11] De Raedt, L.; Manhaeve, R.; Dumancic, S.; Demeester, T.; Kimmig, A., Neuro-symbolic= neural+ logical+ probabilistic, (NeSy’19@ IJCAI, the 14th International Workshop on Neural-Symbolic Learning and Reasoning (2019)), 1-4
[12] De Raedt, L.; Kimmig, A.; Toivonen, H., ProbLog: a probabilistic prolog and its application in link discovery, (IJCAI (2007)), 2462-2467
[13] LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient-based learning applied to document recognition, Proc. IEEE, 86, 2278-2324 (1998)
[14] Lloyd, J. W., Foundations of Logic Programming (1989), Springer · Zbl 0547.68005
[15] Van Gelder, A.; Ross, K. A.; Schlipf, J. S., The well-founded semantics for general logic programs, J. ACM, 38, 620-650 (1991) · Zbl 0799.68045
[16] Goodfellow, I.; Bengio, Y.; Courville, A., Deep Learning (2016), MIT Press · Zbl 1373.68009
[17] Pearl, J., Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (1988), Morgan Kaufmann Publishers Inc.
[18] Kimmig, A.; Van den Broeck, G.; De Raedt, L., Algebraic model counting, J. Appl. Log., 22, 46-62 (2017) · Zbl 1436.68335
[19] Skryagin, A.; Stelzner, K.; Molina, A.; Ventola, F.; Yu, Z.; Kersting, K., Sum-product logic: integrating probabilistic circuits into deepproblog, (Working Notes of the ICML 2020 Workshop on Bridge Between Perception and Reasoning: Graph Neural Networks and Beyond (2020))
[20] Fierens, D.; Van den Broeck, G.; Renkens, J.; Shterionov, D.; Gutmann, B.; Thon, I.; Janssens, G.; De Raedt, L., Inference and learning in probabilistic logic programs using weighted Boolean formulas, Theory Pract. Log. Program., 15, 358-401 (2015) · Zbl 1379.68062
[21] Darwiche, A.; Marquis, P., A knowledge compilation map, J. Artif. Intell. Res., 17, 229-264 (2002) · Zbl 1045.68131
[22] Darwiche, A., SDD: a new canonical representation of propositional knowledge bases, (Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence. Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI-11 (2011)), 819-826
[23] Domingos, P.; Lowd, D., Markov logic: an interface layer for artificial intelligence, Synth. Lect. Artif. Intell. Mach. Learn., 3, 1-155 (2009) · Zbl 1202.68403
[24] Frazier, M.; Pitt, L., Learning from entailment: an application to propositional horn sentences, (Machine Learning, Proceedings of the Tenth International Conference. Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts, Amherst, MA, USA, June 27-29, 1993 (1993)), 120-127
[25] Gutmann, B.; Kimmig, A.; Kersting, K.; De Raedt, L., Parameter learning in probabilistic databases: a least squares approach, (Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2008), Springer), 473-488
[26] Kimmig, A.; Van den Broeck, G.; De Raedt, L., An algebraic Prolog for reasoning about possible worlds, (AAAI (2011))
[27] Eisner, J., Parameter estimation for probabilistic finite-state transducers, (Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics (2002)), 1-8
[28] Kingma, D. P.; Ba, J., Adam: a method for stochastic optimization, (Bengio, Y.; LeCun, Y., 3rd International Conference on Learning Representations, Conference Track Proceedings. 3rd International Conference on Learning Representations, Conference Track Proceedings, ICLR 2015, San Diego, CA, USA, May 7-9, 2015 (2015))
[29] Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P., Gradient-based learning applied to document recognition, (Proceedings of the IEEE (1998)), 2278-2324
[30] Solar-Lezama, A., Program sketching, Int. J. Softw. Tools Technol. Transf., 15, 475-495 (2013)
[31] Reed, S.; de Freitas, N., Neural programmer-interpreters, (International Conference on Learning Representations. International Conference on Learning Representations, ICLR (2016))
[32] Roy, S.; Roth, D., Solving general arithmetic word problems, (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (2015)), 1743-1752
[33] Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O., Translating embeddings for modeling multi-relational data, (Advances in Neural Information Processing Systems (2013)), 2787-2795
[34] Sinha, K.; Sodhani, S.; Dong, J.; Pineau, J.; Hamilton, W. L., Clutrr: a diagnostic benchmark for inductive reasoning from text, (Empirical Methods of Natural Language Processing. Empirical Methods of Natural Language Processing, EMNLP (2019))
[35] Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y., Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014), arXiv preprint
[36] Dries, A.; Kimmig, A.; Meert, W.; Renkens, J.; Van den Broeck, G.; Vlasselaer, J.; De Raedt, L., Problog2: probabilistic logic programming, (Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2015), Springer), 312-315
[37] Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; Lerer, A., Automatic differentiation in pytorch, (Proceedings of the Workshop on the Future of Gradient-Based Machine Learning Software and Techniques, Co-Located with the 31st Annual Conference on Neural Information Processing Systems. Proceedings of the Workshop on the Future of Gradient-Based Machine Learning Software and Techniques, Co-Located with the 31st Annual Conference on Neural Information Processing Systems, NIPS 2017 (2017))
[38] Hammer, B.; Hitzler, P., Perspectives of Neural-Symbolic Integration, vol. 8 (2007), Springer: Springer Heidelberg · Zbl 1124.68092
[39] Hölldobler, S.; Kalinke, Y.; Störr, H.-P., Approximating the semantics of logic programs by recurrent neural networks, Appl. Intell., 11, 45-58 (1999)
[40] De Raedt, L.; Manhaeve, R.; Dumancic, S.; Demeester, T.; Kimmig, A., Neuro-symbolic = neural + logical + probabilistic, (NeSy’19@ IJCAI, the 14th International Workshop on Neural-Symbolic Learning and Reasoning (2019))
[41] Rocktäschel, T.; Singh, S.; Riedel, S., Injecting logical background knowledge into embeddings for relation extraction, (NAACL HLT 2015, the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2015)), 1119-1129
[42] Demeester, T.; Rocktäschel, T.; Riedel, S., Lifted rule injection for relation embeddings, (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)), 1389-1399
[43] Minervini, P.; Demeester, T.; Rocktäschel, T.; Riedel, S., Adversarial sets for regularised neural link predictors, (Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, UAI (2017))
[44] Diligenti, M.; Gori, M.; Sacca, C., Semantic-based regularization for learning and inference, Artif. Intell., 244, 143-165 (2017) · Zbl 1404.68100
[45] Donadello, I.; Serafini, L.; d’Avila Garcez, A. S., Logic tensor networks for semantic image interpretation, (Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017 (2017)), 1596-1602
[46] Xu, J.; Zhang, Z.; Friedman, T.; Liang, Y.; den Broeck, G. V., A semantic loss function for deep learning with symbolic knowledge, (Proceedings of the 35th International Conference on Machine Learning. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (2018)), 5498-5507
[47] Weber, L.; Minervini, P.; Münchmeyer, J.; Leser, U.; Rocktäschel, T., Nlprolog: reasoning with weak unification for question answering in natural language, (Proceedings of ACL 2018, Tutorial Abstracts (2019))
[48] Andreas, J.; Rohrbach, M.; Darrell, T.; Klein, D., Neural module networks, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)), 39-48
[49] Evans, R.; Grefenstette, E., Learning explanatory rules from noisy data, J. Artif. Intell. Res., 61, 1-64 (2018) · Zbl 1426.68235
[50] Šourek, G.; Aschenbrenner, V.; Železný, F.; Schockaert, S.; Kuželka, O., Lifted relational neural networks: efficient learning of latent relational structures, J. Artif. Intell. Res., 62 (2018) · Zbl 1444.68163
[51] Kazemi, S. M.; Poole, D., RelNN: a deep neural model for relational learning, (AAAI (2018))
[52] Lippi, M.; Frasconi, P., Prediction of protein β-residue contacts by Markov logic networks with grounding-specific weights, Bioinformatics, 25, 2326-2333 (2009)
[53] Wang, H.; Poon, H., Deep probabilistic logic: a unifying framework for indirect supervision, (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 (2018)), 1891-1902
[54] Richardson, M.; Domingos, P., Markov logic networks, Mach. Learn., 62, 107-136 (2006) · Zbl 1470.68221
[55] Bach, S. H.; Broecheler, M.; Huang, B.; Getoor, L., Hinge-loss Markov random fields and probabilistic soft logic (2015), arXiv preprint
[56] Kalyan, A.; Mohta, A.; Polozov, O.; Batra, D.; Jain, P.; Gulwani, S., Neural-guided deductive search for real-time program synthesis from examples, (ICLR (2018))
[57] Zhang, L.; Rosenblatt, G.; Fetaya, E.; Liao, R.; Byrd, W. E.; Might, M.; Urtasun, R.; Zemel, R., Neural guided constraint logic programming for program synthesis, (NeurIPS (2018))
[58] Ellis, K.; Morales, L.; Sablé-Meyer, M.; Solar-Lezama, A.; Tenenbaum, J., Learning libraries of subroutines for neurally-guided Bayesian program induction, (NeurIPS (2018))
[59] Yi, K.; Wu, J.; Gan, C.; Torralba, A.; Kohli, P.; Tenenbaum, J. B., Neural-symbolic VQA: disentangling reasoning from vision and language understanding, (NeurIPS (2018))
[60] Mao, J.; Gan, C.; Kohli, P.; Tenenbaum, J. B.; Wu, J., The neuro-symbolic concept learner: interpreting scenes, words, and sentences from natural supervision, (ICLR (2019))
[61] Ellis, K.; Ritchie, D.; Solar-Lezama, A.; Tenenbaum, J., Learning to infer graphics programs from hand-drawn images, (Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; Garnett, R., Advances in Neural Information Processing Systems, vol. 31 (2018), Curran Associates, Inc.), 6059-6068
[62] Kazemi, S. M.; Buchman, D.; Kersting, K.; Natarajan, S.; Poole, D., Relational logistic regression: the directed analog of Markov logic networks, (Proceedings of the 13th AAAI Conference on Statistical Relational AI. Proceedings of the 13th AAAI Conference on Statistical Relational AI, AAAIWS’14-13 (2014), AAAI Press), 41-43
[63] Dumančić, S.; Blockeel, H., Clustering-based relational unsupervised representation learning with an explicit distributed representation, (Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17 (2017)), 1631-1637
[64] Dumančić, S.; Guns, T.; Meert, W.; Blockeel, H., Learning relational representations with auto-encoding logic programs, (Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 (2019)), 6081-6087, in press
[65] Cropper, A., Playgol: learning programs through play, (Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 (2019)), 6074-6080, in press
[66] Kimmig, A.; Costa, V. S.; Rocha, R.; Demoen, B.; De Raedt, L., On the efficient execution of problog programs, (International Conference on Logic Programming (2008), Springer), 175-189 · Zbl 1185.68162
[67] Vlasselaer, J.; Van den Broeck, G.; Kimmig, A.; Meert, W.; De Raedt, L., Anytime inference in probabilistic logic programs with TP-compilation, (Twenty-Fourth International Joint Conference on Artificial Intelligence (2015))
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.