Learning in Structured Domains

General Objectives

To investigate on the possibility of extending the computational capabilities of neural networks and related machine learning approaches for the treatment of structured domains (sequences, trees and graphs), making particular attention to adaptive, constructive and contextual models and to the study of their theoretical properties.

This is a traditional research aim of the CIML group in Pisa, with origins since the 90s. CIML members and collaborators were among the pioneers in the processing of structures by Recursive Neural Networks and are active since then for the progress of the field by continuously developing theoretical analysis, new models, and many applications, including pioneering approaches for the chemical domain [4,12,14,15,16]. Together with the historical origins, the  current developments contribute to lead the group to an international scientific leadership in topics for learning in structured domains. The basic of Recursive Neural Networks for adaptive processing of trees, with some historical notes, can be found in [17] and in  RNN-Wikipedia.

The CIML research results include (as regards mainly the historical perspective):

⦁    Supervised neural networks for structures: the proposers have investigated both theoretical issues and real-world applications. They have proposed a first approach to deal with contextual information in structured domains by Recursive Neural Networks (RNN). The proposed model, i.e. Contextual Recursive Cascade Correlation (CRCC) [8], a generalization of the Recursive Cascade Correlation (RCC) model, is able to partially remove the causality assumption by exploiting contextual information. They formally characterize the properties of CRCC showing that it is able to compute contextual transductions and also some causal supersource transductions that RCC cannot compute [5, 6, 12]. Also, comparisons with kernel based methods have been considered [7].

⦁    Unsupervised recursive networks for structures: they define also a general recursive framework for unsupervised processing of structured data [10,11,13]. This general framework offers a uniform notation for training mechanisms of different models and insights into theoretical issues from the SOM literature to the structure processing case.

⦁    The introduction of machine learning models for general structures by Neural Networks. In particular, the NN4G (Neural Network for Graph) model [4] extends the input domain of Neural Networks to general (directed/undirected, cyclic/acyclic) classes of labeled graphs by exploiting contextual information with a constructive and adaptive approach. This approach, presented in the first version at the WIRN conference in June 2005 (and then as journal IEEE TNN version in 2009 [4]), pioneered the field of the spatial approach to graph processing by deep  Neural Networks (also known nowadays in terms of convolutional networks for graphs or, more in general, deep graph networks). See the figure above (top page) that show the model and the graph processing through the layers.

⦁    The introduction of efficient probabilistic modeling of trees [1, 2] and efficient neural network (Reservoir Computing) modeling of trees [3, 9].

⦁    The introduction of deep neural network, Reservoir Computing (RC) and generative based approaches for graphs is an ongoing research topic of CIML. For the references to the current developments, including generative kernels for trees (TNNLS  2018), deep probabilistic models for graphs (ICML 2018, JMLR 2020),  deep RC for trees and  graphs (Information Science 2019, AAAI 2020),  comparative studies (ICLR 2020), and a survey on deep learning for graphs (Neural Networks 2020),  please see into the selected paper session of the CIML site at https://ciml.di.unipi.it/publications/

Contact: Alessio Micheli (Associate Professor) ¦ HomePage ¦ E-mail

See collaborators in CIML People Page

Publications on this Topic (selection restricted to the original developments on model studies before 2010, click on the reference to open the paper site)

1. D. Bacciu, A. Micheli, A. Sperduti. “Compositional Generative Mapping for Tree-Structured Data - Part I: Bottom-Up Probabilistic Modeling of Trees”. IEEE Transactions on Neural Networks and Learning Systems Vol. 23 N. 12, December 2012, Pages: 1987 - 2002
2. D. Bacciu, A. Micheli, A. Sperduti. “Compositional Generative Mapping for Tree-Structured Data - Part II: Topographic Projection Model.”  IEEE Transactions on Neural Networks and Learning Systems, Vol. 24 N. 2, February 2013, Pages: 231 - 247.  ISSN: 2162
3. Gallicchio, A. Micheli.  “Tree Echo State Networks”.  Neurocomputing  - Elsevier, Vol. 101, Pages 319-337. Available online 25 September 2012, ISSN: 0925-2312. DOI: 10.1016/j.neucom.2012.08.017
4. A. Micheli.  “Neural Network for Graphs: A Contextual Constructive Approach”, IEEE Transactions on Neural Networks, Vol. 20 (3): 498 - 511, March 2009. ISSN 1045-9227.
5. B. Hammer, A. Micheli, A. Sperduti. "Adaptive Contextual Processing of Structured Data by Recursive Neural Networks: A Survey of Computational Properties".  Chapter in book: Perspectives of Neural-Symbolic Integration. Springer series: 'Studies in Computational Intelligence', Springer Verlag. 2007. ISSN 1860-949X.
6. B. Hammer, A. Micheli, A. Sperduti. "Universal Approximation Capability of Cascade Correlation for Structures." Neural Computation, Vol. 17, Issue 5 - May 2005, Pages 1109-1159, MIT press. ISSN 0899-7667.
7. A. Micheli, F. Portera, A. Sperduti. "A Preliminary Empirical Comparison of Recursive Neural Networks and Tree Kernel Methods on Regression Tasks for Tree Structured Domains." Neurocomputing, Elsevier. Volume 64, Pages 73-92, March 2005 ©2005 Elsevier B.V
8. A. Micheli, D. Sona, A. Sperduti. "Contextual Processing of Structured Data by Recursive Cascade Correlation." IEEE Transactions on Neural Networks. Vol. 15, n. 6, Pages 1396- 1410, November 2004. ISSN 1045-9227.
9. B. Hammer, P. Tino, A. Micheli. "A Mathematical Characterization of the Architectural Bias of Recursive Models", Technical report 252- 2004 - Universitat Osnabruck- Germany.
10. B. Hammer, A. Micheli, A. Sperduti, M. Strickert. "A General Framework for Unsupervised Processing of Structured Data." Neurocomputing, Elsevier. Volume 57, Pages 3-35, March 2004. Imprint: Elsevier ISSN 0925-2312 (Selected from the contribution at ESANN2002).
11. B. Hammer, A. Micheli, A. Sperduti, M. Strickert. "Recursive Self-organizing Network Models." Neural Networks, Elsevier Science. Vol. 17, Issues 8-9, Pages 1061-1085, October-November 2004, Available online since 8 October 2004 (www.sciencedirect.com) © 2004 Published by Elsevier Ltd. Imprint: Pergamon. ISSN 0893-6080. DOI: 10.1016/j.neunet.2004.06.009
12. A. Micheli. "Recursive Processing of Structured Domains in Machine Learning." PhD Thesis, TD-13/03. Scuola di Dottorato "Galileo Galilei". Dipartimento di Informatica, Università di Pisa, Italy, December 2003.
13. B. Hammer, A. Micheli, A. Sperduti. "A General Framework for Self-Organizing Structure Processing Neural Networks", TR-03-04, Dipartimento di Informatica, Universita`  di Pisa, February 2003.
14. L. Bernazzani, C. Duce, A. Micheli, V. Mollica, A. Sperduti, A. Starita, M. R. Tiné. “Predicting Physical Chemical Properties of Compounds from Molecular Structures by Recursive Neural Networks”.  Journal of Chemical Information and Modeling (formerly Journal of Chemical Information and Computer Sciences), ACS Publications,  Washington, DC. Vol. 46(5): 2030 – 2042, September 2006. ACS Publications. ISSN: 1549-9596. DOI 10.1021/ci060104e.
15. A. Micheli, A. Sperduti, A. Starita, A.M. Bianucci. “Analysis of the Internal Representations Developed by Neural Networks for Structures Applied to Quantitative Structure-Activity Relationship Studies of Benzodiazepines.” Journal of Chemical Information and Computer Sciences (ACS Publications), Vol. 41(1): 202-218, January 2001. ISSN 0095-2338.
16. A.M. Bianucci, A. Micheli, A. Sperduti, A. Starita. “Application of Cascade Correlation Networks for Structures to Chemistry”. Applied Intelligence Journal (Kluwer Academic Publishers), Special Issue on “Neural Networks and Structured Knowledge” Vol. 12 (1/2): 117-146, January 2000.  ISSN 0924-669X.
17. A. Sperduti A., A. Starita: "Supervised Neural Networks for Classification of Structures", IEEE Trans on Neural Networks. Vol. 8, No. 3, pp. 714-735, 1997.