PaGaLiNNeT

Title: PaGaLiNNeT – Parallel Grid-aware Library for Neural Networks Training

Grant: Framework Program 7 International Incoming Fellowship 221524
The financial support of the MC IIF 221524 grant is gratefully acknowledged.

Duration: 01/04/2009 – 31/03/2011

Marie Curie researcher: Dr. Volodymyr Turchenko
Home organization: Research Institute of Intelligent Computer Systems, Ternopil National Economic University, 3 Peremoga Square, Ternopil, 46004, UKRAINE
tel:  begin_of_the_skype_highlighting +380352-475050 end_of_the_skype_highlighting ext.12322,  e-mail: vtu<at>tneu<dot>edu<dot>ua

Scientist-in-charge: Prof. Lucio Grandinetti
Host organization: Department of Electronics, Informatics and Systems, Center of Excellence on High Performance Computing, University of Calabria, via P. Bucci, 41C, 87036, Rende (CS), ITALY
Phone/Fax: +39-0984-494847 begin_of_the_skype_highlighting +39-0984-494847 end_of_the_skype_highlighting,  e-mail: lugran<at>unical<dot>it

The aim of this Marie Curie International Incoming Fellowship 221524 “PaGaLiNNeT – Parallel Grid-aware Library for Neural Networks Training” was to develop the enhanced parallel algorithms of neural networks training with improved parallelization efficiency on heterogeneous computational Grids.

The objectives set out at the beginning of the project (for the period 01/04/2009 – 31/03/2011) were as follows:
1. adapt the BSP cost model of parallel NN training algorithms within single pattern, batch pattern and modular approaches to heterogeneous computational Grid resources of host institution;
2. develop enhanced single pattern and batch pattern parallel NN training algorithms based on improved communication and barrier functions;
3. develop a method of automatic matching of parallelization strategy to architecture of appropriate parallel computing system;
4. develop a parallel Grid-aware library for neural networks training capable to use heterogeneous computational resources;
5. test experimentally a parallel Grid-aware library for neural networks training on a heterogeneous computational Grid system of the host institution within the tasks of one of its active projects;

The main outcomes of the Fellowship are:

  • The enchanced parallel batch pattern back propagation training algorithms of multilayer perceptron and recurrent neural network have been developed. We have researched the parallelization efficiencies of these parallel algorithms for typical training scenarios by increasing the neural network connections from 36 to 5041 and increasing the number of training patterns from 100 to 10000 on a symmetric multiprocessor, multi-core and cluster systems using MPICH2-1.2.1, NEC MPI/EX 1.5.2 and Open MPI 1.4 libraries. Our results show that the developed parallel algorithms are scalable and the parallelization efficiencies are high enough for their efficient use on general-purpose parallel computers and computational clusters;
  • The BSP (Bulk Synchronous Parallelism)-based computational cost model of the parallel algorithms has been created for the prediction of the expected training time of the neural network on different processors of different parallel systems. The application of this model shows good prediction accuracy, for instance, the maximal absolute error of prediction of parallelization efficiency does not exceed 5% on a symmetric multiprocessor computer and 2% on a computational cluster;
  • The resource brokering strategy has been developed based on the training time prediction using BSP-based computational cost model to keep high parallelization efficiency of the neural network parallel training algorithm. Our strategy of resource brokering is based on three criteria: a cost of a parallel system, a predicted execution time and a parallelization efficiency of the parallel algorithm. The strategy of resource brokering is based on Pareto optimality with the weighted sum approach for choosing optimal solutions for efficient parallelization of the algorithm. Our results show good conformity with the desired scheduling policy of minimization of the execution time of the parallel algorithm with maximization of the parallelization efficiency in the most economic way. This approach is proved to be very applicable for the practical implementation of the resource broker when any desired scheduling policy could be described by a user during runtime;
  • The parallel Grid-aware library for neural networks training has been developed in C and deployed on a computational Grid of the host institution and other parallel systems available for the experimental research within the project. The library consists of the C routines of: (i) the enhanced parallel batch pattern back propagation training algorithms of multilayer perceptron and recurrent neural network with advanced intercommunication functions between the parallel parts of the algorithms, (ii) the BSP-based computational cost model of the enhanced batch pattern parallel algorithms, (iv) the coarse-grain modular neural network parallelization strategies of static and dynamic mapping. The resource broker implementation depends on the type of the parallel system used.
  • Three actual scientific tasks sufficient to be solved by using neural networks of the team from the host institution have been identified and solved jointly with the appropriate scientific groups: (i) stock price prediction for financial markets in the field of finance [16], (ii) classification of adverse events for heart failure patients and (iii) detection of micronucleus in human lymphocytes in the field of medical care. The developed parallel library was successfully applied to speedup the training process of neural networks for the task of stock price prediction in financial markets (with 83% of parallelization efficiency) and for the task of classification of adverse events in heart failure patients (with 95% of parallelization efficiency).

The results of the project fulfillment have been published in 14 scientific papers, 2 papers are accepted for publication. The results were presented in 5 scientific seminars and 8 international scientific conferences.

The results of this Fellowship have proved that the batch pattern and modular parallelization schemes of neural networks training are efficient on general-purpose parallel systems when (i) we take care of reducing the communication overhead between parallel processors and (ii) we can schedule the parallelization task to the architecture which best suits the dimension of the input parallelization problem. The socio-economic impact of the project could be considered in (i) enlargement of a number of scientific and computationally intensive applications based on neural networks where the developed library could significantly speed up the training process of neural networks, (ii) effective usage of the existing huge pool of computational Grid resources for new scientific and industry-based applications and (iii) improvement of the quality of scientific research.

Publications resulting directly from this Fellowship:

  1. Turchenko V. Scalability of Parallel Batch Pattern Neural Network Training Algorithm, Artificial Intelligence, Journal of Institute of Artificial Intelligence, National Academy of Sciences of Ukraine, 20/04/2009, Vol. 2., pp. 144-150, ISSN 1561-5359, open access
    http://www.nbuv.gov.ua/portal/natural/II/2009_2/4%5C00_Turchenko.pdf
  2. Turchenko V., Grandinetti L. Efficiency Analysis of Parallel Batch Pattern NN Training Algorithm on General-Purpose Supercomputer. Lecture Notes in Computer Science 5518, IWANN 2009, Part II (S. Omatu et al. (Eds.)), Springer-Verlag, Berlin, Heidelberg, 06/06/2009, pp. 223–226, ISSN 0302-9743 begin_of_the_skype_highlighting 0302-9743 end_of_the_skype_highlighting (Print), 1611-3349 (Online), ISBN 978-3-642-02480-1, electronic link
    http://www.springerlink.com/content/k43w68675k575787/?p=9fd96ee0fb164b70857fce2c4726ad3b&pi=30
  3. Turchenko V., Grandinetti L. Minimal Architecture and Training Parameters of Multilayer Perceptron for its Efficient Parallelization, Proceedings of the 5th International Workshop on Artificial Neural Networks and Intelligent Information Processing ANNIIP2009, 4-5 July 2009, Milan, Italy, pp. 79-87, ISBN: 978-989-674-002-3
  4. Turchenko V., Grandinetti L. Efficiency Research of Batch and Single Pattern MLP Parallel Training Algorithms, Proceedings of the 5th IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS2009), 21-23 September 2009, Rende (Cosenza), Italy, pp. 218-224, Print ISBN:978-1-4244-4901-9,CD ISBN: 978-1-4244-4882-1, electronic link
    http://ieeexplore.ieee.org/search/freesrchabstract.jsp?tp=&arnumber=5342990&queryText%3Dturchenko%26openedRefinements%3D*%26searchField%3DSearch+All
  5. Puhol T., Turchenko V., Vozniak S., Sachenko A. Globus-Middleware Based Grid of Research Institute for Intelligent Computer Systems, Proceedings of the 5th IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS2009), 21-23 September 2009, Rende (Cosenza), Italy, pp. 266-271, Print ISBN: 978-1-4244-4901-9, CD ISBN: 978-1-4244-4882-1, electronic link
    http://ieeexplore.ieee.org/search/freesrchabstract.jsp?tp=&arnumber=5342982&queryText%3Dturchenko%26openedRefinements%3D*%26searchField%3DSearch+All
  6. Turchenko V., Grandinetti L. Investigation of Computational Cost Model of MLP Parallel Batch Training Algorithm, Proceedings of 2009 IEEE Symposium on Industrial Electronics and Applications (ISIEA 2009), October 4-6, 2009, Kuala Lumpur, Malaysia, pp. 983-988, Print ISBN: 978-1-4244-4681-0, CD ISBN 978-1-4244-4682-7, electronic link
    http://ieeexplore.ieee.org/search/freesrchabstract.jsp?tp=&arnumber=5356307&queryText%3Dturchenko%26openedRefinements%3D*%26searchField%3DSearch+All
  7. Turchenko V., Grandinetti L. Parallel Batch Pattern BP Training Algorithm of Recurrent Neural Network, Proceedings of the 14th IEEE International Conference on Intelligent Engineering Systems, May 5-7, 2010, Las Palmas of Gran Canaria, Spain, pp. 25-30, CD ISBN: 978-1-4244-7651-0, electronic link
    http://ieeexplore.ieee.org/search/srchabstract.jsp?tp=&arnumber=5483830&queryText%3Dturchenko%26openedRefinements%3D*%26searchField%3DSearch+All
  8. Turchenko V., Grandinetti L., Bosilca G., Dongarra J. Improvement of parallelization efficiency of batch pattern BP training algorithm using Open MPI, Elsevier Procedia Computer Science 2010, Volume 1, Issue 1, The International Conference on Computational Science ICCS 2010, 31 May-02 June 2010, Amsterdam, the Netherlands, pp. 525-533, ISSN: 1877-0509, open access
    http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B9865-506HM1Y-1Y&_user=10&_rdoc=1&_fmt=high&_orig=mlkt&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=822676add76fd7fa2aeaca908e092802
  9. Turchenko V., Grandinetti L. Strategy of Resource Brokering for Efficient Parallelization of MLP Training, The 2010 International Conference on High Performance Computing & Simulation HPCS 2010, June 28 – July 2, 2010, Caen, France, pp. 140-149, Print ISBN: 978-1-4244-6827-0, CD ISBN: 978-1-4244-6829-4, electronic link
    http://ieeexplore.ieee.org/search/srchabstract.jsp?tp=&arnumber=5547138&queryText%3Dturchenko%26openedRefinements%3D*%26pageNumber%3D2%26searchField%3DSearch+All
  10. Turchenko V., Grandinetti L. Scalability of Enhanced Parallel Batch Pattern BP Training Algorithm on General-Purpose Supercomputers, Advances in Intelligent and Soft Computing, Vol. 79, International Symposium on Distributed Computing and Artificial Intelligence DCAI2010 (Ponce de Leon, F.; De Carvalho, A.; Rodríguez-González, S.; De Paz, J.F.; Corchado, J.M. (eds.)), Sep 7-10, 2010, Valencia, Spain, pp. 518-526, Springer, Heidelberg, Sep. 2010, ISSN: 1867-5662, ISBN: 978-3-642-14882-8, electronic link
    http://www.springer.com/engineering/mathematical/book/978-3-642-14882-8
  11. Turchenko V., Grandinetti L. Application of BSP-Based Computational Cost Model to Predict Parallelization Efficiency of MLP Training Algorithm, Lecture Notes in Computing Science, Vol. 6354, The 20th International Conference of Artificial Neural Networks 2010 (K. Diamantaras, W. Duch, L.S. Iliadis (Eds.)), Sep 15-18, 2010, Thessaloniki, Greece, pp. 327–332, Springer, Berlin, Heidelberg, New York, ISSN: 0302-9743 begin_of_the_skype_highlighting 0302-9743 end_of_the_skype_highlighting, ISBN-10: 3-642-15824-2, ISBN-13: 978-3-642-15824-7, electronic link
    http://www.springerlink.com/content/978-3-642-15824-7/#section=779768&page=1&locus=27
  12. Paliy I., Lamonaca F., Turchenko V., Grimaldi D., Sachenko A. Micro Nucleus Detection in Human Lymphocytes Using Convolutional Neural Network, Lecture Notes in Computing Science, Vol. 6352, The 20th International Conference of Artificial Neural Networks 2010 (K. Diamantaras, W. Duch, L.S. Iliadis (Eds.)), Sep 15-18, 2010, Thessaloniki, Greece, pp. 521-530, Springer, Berlin, Heidelberg, New York, ISSN: 0302-9743 begin_of_the_skype_highlighting 0302-9743 end_of_the_skype_highlighting, ISBN-10: 3-642-15818-8, ISBN-13: 978-3-642-15818-6, electronic link
    http://www.springerlink.com/content/978-3-642-15818-6/#section=779902&page=1&locus=0
  13. Turchenko V.O. Brokering methodology of Grid-resources using Pareto-optimality, Measurement and Computational Techniques in Technological Processes, No. 1, 2011, pp. 312–318, ISSN: 2219-9365 (in Ukrainian).
  14. Paliy I., Lamonaca F., Turchenko V., Grimaldi D., Sachenko A. Detection of Micro Nucleus in Human Lymphocytes Altered by Gaussian Noise Using Convolution Neural Network // Proceedings of 2011 IEEE International Instrumentation and Measurement Technology Conference (I2MTC 2011), Binjiang, Hangzhou, China, May 10-12, 2011, pp. 1097-1102, CD ISBN: 978-1-4244-7934-4.
  15. Turchenko V., Marinaro V., Beraldi P., Grandinetti L. Short-term Stock Price Prediction Using MLP in Moving Simulation Mode, Proceedings of the 6th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS2011), 15-17 September 2011, Prague, Czech Republic, accepted.
  16. Turchenko V., Puhol T., Grandinetti L., Sachenko A. NorduGrid Implementation of Resource Brokering Strategy for Parallel Training of Neural Networks, Proceedings of the 6th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS2011), 15-17 September 2011, Prague, Czech Republic, accepted

The dissemination activities were held during the participation of Dr. Turchenko in the following scientific conferences:

  1. The International Symposium on Distributed Computing and Artificial Intelligence DCAI’09, Salamanca, Spain (10-12 June 2009) with oral presentation of the paper “Efficiency Analysis of Parallel Batch Pattern NN Training Algorithm on General-Purpose Supercomputer”;
  2. The 5th International Workshop on Artificial Neural Networks and Intelligent Information Processing ANNIIP2009, Milan, Italy (4-5 July 2009) with oral presentation of the paper “Minimal Architecture and Training Parameters of Multilayer Perceptron for its Efficient Parallelization”;
  3. The 2009 IEEE Symposium on Industrial Electronics & Applications ISIEA’2009, Kuala Lumpur, Malaysia (4-6 October 2009) with oral presentation of the paper “Investigation of Computational Cost Model of MLP Parallel Batch Training Algorithm”;
  4. The 14th International Conference on Intelligent Engineering Systems INES 2010, Las Palmas of Gran Canaria, Spain (02-10 May 2010) with oral presentation of the paper “Parallel Batch Pattern BP Training Algorithm of Recurrent Neural Network”;
  5. The 10th International Conference on Computational Science ICCS 2010, Amsterdam, the Netherlands (27 May – 03 June 2010) with oral presentation of the paper “Improvement of parallelization efficiency of batch pattern BP training algorithm using Open MPI”;
  6. The 2010 International Conference on High Performance Computing & Simulation HPCS 2010, Caen, France (26 June – 03 July 2010) with oral presentation of the paper “Strategy of Resource Brokering for Efficient Parallelization of MLP Training”;
  7. The International Symposium on Distributed Computing and Artificial Intelligence DCAI’2010, Valencia, Spain (08-10 September 2010) with oral presentation of the paper “Scalability of Enhanced Parallel Batch Pattern BP Training Algorithm on General-Purpose Supercomputers”;
  8. The 20th ENNS International Conference on Artificial Neural Networks ICANN 2010, Thessaloniki, Greece (12-18 September 2010) with oral presentation of the papers “Application of BSP-Based Computational Cost Model to Predict Parallelization Efficiency of MLP Training Algorithm” and “Micro Nucleus Detection in Human Lymphocytes Using Convolutional Neural Network”.