×

zbMATH — the first resource for mathematics

Elasticity management of streaming data analytics flows on clouds. (English) Zbl 1372.68048
Summary: In this paper, we present a framework for resource management of Streaming Data Analytics Flows (SDAF). Using advanced techniques in control and optimization theory, we design an adaptive control system tailored to the data ingestion, analytics, and storage layers of the SDAF that is able to continuously detect and self-adapt to workload changes for meeting the users’ service level objectives. Our experiments based on a real-world SDAF show that, the proposed control scheme is able to reduce the deviation from desired utilization of resources by up to 48% compared to existing techniques.
MSC:
68M20 Performance evaluation, queueing, and scheduling in the context of computer systems
90C29 Multi-objective and goal programming
Software:
MOEA
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Dong, M.; Li, H.; Ota, K.; Yang, L. T.; Zhu, H., Multicloud-based evacuation services for emergency management, IEEE Cloud Comput., 1, 4, 50-59, (2014)
[2] Sumbaly, R.; Kreps, J.; Shah, S., The big data ecosystem at linkedin, (Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, (2013), ACM), 1125-1134
[3] Lim, H., Workload management for data-intensive services, (2013), Duke University, PhD thesis
[4] Bhartia, R., Amazon kinesis and apache storm: building a real-time sliding-window dashboard over streaming data, (2014)
[5] Khoshkbarforoushha, A.; Wang, M.; Ranjan, R.; Wang, L.; Alem, L.; Khan, S. U.; Benatallah, B., Dimensions for evaluating cloud resource orchestration frameworks, Computer, 49, 2, 24-33, (2016)
[6] Islam, S.; Lee, K.; Fekete, A.; Liu, A., How a consumer can measure elasticity for cloud platforms, (Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering, (2012), ACM), 85-96
[7] Herbst, N. R.; Kounev, S.; Reussner, R., Elasticity in cloud computing: what it is, and what it is not, (ICAC, (2013)), 23-27
[8] Lehrig, S.; Eikerling, H.; Becker, S., Scalability, elasticity, and efficiency in cloud computing: a systematic literature review of definitions and metrics, (Proceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures, (2015), ACM), 83-92
[9] Lorido-Botran, T.; Miguel-Alonso, J.; Lozano, J. A., A review of auto-scaling techniques for elastic applications in cloud environments, J. Grid Comput., 12, 4, 559-592, (2014)
[10] Zhu, T.; Gandhi, A.; Harchol-Balter, M.; Kozuch, M. A., Saving cash by using less cache, (USENIX Workshop on Hot Topics in Cloud Computing (HotCloud), (2012))
[11] Li, H.; Dong, M.; Ota, K.; Guo, M., Pricing and repurchasing for big data processing in multi-clouds, IEEE Trans. Emerg. Top. Comput., 4, 2, 266-277, (2016)
[12] Khoshkbarforoushha, A.; Ranjan, R.; Gaire, R.; Jayaraman, P. P.; Hosking, J.; Abbasnejad, E., Resource usage estimation of data stream processing workloads in datacenter clouds, arXiv preprint
[13] Lu, C.; Lu, Y.; Abdelzaher, T. F.; Stankovic, J.; Son, S. H., Feedback control architecture and design methodology for service delay guarantees in web servers, IEEE Trans. Parallel Distrib. Syst., 17, 9, 1014-1027, (2006)
[14] Lama, P.; Zhou, X., Efficient server provisioning with control for end-to-end response time guarantee on multitier clusters, IEEE Trans. Parallel Distrib. Syst., 23, 1, 78-86, (2012)
[15] Lama, P.; Zhou, X., Autonomic provisioning with self-adaptive neural fuzzy control for percentile-based delay guarantee, ACM Trans. Auton. Adapt. Syst. (TAAS), 8, 2, 9, (2013)
[16] Malkowski, S. J.; Hedwig, M.; Li, J.; Pu, C.; Neumann, D., Automated control for elastic n-tier workloads based on empirical modeling, (Proceedings of the 8th ACM International Conference on Autonomic Computing, (2011), ACM), 131-140
[17] Kalyvianaki, E.; Charalambous, T.; Hand, S., Adaptive resource provisioning for virtualized servers using Kalman filters, ACM Trans. Auton. Adapt. Syst. (TAAS), 9, 2, 10, (2014)
[18] Padala, P.; Shin, K. G.; Zhu, X.; Uysal, M.; Wang, Z.; Singhal, S.; Merchant, A.; Salem, K., Adaptive control of virtualized resources in utility computing environments, (ACM SIGOPS Operating Systems Review, vol. 41, (2007), ACM), 289-302
[19] Padala, P.; Hou, K.-Y.; Shin, K. G.; Zhu, X.; Uysal, M.; Wang, Z.; Singhal, S.; Merchant, A., Automated control of multiple virtualized resources, (Proceedings of the 4th ACM European Conference on Computer Systems, (2009), ACM), 13-26
[20] Stonebraker, M.; Çetintemel, U.; Zdonik, S., The 8 requirements of real-time stream processing, ACM SIGMOD Rec., 34, 4, 42-47, (2005)
[21] Nehme, R. V.; Lim, H.-S.; Bertino, E.; Rundensteiner, E. A., Streamshield: a stream-centric approach towards security and privacy in data stream environments, (Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, (2009), ACM), 1027-1030
[22] Nehme, R. V.; Lim, H.-S.; Bertino, E., Fence: continuous access control enforcement in dynamic data stream environments, (Proceedings of the Third ACM Conference on Data and Application Security and Privacy, (2013), ACM), 243-254
[23] Puthal, D.; Nepal, S.; Ranjan, R.; Chen, J., A dynamic prime number based efficient security mechanism for big sensing data streams, J. Comput. Syst. Sci., 83, 1, 22-42, (2016) · Zbl 1367.94340
[24] Puthal, D.; Nepal, S.; Ranjan, R.; Chen, J., Dlsef: a dynamic key length based efficient real-time security verification model for big data stream, ACM Trans. Embed. Comput. Syst., 15, 4, (2016)
[25] Urgaonkar, B.; Shenoy, P.; Chandra, A.; Goyal, P., Dynamic provisioning of multi-tier Internet applications, (Second International Conference on Autonomic Computing (ICAC), (2005), IEEE), 217-228
[26] Xu, J.; Zhao, M.; Fortes, J.; Carpenter, R.; Yousif, M., On the use of fuzzy modeling in virtualized data center management, (Fourth International Conference on Autonomic Computing, (2007), IEEE), 25-35
[27] Tsoumakos, D.; Konstantinou, I.; Boumpouka, C.; Sioutas, S.; Koziris, N., Automated, elastic resource provisioning for nosql clusters using TIRAMOLA, (13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), (2013), IEEE), 34-41
[28] Hwang, J.; Wood, T., Adaptive performance-aware distributed memory caching, (ICAC, (2013)), 33-43
[29] Lim, H. C.; Babu, S.; Chase, J. S., Automated control for elastic storage, (Proceedings of the 7th International Conference on Autonomic Computing, (2010), ACM), 1-10
[30] Fernandez, H.; Pierre, G.; Kielmann, T., Autoscaling web applications in heterogeneous cloud infrastructures, (IEEE International Conference on Cloud Engineering (IC2E), (2014), IEEE), 195-204
[31] Jamshidi, P.; Ahmad, A.; Pahl, C., Autonomic resource provisioning for cloud-based software, (Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, (2014), ACM), 95-104
[32] Jamshidi, P.; Sharifloo, A. M.; Pahl, C.; Metzger, A.; Estrada, G., Self-learning cloud controllers: fuzzy q-learning for knowledge evolution, (International Conference on Cloud and Autonomic Computing (ICCAC), (2015), IEEE), 208-211
[33] Farokhi, S.; Lakew, E. B.; Klein, C.; Brandic, I.; Elmroth, E., Coordinating CPU and memory elasticity controllers to meet service response time constraints, (International Conference on Cloud and Autonomic Computing (ICCAC), (2015), IEEE), 69-80
[34] Slagter, K.; Hsu, C.-H.; Chung, Y.-C., An adaptive and memory efficient sampling mechanism for partitioning in mapreduce, Int. J. Parallel Program., 43, 3, 489-507, (2015)
[35] Bandyopadhyay, S.; Saha, S., Some single- and multiobjective optimization techniques, (Unsupervised Classification, (2013), Springer Berlin, Heidelberg), 17-58
[36] Khalil, H. K., Nonlinear systems, (1996), Prentice Hall New Jersey · Zbl 0626.34052
[37] Slotine, J.-J. E.; Li, W., Applied nonlinear control, vol. 199, (1991), Prentice Hall Englewood Cliffs, NJ
[38] Sastry, S.; Bodson, M., Adaptive control: stability, convergence and robustness, (2011), Courier Corporation
[39] Ogata, K., Modern control engineering, (2001), Prentice Hall PTR
[40] Hadka, D., MOEA framework a free and open source Java framework for multiobjective optimization, (2012)
[41] Deb, K.; Pratap, A.; Agarwal, S.; Meyarivan, T., A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans. Evol. Comput., 6, 2, 182-197, (2002)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.