Scalable and Efficient Data Processing in Distributed Infrastructure

Title: Scalable and Efficient Data Processing in Distributed Infrastructure

Abstract:
The development of computer science and technology is expanding our view of the world. As a result, the amount of data observed from the world that needs to be stored and processed has also become extremely large (e.g., the evolution of big data). The potential benefits of big data are real, but to realize them in practice, we need to address the key challenges of how to effectively and efficiently manage, process, and understand this tsunami of data. Meanwhile, over the last two decades, we have witnessed new trends and practices in computing infrastructure (cloud, fog, edge, serverless computing, and convergence between big data and HPC) and distributed and parallel computation (MapReduce, Apache Hadoop, Apache Spark, Apache Flick), as well as the introduction of new data-intensive applications with different characteristics and performance requirements (data stream, graph, and machine learning applications).
This manuscript provides a synthetic overview of my research activities since I was hired as a permanent researcher at Inria in October 2013. It does not claim to be an exhaustive presentation of my work. Rather, it focuses on the main contributions to big data processing by addressing the challenges underlying the design and development of new optimization and data management systems that directly respond to trends in infrastructure, platforms, and applications, and improve the effectiveness and efficiency of data-intensive applications.
The work presented here covers several topics in big data processing at scale, including (1) addressing the challenges of performing big data processing on clouds and HPC, with a focus on performance variability in clouds and shared architecture design of HPC systems; (2) optimizing data stream and graph processing in the face of data volatility, resource dynamics, and data geo-distribution; and (3) storage-centric solutions for cost-effective big data processing. Finally, I put these contributions into the perspective of data management abstractions for a sustainable and scalable Edge-Cloud Continuum.