Designing domain specific language to build large scale distributed systems

Modern distributed applications are becoming increasing large and complex. They often bring together independently developed sub-systems~(e.g. for storage, batch processing, streaming, application logic, logging, caching) into large, geo-distributed and heterogeneous architectures. Combining, configuring, and deploying
these architectures is a difficult and multifaceted task: individual services have their own requirements, configuration spaces, programming models, distribution logic, which must be carefully tuned to insure the overall performance, resilience, and evolvability of the resulting system. This integration effort remains today largely an ad-hoc activity, that is either manual or uses tool-specific scripting capabilities. This low-level approach unfortunately scales poorly in the face of the increasingly complex deployment requirements and topologies of the involved services. For instance, Netflix leverages on Infrastructures as a Service (IaaS) clouds, such as Amazon Elastic Compute Cloud (EC2), to setup a complex software architecture to provision and instantiate on the fly the adequate numbers of Virtual Machines (VMs), dedicated to the transcoding of video streams depending on the incoming load. In particular, nowadays, companies such as Netflix, Twitter, Google setup micro-service architectures deployed across geo-distributed datacenters to schedule bench of interleaved tasks resulting in increasingly complex distributed applications, that can be hard to describe, monitor, and adapt.

Easing the development of complex distributed systems has been a long-running and recurrent objective of middleware research. Most of these efforts have however focused on the local behavior of individual nodes (e.g.~with protocol kernels, or component frameworks), rather than on the programmatic means to describe a system's global structure and behavior. As a result, most of these programming frameworks offer little or no support for the flexible integration of individual systems into a larger whole. As a result, most existing approaches do not take into account that the resulting system may, in turn, serves as the constituent element of another larger system.

Consequently, our main objective in this thesis is to provide an assembly-based programming framework for the implementation of complex distributed application to be deployed in the cloud. Specifically, our aim is to provide a domain specific language (DSL) that exploits self-organizing overlays to map at runtime a developer's high-level description of a complex distributed application onto a concrete infrastructure. Our approach relies on the scalability, resilience, and adaptability of self-organizing overlays to maintain a developer's target distributed application in the face of failures, scaling and dynamic adaptations. Further, our approach goes beyond traditional framework for distributed systems in that itconsiders individual systems as collective distributed entities} enforcing a given internal structure (a star,  a tree, a ring) which developers can assemble programmatically to realize more complex topologies. It also goes beyond existing self-organizing overlays by supporting the description of a target
 topology as a composition of more elementary shapes, breaking away from the monolithic design of typical self-organizing overlay protocols.

This thesis has a key impact on both the research community and industry related to cloud infrastructures. Provisioning, deploying and maintaining large scale distributed topologies in the cloud is unfortunately a cumbersome and error-prone task. Developers must provision nodes, specify the local services they should execute, and connect these local services into an appropriate topology, according to the service and application’s needs. Further, they need to take into account the underlying resources provided by the different provisioned node.
These last years, huge efforts have been done in both the research community and industry to provide
deployment automation tools such as Borg, Kubernetes, Aurora or Mesos. However, these tools offer essential self-healing and scaling capabilities, but still require developers to manually configure the system’s actual topology, potentially aided by low-level scripting tools. This ad-hoc approach results in makeshift, tedious and error-prone code, as this code must take into account the specificity of individual services, must account for the volatility of the cloud infrastructures in which these services typically operate, and must handle service-specific roll-back and recovery operations in case of failures or partitioning.