ACTURUS: Designing domain specific language to build large scale distributed systems

Context.

Modern distributed applications are becoming increasing large and complex. They often bring together independently developed sub- systems~(e.g. for storage, batch processing, streaming, application logic, logging, caching) into large, geo-distributed and heterogeneous architectures. Combining, configuring, and deploying these architectures is a difficult and multifaceted task: individual services have their own requirements, configuration spaces, programming models, distribution logic, which must be carefully tuned to insure the overall performance, resilience, and evolvability of the resulting system. This integration effort remains today largely an ad-hoc activity, that is either manual or uses tool-specific scripting capabilities. This low-level approach unfortunately scales poorly in the face of the increasingly complex deployment requirements and topologies of the involved services. For instance, Netflix leverages on Infrastructures as a Service (IaaS) clouds, such as Amazon Elastic Compute Cloud (EC2), to setup a complex software architecture to provision and instantiate on the fly the adequate numbers of Virtual Machines (VMs), dedicated to the transcoding of video streams depending on the incoming load. In particular, nowadays, companies such as Netflix, Twitter, Google setup micro-service architectures deployed across geo-distributed datacenters to schedule bench of interleaved tasks resulting in increasingly complex distributed applications, that can be hard to describe, monitor, and adapt [14-23].

Objectives: Easing the development of complex distributed systems has been a long-running and recurrent objective of middleware research. Most of these efforts have however focused on the local behavior of individual nodes (e.g.~with protocol kernels, or component frameworks), rather than on the programmatic means to describe a system's global structure and behavior. As a result, most of these programming frameworks offer little or no support for the flexible integration of individual systems into a larger whole. As a result, most existing approaches do not take into account that the resulting system may, in turn, serves as the constituent element of another larger system.

Consequently, our main objective in this thesis is to provide an assembly-based programming framework for the implementation of complex distributed application to be deployed in the cloud. Specifically, our aim is to provide a domain specific language (DSL) that exploits self-organizing overlays to map at runtime a developer's high-level description of a complex distributed application onto a concrete infrastructure. Our approach relies on the scalability, resilience, and adaptability of self-organizing overlays to maintain a developer's target distributed application in the face of failures, scaling and dynamic adaptations. Further, our approach goes beyond traditional framework for distributed systems in that itconsiders individual systems as collective distributed entities} enforcing a given internal structure (a star, a tree, a ring) which developers can assemble programmatically to realize more complex topologies. It also goes beyond existing self-organizing overlays by supporting the description of a target topology as a composition of more elementary shapes, breaking away from the monolithic design of typical self-organizing overlay protocols.