Vous êtes ici

Learning to survive: Dynamic programming for autonomous and heterogeneous IoT networks

Equipe et encadrants
Département / Equipe: 
Site Web Equipe: 
Directeur de thèse
Olivier Berder
Co-directeur(s), co-encadrant(s)
Matthieu Gautier
Sujet de thèse

Due to the variety of Internet of Things (IoT) applications, wireless nodes form a very heterogeneous network, since they embed different sensors and radios, and therefore different computation abilities. To guarantee its sustainability, each node has to organize its different tasks (sense, process, transmit, receive, relay…) with respect to its environment, especially the energy that can be harvested. As a node has no prior knowledge of the energy cost of these controlled or event-based tasks, the latter has to be learned in real time. The role of the power manager is then to prioritize these tasks and allocate an energy budget to each of them, such that the consumed energy is equal to the harvested energy over a long period, which leads to Energy Neutral Operations (ENO).

In previous works, we designed generic power managers able to deal with the different energy sources [le13pimrc,le13enssys]. Our power manager adapts the duty cycle of the node according to the estimation of harvested energy and the consumed energy. To deal with the uncertainty linked to the energy sources, we recently proposed a new power manager based on fuzzy control [aitaoudia16icc, whose outstanding performances were evaluated through network simulations (Castalia/OMNeT++) and experiments on the PowWow WSN platform developed by the GRANIT team [granit13powwow].

Another way to design the power manager is to base on experience, especially thanks to reinforcement learning, which differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. There is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge), which particularly suits to energy harvesting mechanisms. The node will thus be able to track the harvested energy but also the energy costs to execute its different tasks (that can also be time-varying).

The first phase of this work will be dedicated to the bibliography study on Reinforcement Learning principles and its applicability to energy harvesting nodes [hsu12ieeetetc,xiao15icc]. The second phase will be the design of a RL-based power manager, that will adapt the energy budget to the residual energy in the battery. A particular attention will be given to the approximates of value functions, exploring different forms from weighted linear functions to outputs of neural networks. These different possibilities will be evaluated by simulations, and compared to state-of-the art power managers. Finally, the main contribution of this work will be to extend the proposed approach at a network level by learning the costs of the different tasks and by collaboratively deciding the best policies. The best strategy will be implemented on a real wireless sensor platform. A network composed of heterogeneous nodes will be deployed to confirm this superiority in terms of energy management and quality of service .


[akyildiz02jcn] I. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci.  Wireless sensor networks:  a survey.   Journal on Computer Networks, 38(4):393–422, 2002.

[alam11jes] M.-M. Alam,  O. Berder,  D. Ménard  et  O. Sentieys, A  Hybrid  Model  for  Accurate  Energy Analysis for WSN, EURASIP Journal on Embedded Systems, 2011.

[le13pimrc] T.N. Le,  A. Pégatoquet,  O. Sentieys, O.  Berder, C. Belleudy. Duty-Cycle Power Manager for Thermal-Powered  Wireless  Sensor  Networks,  in  24th  IEEE  International  Symposium  on  Personal,  Indoor and Mobile Radio Communications (PIMRC), Londres, United Kingdom, September 2013, pp. 1645-1649.

[granit13powwow] http://powwow.gforge.inria.fr/

[le13enssys] T.N. Le, M. Magno, A. Pégatoquet, O.  Berder, O. Sentieys, E. popovici. Ultra Low Power Asynchronous MAC Protocol using Wake-Up Radio for Energy Neutral Wireless Sensor Networks, in 1st International Workshop on Energy-Neutral Sensing Systems (ENSsys), Rome, Italy, November 2013

[alam12ieeejetcas] M.-M. Alam,  O. Berder,  D. Ménard  et  O. Sentieys, TAD-MAC: Traffic-Aware  Dynamic  MAC Protocol  for  Wireless  Body  Area  Sensor  Networks,  IEEE  Journal  on  Emerging  and  Selected Topics in Circuits and Systems, 2(1) :109-119, mars 2012.

[xiao15icc] Y. Xiao, Z. Han, D. Niyato, C. Yuen, Bayesian Reinforcement Learning for Energy Harvesting Communication Systems with Uncertainty, IEEE International Conference on Communications (ICC15), May 2015

[hsu12ieeetetc] R. C. Hsu, C. T. Liu, and H. L. Wang, “A Reinforcement Learning-Based ToD Provisioning Dynamic Power Management for Sustainable Operation of Energy Harvesting Wireless Sensor Node,” IEEE Transactions on Emerging Topics in Computing, vol. 2, no. 2, pp. 181–191, June 2014.

[park16ieeeaccess] T. Park, N. Abuzainab and W. Saad, "Learning How to Communicate in the Internet of Things: Finite Resources and Heterogeneity," in IEEE Access, vol. 4, no. , pp. 7063-7073, 2016

[aitaoudia16icc] F Aït Aoudia, M Gautier, O Berder, Fuzzy Power Management for Energy Harvesting Wireless Sensor Nodes, IEEE International Conference on Communications (ICC16), May 2016.

[aitaoudia16js] F. Aït Aoudia, M Gautier, O Berder, OPWUM: Opportunistic MAC Protocol Leveraging Wake-Up Receivers in WSNs, Journal of sensors, 2016

Début des travaux: 
Mots clés: 
Reinforcement learning, wireless sensor networks, energy harvesting, MAC protocols, power management
IRISA - Campus universitaire de Beaulieu, Rennes