Thinking out of the (Black)-Box Tools for audits in the presence of malicious model providers

Prediction services based on Machine Learning are now in use across industries, by companies, governments and individuals. Yet, these services often rely on a complex AI supply chain, whose components (training data, models, infrastructure), while critical to their performance, are partially or completely hidden to the final users. Thus, from the point of view of an external user or regulator, these prediction services appear as black boxes, complicating their evaluation and opening avenues for manipulations. In the presence of malicious model providers, this thesis aims at understanding the fundamental limits to back box auditing and designing protocols to provide guarantees beyond the black box interaction model. This manuscript presents three contributions towards that goal. First, I study the benefits of requesting the hypothesis class used by the platform to inform the audit. Then I present a formalization of this quest for the minimal assumption beyond the black box as a prior construction problem and provide a new audit method leveraging the labeled data available to the auditor. Finally, in an attempt to cheaply detect post audit attacks, I introduce a new model fingerprint baseline and theoretical analysis to detect model change.