Exposé : Séminaire de Statistique (STATQAM)

I am delighted to be invited to present our recent work with Agathe Fernandes Machado, Arthur Charpentier, François Hu and Emmanuel Flachaire at the StatQAM statistical seminar.

The presentation is mostly based on our working paper From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration.

To introduce the subject, I took an example from Arthur’s course on Fairness. Consider some pictures generated with a GAN (from https://www.nytimes.com/interactive/2020/11/21/science/artificial-intelligence-fake-people-faces.html), where the “gender” component varied. These pictures were then submitted to a classifier (https://www.picpurify.com/demo-face-gender-age.html) which predicts the gender of “people” it recognizes on images. The results are shown in Figure 1.

Figure 1: Male/Female Classification, using picpurify with images from Hill and White (2020).

What is striking with this example, is that the algorithm seems to be very confident in its predictions. Can the scores it returns be considered as probabilities?

In this talk, I present different ways to visualize calibration vor a binary classifier. Then I introduce our new metric based on local regression: the Local Calibration Score. I then show the impact of poor calibration on standard performance metrics. Lastly, I talk about an ongoing project related to calibration for tree-based methods.

The slides are available on my website: Slides

References

Hill, K., and White, J. (2020). Designed to deceive: Do these people look real to you? The New York Times 11.