Using deep learning to predict ideology from facial photographs: expressions, beauty, and extra-facial information

“Our results confirmed the threat to privacy posed by deep learning approaches. Using a pre-developed and readily available network that was trained and validated exclusively on publicly available data, we were able to predict the ideology of the pictured person roughly 60% of the time in two samples. We also found that existing practices, in which non-facial information is typically included in the small boxes surrounding a face, could produce inflated accuracies: Heat mapping results showed our CNN was originally using information other than the face to classify males. Had we not deleted this information our accuracy would have been higher.

We also provide the first demonstration that model-predicted ideology connects to independently classifiable features of the face. For females (though not males), high attractiveness scores were found among those the model identified as likely to be conservative. These results are credible given that previous research using human raters has also highlighted a link between attractiveness and conservatism¹⁰. Recall that we use photographs provided by the candidates themselves while campaigning for (low-level) political office. Because attractiveness generally helps electoral success, all candidates are incentivized to provide an attractive photograph¹⁰. Further, the amateur nature of these candidates is a strength because the photos represent candidates for office where selective pressures are rather low. Consequently, we are less (which is not to say zero) concerned that the relationship results from a particularly intensive selection of attractive candidates among those on the right.

Attractiveness was not the only correlate of model-predicted ideology. We also found that expressing happiness is associated with conservatism for both genders. Previous work has found smiling in photographs to be a valid indicator of extraversion³², and while extraversion is not broadly associated with ideology³³, some studies have found that right-wing politicians are more extraverted³⁴; though see³⁵. Self-presentation should also be considered. Politicians on the left and right may have different incentives for smiling—for example, smiling faces have been found to look more attractive³⁶, which is comparatively important for conservative politicians. Future work is needed to explore the extent to which happy faces are indicative of conservatism outside of samples of politicians.

While the use of a politician sample represents a potential constraint on generality, the use of candidates for local office in Denmark, attenuates this constraint. First, because selective pressures are relatively small on this candidates and secondly because prior work on attractiveness and politics reports similar findings across different highly developed political contexts¹⁰.

“Future work might explore why our heat maps showed the eyes and mouth regions to be important for the classification of female ideology. While the relevance of these regions to identify things such as whether a face is happy may seem straightforward, happiness correlated with model predicted ideology similarly for males and females, so why a given facial region would be more diagnostic for one gender than another is not immediately obvious.” Future work might explore why our heat maps showed the eyes and mouth regions to be important for the classification of female ideology. While the relevance of these regions to identify things such as whether a face is happy may seem straightforward, happiness correlated with model predicted ideology similarly for males and females, so why a given facial region would be more diagnostic for one gender than another is not immediately obvious.”