Is diversity a recipe for success?

Why this matters

Television and especially films play a crucial role in our modern society. The main characters in films serve as role models and inspire the younger generation, who may spend up to 78,000 hours watching television in their lifetime, as a survey of 2,000 British adults has shown.
Furthermore, the growing public interest in the issue of representativeness has led us to study it through the cinema, believing that it plays a crucial role in achieving a more inclusive society.
To do this, we use a large database of films, and construct two indicators to measure gender and ethnic representation within a film. We then look at the evolution of the latter over time, across the main film genres, as well as across the different countries of the world. Finally, we will try to show how representativeness is perceived by the general public, by analysing the reviews and the box-office revenues of films.

Research questions

We focus our analysis on two main interrogations :

How is representativity distributed across cinematography genre, countries of production and time ?

The theme of representativeness has become increasingly important in the public sphere in recent years, but can this trend be seen in film? Are there regional anomalies? Are there discrepancies between film genres? We will try to answer these questions. In this section we will take a descriptive approach in order to highlight the different aspects of representativeness in the cinema.

How does representativity impact a movie’s critical and commercial success?

Our first question looks at differences in representation within the film industry, but how are they perceived? Do they affect public opinion? Does this affect the box-office scores of the films? These are the questions we will try to answer in this second part. With an analytical approach this time, we will try to measure the impact that representativeness can have on the cinema and on the public.

Our data

To conduct our analysis, we used the CMU Movie Summary Corpus. This is a huge dataset with information on more than 80'000 films, 450'000 characters and more than 40'000 summaries. We will guide you through the type of information available and all the manipulations we have done using the Christmas classic Love Actually as an example. The dataset is separated in three categories:

Movies metadata

A table with general information about the films. Moreover, it contains wikidata identifiers, which allowed us to enrich the data with an additional entry: The Rotten Tomato score. Rotten Tomatoes is an American website that gathers professional reviews of a film, then classifies them between fresh and rotten. The Tomatometer is then the freshness attributed to the film. In order to ensure that we had enough data to carry out an analysis, we also collected the number of reviews made.

Characters metadata

A table with different attributes about the actors and their film characters. However, the ethnicities given for the actors were sometimes extremely specific and other times far too general. They were not usable for a meaningful analysis. We therefore grouped them into nine broad geographical categories, and a tenth to take into account unclassifiable minorities.

Plot summaries

English synopses from Wikipedia. In addition, in order to refine our future analyses, we analysed the lexical field used in the summaries in order to assign a masculinity and femininity score to the summary. For this purpose, we used the empath python library.

Metrics

To answer our research questions, we first need to have a way to quantify the diversity in movies. For this purpose we created some custom metrics :

The actor score

The first thing we needed to measure ethnic and gender diversity was to be able to quantify the importance of the role played by an actor. Do they play a leading role, or is it only a minor role? To do this, we analysed the summary of the film and awarded points each time a character was mentioned, and proportionally to the order of appearance. In addition, we used the natural language processing tool neuralcoref based on spaCy to analyse the summaries, find the character's co-references and replace them with their names, as highlighted in the example below.

Mathematical details

We construct the actor score as follows:

- Each character starts with 2 points (we will perform a logarithm and we did not want to have the case log(1) = 0).

- We then give additional points each time a character is referenced in the summary. 1 point if it is the last reference in the text, two points if it is the second last, etc.

- We calculate the actor score for each character as follows: [log(number of points)]/[total points], in order to have :

1. actor score ∈ [0,1]
2. ∑ actor score = 1

Original summary

On his eleventh birthday, Harry Potter discovers that he is no ordinary boy. Hagrid, a beetle-eyed giant, tells Harry that he is a wizard and has a place at Hogwarts School of Witchcraft and Wizardry, directed by the mysterious professor Dumbledore. In his first year of magical education, with his friends Hermione and Ron, Harry tackles a fully grown mountain troll, learns to play Quidditch, and participates in a thrilling "live" game of chess.

Processed summary

On his eleventh birthday, Harry discovers that Harry is no ordinary boy. Hagrid, a beetle-eyed giant, tells Harry that Harry is a wizard and has a place at Hogwarts School of Witchcraft and Wizardry, directed by the mysterious professor Dumbledore. In his first year of magical education, with his friends Hermione and Ron, Harry tackles a fully grown mountain troll, learns to play Quidditch, and participates in a thrilling "live" game of chess.

Gender metric

The gender metric represents the parity of gender in a film. The more equitable a film is in its gender representation, the higher its score will be. Note that we only considered the genders already present in the dataset, i.e. male and female. We used three factors to construct this score :

• The ratio of male/female actors
• The ratio of the importance of female/male roles
• The difference in the masculinity/femininity scores of the summaries

Mathematical details

g ∈ [0,1] : the gender score asociated to a movie i

Number of male actor in movie i : m_i
Number of female actress in movie i : f_i
Ratio of female vs total number of actor in movie i : r_i ∈ [0,1]

Cummulated importance of male actor in movie i : mp_i
Cummulated importance of female actress in movie i : fp_i
Ratio of cumulated importance of female actress vs total roles importance rp_i

Masculinity of speech in summary of movie i : ms_i
Feminility of speech in summary of movie i : fs_i

r_i ∈ [0,1] : maximum gender diversity is at 0.5
(1 - 2 ∙ r_i) ∈[-1,1] : maximum gender diversity is at 0
|(1 - 2 ∙ r_i)| ∈ [0,1] : maximum gender diversity is at 0
1 - |(1 - 2 ∙ r_i)| ∈ [0,1] : maximum gender diversity is at 1
This part pounder gender score by proportion of gender represented

rp_i ∈ [0,1] : maximum gender diversity is at 0.5 /
(1 - 2 ∙ rp_i) ∈ [-1,1] : maximum gender diversity is at 0 /
|(1 - 2 ∙ rp_i)| ∈ [0,1] : maximum gender diversity is at 0 \ /
1 - |(1 - 2 ∙ rp_i)| ∈ [0,1] : maximum gender diversity is at 1 /\
This part pounder gender score by relative importance of the role they play

fs_i - ms_i : maximum diversity is at 0 /
|fs_i - ms_i| : maximum diversity is at 0 \/
1 - |fs_i - ms_i| : maximum diversity is at 1 /\
This part pounder gender score by the empath score of the summary

g = (1 - |(1 - 2 ∙ r_i)|) ∙ (1 - |(1 - 2 ∙ rp_i)|) ∙ (1 - |fs_i - ms_i|)

Ethnicity metric

This metric aims to illustrate the ethnic diversity represented in a film. To calculate this value, we take into account three main factors

• The number of different ethnicities present in the film in ratio to the number of actors
• The relative importance of these ethnicities in terms of the number of actors (representing a minority is worth more points than representing a majority)
• The visibility of minorities in leading roles

Mathematical details

ethnicity score : s ∈ [0,1]
number of ethnicity in a movie i : e_i
number of actor in a movie i : a_i
ethnicity particular score : p ∈ [0,1]

e_i / (min(e_tot, a_i)) : This term account for number of ethnicity represented (linear in number) in a movie ∈ [0,1]

∑p_j ∈ [0,1] : This termn account for ethnicity relative importance. The more an ethnicity is represented, the less representative value it has.

∑ (min(p_j - |∑ r_jki - p_j|,0)) : This term account for minority visibility in main roles.

s_i = [ (e_i / min(e_tot, a_i)) + ∑ p_j+ ∑ (max(p_j - |∑ r_jki - p_j|,0)) ] / 3

ethnicity particluar score :
p_j ∈ [0,1] , p_j is the largest for ethnicity that are the least represented.
a_j : number of roles/characters of a specific ethnicity (in the whole dataset)
r_jk : Importance of the role k

p_j = [1/ln(a_j)] / [∑ 1/ln(a_l)] ≡ p_j = 1 / [ ln(a_j) ∙ ∑ (1 / ln(a_l) ) ]

Thus we have : p_j ∈ [0,1] and ∑ p_j = 1

Importance of actor role :
r_ki : Importance of the role k in the movie i
∑r_jki : Cumulated importance of the roles played by acotors of ethnicity j in the movie i
∑ r_jki - p_j : Maximum ponderation for specific ethnicity at 0 /
|∑ r_jki - p_j| : Maximum ponderation for specific ethnicity at 0 \ /
p_j - |∑ r_jki - p_j| : Maximum ponderation for specific ethnicity at p_j / \
max(p_j - |∑ r_jki - p_j|,0) : Maximum ponderation for specific ethnicity at p_j / \

How is representativity distributed across cinematography genre, countries of production and time?

Release date's influence

We are first interested at how the scores evolved through time. The average values of the gender diversity metric and ethnicity diversity metric are displayed in the graph below.
Two trends can be seen to emerge. On the one hand, ethnic diversity seems to be decreasing slightly or maybe even stabilising over the years, while on the other hand, we can clearly see the diversity of the genre increasing, having almost tripled in less than a century.
An attempt can be made to explain some of the decline in ethnic diversity by the growth of Indian cinema with the rise of Bollywood in particular and the resulting large increase in actors of South Asian ethnicity. This cinema is naturally at a disadvantage in terms of ethnic representation because it has only one predominant ethnicity, unlike American cinema, which can have two: North American and African-American. In addition, Indian cinema has a large number of actors, thus it does not represent a minority and therefore makes especially few points.
On the contrary, we did not find an obvious explanation for the increase in gender diversity, so this would rather suggest a societal change, which is good news for the representation of diversity. Later we shall try to measure if this effect is visible in commercial or critical success of a movie.

To visualise the evolution of the different genres of films produced, the ethnicity of the actors involved and the main countries of production over time, we have created the interactive graph below. By double clicking on the title of a data item, it is possible to select it in order to better visualise its evolution over time. Also it is normalised to see the percentage distribution of our data.
This graph represents the evolution of world cinema, with all the biases associated with the method and completeness with which the CMU Movie Summary Corpus data was collected. The emergence of Bollywood cinema, and the associated growth in the number of South Asian actors, can easlily be observed, as well as the decline of black and white films. Interestingly enough, one can see the golden age of Italian cinema in the mid sixties where at its peak it represented 25% of the world film production.

How does representativity impact a movie’s critical and commercial success?

In order to do the following analysis, we will use two additional features : the Rotten Tomatoes scores and the box office revenue.
The Rotten Tomatoes scores represent the success of the movies among critics. Whereas, the box office revenue represents the commercial success.

We will conduct an observational study to understand if the representativity in movies influences their success.
The goal of the observational study is to eliminate all the elements that could influence the success of movies except the one we're interseted in : representativity of course. These parasitic elements are called cofounders and the method to remove them is called matching. In our case, we will remove the following cofounders: the country of production of a movie, the year of the production and the genre of the movie.
Once the cofounders are eliminated, we will finally be able to know if diversity in movies influence their success.

The effect of representativity

In this first part, we'll use exact matching : a movie with low diversity and a movie with high diversity are matched if they have exactly the same genre, country of production and year of production, in order to make an unbiased comparison.

The left-side graph shows the impact of diversity on the Rotten Tomatoes score, which represents the film's critical reviews. It can be seen that on average films with greater diversity are more successful, with for example an average of 4 points more on reviews of films with high ethnic diversity. However, it is important to note that this effect is not statistically significant, as the uncertainties are far too great.
The right-hand chart illustrates the impact of diversity on box-office revenues. This time the trend is clearer and the uncertainties lower. There is even a statistically significant difference for the impact of gender diversity, suggesting that film with higher gender diversity are indeed more successful, at least in term of commercial success.

You can hover over the plot to see if you find your favorite movie!

Time evolution of the effet of representativity

We now want to analyze if the influence of representativity in the success of movies evolves with time. In order to do so, we need to consider movies year by year. Therefore, we manipulate less data at once. As we can not afford to loose much data points, we will use propensity score matching. It is a more complex but less strict method than exact matching.

The graph below shows the evolution of the difference in Rotten Tomatoes score between films with high diversity versus films with low diversity. The scale is directly the number of additional percent obtained by movies with higher diversity. One can observe that there is no clear trend of a potential effect of ethnic or gender diversity on the film reviews given by the Rotten Tomatoes score. We see fluctuations from year to year, linked to the limited number of films available, while the average seems to remain stable around zero.
We can see that the effect on ethnic diversity this time is slightly below zero, which contrasts with our previous results. We explain this slight difference by the method of matching (propensity and no longer exact) which has changed. In any case, the effect is not statistically significant enough for us to conclude that the effect is indeed present.

The graph below represents the difference in the average log revenue of high diversity films compared to low diversity films. For example, a difference of 0.3 means that films with high diversity, on average, earned about 3 million dollars more in revenue.
Despite the annual differences, it can be seen that there is no significant change in the effects over the twenty years considered. On the other hand, although there is no trend, we observe that the average is systematically higher than zero, which supports our previous analysis suggesting the presence of a diversity effect on the commercial success of a film.

Conclusion

Through this project, we explored the influence of gender and ethnic diversity in the movie industry. To this end, we built two metrics: the gender diversity metric and the ethnicity diversity metric. We then answered two broad research questions:

How is representativity distributed across cinematography genre, countries of production and time ?

The most interesting result we obtained when answering this question is the time evolution of the representativity. Indeed, in the last 50 years, the average gender representativity has been multiplied by 3 ! It was an interesting yet pleasing discovery, even though the way to equity is still long. On the other hand, the ethnicity score has slightly reduced over the same time window.

How does representativity impact a movie’s critical and commercial success?

All our results were in the encouraging direction of a more diverse and equitable cinema, although the only statistically significant result we obtained is the impact of gender diversity on box office revenue.

These two results combined give us some very interesting insights.
As we said, the success of movies is more impacted by gender diversity than it is by ethnicity diversity. We also observed that only the gender score increases over time. This is good news for gender diversity! There is still plenty of work to be done because we're still far from the objective, but we are going in the right direction. On the contrary, our results show that the problem of the lack of ethnicity diversity is not yet ready to be solved in the cinema industry.