Predicting the Oscars with social media (and the problem with Her)

Her joaquin phoenixThere is a problem with Her. No, not her. I mean… Her. You see Her there, but she’s hard to figure out. You’ve been nervous about Her. At first you try not to think about Her, but you can’t ignore Her…you’re not allowed to ignore Her. So what do you do? You start to think of Her in a way you don’t think about the others. Then, you try a new version of Her. Or just forget about Her completely, and move on.

And so begins the quest of many social media analysts doing research on the 2014 Academy Awards.

Every year, tech and entertainment blogs as well as social media companies use social media to try to predict the Oscars winners. Using a variety of tools and resources, analysts usually make their predictions based on movie or actor mentions. Sometimes they build out extended Boolean or keyword queries for searches.

Most of the time, movie titles are unique enough to do this. But this year, several movies like Her and Gravity have generic names, making this task close to impossible. And that’s only the beginning…

The pitfalls of social media research

Inaccuracy and dissimilar data-sets spoil sentiment
A number of analysts use metrics like Facebook fan base sizes or automated sentiment tools – to no avail. Some movies do not even have social accounts, and others that have been in theatres longer might naturally have larger followings due to their life-span. Apart from multiple accuracy issues, automated sentiment predictions are usually based on similar title-only queries that can prove difficult to search against, like Her.

More query issues!
Popular movies are more likely to receive more mentions than other nominees, no matter what. When in 2009, Avatar became the highest grossing movie of all-time; it had many more Twitter mentions than competitor Hurt Locker, because of its blockbuster scale. But it still lost out at the Oscars that year.

Most social research around nominees comes from just the title of the movie or actor mentions. This only proves the popularity of the movie or actor individually over the award context.

In addition to this conundrum, social media analysts build their own search queries, resulting in different results from report to report. Just think of the issue with Her and the millions of random messages that would show up. This one, for instance includes mentions of “her” and “best picture”.

While various queries from different analysts cannot be avoided, the queries used in predictions can certainly be improved. Rather than basic movie title searches or hashtag mentions (which some companies are using for Her and Gravity this year); queries used in predictions should be contextually based on the Oscar category they are in. For instance, Gravity’s Emmanuel Lubezki’s cinematography nomination may be searched for as: (gravity OR lubezki) AND (cinematography OR bestcinematography).

Timeframes can affect everything
Collecting data before nominations are announced is one mistake. Another is to collect on any days that immediately follow the nomination announcement. The former does not reflect actual award discussions, and the latter leads to massively skewed numbers that are mostly reflective of news sharing, and not opinions.

A safe practice is to have any collection timeframe start the week after announcements to avoid these spikes. As for the ending of the timeframe, that’s up to the researcher. When deciding upon this, think about other award shows that occur before the Oscars, especially the BAFTAs, as those can affect both online opinions and actual Academy voting.

Overall mentions aren’t always the best for Twitter predictions
A final thing to consider, when collecting data, is how to make your final picks. While most predictions are based on total message counts, a linear trend of mentions is more likely to show hot picks and movers. It also takes into account more recent award shows. Therefore, a combination of linear trends and overall totals might be your safest bet.

Where to go from here?

All of these concerns show that any prediction based on social media should be understood as an educated guess. That being said, here’s the Essence methodology for the Oscar predictions below. And like all “predictions” based on Twitter data, these are more similar to findings than actual predictions (unless we get all of them right; then they were totally predictions)!

Using Twitter data and context-based categorical queries built using tools from the social monitoring company Sysomos, Essence looked at mentions of each nominee in two timeframes between the announcements and BAFTAs to capture data. For each of the two time frames, we used a linear trend metric and one based on overall mention count, coming up with four total points, this way capturing early and late favourites. Whichever nominee led most of those four points was our “prediction.”

Also, we didn’t just do this for the standard four or five categories, we went big and did this for all twenty-four Academy Award categories. We also included the top choice from EasyOdds, which compiles odds from up to twelve other betting sites: so we could compare our Twitter results against real-life odds on favourites.

So take a look, share your own predictions, and leave a comment about any other social metrics that you think could or should not be considered for predictions.

Most of all though, enjoy the show on Sunday!

And without further ado…

 Category

1st Pick

2nd Pick

Odds

Picture

12 Years a Slave

Gravity

12 Years a Slave

Actor

DiCaprio

Ejiofor

McConaughey

Actress

Blanchett

Bullock

Blanchett

Supporting Actor

Leto

Abdi

Leto

Supporting Actress

Lawrence

Nyong’o

Nyong’o

Animated Feature

Frozen

The Wind Rises

Frozen

Cinematography

Gravity

Prisoners

Gravity

Costume Design

Great Gatsby

American Hustle

Great Gatsby

Director

Cuaron

McQueen

Cuaron

Documentary

The Act of Killing

The Square

The Act of Killing

Documentary Short

Prison Terminal

The Cave Digger

The Lady in Number 6

Film Editing

Gravity

12 Years a Slave

Gravity

Foreign Language Film

The Great Beauty

The Missing Picture

The Great Beauty

Makeup and Hair

Dallas Buyers Club

Bad Grandpa

Dallas Buyers Club

Original Score

Her

Gravity

Gravity

Original Song

Happy

Let it Go

Let it Go

Production Design

The Great Gatsby

Her

The Great Gatsby

Animated Short

Get a Horse

Feral

Get a Horse

Live Short

Aquel No Era Yo

Helium

The Voorman Problem

Sound Editing

Gravity

Captain Phillips

Gravity

Sound Mixing

Gravity

Lone Survivor

Gravity

Visual Effects

Gravity

Iron Man 3

Gravity

Adapted Screenplay

Philomena

Captain Phillips

12 Years a Slave

Original Screenplay

Her

American Hustle

Her

Andrew Panos is a social media analyst at Essence.