Newsbox

The Big Idea

There are a lot of topics that people from different regions react differently to. Whether it is a political topic, a slang that is local to certain region, etc, the chance is people from different region will have different attitude toward certain topics. NewsBox is a program that captures this information in visually compelling way, through a choropleth map.

Getting Started

NewsBox uses a number of dependencies that requires installation before running the program. First install, python-twitter module, which is used to fetch tweets.

$ pip install python-twitter

Then follow install indico PyPi package, which is used to run a sentiment analysis.

$ pip install indicoio

You will have to get api keys for Twitter API and Indico API. Create your own application in Twitter Apps page and get API key. You can follow indico's installation guide to get indico api key. Last step in setting up NewsBox is to create a configuration file. Create config.py with following code in it.

            indico_key = '<Your Indico API Key>'
            consumer_key = '<Your Twitter Consumer Key (API Key)>'
            indico_secret = '<Your Twitter Consumer Secret (API Secret)>'
            access_token_key = '<Your Twitter Access token>'
            access_token_secret = '<Your Twitter Access token secret>'
            

Running program

How NewsBox Works

The beauty of NewsBox is its structural simplicity. As you can see in the very high level code structure diagram below, there are only three major components of the program. First component of the program is collection of tweets based on specific search term and geographical region, performed using Twitter API and python-twitter Twitter API wrapper for python. After compiling tweets using Twitter API, Indico's machine learning API is used to run a sentiment analysis on tweets. Lastly, NewsBox generates a SVG (scalable vector graphics) file of a choropleth map that represents the different sentiment toward certain keyword in the different regions of the US.

Results

When you give a NewsBox a keyword to run sentiment analysis on, it will display its result in a choropleth map. Different shading of the 4 regions of the US (Northeast, West, South, and Midwest) represent different level of sentiment (how positive that region is) toward the keyword. Red corresponds to positive sentiment and blue corresponds to negative sentiment. The map also displays the numerical value of how the sentiment, ranging from 0 (negative) to 1 (positive).
Following is the few examples generated by NewsBox.
Steph Curry
Democrat
Tesla

Evolution of NewsBox

NewsBox wasn't always the glamorous tweet-sentiment-choropleth-map-producing program you see now. Once upon a time, NewsBox had a very different mission: We wanted to see if there was a causal relationship between the positivity of news sources and public opinion towards different topics. We even created a mock-up of what our program might produce.

After our first technical review, we came up with a better idea. Rather than produce line graphs of average media and Twitter polarity over time towards a certain search term, we could produce a choropleth map that compared the polarity of tweets from different regions in the US. We scrapped the News part of NewsBox and began working on our new idea. We started with a blank map of the US in the form of an SVG file.

We made changes to our original code so we could collect Tweets based on location, and we wrote new code to produce a choropleth map given average polarity values for the four main regions of the US.

We went into our second Technical Review looking mainly for feedback on how we should improve the data visualization part of our project (How can we make the map pretty?). We received and incorporated a number of good suggestions, such as adding numberical values to the map, making the map pop up when the program runs, and adding the search term to the map picture (so when you save the map, you know what search term it represents). These suggestions resulted in a much improved visualization of our data, as you can see below.

Our stretch goal was to make a web application, and since we still had some time left before our project was due, we decided to give it a try. We created a simple web app; the relatively tricky part was that the picture we wanted to display (the map) changed every time the program ran, but after much googling, we figured out how to make it work. The first picture below shows the first page of the web application, and the second page shows the page that pops up after someone entered the search term 'Trump'.

Thanks for reading about NewsBox! If you have any questions, feel free to contact us, Mary Keenan and Sung Woo Park, at mary.keenan@students.olin.edu and sungwoo.park@students.olin.edu, respectively.

Credits

• Twitter's API wrapper for Python, python-twitter fetches the tweets.
• Indico's API analyzes the sentiment of tweets.
• Beautiful Soup edits the SVG file.
• Flask creates the web application.