Subscribe To Our Newsletter

Get tips and tools to tell your data story better.

No, thanks

 In Uncategorized

If you’re interested in data and content marketing, you’ve likely heard about “scraping Twitter.” This is a term that means using some type of tool (usually a computer program) to extract useful data from Twitter and getting that data into a format that is more useful to you. Most of the time this is accomplished by writing code in a programming language. But there are ways to do it with no programming at all!  Here’s a guide on how to get data out of Twitter using skills no more complex than searching on Google and cutting and pasting in Excel. We also give you an example of why you might want to get data out of Twitter.

There are many great things you can do with Twitter data in terms of understanding global conversations on topics that matter to you, getting the word out on your own mission, and understanding your community. For our tutorial we’re going to scrape Twitter for data that we can use to promote our awareness campaign on the underfunding of Alzheimer’s disease. We have a series of infographics that we would like to get seen by a target audience of funders, policy makers and advocates in the Alzheimer’s community. However, we don’t have a list of these people and we need to find them on Twitter and create a targeted audience campaign including their data.

Step 1: Google Search
Do a Google search that searches the Twitter site for curated lists containing your key audience.
Simply cut and paste the following text into the Google search engine substituting your keywords for ours.

site:twitter.com  inurl:lists  inurl:members  inurl:your keyword

1 Twitter Scraping-Google Search

You will get a few pages of results that look something like this. It will be a series of Twitter lists that have been pre-curated to include influential and active people in your area of interest. These lists contain the data on the people you’re looking for. Now all you have to do is to get that data!

2 Twitter Scraping-Google Search

An example of the valuable information inside a curated Twitter list…

3 Twitter Scraping-Search results


Step 2: Copy the web address of your search
At the top of your web browser is the specific “web address” of the search you just did. It’s officially called the URL. Copy this entire string of text. You don’t have to understand what it means for our purposes. Just copy the whole thing.
(I’m using Chrome here. If you want your screen to look exactly like mine, use Chrome too. But it should work on any browser).

4 Twitter Scraping-Copy Search URL

Step 3: Open a website called Import.io
Open a new tab in your browser and navigate to Iport.io (https://import.io/) It’s free and there’s nothing to download or sign up for in terms of this tutorial. It does have other advanced features we can use in a later tutorial.

5 Twitter Scraping-Import

Step 4: Paste your search url into the Iport.io page
Take the web address you copied in Step 2 and paste it into the large white box on the Import.io page.

6 Twitter Scraping-Past iPort

Step 5: Click on the pink “Get Data” button
You will now have a beautifully organized list of links to curated Twitter lists of key influencers in your area.

7 Twitter Scraping-Get Data

Step 6: Download the Data
At the bottom left of the page there are several buttons. One says “Download”. This is the one to click.

8 Twitter Scraping-Download
Import.io will ask you if you’d like to download only the first page or all the pages of data you have. Choose all the pages.

9 Twitter Scraping-Choose All Pages

 

 

10 Twitter Scraping-Download
The data you’ve scraped will download as a csv file to wherever your computer stores downloads. It will be called “magic” so if you can’t find the file, search your computer for files called “magic”.

11 Twitter Scraping-Search Magic file


Step 7: Open the scraped data in Excel
Using Excel, open the “magic” file of scraped data. This file is a csv file that Excel can easily read and open as a familiar looking spreadsheet. Take a quick look at the spreadsheet – it has all the information you need for the next step. The first column is the list of links that go directly to each list of key influencers. These lists contain the Twitter usernames that you ultimately need to know.

12 Twitter Scraping-Excel output


Step 8: Copy the first link
The direct links to the lists are found in column A of the spreadsheet. For this example, we’re using the first link in the column. You can choose whichever links in column A that seem the most useful to you. You don’t have to use the first one.

13 Twitter Scraping-Copy first link


Step 9: Paste that link address into Import.io
Return to the tab with the Import.io website open. Paste the direct link you just copied into the same white box as last time, replacing the previous text you’d pasted in.

14 Twitter Scraping-Paste Link


Step 10: Click on the pink “Get Data” button
Now you have a detailed list of data on all the members of the first list in your search. This data includes the Twitter usernames of handpicked key influencers in your field. This username data is all you need to create a targeted audience for yourself on twitter.

15 Twitter Scraping-Get Data


Step 11: Download the Data
This step is the same as before. At the bottom left of the page there are several buttons. One says “Download”. This is the one to click.

16 Twitter Scraping-Download


Step 12: Open the scraped data in Excel
This round of data is also saved to your computer with the filename “magic”.

17 Twitter Scraping-Open Data


Step 13: Copy the usernames and paste into new Excel sheet
The data you will use from this spreadsheet is the Twitter usernames. In this example they are found in column J of the spreadsheet you just opened.  Copy this column and paste the usernames into a new, blank Excel spreadsheet.

18 Twitter Scraping-Copy-Paste users

Step 14: Save as a .csv file
Once you’ve pasted the list of usernames into the spreadsheet, give it a useful name and save as a .csv. This is an option in the Save As menu.

19 Twitter Scraping-User names


Step 15: Repeat Scraping until you have all the Twitter usernames you’d like
Steps 8 through 13 can be easily repeated, with you copying the direct link to a new curated Twitter list each time. You can choose the lists that look the most useful for your purposes or you can copy them all in order. These steps only involve copying and pasting links. If you plan to use the data to create a custom targeted audience in Twitter, you need at least 500 usernames. In our experiments, this took about 20 minutes.

Step 16: Import your scraped data into Twitter!
Now that you’ve scraped all the data you need, you can use it in any way that works best for you. One option is to upload the list of Twitter usernames into the Twitter targeted audiences feature. Then you can use this to send custom-designed tweets to influencers in your field.

To import your data and create a custom target audience, go to the Twitter Ads section -> Create a New Campaign. Within the New Campaign, choose to “Add Tailored Audience” and “Upload your own list”.

Then name your audience, choose the “Twitter usernames” option and upload the csv file containing all your scraped Twitter names. Uncheck the box that says “The records in this file are already normalized…..”

20 Twitter Scraping-Twitter import

21 Twitter Scraping-all campaigns22 Twitter Scraping-Campaign detail23 Twitter Scraping-audiences 24 Twitter Scraping-audience list
Congratulations!
You’ve successfully scraped Twitter and made good use of your new data!!

Stay tuned for more “How to” tutorials each month here at Datassist…and take time right now to:
Sign up for monthly data science and visualization resources.
Take a look at some of the work we have done.
Friend us on Facebook for a personal connection with what we do.
Follow us on Twitter for the latest news.
Get inspired with us on Pinterest.

Recommended Posts

Start typing and press Enter to search