watty62 / Count_twitter_followers

This is a scraper that runs on Morph. To get started see the documentation.

The purpose of this scraper is to scrape the number of followers we have for a number of our corporate accounts and store these in a database table.

The scraper.py file is in two parts.

Section A is a set-up routine which creates the database table, assigns columns, then writes in data in the new format from some lists which I generated from the followers.csv file. I chose to do this as I was unable to import and use the CSV library that I had used on my home laptop.

If you want to experiment with this, uncomment Section A, and comment out (using ''') section B.

Section B is the scraper itself.

It first checks the date - and only runs if it is the 1st of the month.

Then it processes a list of twitter account names, constructing full URLs and scraping the number of followers for each from the twitter page. The actual figure only appears as a tooltip mouse-over - and I had some help working out how to get it out of the code!

Once it has the number of followers per account it writes these off to the database.

When I wrote this (30/08/14 - or 20140830 as the script would call it!) the code hadn't executed for real - although it has all been tested in chunks.

Since then it ran as planned in Sept and Oct 2014!

It subsequently failed in November. I traced it to a Twitter account having being closed down. The result was that there was no valid page available for checking followers. So better error trapping would be good - someday!

If you have any questions you can contact me on Twitter: @watty62