nloveladyallen / github_trending

Github Trending Repositories


GitHub Trending Scraper

It scrapes the GitHub trending page!

This script uses Nokogiri to scrape https://github.com/trending, then gets info on each repo using the GitHub API. It is a morph.io-compatible scraper, meaning you can use it as an API as well as locally.

Setup

$ git clone https://github.com/nloveladyallen/github_trending
$ cd github_trending
$ bundle install

Usage

Locally

$ export MORPH_UNAME='yourgithubusername'
$ export MORPH_PWD='yourgithubpassword'
$ ./scraper.rb

Creates data.sqlite, containing one table, data, with one column for every part of the GitHub API and one row for every trending item (there are 25 at any given time).

API

Data from this scraper is available from morph.io. You must have a morph.io API key. Signing up is free, and only requires a GitHub account. A morph.io request looks like this:

GET https://api.morph.io/nloveladyallen/github_trending/data.[format]?key=[api_key]&query=[sql]

See the morph.io API documentation for more details.

Contributors nloveladyallen

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling...

Statistics

Average successful run time: less than a minute

Total run time: about 1 month

Total cpu time used: 9 minutes

Total disk space used: 48.2 KB

History

  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    17 pages scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    566 pages scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    25 records removed in the database
    1 page scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    742 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    56 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    53 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    4 records added, 21 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    4 records removed, 21 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    2 records added, 23 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    10 records added, 13 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    5 records added, 8 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    17 records removed, 8 records updated in the database
    52 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    49 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    16 records added, 9 records updated in the database
  • Auto ran revision 57c8733b and failed .
    16 records removed, 9 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    9 records added, 16 records updated in the database
  • Auto ran revision 57c8733b and failed .
    5 records added, 11 records updated in the database
  • Auto ran revision 57c8733b and failed .
    14 records removed, 11 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    1 record added, 24 records updated in the database
  • Auto ran revision 57c8733b and failed .
    1 record removed, 24 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    17 records added, 8 records updated in the database
  • Auto ran revision cccc3892 and failed .
    10 records removed, 8 records updated in the database
  • Auto ran revision cccc3892 and failed .
    7 records removed, 18 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    5 records added, 20 records updated in the database
  • Auto ran revision cccc3892 and failed .
    5 records removed, 20 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    5 records added, 20 records updated in the database
  • Auto ran revision cccc3892 and failed .
    5 records removed, 20 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    6 records added, 19 records updated in the database
  • Auto ran revision cccc3892 and failed .
    3 records removed, 19 records updated in the database
  • Auto ran revision cccc3892 and failed .
    3 records removed, 22 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Manually ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Manually ran revision cccc3892 and completed successfully .
    25 records added in the database
  • Manually ran revision cccc3892 and failed .
    nothing changed in the database
  • Created on morph.io

Scraper code

Ruby

github_trending / scraper.rb