nloveladyallen / github_trending

Github Trending Repositories

Scrapes github.com

Build software better, together.


GitHub Trending Scraper

It scrapes the GitHub trending page!

This script uses Nokogiri to scrape https://github.com/trending, then gets info on each repo using the GitHub API. It is a morph.io-compatible scraper, meaning you can use it as an API as well as locally.

Setup

$ git clone https://github.com/nloveladyallen/github_trending
$ cd github_trending
$ bundle install

Usage

Locally

$ export MORPH_UNAME='yourgithubusername'
$ export MORPH_PWD='yourgithubpassword'
$ ./scraper.rb

Creates data.sqlite, containing one table, data, with one column for every part of the GitHub API and one row for every trending item (there are 25 at any given time).

API

Data from this scraper is available from morph.io. You must have a morph.io API key. Signing up is free, and only requires a GitHub account. A morph.io request looks like this:

GET https://api.morph.io/nloveladyallen/github_trending/data.[format]?key=[api_key]&query=[sql]

See the morph.io API documentation for more details.

Contributors nloveladyallen

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.2.4 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/..  Installing mini_portile2 2.0.0  Installing sqlite3 1.3.11 with native extensions  Installing json 1.8.3 with native extensions  Using bundler 1.11.2  Installing nokogiri 1.6.7.2 with native extensions  Bundle complete! 3 Gemfile dependencies, 5 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (15.67s)  Cleaning up the bundler cache.   ###### WARNING:  You have not declared a Ruby version in your Gemfile.  To set your Ruby version add this line to your Gemfile:  ruby '2.2.4'  # See https://devcenter.heroku.com/articles/ruby-versions for more information.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... scraper.rb:73:in `<main>': undefined method `api_sort' for nil:NilClass (NoMethodError)

Statistics

Average successful run time: less than a minute

Total run time: about 6 hours

Total cpu time used: 9 minutes

Total disk space used: 46.2 KB

History

  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    17 pages scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    566 pages scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 57c8733b and failed .
    25 records removed in the database
    1 page scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    742 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    56 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    53 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    4 records added, 21 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    4 records removed, 21 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    2 records added, 23 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    10 records added, 13 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    5 records added, 8 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and failed .
    17 records removed, 8 records updated in the database
    52 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    49 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    nothing changed in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
    51 pages scraped
  • Auto ran revision 57c8733b and completed successfully .
    16 records added, 9 records updated in the database
  • Auto ran revision 57c8733b and failed .
    16 records removed, 9 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and failed .
    nothing changed in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    9 records added, 16 records updated in the database
  • Auto ran revision 57c8733b and failed .
    5 records added, 11 records updated in the database
  • Auto ran revision 57c8733b and failed .
    14 records removed, 11 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    1 record added, 24 records updated in the database
  • Auto ran revision 57c8733b and failed .
    1 record removed, 24 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    25 records updated in the database
  • Auto ran revision 57c8733b and completed successfully .
    17 records added, 8 records updated in the database
  • Auto ran revision cccc3892 and failed .
    10 records removed, 8 records updated in the database
  • Auto ran revision cccc3892 and failed .
    7 records removed, 18 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    5 records added, 20 records updated in the database
  • Auto ran revision cccc3892 and failed .
    5 records removed, 20 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    5 records added, 20 records updated in the database
  • Auto ran revision cccc3892 and failed .
    5 records removed, 20 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    6 records added, 19 records updated in the database
  • Auto ran revision cccc3892 and failed .
    3 records removed, 19 records updated in the database
  • Auto ran revision cccc3892 and failed .
    3 records removed, 22 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Auto ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Manually ran revision cccc3892 and completed successfully .
    25 records updated in the database
  • Manually ran revision cccc3892 and completed successfully .
    25 records added in the database
  • Manually ran revision cccc3892 and failed .
    nothing changed in the database
  • Created on morph.io

Scraper code

Ruby

github_trending / scraper.rb