loren / fbopen_data

FBO (Federal Business Opportunity) data from FBOpen

Scrapes api.data.gov

api.data.gov


This is a scraper of FBOpen data that runs on Morph.

Contributors loren

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.2.5 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/............  Fetching version metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Installing htmlentities 4.3.4  Installing crass 1.0.2  Installing json 1.8.3 with native extensions  Installing multi_xml 0.5.5  Installing httpclient 2.8.0  Installing pkg-config 1.1.7  Installing mini_portile2 2.1.0  Using bundler 1.11.2  Installing sqlite3 1.3.11 with native extensions  Installing nokogiri 1.6.8 with native extensions  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing nokogumbo 1.4.7 with native extensions  Installing httparty 0.13.7  Installing sanitize 4.0.1  Bundle complete! 4 Gemfile dependencies, 15 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from httparty:  When you HTTParty, you must party hard!  Bundle completed (30.30s)  Cleaning up the bundler cache.  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Fetching results 0..499 at 2017-02-26 07:51:11 +0000 Fetching results 0..499 at 2017-02-26 07:51:11 +0000 Fetching results 0..499 at 2017-02-26 07:51:11 +0000 /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/protocol.rb:158:in `rescue in rbuf_fill': Net::ReadTimeout (Net::ReadTimeout) from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/protocol.rb:152:in `rbuf_fill' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/protocol.rb:134:in `readuntil' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/protocol.rb:144:in `readline' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http/response.rb:39:in `read_status_line' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http/response.rb:28:in `read_new' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:1414:in `block in transport_request' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:1411:in `catch' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:1411:in `transport_request' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:1384:in `request' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:1377:in `block in request' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:853:in `start' from /app/vendor/ruby-2.2.5/lib/ruby/2.2.0/net/http.rb:1375:in `request' from /app/vendor/bundle/ruby/2.2.0/gems/httparty-0.13.7/lib/httparty/request.rb:117:in `perform' from /app/vendor/bundle/ruby/2.2.0/gems/httparty-0.13.7/lib/httparty.rb:545:in `perform_request' from /app/vendor/bundle/ruby/2.2.0/gems/httparty-0.13.7/lib/httparty.rb:476:in `get' from /app/vendor/bundle/ruby/2.2.0/gems/httparty-0.13.7/lib/httparty.rb:583:in `get' from scraper.rb:60:in `fetch_results' from scraper.rb:71:in `<main>'

Data

Downloaded 546 times by loren bilelensi davidpranata

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (29 MB) Use the API

rows 0 / 0

Statistics

Average successful run time: about 1 hour

Total run time: 7 days

Total cpu time used: about 2 hours

Total disk space used: 29 MB

History

  • Auto ran revision bcaa76e9 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision bcaa76e9 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision bcaa76e9 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision bcaa76e9 and failed .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision bcaa76e9 and failed .
    nothing changed in the database
    2 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

fbopen_data / scraper.rb