loren / state_data

Leads from the US Department of State BIDS (Business Information Database System)

Scrapes bids.state.gov

BIDS: Business Information Database System


This is a scraper of BIDS data that runs on Morph.

Contributors loren

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.2.5 -----> Installing dependencies using bundler 1.11.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/............  Fetching version metadata from https://rubygems.org/..  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Installing json 1.8.3 with native extensions  Installing multi_xml 0.5.5  Installing httpclient 2.6.0.1  Installing sqlite3 1.3.10 with native extensions  Using bundler 1.11.2  Installing sqlite_magic 0.0.3  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Installing httparty 0.13.7  Bundle complete! 2 Gemfile dependencies, 8 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from httparty:  When you HTTParty, you must party hard!  Bundle completed (9.60s)  Cleaning up the bundler cache.  Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... /app/vendor/bundle/ruby/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `parse': 757: unexpected token at '<!DOCTYPE html> (JSON::ParserError) <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" /> <meta name="description" content=""> <meta name="author" content=""> <link rel="shortcut icon" href="../../assets/ico/favicon.png"> <title>BIDS</title> <script src="/bower_components/jquery/dist/jquery.min.js"></script> <script src="/bower_components/lodash/lodash.min.js"></script> <script src="/bower_components/select2/dist/js/select2.min.js"></script> <script src="/bower_components/jquery-validation/dist/jquery.validate.min.js"></script> <script src="/bower_components/datatables/media/js/jquery.dataTables.min.js"></script> <script src="/bower_components/datatables/media/js/dataTables.bootstrap.min.js"></script> <!--<script src="/bower_components/leaflet/dist/leaflet.js"></script>--> <!-- CSS dependencies --> <link rel="stylesheet" href="/bower_components/datatables/media/css/dataTables.bootstrap.min.css"> <!-- Bootstrap JS --> <script src="/bower_components/bootstrap/dist/js/bootstrap.min.js"></script> <script src="/bower_components/bootstrap-multiselect/dist/js/bootstrap-multiselect.js"></script> <!-- Bootstrap core CSS --> <link rel="stylesheet" href="/bower_components/bootstrap/dist/css/bootstrap.min.css"> <link rel="stylesheet" href="/bower_components/select2/dist/css/select2.min.css"> <link rel="stylesheet" href="/bower_components/select2/dist/css/select2.min.css"> <link rel="stylesheet" href="/bower_components/bootstrap-multiselect/dist/css/bootstrap-multiselect.css"> <!--<link rel="stylesheet" href="/bower_components/leaflet/dist/leaflet.css">--> <!-- Including Async await files --> <script src="/bower_components/async/dist/async.min.js"></script> <!-- <link rel="stylesheet" href="https://cdn.jsdelivr.net/flexboxgrid/6.3.0/flexboxgrid.min.css" type="text/css" > --> <!-- Custom styles for this template --> <link href="/css/bids-style.css" rel="stylesheet"> <link href="/css/app.css" rel="stylesheet"> <!-- Application javascript --> <script src="/scripts/bids.js"></script> <script src="/scripts/navbar.js"></script> <!--[if lt IE 9]><script src="../../docs-assets/js/ie8-responsive-file-warning.js"></script><![endif]--> <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries --> <!--[if lt IE 9]> <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script> <script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script> <![endif]--> </head> <body> <main class="main-container"> <nav id="navbar" class="navbar nav navbar-default section--navbar"> <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand bids-primary-text logo-display" href="/"> <img src="../../assets/img/words-logo.jpg" height="50"> </a> <div class="navbar-collapse collapse"> <div class="navbar-left"> </div> <ul id="nav-bar" class="nav navbar-nav navbar-right"> <li class="active"><a href="/map">Map</a></li> <li><a href="/leadform">Add Lead</a></li> <li><a href="/faq">About</a></li> <li><a href="/feedback">Feedback</a></li> <li><a href="#" data-toggle="modal" data-target="#login-modal">Login</a></li> </ul> </div><!--/.nav-collapse --> </nav> <div class="alert alert-danger note map-alerts">Not Found</div> <div class="beta-note">beta</div> <div class="modal fade" id="login-modal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true" style="display: none;"> <div class="modal-dialog"> <div class="loginmodal-container"> <h1>Login to Your Account</h1><br> <form action="/login" method="post"> <input type="text" name="email" placeholder="Email"> <input type="password" name="password" placeholder="Password"> <input type="submit" name="login" class="login loginmodal-submit" value="Login"> </form> <div class="login-help"> <a href="#" data-toggle="modal" data-target="#signup-modal" data-dismiss="modal">Register</a> - <a href="#" data-toggle="modal" data-target="#notyet-modal" data-dismiss="modal">Forgot Password</a> </div> </div> </div> </div> <div class="modal fade" id="signup-modal" tabindex="-1" role="dialog" aria-labelledby="signupLabel" aria-hidden="true" style="display: none;"> <div class="modal-dialog"> <div class="signup-modal-container"> <!-- show any messages that come back with authentication --> <div class="alert alert-danger">Not Found</div> <h1>Create an Account</h1><br> <form action="/signup" method="post" id="signup-form"> <input type="text" name="email" placeholder="Email"> <input type="password" name="password" id="password" placeholder="Password"> <input type="password" name="password_again" id="password_again" placeholder="Re-type Password"> <input type="submit" name="signup" class="signup signup-modal-submit" value="Create Account"> </form> </div> </div> </div> <script> $("#signup-form").validate({ rules: { password: 'required', password_again: { required: true, equalTo: "#password" }, email: { required: true, email: true, govEmails: true } } }); </script> <div class="modal fade" id="notyet-modal" tabindex="-1" role="dialog" aria-labelledby="notyetLabel" aria-hidden="true" style="display: none;"> <div class="modal-dialog"> <div class="signup-modal-container"> <h1>Not Yet Implemented</h1><br> </div> </div> </div> <div class="container-fluid"> <div class="container section--error"> <h1 class="text-center">Not Found</h1> <br/> <div class="carousel slide" data-ride="carousel"> <div class="carousel-inner"> <div class="item active"> <img src="/images/error.jpg" class="img-thumbnail center-block error-image" alt="Error happened!"> <div class="carousel-caption"> <p class="error-full-message">Error: Not Found</p> </div> </div> </div> </div> </div> </div> </main> <footer> </footer> </body> </html> ' from /app/vendor/bundle/ruby/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `parse' from scraper.rb:19:in `fetch_results' from scraper.rb:42:in `<main>'

Data

Downloaded 177 times by loren

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.26 MB) Use the API

rows 0 / 0

Statistics

Average successful run time: half a minute

Total run time: 1 day

Total cpu time used: 3 minutes

Total disk space used: 3.29 MB

History

  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    81 pages scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and failed .
    652 records removed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    2 records added, 650 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    450 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    280 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    455 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    82 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    449 records updated in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    2 records added, 648 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 369 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 139 records updated in the database
    4 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    4208 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    623 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 336 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record added, 650 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 564 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    81 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    24 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    446 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 626 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    35 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 652 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 71 records updated in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    19 pages scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    627 records updated in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 215 records updated in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    431 records updated in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    1 record added, 653 records updated in the database
  • Auto ran revision bd5d4176 and completed successfully .
    1 record removed, 654 records updated in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Auto ran revision bd5d4176 and completed successfully .
    nothing changed in the database
  • Manually ran revision bd5d4176 and completed successfully .
    574 records removed, 654 records updated in the database
  • Manually ran revision d1d0476a and completed successfully .
    nothing changed in the database
  • Manually ran revision 3dda2d8e and completed successfully .
    1229 records added in the database
  • Created on morph.io

Scraper code

Ruby

state_data / scraper.rb