austccr / link_and_attachment_archiver

Parse links out of HTML in a morph.io scraper and do some archiving actions on each of them. This is kind of a meta-archiver really.


Contributors equivalentideas

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.6.3 -----> Installing dependencies using bundler version 1.17.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/......  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.17.2  Fetching coderay 1.1.2  Fetching ffi 1.10.0  Fetching httpclient 2.8.3  Installing ffi 1.10.0 with native extensions  Installing coderay 1.1.2  Installing httpclient 2.8.3  Fetching method_source 0.9.2  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching sqlite3 1.4.0  Installing method_source 0.9.2  Installing sqlite3 1.4.0 with native extensions  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Fetching pry 0.12.2  Installing pry 0.12.2  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching ethon 0.12.0  Installing ethon 0.12.0  Fetching typhoeus 1.3.1  Installing typhoeus 1.3.1  Bundle complete! 8 Gemfile dependencies, 13 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into `./vendor/bundle`  Removing bundler (1.15.2)  Bundle completed (29.03s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Searching for records at https://api.morph.io/austccr/bca_media_releases_scraper/data.json Requesting records 1 to 5 with 'select * from "data" limit 5 offset 0' /app/vendor/ruby-2.6.3/lib/ruby/2.6.0/json/common.rb:156:in `parse': 767: unexpected token at '' (JSON::ParserError) from /app/vendor/ruby-2.6.3/lib/ruby/2.6.0/json/common.rb:156:in `parse' from scraper.rb:29:in `archive_links_from_morph_results' from scraper.rb:81:in `work_through_morph_results' from scraper.rb:95:in `block in <main>' from scraper.rb:90:in `each' from scraper.rb:90:in `<main>'

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.89 MB) Use the API

rows 10 / 2710

url syndication errors source_url archived_at
2019-03-19 02:47:06 UTC
2019-03-24 02:15:41 UTC
302: Found
2019-03-24 02:15:42 UTC
2019-04-02 22:26:39 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190402%20Australia%27s%20mining%20industry%20delivers%20Budget%20surplus%20and%20a%20stronger%20economy.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX
2019-04-02 22:26:40 UTC
2019-04-17 17:48:14 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190519%20Australia%27s%20minerals%20industry%20will%20work%20with%20Coalition%20government%20for%20a%20stronger%20Australia.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.Valid
2019-05-20 20:25:51 UTC
2019-06-07 04:10:03 UTC
400: Invalid URI: noSlash
2019-06-12 05:11:00 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/181012%20Commodity%20Insights%20Met%20Coal%20Report.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath
2019-07-01 06:12:15 UTC

Statistics

Average successful run time: about 3 hours

Total run time: about 1 month

Total cpu time used: about 1 hour

Total disk space used: 4.14 MB

History

  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history