austccr / link_and_attachment_archiver

Parse links out of HTML in a morph.io scraper and do some archiving actions on each of them. This is kind of a meta-archiver really.

Contributors equivalentideas

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Ruby app detected [1G-----> Compiling Ruby [1G-----> Using Ruby version: ruby-2.6.3 [1G-----> Installing dependencies using bundler version 1.17.2 [1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment [1G Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [1G Fetching gem metadata from https://rubygems.org/...... [1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git [1G Using bundler 1.17.2 [1G Fetching coderay 1.1.2 [1G Fetching ffi 1.10.0 [1G Fetching httpclient 2.8.3 [1G Installing ffi 1.10.0 with native extensions [1G Installing coderay 1.1.2 [1G Installing httpclient 2.8.3 [1G Fetching method_source 0.9.2 [1G Fetching mini_portile2 2.4.0 [1G Installing mini_portile2 2.4.0 [1G Fetching sqlite3 1.4.0 [1G Installing method_source 0.9.2 [1G Installing sqlite3 1.4.0 with native extensions [1G Fetching nokogiri 1.10.1 [1G Installing nokogiri 1.10.1 with native extensions [1G Fetching pry 0.12.2 [1G Installing pry 0.12.2 [1G Fetching sqlite_magic 0.0.6 [1G Installing sqlite_magic 0.0.6 [1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176) [1G Fetching ethon 0.12.0 [1G Installing ethon 0.12.0 [1G Fetching typhoeus 1.3.1 [1G Installing typhoeus 1.3.1 [1G Bundle complete! 8 Gemfile dependencies, 13 gems now installed. [1G Gems in the groups development and test were not installed. [1G Bundled gems are installed into `./vendor/bundle` [1G Removing bundler (1.15.2) [1G Bundle completed (29.03s) [1G Cleaning up the bundler cache. [1G Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. [1G-----> Detecting rake tasks [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... Searching for records at https://api.morph.io/austccr/bca_media_releases_scraper/data.json Requesting records 1 to 5 with 'select * from "data" limit 5 offset 0' /app/vendor/ruby-2.6.3/lib/ruby/2.6.0/json/common.rb:156:in `parse': 767: unexpected token at '' (JSON::ParserError) from /app/vendor/ruby-2.6.3/lib/ruby/2.6.0/json/common.rb:156:in `parse' from scraper.rb:29:in `archive_links_from_morph_results' from scraper.rb:81:in `work_through_morph_results' from scraper.rb:95:in `block in <main>' from scraper.rb:90:in `each' from scraper.rb:90:in `<main>'

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.89 MB) Use the API

rows 10 / 2710

url syndication errors source_url archived_at
http://www.minerals.org.au/file_upload/files/publications/Australias_investment_challenge_in_wake_of_2018_US_tax_reform_March_2018_FINAL.pdf
https://web.archive.org/web/20190319024704/http://www.minerals.org.au/file_upload/files/publications/Australias_investment_challenge_in_wake_of_2018_US_tax_reform_March_2018_FINAL.pdf
http://minerals.org.au/news/global-tax-expert-warns-australia-don%E2%80%99t-get-left-out-cold-company-tax
2019-03-19 02:47:06 UTC
http://minerals.org.au/files/190319-mca-ceo-tax-conference-opening-speechpdf
https://web.archive.org/web/20190324021540/http://minerals.org.au/files/190319-mca-ceo-tax-conference-opening-speechpdf
http://minerals.org.au/news/opening-speech-mca-2019-tax-conference
2019-03-24 02:15:41 UTC
http://minerals.org.au/sites/default/files/190319%20MCA%20CEO%20Tax%20Conference%20Opening%20Speech.pdf
302: Found
http://minerals.org.au/news/opening-speech-mca-2019-tax-conference
2019-03-24 02:15:42 UTC
http://minerals.org.au/files/190402-australias-mining-industry-delivers-budget-surplus-and-stronger-economypdf
https://web.archive.org/web/20190402222638/http://minerals.org.au/files/190402-australias-mining-industry-delivers-budget-surplus-and-stronger-economypdf
http://minerals.org.au/news/australia%E2%80%99s-minerals-industry-delivers-forecast-budget-surplus-and-stronger-economy
2019-04-02 22:26:39 UTC
https://minerals.org.au/sites/default/files/190402%20Australia%27s%20mining%20industry%20delivers%20Budget%20surplus%20and%20a%20stronger%20economy.pdf
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190402%20Australia%27s%20mining%20industry%20delivers%20Budget%20surplus%20and%20a%20stronger%20economy.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX
http://minerals.org.au/news/australia%E2%80%99s-minerals-industry-delivers-forecast-budget-surplus-and-stronger-economy
2019-04-02 22:26:40 UTC
https://www.2gb.com/podcast/april-9th-full-show-2/
https://www.2gb.com/podcast/april-9th-full-show-2/
2019-04-17 17:48:14 UTC
https://minerals.org.au/sites/default/files/190519%20Australia%27s%20minerals%20industry%20will%20work%20with%20Coalition%20government%20for%20a%20stronger%20Australia.pdf
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190519%20Australia%27s%20minerals%20industry%20will%20work%20with%20Coalition%20government%20for%20a%20stronger%20Australia.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.Valid
http://minerals.org.au/news/australia%E2%80%99s-minerals-industry-will-work-coalition-government-stronger-australia
2019-05-20 20:25:51 UTC
https://www.theguardian.com/australia-news/2019/apr/01/50-of-new-cars-to-be-electric-vehicles-by-2030-under-labor-climate-change-policy#img-1
https://www.theguardian.com/australia-news/2019/apr/01/50-of-new-cars-to-be-electric-vehicles-by-2030-under-labor-climate-change-policy#img-1
2019-06-07 04:10:03 UTC
https://www.gladstoneobserver.com.au%2Fnews%2Fclimate-policy-uncertainty-over-impact-on-key-indu%2F3708648%2F&usg=AOvVaw2ppGvuUV5O4I4zQnml6eXc
400: Invalid URI: noSlash
https://www.gladstoneobserver.com.au%2Fnews%2Fclimate-policy-uncertainty-over-impact-on-key-indu%2F3708648%2F&usg=AOvVaw2ppGvuUV5O4I4zQnml6eXc
2019-06-12 05:11:00 UTC
https://minerals.org.au/sites/default/files/181012%20Commodity%20Insights%20Met%20Coal%20Report.pdf
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/181012%20Commodity%20Insights%20Met%20Coal%20Report.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath
http://minerals.org.au/news/market-demand-study-australian-metallurgical-coal
2019-07-01 06:12:15 UTC

Statistics

Average successful run time: about 3 hours

Total run time: about 1 month

Total cpu time used: about 1 hour

Total disk space used: 4.14 MB

History

  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • Auto ran revision bc449ac8 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history