austccr / link_and_attachment_archiver

Parse links out of HTML in a morph.io scraper and do some archiving actions on each of them. This is kind of a meta-archiver really.

Scrapes api.morph.io and web.archive.org

Get structured data out of the web. Code collaboration through GitHub. Run your scrapers in the cloud.


This script finds links in HTML text and saves a backup copy of them.

Adding your morph.io API key securely

You need your morph.io API key to run this, as it requests records from the morph.io API. Find your key at https://morph.io/documentation/api .

Your morph.io API key should be set to the environment variable MORPH_API_KEY. Accessed as:

ENV['MORPH_API_KEY']

For morph.io runs

On your morph.io scraper's settings page, add the environment variable.

See the morph.io secret values documentation.

On your local machine

We use dotenv to add environment variables from a .env file locally.

Create the .env file by copying .env.example:

cp .env.example .env

Then replace the example string with your API key in your new .env file.

This file is listed in .gitignore and so won't be checked into your git repository. In other words, it will only stay on your local machine, secret from the web.

Now, to run the script with your secret key, execute it with dotenv:

bundle exec dotenv ruby scraper.rb

TODOs

  • [ ] handle updates
    • [ ] if we don't have it, archive it
    • [ ] if we already have the file for this source_url, don't update
    • [ ] if we have it are there are errors, try again
  • [ ] investigate 302 records
  • [ ] if we get a cert missing https error from archive.org, try the http version of the links.
  • [ ] write a proper description in the readme
  • [ ] point this at more sources, like the main lobbywatch archive for example

Contributors equivalentideas

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Searching for records at https://api.morph.io/austccr/mca_media_releases_scraper/data.json Requesting records 1 to 5 with 'select * from "data" limit 5 offset 0' Extracting and archiving links from http://minerals.org.au/news/exceptional-victorian-women-resources-sought-2019-awards Extracting and archiving links from http://minerals.org.au/news/strong-result-australian-mining-exploration-expenditure Extracting and archiving links from http://minerals.org.au/news/stronger-mining-industry-supports-faster-return-budget-surplus Extracting and archiving links from http://minerals.org.au/news/peru-australia-free-trade-agreement-will-deliver-investment-and-jobs-australia Extracting and archiving links from http://minerals.org.au/news/one-step-closer-meeting-australia%E2%80%99s-paris-commitments Requesting records 6 to 10 with 'select * from "data" limit 5 offset 5' Extracting and archiving links from http://minerals.org.au/news/health-and-safety-first-priority-australian-minerals-sector Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-energy-tipping-point-speech-energy-policy-institute-australia Extracting and archiving links from http://minerals.org.au/news/mca-welcomes-transparency-policy Extracting and archiving links from http://minerals.org.au/news/mca-calls-parliament-support-regional-communities-and-jobs Extracting and archiving links from http://minerals.org.au/news/powering-future-australian-mining-people-innovation-and-modern-education Requesting records 11 to 15 with 'select * from "data" limit 5 offset 10' Extracting and archiving links from http://minerals.org.au/news/australian-minerals-industry-research-scholarship-winners-announced-1 Extracting and archiving links from http://minerals.org.au/news/policy-reform-required-secure-australia%E2%80%99s-place-critical-commodity-supply-chains Extracting and archiving links from http://minerals.org.au/news/future-work-changing-skills-landscape-miners Extracting and archiving links from http://minerals.org.au/news/future-work-economic-implications-technology-and-digital-mining Extracting and archiving links from http://minerals.org.au/news/next-frontier-australian-mining-policy-priorities Requesting records 16 to 20 with 'select * from "data" limit 5 offset 15' Extracting and archiving links from http://minerals.org.au/news/reform-overdue-so-australian-minerals-industry-can-invest-grow-and-support-regional-australia Extracting and archiving links from http://minerals.org.au/news/new-campaign-shows-there%E2%80%99s-more-australian-mining Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-world-class-mining-industry-sets-new-records-2018 Extracting and archiving links from http://minerals.org.au/news/strong-industry-interest-underwriting-new-generation-investments-program Extracting and archiving links from http://minerals.org.au/news/occupational-health-and-safety-laws-must-focus-prevention-and-reporting Requesting records 21 to 25 with 'select * from "data" limit 5 offset 20' Extracting and archiving links from http://minerals.org.au/news/minerals-council-australia-pre-budget-submission-2019-20 Extracting and archiving links from http://minerals.org.au/news/submission-expert-review-australias-vocational-education-and-training-system Extracting and archiving links from http://minerals.org.au/news/delivering-more-competition-coastal-shipping-will-cut-costs-business-and-consumers Extracting and archiving links from http://minerals.org.au/news/heatwaves-proof-positive-australia-needs-nuclear Extracting and archiving links from http://minerals.org.au/news/using-australian-coal-power-hydrogen%E2%80%99s-future Requesting records 26 to 30 with 'select * from "data" limit 5 offset 25' Extracting and archiving links from http://minerals.org.au/news/populists-multi-billion-blunder-foreign-investment-australia Extracting and archiving links from http://minerals.org.au/news/jobs-and-investment-under-threat-nt-government%E2%80%99s-mining-tax-grab Extracting and archiving links from http://minerals.org.au/news/mca-nt-submission-environment-protection-bill-and-regulations Extracting and archiving links from http://minerals.org.au/news/submission-galilee-basin-coal-prohibition-bill-2018 Extracting and archiving links from http://minerals.org.au/news/senate-must-reject-greens-bid-ban-thermal-coal-mining-queensland%E2%80%99s-galilee-basin Requesting records 31 to 35 with 'select * from "data" limit 5 offset 30' Extracting and archiving links from http://minerals.org.au/news/measured-response-vital-reducing-australia%E2%80%99s-emissions-without-damaging-economy-jobs Extracting and archiving links from http://minerals.org.au/news/answer-poor-regulation-not-more-regulation Extracting and archiving links from http://minerals.org.au/news/more-work-needed-low-cost-offsets-despite-cop24-agreement Extracting and archiving links from http://minerals.org.au/news/labor%E2%80%99s-federal-epa-will-cost-jobs-and-investment Extracting and archiving links from http://minerals.org.au/news/generation-underwriting-program-long-overdue-first-step-securing-low-cost-reliable-power Requesting records 36 to 40 with 'select * from "data" limit 5 offset 35' Extracting and archiving links from http://minerals.org.au/news/veac-central-west-investigation-draft-proposal-paper-public-comment Extracting and archiving links from http://minerals.org.au/news/submission-inquiry-effectiveness-current-temporary-skilled-visa-system-targeting-genuine-skills Extracting and archiving links from http://minerals.org.au/news/modern-responsive-tax-system-needed-support-world-class-australian-minerals-industry Extracting and archiving links from http://minerals.org.au/news/indias-rapid-rise-and-growing-middle-class-creates-hunger-commodities Extracting and archiving links from http://minerals.org.au/news/adani-carmichael-mine-will-boost-regional-jobs-fund-services-and-open-new-mining-region Requesting records 41 to 45 with 'select * from "data" limit 5 offset 40' Extracting and archiving links from http://minerals.org.au/news/more-opportunities-australia-and-peru-rapid-implementation-free-trade-agreement Extracting and archiving links from http://minerals.org.au/news/government-puts-mining-lead-response-india-economic-strategy Extracting and archiving links from http://minerals.org.au/news/reality-check-needed-labor%E2%80%99s-renewables-pledge Extracting and archiving links from http://minerals.org.au/news/greens-policy-thermal-coal-exports-should-be-rejected Extracting and archiving links from http://minerals.org.au/news/submission-department-environment-and-energy-underwriting-new-generation-investment Requesting records 46 to 50 with 'select * from "data" limit 5 offset 45' Extracting and archiving links from http://minerals.org.au/news/mca-victorian-division-welcomes-initiative-bolster-state%E2%80%99s-power-generation-500-mw Extracting and archiving links from http://minerals.org.au/news/tpp-11-ratification-great-news-australian-jobs-and-workers Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-world-class-minerals-industry-shows-global-leadership-sustainability Extracting and archiving links from http://minerals.org.au/news/regional-jobs-communities-benefit-new-victorian-mining-exploration-investment Extracting and archiving links from http://minerals.org.au/news/submission-senate-inquiry-australia%E2%80%99s-faunal-extinction-crisis Requesting records 51 to 55 with 'select * from "data" limit 5 offset 50' Extracting and archiving links from http://minerals.org.au/news/fosterville-gold-mine-takes-out-26th-annual-gastech-australia-victorian-mine-rescue-competition Extracting and archiving links from http://minerals.org.au/news/bright-future-ahead-australia%E2%80%99s-steel-making-met-coal-exports Extracting and archiving links from http://minerals.org.au/news/new-frontiers-india Extracting and archiving links from http://minerals.org.au/news/big-future-opportunities-australia-india-trade-investment-despite-bumpy-ride-ahead Extracting and archiving links from http://minerals.org.au/news/market-demand-study-australian-metallurgical-coal Requesting records 56 to 60 with 'select * from "data" limit 5 offset 55' Extracting and archiving links from http://minerals.org.au/news/speedy-passage-tpp-11-legislation-critical-exporters Extracting and archiving links from http://minerals.org.au/news/discussion-paper-women-stem-decadal-plan Extracting and archiving links from http://minerals.org.au/news/mining-and-epbc-act-nuclear-action-trigger-0 Extracting and archiving links from http://minerals.org.au/news/time-end-discrimination-against-uranium-epbc-act Extracting and archiving links from http://minerals.org.au/news/mtec-key-performance-measures-report-2018 Requesting records 61 to 65 with 'select * from "data" limit 5 offset 60' Extracting and archiving links from http://minerals.org.au/news/mining%E2%80%99s-role-powering-modern-australian-life-confirmed-resources-2030-taskforce-report Extracting and archiving links from http://minerals.org.au/news/winning-women-australian-resources-recognised-national-awards Extracting and archiving links from http://minerals.org.au/news/submission-safe-work-australia-consultation-ris-wes Extracting and archiving links from http://minerals.org.au/news/new-indigenous-education-and-employment-partnership-australian-minerals-industry Extracting and archiving links from http://minerals.org.au/news/new-modelling-confirms-boost-australian-jobs-wages-tpp-11-trade-deal Requesting records 66 to 70 with 'select * from "data" limit 5 offset 65' Extracting and archiving links from http://minerals.org.au/news/indonesia-australia-free-trade-agreement-brings-benefits-both-resource-sectors Extracting and archiving links from http://minerals.org.au/news/australian-minerals-industry-offers-scholarships-talented-researchers Extracting and archiving links from http://minerals.org.au/news/stronger-victorian-minerals-sector-means-more-jobs-and-growing-regions Extracting and archiving links from http://minerals.org.au/news/australian-minerals-industry-recognises-its-female-stars Extracting and archiving links from http://minerals.org.au/news/australia-needs-stable-government-and-decisive-action-strong-economy Requesting records 71 to 75 with 'select * from "data" limit 5 offset 70' Extracting and archiving links from http://minerals.org.au/news/submission-how-mining-sector-can-support-businesses-regional-economies Extracting and archiving links from http://minerals.org.au/news/back-neg-and-end-energy-wars-or-face-consequences Extracting and archiving links from http://minerals.org.au/news/submission-review-national-pollutant-inventory Extracting and archiving links from http://minerals.org.au/news/mining-industry-favourability-rebounds-six-year-high Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-resources-sector-drives-record-exports-2017-18 Requesting records 76 to 80 with 'select * from "data" limit 5 offset 75' Extracting and archiving links from http://minerals.org.au/news/labor%E2%80%99s-plan-will-erode-competitiveness-and-jeopardise-jobs Extracting and archiving links from http://minerals.org.au/news/energy-security-board-national-energy-guarantee-draft-detailed-consultation-paper Extracting and archiving links from http://minerals.org.au/news/accc-proposes-sensible-approach-secure-more-affordable-energy Extracting and archiving links from http://minerals.org.au/news/national-energy-guarantee-draft-detailed-design-consultation-commonwealth-elements Extracting and archiving links from http://minerals.org.au/news/submission-department-foreign-affairs-and-trade-proposed-pacific-alliance-free-trade-agreement Requesting records 81 to 85 with 'select * from "data" limit 5 offset 80' Extracting and archiving links from http://minerals.org.au/news/coal-overtakes-iron-ore-government-forecasts-australia%E2%80%99s-largest-export-2018-19 Extracting and archiving links from http://minerals.org.au/news/australian-mining-industry-annual-lecture-2018 Extracting and archiving links from http://minerals.org.au/news/victoria%E2%80%99s-first-female-mine-site-general-manager-takes-out-2018-victorian-women-resources Extracting and archiving links from http://minerals.org.au/news/robust-asian-thermal-coal-demand-creates-opportunities-australia Extracting and archiving links from http://minerals.org.au/news/further-clarity-needed-australia%E2%80%99s-energy-policy-framework Requesting records 86 to 90 with 'select * from "data" limit 5 offset 85' Extracting and archiving links from http://minerals.org.au/news/inspirational-australian-mining-women-recognised-queen%E2%80%99s-birthday-honours Extracting and archiving links from http://minerals.org.au/news/tax-regulatory-reform-needed-spur-investment-and-jobs-australian-mining Extracting and archiving links from http://minerals.org.au/news/australia-eu-fta-would-boost-australia%E2%80%99s-network-free-trade-agreements Extracting and archiving links from http://minerals.org.au/news/support-mining-jobs-lessen-regional-inequality Extracting and archiving links from http://minerals.org.au/news/new-uses-gold-mean-more-opportunities-world-class-australian-gold-industry Requesting records 91 to 95 with 'select * from "data" limit 5 offset 90' Extracting and archiving links from http://minerals.org.au/news/australian-mining-sector-must-work-harder-promote-industry-careers Extracting and archiving links from http://minerals.org.au/news/miners-work-australias-work-class-mining-workforce Extracting and archiving links from http://minerals.org.au/news/early-implementation-tpp-11-will-deliver-jobs-growth-and-opportunity-australia Extracting and archiving links from http://minerals.org.au/news/submission-joint-standing-committee-treaties-inquiry-comprehensive-and-progressive-agreement Extracting and archiving links from http://minerals.org.au/news/mca-submission-house-representatives-standing-committee-economics-inquiry-impediments-business Requesting records 96 to 100 with 'select * from "data" limit 5 offset 95' Extracting and archiving links from http://minerals.org.au/news/structural-reforms-needed-deliver-long-term-economic-prosperity Extracting and archiving links from http://minerals.org.au/news/australian-mining-industry-thanks-tim-hammond-mp-his-contribution Extracting and archiving links from http://minerals.org.au/news/real-world-experience-counters-actu-claims-over-trade-agreements Extracting and archiving links from http://minerals.org.au/news/nt-royalty-proposal-hit-mining-investment Extracting and archiving links from http://minerals.org.au/news/mca-welcomes-progress-national-energy-guarantee Requesting records 101 to 105 with 'select * from "data" limit 5 offset 100' Extracting and archiving links from http://minerals.org.au/news/mining%E2%80%99s-company-tax-and-royalty-payments-support-australia%E2%80%99s-economy-essential-services Extracting and archiving links from http://minerals.org.au/news/ms-tania-constable-psm-appointed-mca-chief-executive Extracting and archiving links from http://minerals.org.au/news/make-neg-work-reduce-power-costs-and-improve-system-reliability Extracting and archiving links from http://minerals.org.au/news/submission-national-energy-guarantee-draft-design-consultation-paper Extracting and archiving links from http://minerals.org.au/news/why-australia-should-support-revived-trans-pacific-partnership Requesting records 106 to 110 with 'select * from "data" limit 5 offset 105' Extracting and archiving links from http://minerals.org.au/news/australian-mining-sector-calls-urgent-native-title-amendments-end-uncertainty Extracting and archiving links from http://minerals.org.au/news/submission-northern-territory-revenue-discussion-paper Extracting and archiving links from http://minerals.org.au/news/pacer-plus-trade-agreement-good-news-australia-and-our-pacific-neighbours Extracting and archiving links from http://minerals.org.au/news/decline-mining-investment-attractiveness-wake-call-state-governments-0 Extracting and archiving links from http://minerals.org.au/news/submission-attorney-generals-department-native-title-act-reforms Requesting records 111 to 115 with 'select * from "data" limit 5 offset 110' Extracting and archiving links from http://minerals.org.au/news/modernising-stem-education-australia%E2%80%99s-future-mining-workforce Extracting and archiving links from http://minerals.org.au/news/australian-minerals-sector-calls-workplace-relations-education-reform-build-future-workforce Extracting and archiving links from http://minerals.org.au/news/submission-senate-select-committee-future-work-and-workers Extracting and archiving links from http://minerals.org.au/news/submission-optimising-stem-industry-school-partnerships-inspiring-australias-next-generation Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-resource-governance-expertise-promoted-boost-investment-african-mining Requesting records 116 to 120 with 'select * from "data" limit 5 offset 115' Extracting and archiving links from http://minerals.org.au/news/mining-boom-delivers-record-resource-export-revenues Extracting and archiving links from http://minerals.org.au/news/coal-exports-set-all-time-record-2017 Extracting and archiving links from http://minerals.org.au/news/responsible-minerals-industry-uses-water-create-economic-value-australians Extracting and archiving links from http://minerals.org.au/news/tpp-breakthrough-boost-australian-economy Extracting and archiving links from http://minerals.org.au/news/water-use-extractive-industry Requesting records 121 to 125 with 'select * from "data" limit 5 offset 120' Extracting and archiving links from http://minerals.org.au/news/reform-overdue-drive-growth-and-protect-australia-future-downturn Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-innovative-minerals-sector-promotes-drone-technology-new-ad Extracting and archiving links from http://minerals.org.au/news/independent-study-shows-australian-minerals-companies-paying-their-fair-share-tax Extracting and archiving links from http://minerals.org.au/news/australian-mining-and-mining-services-supporting-growth-and-prosperity Extracting and archiving links from http://minerals.org.au/news/australian-mining-education-summit-help-build-tomorrow%E2%80%99s-workforce Requesting records 126 to 130 with 'select * from "data" limit 5 offset 125' Extracting and archiving links from http://minerals.org.au/news/high-australian-power-prices-threaten-jobs-hurt-global-competitiveness Extracting and archiving links from http://minerals.org.au/news/mca-victoria-lauds-long-overdue-reforms-earth-resources-regulation-approval-0 Extracting and archiving links from http://minerals.org.au/news/australia-well-placed-meet-growing-asian-coal-demand-2 Extracting and archiving links from http://minerals.org.au/news/myefo-shows-australian-mining-pays-more-its-fair-share-tax-0 Extracting and archiving links from http://minerals.org.au/news/mca-welcomes-pc-findings-benefits-regional-communities-mining Requesting records 131 to 135 with 'select * from "data" limit 5 offset 130' Extracting and archiving links from http://minerals.org.au/news/pre-budget-submission-2018-19 Extracting and archiving links from http://minerals.org.au/news/new-frontiers-australian-mining-and-mining-services-exports Extracting and archiving links from http://minerals.org.au/news/submission-productivity-commissions-initial-report-transitioning-regional-economies Extracting and archiving links from http://minerals.org.au/news/restoring-equity-and-fairness Extracting and archiving links from http://minerals.org.au/news/submission-productivity-commission-inquiry-horizontal-fiscal-equalisation-australia Requesting records 136 to 140 with 'select * from "data" limit 5 offset 135' Extracting and archiving links from http://minerals.org.au/news/high_energy_prices_are_hurting_consumers Extracting and archiving links from http://minerals.org.au/news/minerals_research_scholarships_to_boost_exploration_knowledge_base_1 Extracting and archiving links from http://minerals.org.au/news/nt_sharing_its_mining_royalty_revenue_with_other_states Extracting and archiving links from http://minerals.org.au/news/new_frontiers_for_australian_mining_and_mining_services_exports Extracting and archiving links from http://minerals.org.au/news/submission_to_the_productivity_commission_inquiry_into_the_progress_of_water_reforms_draft_report Requesting records 141 to 145 with 'select * from "data" limit 5 offset 140' Extracting and archiving links from http://minerals.org.au/news/submission_to_the_escosa_review_of_guidelines_for_the_access_regime_for_the Extracting and archiving links from http://minerals.org.au/news/submission_to_the_inquiry_into_the_retirement_of_coal_fired_power_stations Extracting and archiving links from http://minerals.org.au/news/submission_to_2017_review_of_climate_change_policies Extracting and archiving links from http://minerals.org.au/news/submission_on_coastal_shipping_reforms_discussion_paper Extracting and archiving links from http://minerals.org.au/news/updated_land_access_guide Requesting records 146 to 150 with 'select * from "data" limit 5 offset 145' Extracting and archiving links from http://minerals.org.au/news/mca_submission_to_the_senate_inquiry_into_mine_rehabilitation Extracting and archiving links from http://minerals.org.au/news/submission_on_the_ratification_of_the_minamata_convention_on_mercury Extracting and archiving links from http://minerals.org.au/news/changes_still_required_to_improve_workability_of_victorias_native_vegetatio Extracting and archiving links from http://minerals.org.au/news/iea_confirms_fundamental_importance_of_minerals_sector_for_meeting_future_energy_needs Extracting and archiving links from http://minerals.org.au/news/a_new_framework_for_victorias_environment_review_committees Requesting records 151 to 155 with 'select * from "data" limit 5 offset 150' Extracting and archiving links from http://minerals.org.au/news/submission_to_the_inquiry_into_national_freight_and_supply_chain_priorities Extracting and archiving links from http://minerals.org.au/news/a_modern_licencing_framework_for_victoria Extracting and archiving links from http://minerals.org.au/news/mca-annual-report-2016 Extracting and archiving links from http://minerals.org.au/news/submission_on_tax_expenditures_statement Extracting and archiving links from http://minerals.org.au/news/submission_on_tax_deductible_gift_recipient_reform_opportunities_discussion_paper Requesting records 156 to 160 with 'select * from "data" limit 5 offset 155' Extracting and archiving links from http://minerals.org.au/news/MCA_SUBMISSION_TO_THE_INDEPENDENT_REVIEW_INTO_THE_FUTURE_SECURITY_OF_THE_NATIONAL_ELECTRICITY_MARKET_6_March_2017_%2528including_BAE_paper%2529.pdf Extracting and archiving links from http://minerals.org.au/news/submission_on_the_register_of_foreign_ownership_of_water_and_agricultural_l Extracting and archiving links from http://minerals.org.au/news/submission_on_the_draft_national_standard_for_environmental_risk_management Extracting and archiving links from http://minerals.org.au/news/submission_to_joint_standing_committee_on_foreign_affairs_defence_and_trade Extracting and archiving links from http://minerals.org.au/news/mca-victoria-pre-budget-submission-2017 Requesting records 161 to 165 with 'select * from "data" limit 5 offset 160' Extracting and archiving links from http://minerals.org.au/shaping_economic_opportunities_for_indigenous_australians Extracting and archiving links from http://minerals.org.au/news/strong_signal_for_australian_coal_demand Extracting and archiving links from http://minerals.org.au/bipartisan_support_for_trade_will_boost_growth_prospects Extracting and archiving links from http://minerals.org.au/news/uneps_mixed_signals_on_climate_change Extracting and archiving links from http://minerals.org.au/news/free_trade_deal_with_peru_will_boost_australian_economy Requesting records 166 to 170 with 'select * from "data" limit 5 offset 165' Extracting and archiving links from http://minerals.org.au/news/playing_politics_with_energy_means_higher_prices Extracting and archiving links from http://minerals.org.au/getting_a_start_in_the_minerals_industry_national_skills_week Extracting and archiving links from http://minerals.org.au/former_labor_environment_minister_backs_modern_coal_fired_power Extracting and archiving links from http://minerals.org.au/news/its_time_to_kick_australias_nuclear_prohibition_into_the_bin Extracting and archiving links from http://minerals.org.au/victorians_make_their_mark_at_national_women_in_resources_awards Requesting records 171 to 175 with 'select * from "data" limit 5 offset 170' Extracting and archiving links from http://minerals.org.au/leaders_converge_in_canberra_for_women_in_mining_breakfast_1 Extracting and archiving links from http://minerals.org.au/news/gold_royalties_hike_highlights_need_for_gst_distribution_reform Extracting and archiving links from http://minerals.org.au/mtec_key_performance_measures_report_2017_1 Extracting and archiving links from http://minerals.org.au/latrobe_valley_hele_plant_would_deliver_reliable_affordable_power Extracting and archiving links from http://minerals.org.au/minerals_council_of_australia_ceo_to_step_down Requesting records 176 to 180 with 'select * from "data" limit 5 offset 175' Extracting and archiving links from http://minerals.org.au/mca_welcomes_ministerial_appointment Extracting and archiving links from http://minerals.org.au/policy_needs_to_deliver_reliable_and_affordable_energy_supply Extracting and archiving links from http://minerals.org.au/support_for_reform_of_project_approval_processes Extracting and archiving links from http://minerals.org.au/marking_25_years_of_the_victorian_mine_rescue_competition Extracting and archiving links from http://minerals.org.au/new_report_confirms_mining_receives_negligible_government_assistance Requesting records 181 to 185 with 'select * from "data" limit 5 offset 180' Extracting and archiving links from http://minerals.org.au/workplace_relations_the_case_for_reform Extracting and archiving links from http://minerals.org.au/uranium_industry_welcomes_senate_support_for_ansto_bill Extracting and archiving links from http://minerals.org.au/how_gst_is_killing_our_productivity Extracting and archiving links from http://minerals.org.au/news/hele-coal-can-contribute-australias-emissions-reduction-targets Extracting and archiving links from http://minerals.org.au/news/victorian-coal-policy-higher-prices-fewer-jobs Requesting records 186 to 190 with 'select * from "data" limit 5 offset 185' Extracting and archiving links from http://minerals.org.au/news/independent-report-backs-modern-coal-generation-australia-0 Extracting and archiving links from http://minerals.org.au/news/productivity-commission-inquiry-horizontal-fiscal-equialisation Extracting and archiving links from http://minerals.org.au/node/5696 Extracting and archiving links from http://minerals.org.au/news/engineers-take-out-top-honours-first-victorian-resources-awards Extracting and archiving links from http://minerals.org.au/minerals_sector_response_to_the_energy_security_review Requesting records 191 to 195 with 'select * from "data" limit 5 offset 190' Extracting and archiving links from http://minerals.org.au/legislation_restores_certainty_for_indigenous_land_use_agreements Extracting and archiving links from http://minerals.org.au/wa_labor_confirms_approved_uranium_projects_to_proceed Extracting and archiving links from http://minerals.org.au/australians_strongly_support_free_trade Extracting and archiving links from http://minerals.org.au/ositive_steps_to_reduce_energy_costs_and_boost_reliability Extracting and archiving links from http://minerals.org.au/stark_disparities_highlght_flaws_in_gst_distribution_system Requesting records 196 to 200 with 'select * from "data" limit 5 offset 195' Extracting and archiving links from http://minerals.org.au/adani_decision_welcomed Extracting and archiving links from http://minerals.org.au/without_coal_its_a_pathway_to_higher_prices Extracting and archiving links from http://minerals.org.au/government_takes_balanced_view_with_low_emission_strategy Extracting and archiving links from http://minerals.org.au/mining_stands_to_benefit_from_fta_with_peru Extracting and archiving links from http://minerals.org.au/mining_industry_committed_to_working_with_indigenous_australians Requesting records 201 to 205 with 'select * from "data" limit 5 offset 200' Extracting and archiving links from http://minerals.org.au/Media_Release_BP_TPP_revival_good_news_for_exports_22_May_2017.pdf Extracting and archiving links from http://minerals.org.au/new_report_low_emission_coal_technology_key_to_growth_in_south-east_asia Extracting and archiving links from http://minerals.org.au/bank_decision_on_new_coal_basin_developments Extracting and archiving links from http://minerals.org.au/another_missed_opportunity_to_improve_victorias_investment_attractiveness Extracting and archiving links from http://minerals.org.au/new_boost_for_australian_steel_highlights_benefits_of_mining Requesting records 206 to 210 with 'select * from "data" limit 5 offset 205' Extracting and archiving links from http://minerals.org.au/anti-trade_myths_punctured_by_reality Extracting and archiving links from http://minerals.org.au/budget_investment_focus_welcome_but_reform_challenge_remains Extracting and archiving links from http://minerals.org.au/us_tax_plan_a_wake-up_call_for_australia Extracting and archiving links from http://minerals.org.au/the_facts_about_australian_trade Extracting and archiving links from http://minerals.org.au/national_library_of_australia_fellowships_to_tell_minings_australian_story Requesting records 211 to 215 with 'select * from "data" limit 5 offset 210' Extracting and archiving links from http://minerals.org.au/mining_notches_up_strong_export_growth Extracting and archiving links from http://minerals.org.au/new_research_shows_the_mining_and_mets_sector_accounts_for_15_of_gdp Extracting and archiving links from http://minerals.org.au/news/international_priorities_for_australias_resources_sector Extracting and archiving links from http://minerals.org.au/news/mining-and-mets-engines-economic-growth-and-prosperity-australians Extracting and archiving links from http://minerals.org.au/flawed_diverted_profits_tax_is_bad_law Requesting records 216 to 220 with 'select * from "data" limit 5 offset 215' Extracting and archiving links from http://minerals.org.au/news/native_title_amendment_senate_report Extracting and archiving links from http://minerals.org.au/stop_adani_alliance_a_futile_vanity_pose Extracting and archiving links from http://minerals.org.au/it_is_time_to_end_the_ban_on_nuclear_power_in_australia Extracting and archiving links from http://minerals.org.au/uranium_industry_congratulates_wa_labor_on_election_victory Extracting and archiving links from http://minerals.org.au/new_report_highlights_tax_uncompetitiveness Requesting records 221 to 225 with 'select * from "data" limit 5 offset 220' Extracting and archiving links from http://minerals.org.au/news/minerals_council_submission_to_finkel_review Extracting and archiving links from http://minerals.org.au/news/submission_to_inquiry_into_the_native_title_amendment_indigenous_land_use_agreements_bill_2017 Extracting and archiving links from http://minerals.org.au/pre-budget_submission_2017-18 Extracting and archiving links from http://minerals.org.au/news/submission_on_diverted_profits_tax_exposure_draft_legislation Extracting and archiving links from http://minerals.org.au/mca_welcomes_a_new_life_for_mmgs_century_mine_1 Requesting records 226 to 230 with 'select * from "data" limit 5 offset 225' Extracting and archiving links from http://minerals.org.au/recognising_gender_diversity_across_victorias_minerals_industry_launch_of_victorian_awards Extracting and archiving links from http://minerals.org.au/new_global_report_highlights_risk_of_policy_complacency Extracting and archiving links from http://minerals.org.au/mining_remains_major_contributor_to_nations_economic_stability Extracting and archiving links from http://minerals.org.au/wto_breakthrough_on_trade_facilitation_agreement Extracting and archiving links from http://minerals.org.au/coal_fired_generation_underpins_energy_security Requesting records 231 to 235 with 'select * from "data" limit 5 offset 230' Extracting and archiving links from http://minerals.org.au/launch_of_new_coal_advertising_campaign Extracting and archiving links from http://minerals.org.au/new_report_backs_carbon_capture_and_storage_in_australia Extracting and archiving links from http://minerals.org.au/government_takes_balanced_view_with_low_emissions_strategy Extracting and archiving links from http://minerals.org.au/new_generation_coal_technology_report_released Extracting and archiving links from http://minerals.org.au/victoria_goes_backwards_as_power_bills_go_up Requesting records 236 to 240 with 'select * from "data" limit 5 offset 235' Extracting and archiving links from http://minerals.org.au/http%3A//www.minerals.org.au/news/mining_industry_an_essential_part_of_australias_future_new_campaign Extracting and archiving links from http://minerals.org.au/news/minerals_sector_supports_ambitious_and_pragmatic_trade_policy Extracting and archiving links from http://minerals.org.au/http%3A//www.minerals.org.au/file_upload/files/media_releases/Media_Release_GE_Strong_support_for_Aust_coal_from_PM_1_February_2017.pdf Extracting and archiving links from http://minerals.org.au/news/new_generation_coal_reliable_secure_and_low_emissions Extracting and archiving links from http://minerals.org.au/news/wa_government_approves_three_uranium_projects Requesting records 241 to 245 with 'select * from "data" limit 5 offset 240' Extracting and archiving links from http://minerals.org.au/news/minister_canavan_on_target_in_supporting_coal_industry_and_jobs Extracting and archiving links from http://minerals.org.au/act_labor-greens_attack_on_adanis_mine_a_futile_gesture Extracting and archiving links from http://minerals.org.au/news/pernicious_coal_royalty_increase_to_hurt_households_and_industries_again Extracting and archiving links from http://minerals.org.au/news/conflict-cooperation-prof-marcia-langton Extracting and archiving links from http://minerals.org.au/news/austmine-mca-mou-strengthens-minings-sustainability Requesting records 246 to 250 with 'select * from "data" limit 5 offset 245' Extracting and archiving links from http://minerals.org.au/news/workforce-skills-education-training Extracting and archiving links from http://minerals.org.au/news/australian-mining-delivers-record-tax-and-royalty-payments-benefit-communities-and-families Extracting and archiving links from http://minerals.org.au/news/offsets-crucial-australia-meet-emissions-targets Extracting and archiving links from http://minerals.org.au/news/australia%E2%80%99s-competitors-leave-us-standing-still-global-tax-race Extracting and archiving links from http://minerals.org.au/news/proposed-perth-based-australian-future-mines-centre-would-strengthen-exploration-productivity Requesting records 251 to 255 with 'select * from "data" limit 5 offset 250' Extracting and archiving links from http://minerals.org.au/news/submission-review-australian-qualifications-framework Extracting and archiving links from http://minerals.org.au/news/global-tax-expert-warns-australia-don%E2%80%99t-get-left-out-cold-company-tax Extracting and archiving links from http://minerals.org.au/news/opening-speech-mca-2019-tax-conference Finished... Archived 501 urls from 253 records.

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (829 KB) Use the API

rows 10 / 504

url syndication errors source_url archived_at
2019-03-19 02:47:06 UTC
2019-03-24 02:15:41 UTC
302: Found
2019-03-24 02:15:42 UTC
2019-03-25 10:50:38 UTC
2019-03-25 10:50:39 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190308%20Exceptional%20Victorian%20women%20in%20resources%20sought%20for%202019%20awards_0.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building
2019-03-25 10:50:40 UTC
2019-03-25 10:50:43 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190305%20Strong%20result%20for%20Australian%20mining%20exploration%20expenditure_0.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:
2019-03-25 10:50:44 UTC
2019-03-25 10:50:47 UTC
502: Bad GatewayLiveDocumentNotAvailableException: https://minerals.org.au/sites/default/files/190226%20Peru-Australia%20Free%20Trade%20Agreement%20will%20deliver%20investment%20and%20jobs%20for%20Australia.pdf: live document unavailable: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorExceptio
2019-03-25 10:50:48 UTC

Statistics

Average successful run time: 17 minutes

Total run time: about 4 hours

Total cpu time used: less than a minute

Total disk space used: 1.01 MB

History

  • Auto ran revision 45bec376 and completed successfully .
    501 records added, 499 records removed in the database
    552 pages scraped
  • Auto ran revision 45bec376 and completed successfully .
    501 records added, 501 records removed in the database
    552 pages scraped
  • Auto ran revision 45bec376 and completed successfully .
    501 records added, 501 records removed in the database
    552 pages scraped
  • Auto ran revision 45bec376 and completed successfully .
    501 records added, 501 records removed in the database
    552 pages scraped
  • Auto ran revision 45bec376 and completed successfully .
    501 records added, 501 records removed in the database
    552 pages scraped
  • ...
  • Created on morph.io

Show complete history