openaustralia / australian-newspapers

Basic information about newspapers around Australia


This is a scraper that runs on Morph. To get started see the documentation

Contributors mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Installing connection_pool 2.2.2  Fetching mime-types-data 3.2018.0812  Installing unf_ext 0.0.7.5 with native extensions  Installing httpclient 2.8.3  Installing mime-types-data 3.2018.0812  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Fetching mini_portile2 2.3.0  Installing mini_portile2 2.3.0  Fetching ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Fetching net-http-persistent 3.0.0  Installing sqlite3 1.3.13 with native extensions  Installing net-http-persistent 3.0.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.8.5  Installing nokogiri 1.8.5 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20180417  Installing domain_name 0.5.20180417  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (24.62s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Adelaide Advertiser Advocate Age Al Ankabout Al Furat Al Moharer Alice Springs News Allora Advertiser Along the Grapevine ANU Reporter Ararat Advertiser Area News Armidale Express Armidale Independent Augusta Margaret River Mail Australia Business Review Australian Australian Financial Review Australian Jewish News Australian News Network Avon Valley Advocate Ballarat Courier Bangalow's Heartbeat Barossa and Light Herald Barrier Daily Truth Batemans Bay Post Bayside Bulletin Beaudesert Times Bega District News Bellingen Courier Sun Benalla Ensign Bendigo Advertiser Bendigo Weekly Bharat Times Blacktown Advocate Blacktown Sun Blaney Chronicle Blaze Blue Mountains Gazette Bombala Times Boorawa News Border Chronicle Border Mail Border Watch Braidwood Times Bright Observer Brimbank Star Weekly Brisbane News Brisbane Times Bunbury Mail Bundaberg News Mail Bunyip Business News Busselton Dunsborough Mail Byron News Cairns Post Camden Advertiser Camden Haven Courier Campaspe News Campbelltown Mcarthur Advertiser Camperdown Chronicle Canberra City News Canberra Times Canowindra News Catholic Leader Catholic Outlook Central Coast Express Advocate Central Midlands and Coastal Advocate Central Queensland News Central Western Daily Cessnock Advertiser Chinese Herald Clarence Valley Review Coastal Times Cobar Weekly Cobram Courier Coffs Coast Advocate Collie Mail Coly Point Observer Coober Pedy News Coober Pedy Regional Times Cooma Monaro Express Cootamundra Herald Country News Countryman Courier Mail Cowra Community News Cowra Guardian Crikey Crookwell Gazette Daily Examiner Daily Mercury Daily Telegraph Dandenong Journal Dandenong Journal Deniliquin Pastoral Times Donnybrook Bridgetown Mail Dubbo Daily Liberal Dungog Chronicle Dunoon & District Gazette Dutch Courier East Coast News East Gippsland Newspapers East News Eastern Riverina Chronicle Echonet Daily Eden Magnet Elmassry Esperance Express Examiner Eyre Peninsula Tribune Fairfield Advance Fairfield City Champion Farm Weekly Farrago Flinders News Forbes Advocate Foster Mirror Fraser Coast Chronicle Fremantle Gazette Gay News Network Geelong Advertiser Geraldton Guardian Gippsland Times Gladstone Observer Glen Innes Examiner Gloucester Advocate Gold Coast Bulletin Goondiwindi Argus Goulburn Post Great Lakes Advocate Great Southern Star Great Southern Weekender Grenfell Record Guardian Australia Gujarat Times Guyra Argus Gympie Times Harden Murrumburrah Express Hawkesbury Gazette Hepburn Advocate Herald Sun Herald Sun Hervey Bay Independent Hills News Hills Shire Times Hinterland Times Hunter Valley News Hutt River Independent Illawara Mercury Independent Independent Newsgroup Indian Link Indian Voice Inner West Courier Inverell Times Ipswich Satellite Irish Echo Irrigator Islander Jimboomba Times Junee Southern Cross Kalgoorlie Miner Katherine Times Kiama Independent/Lake Times Knox Journal Koori Mail Kyabram Free Press Lakes Mail LaTrobe Valley Express Launceston Times Lithgow Mercury Liverpool City Champion Logan West Leader Lot's Wife (Monash) Loxton News Macarthur Chronicle Macleay Argus Macquarie Grapeshot Magnetic Times Maitland Mercury Malaysia Sun Mandurah Mail Manly Daily Manning River Times Mansfield Courier Mareeba Express Maribyrnong Weekly Meander Valley News Melbourne Bayside News Melbourne Central News Melbourne East News Melbourne News Melbourne North News Melbourne Observer Melton Express Telegraph Mercury Merimbula News Weekly Merredin Wheatbelt Mercury Midland Express Milton Ulladulla Times Monash Journal Moonee Valley Weekly Review Moorabin Leader Moree Champion Moreton News Morning Bulletin Mornington Peninsula Weekly Mount Barker Courier Moyne Gazette Mt Buller News Mt Hotham Falls Creek News Mudgee Guardian Murray Pioneer Murray Valley Standard Muswellbrook Chronicle Myall Coast Nota Nambucca Guardian News Naracoorte Herald Narooma News Narromine News New Matilda Newcastle Herald Newcastle Star Newcastle U Opus News Classifieds Noosa News North East & Goulburn Murray Farmer North Queensland Register North Shore Times North West Star Northen Rivers Echo Northern Argus Northern Daily Leader Northern Herald Northern Midlands News Northern Star Northern Star Weekly Hume and Whittlesea Northern Times Northwest News NT News Nyngan Observer Oberon Review Ocean Grove Voice Ovens & Murray Advertiser Parkes Champion Post Parramatta Advertiser Parramatta Sun Peninsula News Penrith City Gazette Pilbara Echo Plains Producer Port Lincoln Times Port Macquarie News Port Pirie Recorder Portland Observer Post Newspapers Queanbeyan Age Queensland Times Rabelais (La Trobe) Range News River News Riverina Leader Riverine Herald Rouse Hill Courier Roxby Downs Monitor Sada E Watan Sangbad Scone Advocate Seymour Telegraph Shepparton News Singleton Argus South Coast Register South Gippsland Sentinel Times South West Advertiser Southern Free Times Southern Highlands News Southern Riverina News Southern Star Southern Weekly St George Sutherland Leader St Marys Star Stanthorpe Border Post Stawell Times News Stock & Land Student View Summit Sun Sun City News Sun Herald Sunday Mail (Qld) Sunday Mail (SA) Sunday Telegraph Sunday Times Sunraysia Daily Sunshine Coast Daily Sunshine Coast News Swan Hill Guardian Swinburne Swine Sydney Morning Herald Sydney News Sydney Star Observer Tamar Community Times Tenterfield Star The Rural Toodyay Herald Toowoomba Chronicle Townsville Bulletin Trading Room Transcontinental Tribune International Tweed Daily News Tweed Heads News UNE Nucleus UNSW Tharunka UOW Tertangula USyd Honi Soit UTS Vertigo UWS Cruwsible Victor Harbor Times Viet Luan Village Journal WA Business News WA Today Wagga Wagga Daily Advertiser Wagin Argus Walcha News Walpole Weekly Wangaratta Chronicle Wanneroo Times Warnambool Extra Warnambool Standard Warrandyte Diary Warwick Daily News Wauchope Gazette Weekend Australian Weekly Times Weekly Times Wellington Times West Australian West Australian Great Southern West Australian Mid West & Gascoyne West Australian South West West Coast Sentinel West Side News Westender Western Advocate Western Echo Western Weekender Westerner Whitsunday Times Whyalla News Wimmera Mail Times Wingham Chronicle Wollondilly Advertiser Woroni (ANU) Wyndham Star Weekly Yass Tribune Yorke Peninsula Country Times Young Witness

Data

Downloaded 1 time by mlandauer

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (84 KB) Use the API

rows 10 / 356

Statistics

Average successful run time: 2 minutes

Total run time: 2 minutes

Total cpu time used: less than 5 seconds

Total disk space used: 109 KB

History

  • Manually ran revision eb07f3a2 and completed successfully .
    356 records added in the database
  • Created on morph.io

Scraper code

Ruby

australian-newspapers / scraper.rb