planningalerts-scrapers / multiple_horizon

Scrapes multiple authorities with the horizon system


This is a scraper that runs on Morph. To get started see the documentation

Contributors mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.5.1 -----> Installing dependencies using bundler version 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/planningalerts-scrapers/horizon_xml.git  Fetching concurrent-ruby 1.1.5  Fetching minitest 5.11.3  Fetching thread_safe 0.3.6  Installing minitest 5.11.3  Installing thread_safe 0.3.6  Installing concurrent-ruby 1.1.5  Fetching public_suffix 3.1.1  Installing public_suffix 3.1.1  Using bundler 1.15.2  Fetching connection_pool 2.2.2  Fetching safe_yaml 1.0.5  Installing connection_pool 2.2.2  Installing safe_yaml 1.0.5  Fetching unf_ext 0.0.7.6  Installing unf_ext 0.0.7.6 with native extensions  Fetching hashdiff 0.4.0  Installing hashdiff 0.4.0  Fetching mime-types-data 3.2019.0331  Fetching net-http-digest_auth 1.4.1  Installing net-http-digest_auth 1.4.1  Installing mime-types-data 3.2019.0331  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing ntlm-http 0.1.1  Installing webrobots 0.1.2  Fetching httpclient 2.8.3  Fetching sqlite3 1.4.1  Installing sqlite3 1.4.1 with native extensions  Installing httpclient 2.8.3  Fetching timecop 0.9.1  Installing timecop 0.9.1  Fetching vcr 5.0.0  Installing vcr 5.0.0  Fetching tzinfo 1.2.5  Installing tzinfo 1.2.5  Fetching addressable 2.6.0  Installing addressable 2.6.0  Fetching net-http-persistent 3.0.1  Installing net-http-persistent 3.0.1  Fetching crack 0.4.3  Installing crack 0.4.3  Fetching i18n 1.6.0  Installing i18n 1.6.0  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.3  Installing nokogiri 1.10.3 with native extensions  Fetching webmock 3.6.0  Installing webmock 3.6.0  Fetching activesupport 5.2.3  Installing activesupport 5.2.3  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20190701  Installing domain_name 0.5.20190701  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.6  Installing mechanize 2.7.6  Using horizon_xml 1.1.1 from https://github.com/planningalerts-scrapers/horizon_xml.git (at master@c0ccb8a)  Bundle complete! 5 Gemfile dependencies, 34 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Post-install message from hashdiff:  The HashDiff constant used by this gem conflicts with another gem of a similar name. As of version 1.0 the HashDiff constant will be completely removed and replaced by Hashdiff. For more information see https://github.com/liufengyun/hashdiff/issues/45.  Bundle completed (27.70s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.2). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... The HashDiff constant used by this gem conflicts with another gem of a similar name. As of version 1.0 the HashDiff constant will be completely removed and replaced by Hashdiff. For more information see https://github.com/liufengyun/hashdiff/issues/45. Scraping authorities: cowra, liverpool_plains, uralla, walcha, weddin, maitland Collecting feed data for cowra... Saving record CDC 2019/15.1, 55 River Park Road COWRA 2794 NSW Saving record CC 2019/38.1, 27 Vaux Street COWRA 2794 NSW Saving record DA 2018/37.2, 31-29 Grenfell Road COWRA 2794 NSW Saving record CC 2019/68.1, 18 Thurburn Street WATTAMONDARA 2794 NSW Saving record DA 2019/67.1, 142 Clearview Road DARBYS FALLS 2793 NSW Saving record DA 2019/68.1, 18 Thurburn Street WATTAMONDARA 2794 NSW Saving record CC 2019/66.1, 9 Argoon Street COWRA 2794 NSW Saving record DA 2019/64.1, 507 Stoney Hill Road COWRA 2794 NSW Saving record DA 2019/65.1, 22 Taronga Drive COWRA 2794 NSW Saving record DA 2019/66.1, 9 Argoon Street COWRA 2794 NSW Saving record OC 2018/14.1, 113 Willowvale Road COWRA 2794 NSW Saving record CDC 2019/14.1, 47 Mulyan Street COWRA 2794 NSW Saving record FS 2018/50.1, Young Road COWRA 2794 NSW Saving record FS 2019/41.1, 264 Cucumber Lane CANOWINDRA 2804 NSW Saving record CC 2019/63.1, Killara Road COWRA 2794 NSW Saving record DA 2019/63.1, Killara Road COWRA 2794 NSW Saving record DA 2019/61.1, 28 Lachlan Street COWRA 2794 NSW Saving record DA 2019/62.1, 12 Casuarina Close COWRA 2794 NSW Saving record CC 2019/61.1, 28 Lachlan Street COWRA 2794 NSW Collecting feed data for liverpool_plains... Collecting feed data for uralla... Saving record DA2019/52.1, Court Street BUNDARRA 2359 NSW Saving record CD2019/7.1, 2 Prince Avenue URALLA 2358 NSW Saving record DA2019/51.1, 52 John Street URALLA 2358 NSW Saving record S682019/51.1, 52 John Street URALLA 2358 NSW Saving record OC1985/136.1, 14 Rowan Avenue URALLA 2358 NSW Saving record CC2019/31.1, 54 Hill Street URALLA 2358 NSW Saving record DA2019/50.1, Barleyfields Road URALLA 2358 NSW Saving record OSSM2019/1729.1, 24 Everton Vale Road ROCKY RIVER 2358 NSW Saving record S682019/48.1, 36 Bridge Street URALLA 2358 NSW Saving record DA2019/48.1, 36 Bridge Street URALLA 2358 NSW Saving record S682019/47.1, Court Street BUNDARRA 2359 NSW Saving record DA2019/49.1, 41 Saumarez War Service Road SAUMAREZ 2350 NSW Saving record CC2019/11.1, 24 McDonalds Lane ROCKY RIVER 2358 NSW Saving record OSSM2019/1730.1, 21 Sawmill Lane KINGSTOWN 2358 NSW Saving record S682019/34.1, 158 Gostwyck Road GOSTWYCK 2358 NSW Collecting feed data for walcha... Saving record S68 2019/25.1, 168W Evans Street WALCHA 2354 NSW Saving record CC 2019/243.1, 168W Evans Street WALCHA 2354 NSW Saving record DA 2019/51.1, 168W Evans Street WALCHA 2354 NSW Saving record S68 2019/24.1, 132S Derby Street WALCHA 2354 NSW Saving record CC 2019/241.1, 132S Derby Street WALCHA 2354 NSW Saving record CC 2019/242.1, 58W Fitzroy Street WALCHA 2354 NSW Saving record DA 2019/49.1, 132S Derby Street WALCHA 2354 NSW Collecting feed data for weddin... Saving record FS2018/68.1, 80 George Street GRENFELL 2810 NSW Saving record DA2019/24.1, NULL Grafton Street GRENFELL 2810 NSW Saving record DA2019/23.1, 21 Bradley Street GRENFELL 2810 NSW Saving record OC2019/11.1, 4951 Henry Lawson Way GRENFELL 2810 NSW Saving record DA2019/21.1, 70-72 Gooloogong Road GRENFELL 2810 NSW Saving record CC2019/21.1, 70-72 Gooloogong Road GRENFELL 2810 NSW Saving record OC2019/21.1, 70-72 Gooloogong Road GRENFELL 2810 NSW Saving record DA2019/22.1, 30 Stan McCabe Drive GRENFELL 2810 NSW Saving record CC2019/11.1, 4951 Henry Lawson Way GRENFELL 2810 NSW Collecting feed data for maitland... Saving record DA/2019/608, 14 Irma Place LOCHINVAR NSW 2321 Saving record DA/2018/1587:2, 51 Shipley Drive RUTHERFORD NSW 2320 Saving record CDC/2019/333, 50 Dunmore Road LARGS NSW 2320 Saving record CDC/2019/334, 12 Cascade Close LOUTH PARK NSW 2320 Saving record CC/2019/573, 14 Irma Place LOCHINVAR NSW 2321 Saving record DA/2019/605, 63 Laurie Drive RAWORTH NSW 2321 Saving record DA/2019/607, 24 Murray Street EAST MAITLAND NSW 2323 Saving record CDC/2019/331, 34 Mapplewell Circuit FARLEY NSW 2320 Saving record CDC/2019/332, 105 Billabong Parade CHISHOLM NSW 2322 Saving record CC/2019/569, 4 Hedda Close LOCHINVAR NSW 2321 Saving record CC/2019/570, 21 King Street LORN NSW 2320 Saving record CC/2019/572, 24 Murray Street EAST MAITLAND NSW 2323 Saving record DA/2019/602, 39 Sandstone Drive ANAMBAH NSW 2320 Saving record DA/2019/230:1, 6 Waterford Close ASHTONFIELD NSW 2323 Saving record DA/2019/603, 10 Tarragon Way CHISHOLM NSW 2322 Saving record DA/2019/604, 41 Mapplewell Circuit FARLEY NSW 2320 Saving record CDC/2019/330, 29 Percher Street CHISHOLM NSW 2322 Saving record CC/2019/566, 39 Sandstone Drive ANAMBAH NSW 2320 Saving record CC/2019/568, 51 Shipley Drive RUTHERFORD NSW 2320 Saving record DA/2019/596, 17 Somerset Drive THORNTON NSW 2322 Saving record DA/2019/597, 26 Harvest Boulevard CHISHOLM NSW 2322 Saving record DA/2019/598, 54 Martins Wharf Road MILLERS FOREST NSW 2324 Saving record DA/2019/599, 28 Waterworks Road RUTHERFORD NSW 2320 Saving record DA/2019/601, 6 Irma Place LOCHINVAR NSW 2321 Saving record CDC/2019/329, 40 Golden Whistler Avenue ABERGLASSLYN NSW 2320 Saving record CC/2019/562, 17 Somerset Drive THORNTON NSW 2322 Saving record CC/2019/563, 104 Mustang Drive RUTHERFORD NSW 2320 Saving record CC/2019/564, 15 Holliday Close RUTHERFORD NSW 2320 Saving record CC/2019/565, 28 Waterworks Road RUTHERFORD NSW 2320 Saving record DA/2019/592, 21 Firebrick Drive THORNTON NSW 2322 Saving record DA/2019/593, 48 Ardennes Circuit GILLIESTON HEIGHTS NSW 2321 Saving record DA/2019/594, 52A Kensington Road BOLWARRA NSW 2320 Saving record DA/2019/595, 26 Mapplewell Circuit FARLEY NSW 2320 Saving record CDC/2019/326, 38 Park Street EAST MAITLAND NSW 2323 Saving record CDC/2019/327, 85 Arrowtail Street CHISHOLM NSW 2322 Saving record CDC/2019/328, 155 Billabong Parade CHISHOLM NSW 2322 Saving record CC/2019/557, 20 Corella Close ABERGLASSLYN NSW 2320 Saving record CC/2019/559, 71 Turton Street METFORD NSW 2323 Saving record CC/2019/560, 48 Ardennes Circuit GILLIESTON HEIGHTS NSW 2321 Saving record CC/2019/561, 26 Mapplewell Circuit FARLEY NSW 2320 Saving record DA/2019/587, 8 Vantage Court BOLWARRA NSW 2320 Saving record DA/2019/588, 12 Duckenfield Road MORPETH NSW 2321 Saving record DA/2019/589, 30 Sandstone Drive WINDELLA NSW 2320 Saving record DA/2019/590, 26 Auburn Street GILLIESTON HEIGHTS NSW 2321 Saving record CDC/2019/325, 22 Harlington Avenue FARLEY NSW 2320 Saving record DA/2019/583, 37 Page Street EAST MAITLAND NSW 2323 Saving record DA/2019/584, 552 Luskintyre Road LUSKINTYRE NSW 2321 Saving record DA/2018/1691:1, 27 Percher Street CHISHOLM NSW 2322 Saving record DA/2019/585, 32 Kensington Road BOLWARRA NSW 2320 Saving record DA/2019/586, 73 Owlpen Lane FARLEY NSW 2320 Saving record CDC/2019/321, 16 Whiteley Avenue METFORD NSW 2323 Saving record CDC/2019/322, 4 Ibis Parade WOODBERRY NSW 2322 Saving record CDC/2016/2221:1, 76 Dragonfly Drive CHISHOLM NSW 2322 Saving record CDC/2019/324, 2 Angelica Road CHISHOLM NSW 2322 Saving record CC/2019/554, 30 Lancaster Street THORNTON NSW 2322 Saving record CC/2019/555, 34 Lancaster Street THORNTON NSW 2322 Saving record CC/2019/556, 12 Hunter Close LOCHINVAR NSW 2321 Saving record DA/2019/579, 3 Eldon Drive LOUTH PARK NSW 2320 Saving record DA/2019/580, 12 Irma Place LOCHINVAR NSW 2321 Saving record DA/2018/1846:1, 54 Martins Wharf Road MILLERS FOREST NSW 2324 Saving record DA/2019/581, 21 Belmore Road LORN NSW 2320 Saving record DA/2019/582, 8 Lambert Close BOLWARRA NSW 2320 Saving record CDC/2019/320, 14 Corella Close ABERGLASSLYN NSW 2320 Saving record CC/2019/549, 3 Eldon Drive LOUTH PARK NSW 2320 Saving record CC/2019/550, 79 Tarragon Way CHISHOLM NSW 2322 Saving record CC/2019/551, 44 Tournament Street RUTHERFORD NSW 2320 Saving record CC/2019/552, 8 Lambert Close BOLWARRA NSW 2320 Saving record CC/2019/553, 17 Carnival Close WINDELLA NSW 2320 Saving record DA/2019/573, 547 Raymond Terrace Road CHISHOLM NSW 2322 Saving record DA/2019/574, 547 Raymond Terrace Road CHISHOLM NSW 2322 Saving record DA/2017/1173:1, 2 Watervale Circuit CHISHOLM NSW 2322 Saving record DA/2019/575, 361 Morpeth Road MORPETH NSW 2321 Saving record DA/2019/576, 12 Cornwell Street THORNTON NSW 2322 Saving record DA/2019/577, 4 Lancaster Street THORNTON NSW 2322 Saving record DA/2019/578, 82 Ballydoyle Drive ASHTONFIELD NSW 2323 Saving record CC/2019/545, 177A Bolwarra Park Drive BOLWARRA HEIGHTS NSW 2320 Saving record CC/2019/546, 6 Cockatoo Ridge ABERGLASSLYN NSW 2320 Saving record CC/2019/547, 4 Newfield Street RUTHERFORD NSW 2320 Saving record CC/2019/548, 82 Ballydoyle Drive ASHTONFIELD NSW 2323 Saving record DA/2019/569, 30 John Verge Avenue RUTHERFORD NSW 2320 Saving record DA/2019/570, 2 Collaroy Parade LOUTH PARK NSW 2320 Saving record DA/2019/571, 21 Young Street EAST MAITLAND NSW 2323 Saving record DA/2019/572, 7 Rebecca Close RUTHERFORD NSW 2320 Saving record CDC/2019/317, 18 Friarbird Ridge ABERGLASSLYN NSW 2320 Saving record CDC/2019/318, 15 Collaroy Parade LOUTH PARK NSW 2320 Saving record CDC/2019/319, 7 Vantage Court BOLWARRA NSW 2320 Saving record CC/2019/542, 30 John Verge Avenue RUTHERFORD NSW 2320 Saving record CC/2019/543, 2 Collaroy Parade LOUTH PARK NSW 2320 Saving record CC/2019/544, 21 Young Street EAST MAITLAND NSW 2323 Saving record DA/2019/567, 24 Lancaster Street THORNTON NSW 2322 Saving record DA/2019/568, 5 Collaroy Parade LOUTH PARK NSW 2320 Saving record CDC/2019/316, 65 Barr Promenade THORNTON NSW 2322 Saving record CC/2019/538, 101 Billabong Parade CHISHOLM NSW 2322 Saving record CC/2019/539, 2 Red Arrow Close CHISHOLM NSW 2322 Saving record CC/2019/540, 24 Lancaster Street THORNTON NSW 2322 Saving record CC/2019/541, 5 Collaroy Parade LOUTH PARK NSW 2320 Saving record SM2019/15, 45 Cantwell Road LOCHINVAR NSW 2321 Saving record DA/2019/565, 13 Lucy Place RAWORTH NSW 2321 Saving record DA/2019/566, 3 Norfolk Street ASHTONFIELD NSW 2323 Saving record CC/2019/535, 22 Fitzroy Street EAST MAITLAND NSW 2323 Saving record CC/2019/536, 13 Lucy Place RAWORTH NSW 2321 Saving record CC/2019/537, 47 Grand Parade RUTHERFORD NSW 2320 Saving record DA/2019/558, 3 Angelica Road CHISHOLM NSW 2322 Saving record DA/2019/559, 25 Irma Place LOCHINVAR NSW 2321 Saving record DA/2019/560, 18 Bairds Close RUTHERFORD NSW 2320 Saving record DA/2019/561, 43 Ferraby Drive METFORD NSW 2323 Saving record DA/2019/562, 14 Kilkenny Circuit ASHTONFIELD NSW 2323 Saving record DA/2019/563, 67 Tarragon Way CHISHOLM NSW 2322 Saving record DA/2019/564, 14 Cross Street SOUTH MAITLAND NSW 2320 Saving record CDC/2019/313, 3 Carey Close LOUTH PARK NSW 2320 Saving record CDC/2019/314, 29 Percher Street CHISHOLM NSW 2322 Saving record CDC/2019/315, 11 Harlington Avenue FARLEY NSW 2320 Saving record CC/2019/530, 3 Angelica Road CHISHOLM NSW 2322 Saving record CC/2019/532, 14 High Street MORPETH NSW 2321 Saving record CC/2019/533, 14 Kilkenny Circuit ASHTONFIELD NSW 2323 Saving record CC/2019/534, 67 Tarragon Way CHISHOLM NSW 2322

Data

Downloaded 109 times by DevAppProject openaustralia mattykn

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (66 KB) Use the API

rows 10 / 166

council_reference address description info_url date_scraped date_received authority_label
CDC 2019/15.1
55 River Park Road COWRA 2794 NSW
Alteration and/or Additions to existing Residential Dwelling
2019-07-17
2019-07-17
cowra
CC 2019/38.1
27 Vaux Street COWRA 2794 NSW
Alterations and additions to Cowra Bowling Club
2019-07-17
2019-07-16
cowra
DA 2018/37.2
31-29 Grenfell Road COWRA 2794 NSW
Alteration and/or Additions to existing Storage/Warhouse Building
2019-07-17
2019-07-12
cowra
CC 2019/68.1
18 Thurburn Street WATTAMONDARA 2794 NSW
New Building Residential Garage/Shed
2019-07-17
2019-07-12
cowra
DA 2019/67.1
142 Clearview Road DARBYS FALLS 2793 NSW
Alteration and/or Additions to existing Rural Dwelling
2019-07-17
2019-07-12
cowra
DA 2019/68.1
18 Thurburn Street WATTAMONDARA 2794 NSW
New Building Residential Garage/Shed
2019-07-17
2019-07-12
cowra
CC 2019/66.1
9 Argoon Street COWRA 2794 NSW
Shed
2019-07-17
2019-07-09
cowra
DA 2019/64.1
507 Stoney Hill Road COWRA 2794 NSW
Removal of existing dwelling (transportable) and installation of new
2019-07-17
2019-07-09
cowra
DA 2019/65.1
22 Taronga Drive COWRA 2794 NSW
Garage
2019-07-17
2019-07-09
cowra
DA 2019/66.1
9 Argoon Street COWRA 2794 NSW
Shed
2019-07-17
2019-07-09
cowra

Statistics

Average successful run time: 2 minutes

Total run time: 15 minutes

Total cpu time used: less than 20 seconds

Total disk space used: 91.5 KB

History

  • Auto ran revision cd7ec55f and completed successfully .
    166 records added, 159 records removed in the database
  • Auto ran revision cd7ec55f and completed successfully .
    159 records added, 148 records removed in the database
  • Auto ran revision cd7ec55f and failed .
    140 records added, 133 records removed in the database
  • Auto ran revision cd7ec55f and completed successfully .
    141 records added, 141 records removed in the database
  • Auto ran revision cd7ec55f and completed successfully .
    141 records added, 135 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

multiple_horizon / scraper.rb