Sutherland Shire Council Scraper

This site is a typical .NET site involves the followings.

  • Cookie - No
  • Pagnation - Yes
  • Javascript - Yes
  • Structured HTML - Good

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default)
  • thismonth
  • lastmonth

Enjoy

Contributors mlandauer LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.1 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/......  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Using bundler 1.15.2  Fetching unf_ext 0.0.7.5  Fetching httpclient 2.8.3  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Installing httpclient 2.8.3  Fetching net-http-digest_auth 1.4.1  Installing unf_ext 0.0.7.5 with native extensions  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.3.0  Installing net-http-digest_auth 1.4.1  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Installing mini_portile2 2.3.0  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.8.2  Installing nokogiri 1.8.2 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching domain_name 0.5.20170404  Installing domain_name 0.5.20170404  Fetching http-cookie 1.0.3  Installing http-cookie 1.0.3  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 18 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (22.85s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting 'TM' data, changable via MORPH_PERIOD environment Skipping already saved record DA22/0430 Skipping already saved record DA22/0431 Skipping already saved record DA22/0433 Skipping already saved record DA22/0259 Skipping already saved record DA22/0416 Skipping already saved record DA22/0420 Skipping already saved record DA22/0439 Skipping already saved record MA22/0123 Skipping already saved record DA22/0436 Skipping already saved record MA22/0118 Skipping already saved record DA22/0429 Skipping already saved record DA22/0342 Skipping already saved record DA22/0427 Skipping already saved record MA22/0077 Skipping already saved record DA22/0443 Getting page 2... Skipping already saved record DA22/0437 Skipping already saved record DA22/0442 Skipping already saved record DA22/0441 Skipping already saved record DA22/0421 Skipping already saved record DA22/0452 Skipping already saved record MA22/0122 Skipping already saved record DA22/0448 Skipping already saved record DA22/0449 Skipping already saved record DA22/0450 Skipping already saved record DA22/0435 Skipping already saved record DA22/0440 Skipping already saved record DA22/0283 Skipping already saved record DA22/0447 Skipping already saved record DA22/0451 Skipping already saved record DA22/0458 Getting page 3... Skipping already saved record MA22/0117 Skipping already saved record MA22/0126 Skipping already saved record DA22/0446 Skipping already saved record DA22/0454 Skipping already saved record DA22/0460 Skipping already saved record DA22/0461 Skipping already saved record DA22/0456 Skipping already saved record DA22/0343 Skipping already saved record DA22/0453 Skipping already saved record DA22/0455 Skipping already saved record DA22/0464 Skipping already saved record DA22/0462 Skipping already saved record DA22/0468 Skipping already saved record DA22/0457 Skipping already saved record DA22/0463 Getting page 4... Skipping already saved record MA22/0128 Skipping already saved record DA22/0376 Skipping already saved record MA22/0127 Skipping already saved record DA22/0267 Skipping already saved record DA22/0474 Skipping already saved record MA22/0129 Skipping already saved record DA22/0488 Skipping already saved record DA22/0483 Skipping already saved record DA22/0438 Skipping already saved record DA22/0466 Skipping already saved record DA22/0470 Skipping already saved record DA22/0481 Skipping already saved record DA22/0482 Skipping already saved record DA22/0469 Skipping already saved record DA22/0484 Getting page 5... Skipping already saved record DA22/0494 Skipping already saved record DA22/0486 Saving record DA22/0478, 39 Struen Marie Street KAREELA NSW 2232 Skipping already saved record MA22/0125 Skipping already saved record DA22/0479 Skipping already saved record DA22/0476 Skipping already saved record DA22/0485 Skipping already saved record DA22/0490 Skipping already saved record DA22/0475 Skipping already saved record DA22/0489 Skipping already saved record DA22/0467 Saving record DA22/0493, 79 Scarborough Street BUNDEENA NSW 2230 Saving record DA22/0471, 33 Ingara Avenue MIRANDA NSW 2228 Saving record DA22/0491, 86 Gannons Road CARINGBAH SOUTH NSW 2229 Saving record DA22/0496, 16 Namatjira Street BARDEN RIDGE NSW 2234 Getting page 6... Saving record DA22/0477, 5 Dover Place ENGADINE NSW 2233 Saving record DA22/0497, 13 Field Place ILLAWONG NSW 2234

Data

Downloaded 1544 times by LoveMyData dcalde

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.6 MB) Use the API

rows 10 / 5517

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
DA17/0569
133 Fowler Rd ILLAWONG 2234
Construction & installation of shared jetty extending along the prolongation of the shared boundary & installation of ramp and pontoon
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23
2017-05-24
2017-06-07
DA17/0598
121 Linden St SUTHERLAND 2232
Construction of a cremator facility at Woronora Cemetary
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-22
DA17/0600
Construction of new dwelling
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-22
DA17/0601
12 Drummond Rd OYSTER BAY 2225
Construction of swimming pool, deck, retaining walls and pergola
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-22
2017-05-26
2017-06-09
DA17/0602
468 Port Hacking Rd CARINGBAH 2229
Construction of 2 swimming pools in approved dual occupancy
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23
DA17/0603
Construction of new dwelling and retaining walls
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23
DA17/0604
117 Jannali Ave SUTHERLAND 2232
Two lot torrens title subdivision of approved dual occupancy
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23
DA17/0605
10 Warrangarree Dr WORONORA HEIGHTS 2233
Deck addition to existing residential dwelling
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23
DA17/0606
10 Wategos St GREENHILLS BEACH 2230
Construction of swimming pool
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23
DA17/0607
24 Stephen Rd ENGADINE 2233
Construction of retaining wall
mailto:ssc@ssc.nsw.gov.au
2017-05-25
2017-05-23

Statistics

Average successful run time: 1 minute

Total run time: 1 day

Total cpu time used: 29 minutes

Total disk space used: 2.63 MB

History

  • Auto ran revision 87d82a1d and completed successfully .
    7 records added in the database
  • Auto ran revision 87d82a1d and completed successfully .
    8 records added in the database
  • Auto ran revision 87d82a1d and completed successfully .
    nothing changed in the database
  • Auto ran revision 87d82a1d and completed successfully .
    nothing changed in the database
  • Auto ran revision 87d82a1d and completed successfully .
    2 records added in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

sutherland / scraper.rb