LoveMyData / maitland

Maitland Development Applications

Scrapes myhorizon.maitland.nsw.gov.au

Horizon - Simplifying Solutions for Local Government - SolOrient


Maitland City Council Scraper

This council involves the followings * Server - nginx/1.8.0 * Cookie tracking - Yes * Pagnation - No - However, modified query to pull max 500 records * XML - Yes * Horizon system

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default, today -7 days) * thismonth * lastmonth

Enjoy

Contributors CloCkWeRX LoveMyData equivalentideas

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.0.0-p481 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Rubygems 2.0.14 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.  Using bundler 1.15.2  Fetching unf_ext 0.0.7.2  Installing unf_ext 0.0.7.2 with native extensions  Fetching httpclient 2.8.2.4  Installing httpclient 2.8.2.4  Fetching mime-types-data 3.2016.0521  Installing mime-types-data 3.2016.0521  Fetching net-http-digest_auth 1.4  Installing net-http-digest_auth 1.4  Fetching net-http-persistent 2.9.4  Installing net-http-persistent 2.9.4  Fetching mini_portile2 2.1.0  Installing mini_portile2 2.1.0  Fetching pkg-config 1.1.7  Installing pkg-config 1.1.7  Fetching ntlm-http 0.1.1  Installing ntlm-http 0.1.1  Fetching webrobots 0.1.2  Installing webrobots 0.1.2  Fetching sqlite3 1.3.11  Installing sqlite3 1.3.11 with native extensions  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching mime-types 3.1  Installing mime-types 3.1  Fetching nokogiri 1.6.8  Installing nokogiri 1.6.8 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Fetching domain_name 0.5.20160826  Installing domain_name 0.5.20160826  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching http-cookie 1.0.2  Installing http-cookie 1.0.2  Fetching mechanize 2.7.5  Installing mechanize 2.7.5  Bundle complete! 2 Gemfile dependencies, 19 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Bundle completed (24.06s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Scraping from 14/04/2018 to 21/04/2018, changable via MORPH_PERIOD variable Saving record CCP-2018-0994, 14 Denton Park Drive RUTHERFORD 2320 NSW Saving record CC-2018-0993, 84 Brigantine Street RUTHERFORD 2320 NSW Saving record DA-2018-0992, 84 Brigantine Street RUTHERFORD 2320 NSW Saving record CDP-2018-0989, 38 Collinson Street TENAMBIT 2323 NSW Saving record CC-2018-0981, 35 SANDSTONE DRIVE ANAMBAH 2320 NSW Saving record DA-2018-0980, 35 SANDSTONE DRIVE ANAMBAH 2320 NSW Saving record CCP-2018-0979, 1 Easton Close RUTHERFORD 2320 NSW Saving record CCP-2018-0978, 48 Bonville Avenue THORNTON 2322 NSW Saving record CCP-2018-0976, 19 Anne Street EAST MAITLAND 2323 NSW Saving record CC-2018-0975, 50 TOURNAMENT STREET RUTHERFORD 2320 NSW Saving record DA-2018-0974, 50 TOURNAMENT STREET RUTHERFORD 2320 NSW Saving record DA-2018-0973, 175 Parish Drive THORNTON 2322 NSW Skipping already saved record DA-2018-0972 Skipping already saved record DA-2018-0971 Skipping already saved record CDP-2018-0970 Skipping already saved record CDP-2018-0969 Skipping already saved record CDP-2018-0968 Skipping already saved record DA-2018-0967 Skipping already saved record CDP-2018-0964 Skipping already saved record CCP-2018-0963 Skipping already saved record CDP-2018-0962 Skipping already saved record CDP-2018-0961 Skipping already saved record CDP-2018-0960 Skipping already saved record CDP-2018-0959 Skipping already saved record DA-2018-0958 Skipping already saved record DA-2018-0957 Skipping already saved record CC-2018-0956 Skipping already saved record DA-2018-0955 Skipping already saved record CC-2018-0954 Skipping already saved record DA-2018-0953 Skipping already saved record CC-2018-0952 Skipping already saved record DA-2018-0951 Skipping already saved record CCP-2018-0950 Skipping already saved record DA-2018-0949 Skipping already saved record DA-2018-0948 Skipping already saved record CDP-2018-0947 Skipping already saved record CCP-2018-0946 Skipping already saved record CDP-2018-0945 Skipping already saved record CDP-2018-0944 Skipping already saved record CDP-2018-0943 Skipping already saved record CDP-2018-0942 Skipping already saved record CC-2018-0941 Skipping already saved record DA-2018-0940 Skipping already saved record DA-2018-0939 Skipping already saved record CDP-2018-0938 Skipping already saved record DA-2018-0937 Skipping already saved record DA-2018-0936 Skipping already saved record DA-2018-0935 Skipping already saved record CCP-2018-0934 Skipping already saved record CCP-2018-0933 Skipping already saved record CCP-2018-0932 Skipping already saved record CC-2018-0931 Skipping already saved record DA-2018-0930 Skipping already saved record DA-2018-0921 Skipping already saved record CCP-2018-0920 Skipping already saved record CCP-2018-0919 Skipping already saved record DA-2018-0918 Skipping already saved record CCP-2018-0917 Skipping already saved record CDP-2018-0916 Skipping already saved record CDP-2018-0915 Skipping already saved record CDP-2018-0914 Skipping already saved record CCP-2018-0913 Skipping already saved record CDP-2018-0912 Skipping already saved record DA-2018-0911 Skipping already saved record CDP-2018-0910 Skipping already saved record CDP-2018-0909 Saving record DA-2017-2490-A, 13 2320 NSW Skipping already saved record DA-2016-2641-A Skipping already saved record DA-2016-2210-A Skipping already saved record DA-2011-0940-A

Data

Downloaded 488 times by LoveMyData gdmhunter

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.16 MB) Use the API

rows 10 / 3990

council_reference address description info_url comment_url date_scraped date_received
CCP-2016-2159
5 Lorn Street LORN 2320
Pool
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
DA-2016-2158
7 Justine Parade RUTHERFORD 2320
Tree Application
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
DA-2016-2157
56 Kyle Street RUTHERFORD 2320
Recreation Facility
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
CC-2016-2156
23 McKeachie Drive ABERGLASSLYN 2320
Garage
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
DA-2016-2155
23 McKeachie Drive ABERGLASSLYN 2320
Garage
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
CC-2016-2154
24 Rivergum Drive ABERGLASSLYN 2320
Dwelling House
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
DA-2016-2153
24 Rivergum Drive ABERGLASSLYN 2320
Dwelling House
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
CDP-2016-2152
52 Hillcrest Drive GILLIESTON HEIGHTS 2321
Dwelling House
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
CCP-2016-2151
14 Mitchell Drive EAST MAITLAND 2323
Shop
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15
CDP-2016-2150
28 Queen Street RUTHERFORD 2320
Dwelling alteration/addition
mailto:info@maitland.nsw.gov.au
2016-09-16
2016-09-15

Statistics

Average successful run time: less than a minute

Total run time: about 7 hours

Total cpu time used: 9 minutes

Total disk space used: 1.19 MB

History

  • Auto ran revision 253daa56 and completed successfully .
    13 records added in the database
    2 pages scraped
  • Auto ran revision 253daa56 and completed successfully .
    9 records added in the database
  • Auto ran revision 253daa56 and completed successfully .
    26 records added in the database
    2 pages scraped
  • Auto ran revision 253daa56 and completed successfully .
    1 record added in the database
    2 pages scraped
  • Auto ran revision 253daa56 and completed successfully .
    21 records added in the database
    2 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

maitland / scraper.rb