jamezpolley / newcastle_atdis_test

Newcastle Development (ATDIS) (TEST ONLY)


Newcastle ATDIS scraper (test only)

  • Server - Technology One
  • Pagnation - Yes
  • JSON - Yes - Yay....

We're using the ATDIS feed for this scraper.

The test environment only has data up to early 2018, so we need to manually start a lodgementstartdate otherwise the scraper doesn't see anything in the last 30 days and exits.

For simpler local testing, simply run make on a machine with docker and docker-compose installed. A docker image which (roughly) matches what morph.io will run in production will be created and executed.

If the scraper run was successful, the resulting data.sqlite will be extracted from the container and left in the current directory. The next time make is run, this will be moved to prev-data.sqlite

Enjoy

Contributors henare LoveMyData jamezpolley equivalentideas

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... D, [2019-06-17T04:39:01.650364 #15] DEBUG -- : Fetching https://property.ncc.nsw.gov.au/T1PRTESTBAU/WebAppServices/ATDIS/atdis/1.0/applications.json?lodgement_date_end=2019-06-17&lodgement_date_start=2018-01-01&page=3 D, [2019-06-17T04:39:34.773370 #15] DEBUG -- : Fetching https://property.ncc.nsw.gov.au/T1PRTESTBAU/WebAppServices/ATDIS/atdis/1.0/applications.json?lodgement_date_end=2019-06-17&lodgement_date_start=2018-01-01&page=4 D, [2019-06-17T04:39:59.167077 #15] DEBUG -- : Fetching

Data

Downloaded 4 times by jamezpolley

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (52 KB) Use the API

rows 10 / 93

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
DA2018/00085
60 Portland Place, New Lambton, NSW 2305
Swimming pool & retaining wall
2019-06-17
2018-02-05
DA2018/00100
20 Summit Street, North Lambton, NSW 2299
Alterations and additions to dwelling
2019-06-17
2018-02-07
DA2018/00101
4 Third Street, Adamstown, NSW 2289
Swimming Pool
2019-06-17
2018-02-08
DA2018/00163
41 Carrington Street, Mayfield, NSW 2304
Residential Alterations and Additions
2019-06-17
2018-09-14
DA2018/00162
282 King Street, Newcastle, NSW 2300
Residential Alterations and Additions
2019-06-17
2018-08-22
DA2018/00123
8 Baker Street, New Lambton, NSW 2305
Alterations and additions to existing dwelling
2019-06-17
2018-02-27
DA2018/00160
1 Smith Street, Hamilton South, NSW 2303
Residential Alterations and Additions
2019-06-17
2018-08-22
DA2018/00076
1 Thorn Road, Hamilton North, NSW 2292
Demolition of outbuilding erection of garage
2019-06-17
2018-01-31
DA2018/00156
9 Robinia Close, Elermore Vale, NSW 2287
New residential dwelling and retaining walls
2019-06-17
2018-08-07
DA2018/00159
9 Robinia Close, Elermore Vale, NSW 2287
New residential dwelling and retaining walls
2019-06-17
2018-08-10

Statistics

Average successful run time: 3 minutes

Total run time: about 3 hours

Total cpu time used: 2 minutes

Total disk space used: 102 KB

History

  • Manually ran revision ea4a5108 and completed successfully .
    93 records added, 93 records removed in the database
  • Manually ran revision 406d56a5 and completed successfully .
    nothing changed in the database
  • Manually ran revision 406d56a5 and completed successfully .
    nothing changed in the database
  • Manually ran revision 406d56a5 and completed successfully .
    nothing changed in the database
  • Manually ran revision 406d56a5 and completed successfully .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

newcastle_atdis_test / scraper.rb