jamezpolley / newcastle_atdis_test

Newcastle Development (ATDIS) (TEST ONLY)

README.md

Newcastle ATDIS scraper

  • Server - Technology One
  • ATDIS feed. Very simple to scrape. Thank you Newcastle Council!
  • JSON - Yes - Yay....

For simple local testing, simply run make on a machine with docker and docker-compose installed. A docker image which (roughly) matches what morph.io will run in production will be created and executed.

If the scraper run was successful, the resulting data.sqlite will be extracted from the container and left in the current directory. The next time make is run, this will be moved to prev-data.sqlite

Enjoy

Contributors henare LoveMyData jamezpolley equivalentideas JasonThomasData mercuryfrost nik3daz CloCkWeRX

Last run failed with status code 999.

Console output of last run

The Morph Github App installed on jamezpolley needs access to the repository newcastle_atdis_test. Please go to https://github.com/apps/morph-io/installations/new/permissions?repository_ids%5B%5D=177900144&repository_ids%5B%5D=187941427&repository_ids%5B%5D=229189325&repository_ids%5B%5D=186961522&repository_ids%5B%5D=186960709&repository_ids%5B%5D=189337188&repository_ids%5B%5D=160617981&repository_ids%5B%5D=186969111&repository_ids%5B%5D=237375625&repository_ids%5B%5D=186966763&repository_ids%5B%5D=177900124&repository_ids%5B%5D=128887759&repository_ids%5B%5D=157314243&repository_ids%5B%5D=172443537&repository_ids%5B%5D=185325149&repository_ids%5B%5D=180955387&repository_ids%5B%5D=161434693&repository_ids%5B%5D=186540164&repository_ids%5B%5D=274279424&repository_ids%5B%5D=297501812&repository_ids%5B%5D=246733966&repository_ids%5B%5D=172621382&repository_ids%5B%5D=180928872&repository_ids%5B%5D=185505981&repository_ids%5B%5D=188640212&repository_ids%5B%5D=193009978&repository_ids%5B%5D=190689725&repository_ids%5B%5D=707463452&repository_ids%5B%5D=168463091&repository_ids%5B%5D=141211345&repository_ids%5B%5D=126745721&repository_ids%5B%5D=177900097&repository_ids%5B%5D=187308486&repository_ids%5B%5D=186761613&repository_ids%5B%5D=242864779&repository_ids%5B%5D=164060314&repository_ids%5B%5D=138673644&repository_ids%5B%5D=140508663&suggested_target_id=216918 Why? See https://morph.io/documentation/github_app

Data

Downloaded 7 times by jamezpolley

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (180 KB) Use the API

rows 10 / 395

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
DA2019/00513
26 Meya Crescent, Fletcher, NSW 2287
Dwelling house and associated retaining walls
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00513
2019-07-05
2019-05-08
DA2019/00261
2 Lloyd Street, Merewether, NSW 2291
Demolition of dwelling and erection of two storey dwelling
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00261
2019-07-05
2019-03-14
DA2019/00578
11 Mackie Avenue, New Lambton, NSW 2305
Swimming pool
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00578
2019-07-05
2019-05-29
DA2019/00547
346 Maitland Road, Mayfield, NSW 2304
Demolition of outbuilding erection of garage
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00547
2019-07-05
2019-05-20
DA2019/00393
163 Christo Road, Waratah, NSW 2298
Alterations and additions to dwelling
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00393
2019-07-05
2019-04-11
DA2019/00581
4 Gavey Street, Mayfield, NSW 2304
Demolition of outbuilding erection of garage
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00581
2019-07-05
2019-05-24
DA2019/00118
119 Bruce Street, Cooks Hill, NSW 2300
Dwelling, alterations and additions to
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00118
2019-07-05
2019-02-08
DA2019/00400
9 Pembroke Street, Stockton, NSW 2295
Demolition of outbuilding erection of garage and storage area
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00400
2019-07-05
2019-04-12
DA2019/00468
116 Young Road, Lambton, NSW 2299
Dwelling house, alterations and additions to
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00468
2019-07-05
2019-05-06
DA2019/00360
3 Karoola Road, Lambton, NSW 2299
Single storey dwelling
https://property.ncc.nsw.gov.au/T1PRPROD/WebApps/eProperty/P1/eTrack/eTrackApplicationDetails.aspx?r=TCON.LG.WEBGUEST&f=%24P1.ETR.APPDET.VIW&ApplicationId=DA2019%2f00360
2019-07-05
2019-04-03

Statistics

Average successful run time: 3 minutes

Total run time: about 12 hours

Total cpu time used: 16 minutes

Total disk space used: 236 KB

History

  • Auto ran and failed .
  • Auto ran and failed .
  • Auto ran and failed .
  • Auto ran and failed .
  • Auto ran and failed .
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

newcastle_atdis_test / scraper.rb