planningalerts-scrapers / bogan

Bogan Development Applications


Bogan Shire Council Scraper

  • Server - Unknown but I like it
  • Cookie tracking - Yes
  • Agreement - Yes
  • Pagnation - No
  • JSON - Yes - Yay....

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, today -10 days)
  • thismonth
  • lastmonth
  • year (eg. 2016)

Enjoy

Contributors CloCkWeRX LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> PHP app detected -----> Bootstrapping... -----> Installing platform packages...  NOTICE: No runtime required in composer.json; requirements  from dependencies in composer.lock will be used for selection  - php (7.2.1)  - ext-gd (bundled with php)  - ext-mbstring (bundled with php)  - ext-pdo_sqlite (bundled with php)  - ext-sqlite3 (bundled with php)  - apache (2.4.29)  - nginx (1.8.1) -----> Installing dependencies...  Composer version 1.5.2 2017-09-11 16:59:25  Loading composer repositories with package information  Installing dependencies from lock file  Package operations: 4 installs, 0 updates, 0 removals  - Installing pguardiario/phpuri (1.0): Downloading (connecting...)Downloading (0%) Downloading (100%)  - Installing electrolinux/phpquery (0.9.6): Downloading (connecting...)Downloading (0%) Downloading (5%)Downloading (10%)Downloading (15%)Downloading (20%)Downloading (25%)Downloading (30%)Downloading (35%)Downloading (40%)Downloading (45%)Downloading (50%)Downloading (55%)Downloading (60%)Downloading (65%)Downloading (70%)Downloading (75%)Downloading (80%)Downloading (85%)Downloading (90%)Downloading (95%)Downloading (100%)  - Installing byjg/pgbrowser (1.0.0): Downloading (connecting...)Downloading (0%) Downloading (65%)Downloading (100%)  - Installing openaustralia/scraperwiki (dev-morph_defaults e996fe0): Cloning e996fe0253  Generating optimized autoload files -----> Preparing runtime environment... -----> Checking for additional extensions to install...  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data between 10/01/2018 and 20/01/2018, changable via MORPH_PERIOD environment Skipping already saved record DA 2018/3 Skipping already saved record S96 2017/31/2 Skipping already saved record CONS 2017/31/2 Skipping already saved record OCC 2017/31/2 Skipping already saved record DA 2018/4 Skipping already saved record CONS 2018/3 Skipping already saved record OCC 2018/3

Data

Downloaded 377 times by LoveMyData TheZepto MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (10 KB) Use the API

rows 10 / 22

council_reference address description info_url comment_url date_scraped date_received on_notice_from on_notice_to
010.2016.00000018.001
27 Ellen St, Nyngan, NSW 2825
New Garage/Carport/Pergola/Shed
2016-08-03
2016-08-02
010.2016.00000020.001
13 Mudal St, Nyngan, NSW 2825
New Dual Occupancy
2016-09-06
2016-09-02
010.2016.00000023.001
31 Dandaloo St, Nyngan, NSW 2825
New Garage and Relocation of Pergola
2016-09-15
2016-09-15
010.2016.00000022.001
876 Barrier Hwy, NYNGAN, NSW 2825
New Education Building
2016-09-15
2016-09-12
010.2016.00000024.001
Lawlor St, NYNGAN, NSW 2825
New Industrial Building
2016-09-27
2016-09-26
010.2017.00000006.001
54 Oatley St, Nyngan, NSW 2825
New Transportable Dwelling
2017-04-10
2017-03-14
2017-04-10
2017-04-21
DA 2017/34
51-57 Flashman Ave, Nyngan 2825 NSW
New Single Dwelling
mailto:admin@bogan.nsw.gov.au
2017-10-29
2017-10-27
DA 2017/33
5 Cathundril St, Nyngan 2825 NSW
Mobile Vending Vehicle
mailto:admin@bogan.nsw.gov.au
2017-10-29
2017-10-26
DA 2017/35
50-52 Cannonbar St, Nyngan 2825 NSW
New Garage/Carport/Pergola/Shed
mailto:admin@bogan.nsw.gov.au
2017-11-22
2017-11-22
CONS 2017/35
50-52 Cannonbar St, Nyngan 2825 NSW
New Garage/Carport/Pergola/Shed
mailto:admin@bogan.nsw.gov.au
2017-11-22
2017-11-22

Statistics

Average successful run time: 1 minute

Total run time: about 10 hours

Total cpu time used: 10 minutes

Total disk space used: 45 KB

History

  • Auto ran revision d7f52647 and completed successfully .
    nothing changed in the database
  • Auto ran revision d7f52647 and completed successfully .
    7 records added in the database
  • Auto ran revision d7f52647 and failed .
    nothing changed in the database
  • Auto ran revision d7f52647 and failed .
    nothing changed in the database
  • Auto ran revision d7f52647 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

PHP

bogan / scraper.php