Albury City Council Scraper

  • Server - Unknown but I like it
  • Cookie tracking - Yes
  • Agreement - Yes
  • Pagnation - No
  • JSON - Yes - Yay....

Setup MORPH_PERIOD for data recovery, available options are

  • thisweek (default, today -10 days)
  • thismonth
  • lastmonth
  • year (eg. 2016)

Enjoy

Contributors henare LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> PHP app detected -----> Bootstrapping... -----> Installing platform packages...  NOTICE: No runtime required in composer.json; requirements  from dependencies in composer.lock will be used for selection  - php (7.2.12)  - ext-gd (bundled with php)  - ext-mbstring (bundled with php)  - ext-pdo_sqlite (bundled with php)  - ext-sqlite3 (bundled with php)  - nginx (1.8.1)  - apache (2.4.37) -----> Installing dependencies...  Composer version 1.5.2 2017-09-11 16:59:25  Loading composer repositories with package information  Installing dependencies from lock file  Package operations: 4 installs, 0 updates, 0 removals  - Installing pguardiario/phpuri (1.0): Downloading (connecting...)Downloading (100%)   - Installing electrolinux/phpquery (0.9.6): Downloading (connecting...)Downloading (100%)   - Installing byjg/pgbrowser (1.0.0): Downloading (connecting...)Downloading (100%)   - Installing openaustralia/scraperwiki (dev-morph_defaults e996fe0): Cloning e996fe0253  Generating optimized autoload files -----> Preparing runtime environment... -----> Checking for additional extensions to install...  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data between 08/11/2018 and 18/11/2018, changable via MORPH_PERIOD environment Skipping already saved record 11.2018.36528.1 Skipping already saved record 10.2018.36528.1 Skipping already saved record 13.2018.29086.1 Skipping already saved record 10.2018.36530.1 Skipping already saved record 13.2018.29085.1 Skipping already saved record 10.2018.36527.1 Skipping already saved record 10.2018.36529.1 Skipping already saved record 10.2018.36526.1 Skipping already saved record 13.2018.29084.1 Skipping already saved record 10.2018.36525.1 Skipping already saved record 11.2018.36520.1 Skipping already saved record 10.2018.36520.1 Skipping already saved record 13.2018.29083.1 Skipping already saved record 10.2018.36522.1 Skipping already saved record 10.2018.36521.1 Skipping already saved record 10.2018.36519.1 Skipping already saved record 11.2005.25754.2 Skipping already saved record 17.2018.81.1 Skipping already saved record 17.2016.34950.3 Skipping already saved record 10.2018.36524.1 Skipping already saved record 11.2018.36510.1 Skipping already saved record 11.2018.36515.1 Skipping already saved record 11.2018.36517.1 Skipping already saved record 11.2018.36518.1 Skipping already saved record 10.2018.36518.1 Skipping already saved record 10.2018.36523.1

Data

Downloaded 2329 times by openaustralia LoveMyData jesse-a-reilly TheZepto henare

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (1.97 MB) Use the API

rows 10 / 4360

council_reference description date_received address info_url comment_url date_scraped
010.2014.00032861.001
Residential - Single New Dwelling
2014-03-13
85 Cambridge Drive THURGOONA NSW 2640
2014-03-16
010.2014.00032860.001
Residential - New second occupancy
2014-03-13
391 Dick Road LAVINGTON NSW 2641
2014-03-16
010.2014.00032859.001
Change of Use Residential - Other
2014-03-13
439 Mcdonald Road LAVINGTON NSW 2641
2014-03-16
010.2011.00030892.002
Modification of Consent - One (1) Additional Residence &
2014-03-13
602 Heathwood Avenue LAVINGTON NSW 2641
2014-03-16
010.2014.00032858.001
Commercial Signage - Rivalea Australia
2014-03-12
2 / 576 Kiewa Street ALBURY NSW 2640
2014-03-16
010.2014.00032857.001
Patio
2014-03-12
29 Darrambal Drive SPRINGDALE HEIGHTS NSW 2641
2014-03-16
010.2014.00032856.001
Workshop & Associated Amenities Building - Council Depot
2014-03-12
850 Jelbart Road NORTH ALBURY NSW 2640
2014-03-16
010.2014.00032855.001
Maintenance Shed & Water Tank - Trinity Anglican College
2014-03-11
421 Elizabeth Mitchell Drive THURGOONA NSW 2640
2014-03-16
010.2014.00032854.001
Residence & Garage
2014-03-11
76 Slattery Place THURGOONA NSW 2640
2014-03-16
010.2014.00032853.001
Front Fence
2014-03-11
24 Barling Walk EAST ALBURY NSW 2640
2014-03-16

Statistics

Average successful run time: half a minute

Total run time: 16 days

Total cpu time used: 33 minutes

Total disk space used: 2 MB

History

  • Auto ran revision e9916dea and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision e9916dea and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision e9916dea and completed successfully .
    6 records added in the database
    5 pages scraped
  • Auto ran revision e9916dea and completed successfully .
    1 record added in the database
    5 pages scraped
  • Auto ran revision e9916dea and completed successfully .
    2 records added in the database
    5 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

PHP

albury / scraper.php