planningalerts-scrapers / griffith

Scrapes datracking.griffith.nsw.gov.au

The official Griffith City Council Application Tracker


Griffith City Council scraper

This site is a typical .NET site involves the followings. * Cookie - Yes * Pagnation - No, hardcoded to fetch up to 100 records * Javascript - No * Ajax - Yes * JSON - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default, today - 10 days) * thismonth * lastmonth * custom = yyyy-mm ( eg. 2017-01 )

Enjoy

Contributors mlandauer LoveMyData equivalentideas henare

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> PHP app detected -----> Bootstrapping... -----> Installing platform packages...  NOTICE: No runtime required in composer.json; requirements  from dependencies in composer.lock will be used for selection  - php (7.2.5)  - ext-gd (bundled with php)  - ext-mbstring (bundled with php)  - ext-pdo_sqlite (bundled with php)  - ext-sqlite3 (bundled with php)  - nginx (1.8.1)  - apache (2.4.33) -----> Installing dependencies...  Composer version 1.5.2 2017-09-11 16:59:25  Loading composer repositories with package information  Installing dependencies from lock file  Package operations: 5 installs, 0 updates, 0 removals  - Installing pguardiario/phpuri (1.0): Downloading (connecting...)Downloading (100%)   - Installing electrolinux/phpquery (0.9.6): Downloading (connecting...)Downloading (100%)   - Installing byjg/pgbrowser (1.0.0): Downloading (connecting...)Downloading (100%)   - Installing openaustralia/scraperwiki (dev-morph_defaults e996fe0): Cloning e996fe0253  - Installing sunra/php-simple-html-dom-parser (v1.5.2): Downloading (connecting...)Downloading (0%) Downloading (15%)Downloading (30%)Downloading (45%)Downloading (50%)Downloading (100%)  Generating optimized autoload files -----> Preparing runtime environment... -----> Checking for additional extensions to install...  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data between 10/05/2018 and 20/05/2018, changable via MORPH_PERIOD environment Skipping already saved record 105/2018.1 Skipping already saved record 106/2018.1 Skipping already saved record 511/2006.1 Skipping already saved record 511/2006.2 Skipping already saved record 104/2018.1 Skipping already saved record 36/2018.2 Skipping already saved record 103/2018.1 Skipping already saved record 226/2017.1 Skipping already saved record 102/2018.1 Skipping already saved record 101/2018.1 Skipping already saved record 100/2018.1 Skipping already saved record 65/2018.2 Skipping already saved record 270/2016.1 Skipping already saved record 100/2018.1

Data

Downloaded 1400 times by openaustralia LoveMyData TheZepto MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (469 KB) Use the API

rows 10 / 1491

info_url council_reference date_received description address date_scraped comment_url
DA 67/2014/1
2014-04-04
Alterations and aditions to existing schol
5 Warambol ST GRIFITH, NSW
2014-04-07
mailto:council@griffith.nsw.gov.au
DA 65/2014/1
2014-04-03
New Comercial Car Wash
2 Oakes RD GRIFITH, NSW
2014-04-07
mailto:council@griffith.nsw.gov.au
DA 64/2014/1
2014-04-02
Proposed smal bar within an existing building
20 Bana AV GRIFITH, NSW
2014-04-07
mailto:council@griffith.nsw.gov.au
DA 66/2014/1
2014-04-03
Aditions to existing dweling and a detached shed
8 Hilston ST GRIFITH, NSW
2014-04-07
mailto:council@griffith.nsw.gov.au
DA 207/2013/2
2014-04-03
Single dweling with atached garage and detached shed
2 Veri ST GRIFITH, NSW
2014-04-07
mailto:council@griffith.nsw.gov.au
DA 68/2014/1
2014-04-10
Two attached dwellings
35 Binya ST GRIFFITH, NSW
2014-04-10
mailto:council@griffith.nsw.gov.au
DA 27/2014/2
2014-04-10
New Dwelling - Residential and Alteration/Addition to existing Garage/ New Dwelling - Residential Alteration/Addition to existing Garage/Carport/Shed/Awning - Res/Rural
18 Leonard RD HANWOOD, NSW
2014-04-10
mailto:council@griffith.nsw.gov.au
DA 194/2013/2
2014-04-11
Modify - Detached Shed & Tennis Court
3 Gardenia CT YOOGALI, NSW
2014-04-11
mailto:council@griffith.nsw.gov.au
DA 69/2014/1
2014-04-14
Two lot subdivision
117 Hanwood AV HANWOOD, NSW
2014-04-14
mailto:council@griffith.nsw.gov.au
DA 155/2012/2
2014-04-14
S.96 Modification Part 1 (6) Both sides of road being North & South should only be (North along developed property)
2 Oakes RD GRIFFITH, NSW
2014-04-14
mailto:council@griffith.nsw.gov.au

Statistics

Average successful run time: 1 minute

Total run time: 4 days

Total cpu time used: 26 minutes

Total disk space used: 576 KB

History

  • Auto ran revision 74f16f1b and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 74f16f1b and completed successfully .
    5 records added in the database
    5 pages scraped
  • Auto ran revision 74f16f1b and completed successfully .
    2 records added in the database
    5 pages scraped
  • Auto ran revision 74f16f1b and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 74f16f1b and completed successfully .
    nothing changed in the database
    5 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

PHP

griffith / scraper.php