LoveMyData / griffith

Scrapes datracking.griffith.nsw.gov.au

The official Griffith City Council Application Tracker


Griffith City Council scraper

This site is a typical .NET site involves the followings. * Cookie - Yes * Pagnation - No, hardcoded to fetch up to 100 records * Javascript - No * Ajax - Yes * JSON - Yes

Setup MORPH_PERIOD for data recovery, available options are * thisweek (default, today - 10 days) * thismonth * lastmonth * custom = yyyy-mm ( eg. 2017-01 )

Enjoy

Contributors mlandauer LoveMyData

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> PHP app detected -----> Bootstrapping... -----> Installing platform packages...  NOTICE: No runtime required in composer.json; requirements  from dependencies in composer.lock will be used for selection  - php (7.2.2)  - ext-gd (bundled with php)  - ext-mbstring (bundled with php)  - ext-pdo_sqlite (bundled with php)  - ext-sqlite3 (bundled with php)  - apache (2.4.29)  - nginx (1.8.1) -----> Installing dependencies...  Composer version 1.5.2 2017-09-11 16:59:25  Loading composer repositories with package information  Installing dependencies from lock file  Package operations: 5 installs, 0 updates, 0 removals  - Installing pguardiario/phpuri (1.0): Downloading (connecting...)Downloading (100%)   - Installing electrolinux/phpquery (0.9.6): Downloading (connecting...)Downloading (0%) Downloading (5%)Downloading (10%)Downloading (15%)Downloading (20%)Downloading (25%)Downloading (30%)Downloading (35%)Downloading (40%)Downloading (45%)Downloading (50%)Downloading (55%)Downloading (60%)Downloading (65%)Downloading (70%)Downloading (75%)Downloading (80%)Downloading (85%)Downloading (90%)Downloading (95%)Downloading (100%)  - Installing byjg/pgbrowser (1.0.0): Downloading (connecting...)Downloading (0%) Downloading (65%)Downloading (100%)  - Installing openaustralia/scraperwiki (dev-morph_defaults e996fe0): Cloning e996fe0253  - Installing sunra/php-simple-html-dom-parser (v1.5.2): Downloading (connecting...)Downloading (0%) Downloading (15%)Downloading (30%)Downloading (45%)Downloading (60%)Downloading (75%)Downloading (95%)Downloading (100%)  Generating optimized autoload files -----> Preparing runtime environment... -----> Checking for additional extensions to install...  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Getting data between 10/02/2018 and 20/02/2018, changable via MORPH_PERIOD environment Skipping already saved record 126/2017.1 Saving record 22/2018.1 - 3 Lowde St, Nericon 2680 NSW Skipping already saved record 150/2017.1 Skipping already saved record 21/2018.1 Skipping already saved record 20/2018.1 Skipping already saved record 19/2018.1 Skipping already saved record 18/2018.1 Skipping already saved record 17/2018.1

Data

Downloaded 436 times by LoveMyData henare

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (259 KB) Use the API

rows 10 / 907

id council_reference address description info_url comment_url date_scraped date_received
1
554/2016.1
68 Zirilli Av, Griffith 2680 NSW
Dwelling with attached garage
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-16
2
553/2016.1
21 Zirilli Av, Griffith 2680 NSW
Dwelling with attached garage
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-24
3
176/2016.1
220 Banna Av, Griffith 2680 NSW
Retail fitout for Australian Unity Client Centre
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-19
4
188/2016.1
62 Hyandra St, Griffith 2680 NSW
Shed
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-31
5
239/2015.1
20 Barellan St, Griffith 2680 NSW
Carport
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-30
6
179/2016.1
79 Spence Rd, Griffith 2680 NSW
Dwelling with attached garage
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-23
7
190/2016.1
54 Pilloni Rd, Yoogali 2680 NSW
New Shed/Awning
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-31
8
192/2016.1
38-40 Kidman Wy, Griffith 2680 NSW
Industrial shed for ancillary to existing trucking business
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-31
9
189/2016.1
7 Gordon Av, Griffith 2680 NSW
Shed
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-31
10
76/2016.1
Railway Pde, Yenda 2681 NSW
Patio
mailto:admin@griffith.nsw.gov.au
2016-09-12
2016-08-30

Statistics

Average successful run time: half a minute

Total run time: about 4 hours

Total cpu time used: 2 minutes

Total disk space used: 365 KB

History

  • Auto ran revision 900ebcfa and completed successfully .
    1 record added, 1 record updated in the database
    5 pages scraped
  • Auto ran revision 900ebcfa and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 900ebcfa and completed successfully .
    nothing changed in the database
    5 pages scraped
  • Auto ran revision 900ebcfa and completed successfully .
    1 record added, 1 record updated in the database
    5 pages scraped
  • Auto ran revision 900ebcfa and completed successfully .
    2 records added, 1 record updated in the database
    5 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

PHP

griffith / scraper.php