otherchirps / port_stephens

Port Stephens Development Applications

Scrapes datracker.portstephens.nsw.gov.au, www.google-analytics.com, maps.googleapis.com, and 1 other domain

The official Port Stephens Council Development Application Tracker (DA Tracker)


This is a scraper that runs on Morph. To get started see the documentation

Contributors ishmagel otherchirps mlandauer

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.6  $ pip install -r requirements.txt  Collecting beautifulsoup4==4.3.2 (from -r requirements.txt (line 1))  /app/.heroku/python/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.  SNIMissingWarning  /app/.heroku/python/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.  InsecurePlatformWarning  Downloading beautifulsoup4-4.3.2.tar.gz (143kB)  Collecting python-dateutil==2.4.2 (from -r requirements.txt (line 2))  Downloading python_dateutil-2.4.2-py2.py3-none-any.whl (188kB)  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r requirements.txt (line 4))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to ./.heroku/src/scraperwiki  Collecting selenium==2.46.0 (from -r requirements.txt (line 5))  Downloading selenium-2.46.0.tar.gz (2.6MB)  Collecting six>=1.5 (from python-dateutil==2.4.2->-r requirements.txt (line 2))  Downloading six-1.10.0-py2.py3-none-any.whl  Collecting dumptruck>=0.1.2 (from scraperwiki->-r requirements.txt (line 4))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r requirements.txt (line 4))  Downloading requests-2.12.4-py2.py3-none-any.whl (576kB)  Installing collected packages: beautifulsoup4, six, python-dateutil, dumptruck, requests, scraperwiki, selenium  Running setup.py install for beautifulsoup4: started  Running setup.py install for beautifulsoup4: finished with status 'done'  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Running setup.py install for selenium: started  Running setup.py install for selenium: finished with status 'done'  Successfully installed beautifulsoup4-4.3.2 dumptruck-0.1.6 python-dateutil-2.4.2 requests-2.12.4 scraperwiki selenium-2.46.0 six-1.10.0   ! Hello! It looks like your application is using an outdated version of Python.  ! This caused the security warning you saw above during the 'pip install' step.  ! We recommend 'python-2.7.12', which you can specify in a 'runtime.txt' file.  ! -- Much Love, Heroku.   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Saving: 016-2017-00000012-001 Saving: 016-2015-00000536-003 Saving: 016-2017-00000011-001 Saving: 009-2017-00000004-001 Saving: 009-2017-00000003-001 Saving: 016-2016-00000746-002 Saving: 016-2017-00000010-001 Saving: 016-2017-00000006-001 Saving: 016-2017-00000009-001 Saving: 016-2017-00000007-001 Saving: 016-2017-00000005-001 Skipping: 016-2017-00000004-001 Skipping: 016-2017-00000001-001 Skipping: 016-2017-00000002-001 Skipping: 016-2017-00000003-001 Skipping: 009-2017-00000001-001 Skipping: 009-2017-00000002-001

Data

Downloaded 1 time by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (76 KB) Use the API

rows 10 / 171

council_reference address description info_url comment_url date_scraped date_received
016-2015-00000370-001
9 Scott St, Wallalong 2320 NSW
Dwelling House
2015-06-19
2015-06-19
016-2015-00000374-001
3 Semillon Cl, Hinton 2321 NSW
Storage Shed
2015-06-19
2015-06-19
016-2015-00000373-001
19 Tomaree Rd, Shoal Bay 2315 NSW
Single Storey Dwelling & Secondary Dwelling ( Granny Flat)
2015-06-19
2015-06-19
016-2015-00000372-001
12 Heather St, Heatherbrae 2324 NSW
Commercial Development - Showroom & Caretakers Cottage & Shed
2015-06-19
2015-06-19
016-2015-00000371-001
48 Argyle Ave, Anna Bay 2316 NSW
Storage & Tool Shed
2015-06-19
2015-06-19
016-2015-00000369-001
7 Cecilia Cl, Salt Ash 2318 NSW
Demolish Dwelling & Garage
2015-06-19
2015-06-19
016-2015-00000367-001
1057 Lemon Tree Passage Rd, Tanilba Bay 2319 NSW
Commercial Premises - Demolish Existing
2015-06-19
2015-06-19
016-2015-00000368-001
47 Osprey Cct, Medowie 2318 NSW
Dwelling House
2015-06-19
2015-06-19
016-2004-00000709-002
137 Sandy Point Rd, Corlette 2315 NSW
Section 96(1) to Approved Three (3) Lot Torrens Title Subdivision - Amend Condition 3
2015-06-19
2015-06-19
016-2009-00000709-002
8 Hank St, Heatherbrae 2324 NSW
S.96 Amendment - amendement to condition
2015-06-19
2015-06-19

Statistics

Average successful run time: 13 minutes

Total run time: about 2 hours

Total cpu time used: half a minute

Total disk space used: 129 KB

History

  • Manually ran revision afc61167 and completed successfully .
    11 records added in the database
    52 pages scraped
  • Manually ran revision afc61167 and failed .
    nothing changed in the database
  • Manually ran revision afc61167 and completed successfully .
    6 records added in the database
    52 pages scraped
  • Manually ran revision afc61167 and completed successfully .
    68 records added in the database
    39 pages scraped
  • Manually ran revision afc61167 and completed successfully .
    59 records added in the database
    39 pages scraped
  • Manually ran revision df3de725 and completed successfully .
    nothing changed in the database
    39 pages scraped
  • Manually ran revision df3de725 and completed successfully .
    27 records added in the database
    39 pages scraped
  • Manually ran revision 20f3532a and completed successfully .
    nothing changed in the database
    39 pages scraped
  • Created on morph.io

Scraper code

Python

port_stephens / scraper.py