Wollondilly Shire Council involves the followings
* Server - .NET but Java backend
* Cookie tracking - Yes - JSESSION
* Pagnation - No - However, query is based on 'date' submit via POST
* Clearly defined data within a row - No
* iFrame - Why??
Setup MORPH_PERIOD for data recovery, available options are
* thisweek (default, today -7 days)
Last run completed successfully
Console output of last run
Injecting configuration and compiling...
[1G [1G-----> Ruby app detected
[1G-----> Compiling Ruby
[1G-----> Using Ruby version: ruby-2.0.0-p481
[1G-----> Installing dependencies using bundler 1.13.7
[1G Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
[1G Fetching gem metadata from https://rubygems.org/........
[1G Fetching version metadata from https://rubygems.org/.
[1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git
[1G Rubygems 2.0.14 is not threadsafe, so your gems will be installed one at a time. Upgrade to Rubygems 2.1.0 or higher to enable parallel gem installation.
[1G Installing unf_ext 0.0.7.2 with native extensions
[1G Installing httpclient 220.127.116.11
[1G Installing mime-types-data 3.2016.0521
[1G Installing net-http-digest_auth 1.4
[1G Installing net-http-persistent 2.9.4
[1G Installing mini_portile2 2.1.0
[1G Installing pkg-config 1.1.7
[1G Installing ntlm-http 0.1.1
[1G Installing webrobots 0.1.2
[1G Installing sqlite3 1.3.11 with native extensions
[1G Using bundler 1.13.7
[1G Installing unf 0.1.4
[1G Installing mime-types 3.1
[1G Installing nokogiri 1.6.8 with native extensions
[1G Installing sqlite_magic 0.0.6
[1G Installing domain_name 0.5.20160826
[1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)
[1G Installing http-cookie 1.0.2
[1G Installing mechanize 2.7.5
[1G Bundle complete! 2 Gemfile dependencies, 19 gems now installed.
[1G Gems in the groups development and test were not installed.
[1G Bundled gems are installed into ./vendor/bundle.
[1G Bundle completed (24.49s)
[1G Cleaning up the bundler cache.
[1G-----> Detecting rake tasks
[1G [1G-----> Discovering process types
[1G Procfile declares types -> scraper
Injecting scraper and running...
Scraping from 17/05/2017 to 24/05/2017, changable via MORPH_PERIOD variable
Skipping already saved record 325/2017
Skipping already saved record 313/2017
Skipping already saved record 317/2017
Skipping already saved record 315/2017
Saving record 327/2017, 13 Church Street APPIN, NSW, 2560
Skipping already saved record 314/2017
Skipping already saved record 507/2014
Skipping already saved record 319/2017
Saving record 328/2017, 2 Fitzroy Street WILTON, NSW, 2571
Skipping already saved record 152/2016
Skipping already saved record 321/2017
Skipping already saved record 312/2017
Skipping already saved record 323/2017
Skipping already saved record 322/2017
Skipping already saved record 324/2017
Skipping already saved record 326/2017
Skipping already saved record 320/2017
Skipping already saved record 318/2017
Skipping already saved record 316/2017
Skipping already saved record 311/2017
Poultry Farm inc Sheds,Office,Workers Amenities,Earthworks & Infrastru Development of a Livestock Intensive Industry (Poultry) including 8 x Sheds,2 x Machinery Sheds,Composting & Litter Sheds,Office,Workers Amenity,Bulk Earthworks & Associated Infrastructure