jacksongs / lobbybot

This chirpy fellow checks the Australian lobbyist register every day.


Contributors jacksongs

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.14 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support  Collecting requests  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)  Collecting tweepy  Downloading https://files.pythonhosted.org/packages/36/1b/2bd38043d22ade352fc3d3902cf30ce0e2f4bf285be3b304a2782a767aec/tweepy-3.8.0-py2.py3-none-any.whl  Collecting BeautifulSoup4  Downloading https://files.pythonhosted.org/packages/c5/48/c88b0b390ae1f785942fc83413feb1268a1eb696f343d4d55db735b9bb39/beautifulsoup4-4.8.2-py2-none-any.whl (106kB)  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 4))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  Collecting lxml  Downloading https://files.pythonhosted.org/packages/25/73/730ad7249847f741c7e622f52b971daa5540f6fb87589bc92c717f5aafba/lxml-4.4.2-cp27-cp27mu-manylinux1_x86_64.whl (5.7MB)  Collecting certifi>=2017.4.17  Downloading https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl (156kB)  Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1  Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB)  Collecting idna<2.9,>=2.5  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)  Collecting chardet<3.1.0,>=3.0.2  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)  Collecting six>=1.10.0  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl  Collecting requests-oauthlib>=0.7.0  Downloading https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl  Collecting PySocks>=1.5.7  Downloading https://files.pythonhosted.org/packages/a2/4b/52123768624ae28d84c97515dd96c9958888e8c2d8f122074e31e2be878c/PySocks-1.7.1-py27-none-any.whl  Collecting soupsieve>=1.2  Downloading https://files.pythonhosted.org/packages/81/94/03c0f04471fc245d08d0a99f7946ac228ca98da4fa75796c507f61e688c2/soupsieve-1.9.5-py2.py3-none-any.whl  Collecting dumptruck>=0.1.2  Downloading https://files.pythonhosted.org/packages/15/27/3330a343de80d6849545b6c7723f8c9a08b4b104de964ac366e7e6b318df/dumptruck-0.1.6.tar.gz  Collecting oauthlib>=3.0.0  Downloading https://files.pythonhosted.org/packages/05/57/ce2e7a8fa7c0afb54a0581b14a65b56e62b5759dbc98e80627142b8a3704/oauthlib-3.1.0-py2.py3-none-any.whl (147kB)  Collecting backports.functools-lru-cache; python_version < "3"  Downloading https://files.pythonhosted.org/packages/da/d1/080d2bb13773803648281a49e3918f65b31b7beebf009887a529357fd44a/backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl  Building wheels for collected packages: dumptruck  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-cp27-none-any.whl size=11845 sha256=2223ca8adb453cce8f0625d8030af2db93a5e0b92243f06481bae6b0d9d14324  Stored in directory: /tmp/pip-ephem-wheel-cache-W4ne5s/wheels/57/df/83/32654ae89119876c7a7db66829bbdb646caa151589dbaf226e  Successfully built dumptruck  Installing collected packages: certifi, urllib3, idna, chardet, requests, six, oauthlib, requests-oauthlib, PySocks, tweepy, backports.functools-lru-cache, soupsieve, BeautifulSoup4, dumptruck, scraperwiki, lxml  Running setup.py develop for scraperwiki  Successfully installed BeautifulSoup4-4.8.2 PySocks-1.7.1 backports.functools-lru-cache-1.6.1 certifi-2019.11.28 chardet-3.0.4 dumptruck-0.1.6 idna-2.8 lxml-4.4.2 oauthlib-3.1.0 requests-2.22.0 requests-oauthlib-1.3.0 scraperwiki six-1.14.0 soupsieve-1.9.5 tweepy-3.8.0 urllib3-1.25.7 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Yesterday the current record count for agencies was: 258 Yesterday the current record count for client was: 1759 Yesterday the current record count for lobbyists was: 560 Traceback (most recent call last): File "scraper.py", line 46, in <module> response = requests.get("http://lobbyists.pmc.gov.au/export/export_%s.cfm"%kind) File "/app/.heroku/python/lib/python2.7/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/app/.heroku/python/lib/python2.7/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='lobbyists.pmc.gov.au', port=80): Max retries exceeded with url: /export/export_agencies.cfm (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f845cdfe150>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Data

Downloaded 11 times by jacksongs MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.17 MB) Use the API

rows 10 / 1430

Former Government Representative End Lobbyist Name Lobbyist Position Start Cessation Date Agency Name
Yes
Adam Kilgour
Managing Director
20-Jan-2016
July 1993
Diplomacy Pty Limited
No
Adrienne Day-Hodge
Director
20-Jan-2016
Day & Hodge Associates Pty Ltd
Yes
Alan Evers-Buckland
Owner / Former Adviser
20-Jan-2016
October 2007
A D Evers-Buckland
Yes
Alexander Cramb
Partner
20-Jan-2016
January 2008
GRACosway Pty Ltd
Yes
Alexandra Brook Kennedy
Director
20-Jan-2016
April 1992
The Write Connection Pty Ltd
No
Andres Puig
Director
20-Jan-2016
The Civic Group Holdings Pty Ltd and Highchair Pty Ltd
No
Andrew Butler
Managing Director
20-Jan-2016
Etched Communications Pty Ltd
No
Andrew Graham Farrant
Principal
20-Jan-2016
One Degree Advisory
No
Andrew James King
Director
20-Jan-2016
BroadSpectrum Consultants Pty Ltd
No
Andrew Plumley
Director
20-Jan-2016
Willard Public Affairs Pty Limited

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.17 MB) Use the API

rows 10 / 4224

End Trading Name Start Updated ABN Agency Name
20-Jan-2016
Knowminister
20-Jan-2016
28-Sep-2015
26140833772
KnowMinister.com Pty Ltd
21-Jan-2016
LESL Services Pty Ltd
20-Jan-2016
21-Jun-2015
19096984440
LESL Services Pty Ltd
21-Jan-2016
MacGregor Public Relations
20-Jan-2016
30-Sep-2015
18104844626
MacGregor Public Relations
21-Jan-2016
Parker & Partners Public Affairs
20-Jan-2016
08-Oct-2015
22083475683
Parker & Partners Pty Ltd
21-Jan-2016
Strategic Partnership Group (Aust) Pty Limited
20-Jan-2016
16-Jun-2015
92 089 461 450
Strategic Partnership Group (Aust) Pty Limited
21-Jan-2016
FTI Consulting
20-Jan-2016
24-Nov-2015
69102531457
FTI Consulting (Sydney) Pty Ltd as Trustee for the Third Person Communication Trust
21-Jan-2016
Visionary Enterprising Management Services Pty Ltd
20-Jan-2016
23-Jun-2015
13099902459
Visionary Enterprising Management Services Pty Ltd
21-Jan-2016
WELLS HASLEM STRATEGIC PUBLIC AFFAIRS PTY. LTD.
20-Jan-2016
02-Sep-2015
52159456685
WELLS HASLEM STRATEGIC PUBLIC AFFAIRS PTY. LTD.
26-Jan-2016
A D EVERS-BUCKLAND
20-Jan-2016
09-Nov-2015
72963316425
A D Evers-Buckland
26-Jan-2016
assetpr
20-Jan-2016
18-Jun-2015
37134426716
assetpr

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (2.17 MB) Use the API

rows 10 / 4431

Start Client Name End Agency Name
20-Jan-2016
AA Crown Holdings
The Premier Communications Group Pty Ltd.
20-Jan-2016
AECOM Australia
GRACosway Pty Ltd
20-Jan-2016
Aeroflex a Cobham Company
Vicom Australia Pty Ltd
20-Jan-2016
Aeronautics System Ltd
ABAKUS - Aircraft and Aviation Specialists Pty Limited
20-Jan-2016
AHA
SPIER CONSULTING P/L
20-Jan-2016
AIR CONDITIONING AND REFRIGERATION EQUIPMENT MANUFACTURERS ASSOCIATION OF AUSTRALIA INCORPORATED
Global Issues Group
20-Jan-2016
Air New Zealand
Diplomacy Pty Limited
20-Jan-2016
ALDI Stores
GRACosway Pty Ltd
20-Jan-2016
Alinta Energy
Megan Anwyl
20-Jan-2016
Allergan Australia Pty Limited
Health Communincations Australia Pty Limited

Statistics

Average successful run time: 5 minutes

Total run time: 5 days

Total cpu time used: about 5 hours

Total disk space used: 2.2 MB

History

  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • Auto ran revision a3cc62da and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

lobbybot / scraper.py