drzax / morph-mp-first-speech

Details about the first speech of current Australian politicians.

Scrapes www.aph.gov.au and parlinfo.aph.gov.au

Home – Parliament of Australia


A fairly flaky scraper to get the first parliamentary speech for current Australian parliamentarians.

You might also this MP contact details scraper.

Pull requests welcome.

This is a scraper that runs on Morph. To get started see the documentation.

Contributors drzax

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... -----> Node.js app detected  -----> Reading application state  package.json...  build directory...  cache directory...  environment variables...   Node engine: unspecified  Npm engine: unspecified  Start mechanism: Procfile  node_modules source: package.json  node_modules cached: false   NPM_CONFIG_PRODUCTION=true  NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt  NODE_MODULES_CACHE=true  -----> Installing binaries  Resolving node version (latest stable) via semver.io...  Downloading and installing node 5.9.1...  Using default npm version: 3.7.3  -----> Building dependencies  No cache available  Installing node modules   > sqlite3@3.1.2 install /tmp/build/node_modules/sqlite3  > node-pre-gyp install --fallback-to-build   [sqlite3] Success: "/tmp/build/node_modules/sqlite3/lib/binding/node-v47-linux-x64/node_sqlite3.node" is installed via remote  morph-mp-first-speech@1.0.1 /tmp/build  +-- cheerio@0.20.0  | +-- css-select@1.2.0  | | +-- boolbase@1.0.0  | | +-- css-what@2.1.0  | | +-- domutils@1.5.1  | | `-- nth-check@1.0.1  | +-- dom-serializer@0.1.0  | | `-- domelementtype@1.1.3  | +-- entities@1.1.1  | +-- htmlparser2@3.8.3  | | +-- domelementtype@1.3.0  | | +-- domhandler@2.3.0  | | +-- entities@1.0.0  | | `-- readable-stream@1.1.13  | | +-- core-util-is@1.0.2  | | +-- inherits@2.0.1  | | +-- isarray@0.0.1  | | `-- string_decoder@0.10.31  | +-- jsdom@7.2.2  | | +-- abab@1.0.3  | | +-- acorn@2.7.0  | | +-- acorn-globals@1.0.9  | | +-- cssom@0.3.1  | | +-- cssstyle@0.2.34  | | +-- escodegen@1.8.0  | | | +-- esprima@2.7.2  | | | +-- estraverse@1.9.3  | | | +-- esutils@2.0.2  | | | +-- optionator@0.8.1  | | | | +-- deep-is@0.1.3  | | | | +-- fast-levenshtein@1.1.3  | | | | +-- levn@0.3.0  | | | | +-- prelude-ls@1.1.2  | | | | +-- type-check@0.3.2  | | | | `-- wordwrap@1.0.0  | | | `-- source-map@0.2.0  | | | `-- amdefine@1.0.0  | | +-- nwmatcher@1.3.7  | | +-- parse5@1.5.1  | | +-- sax@1.2.1  | | +-- symbol-tree@3.1.4  | | +-- webidl-conversions@2.0.1  | | +-- whatwg-url-compat@0.6.5  | | | `-- tr46@0.0.3  | | `-- xml-name-validator@2.0.1  | `-- lodash@4.6.1  +-- queue-async@1.2.1  +-- request@2.69.0  | +-- aws-sign2@0.6.0  | +-- aws4@1.3.2  | | `-- lru-cache@4.0.1  | | +-- pseudomap@1.0.2  | | `-- yallist@2.0.0  | +-- bl@1.0.3  | | `-- readable-stream@2.0.6  | | +-- isarray@1.0.0  | | +-- process-nextick-args@1.0.6  | | `-- util-deprecate@1.0.2  | +-- caseless@0.11.0  | +-- combined-stream@1.0.5  | | `-- delayed-stream@1.0.0  | +-- extend@3.0.0  | +-- forever-agent@0.6.1  | +-- form-data@1.0.0-rc4  | | `-- async@1.5.2  | +-- har-validator@2.0.6  | | +-- chalk@1.1.3  | | | +-- ansi-styles@2.2.1  | | | +-- escape-string-regexp@1.0.5  | | | +-- has-ansi@2.0.0  | | | | `-- ansi-regex@2.0.0  | | | +-- strip-ansi@3.0.1  | | | `-- supports-color@2.0.0  | | +-- commander@2.9.0  | | | `-- graceful-readlink@1.0.1  | | +-- is-my-json-valid@2.13.1  | | | +-- generate-function@2.0.0  | | | +-- generate-object-property@1.2.0  | | | | `-- is-property@1.0.2  | | | +-- jsonpointer@2.0.0  | | | `-- xtend@4.0.1  | | `-- pinkie-promise@2.0.0  | | `-- pinkie@2.0.4  | +-- hawk@3.1.3  | | +-- boom@2.10.1  | | +-- cryptiles@2.0.5  | | +-- hoek@2.16.3  | | `-- sntp@1.0.9  | +-- http-signature@1.1.1  | | +-- assert-plus@0.2.0  | | +-- jsprim@1.2.2  | | | +-- extsprintf@1.0.2  | | | +-- json-schema@0.2.2  | | | `-- verror@1.3.6  | | `-- sshpk@1.7.4  | | +-- asn1@0.2.3  | | +-- dashdash@1.13.0  | | | `-- assert-plus@1.0.0  | | +-- ecc-jsbn@0.1.1  | | +-- jodid25519@1.0.2  | | +-- jsbn@0.1.0  | | `-- tweetnacl@0.14.3  | +-- is-typedarray@1.0.0  | +-- isstream@0.1.2  | +-- json-stringify-safe@5.0.1  | +-- mime-types@2.1.10  | | `-- mime-db@1.22.0  | +-- node-uuid@1.4.7  | +-- oauth-sign@0.8.1  | +-- qs@6.0.2  | +-- stringstream@0.0.5  | +-- tough-cookie@2.2.2  | `-- tunnel-agent@0.4.2  `-- sqlite3@3.1.2  +-- nan@2.2.1  `-- node-pre-gyp@0.6.24  +-- mkdirp@0.5.1  | `-- minimist@0.0.8  +-- nopt@3.0.6  | `-- abbrev@1.0.7  +-- npmlog@2.0.3  | +-- ansi@0.3.1  | +-- are-we-there-yet@1.1.2  | | +-- delegates@1.0.0  | | `-- readable-stream@2.0.6  | | +-- core-util-is@1.0.2  | | +-- inherits@2.0.1  | | +-- isarray@1.0.0  | | +-- process-nextick-args@1.0.6  | | +-- string_decoder@0.10.31  | | `-- util-deprecate@1.0.2  | `-- gauge@1.2.7  | +-- has-unicode@2.0.0  | +-- lodash.pad@4.1.0  | | +-- lodash.repeat@4.0.0  | | `-- lodash.tostring@4.1.2  | +-- lodash.padend@4.2.0  | | +-- lodash.repeat@4.0.0  | | `-- lodash.tostring@4.1.2  | `-- lodash.padstart@4.2.0  | +-- lodash.repeat@4.0.0  | `-- lodash.tostring@4.1.2  +-- rc@1.1.6  | +-- deep-extend@0.4.1  | +-- ini@1.3.4  | +-- minimist@1.2.0  | `-- strip-json-comments@1.0.4  +-- request@2.69.0  | +-- aws-sign2@0.6.0  | +-- aws4@1.3.2  | | `-- lru-cache@4.0.0  | | +-- pseudomap@1.0.2  | | `-- yallist@2.0.0  | +-- bl@1.0.3  | | `-- readable-stream@2.0.6  | | +-- core-util-is@1.0.2  | | +-- inherits@2.0.1  | | +-- isarray@1.0.0  | | +-- process-nextick-args@1.0.6  | | +-- string_decoder@0.10.31  | | `-- util-deprecate@1.0.2  | +-- caseless@0.11.0  | +-- combined-stream@1.0.5  | | `-- delayed-stream@1.0.0  | +-- extend@3.0.0  | +-- forever-agent@0.6.1  | +-- form-data@1.0.0-rc4  | | `-- async@1.5.2  | +-- har-validator@2.0.6  | | +-- chalk@1.1.1  | | | +-- ansi-styles@2.2.0  | | | | `-- color-convert@1.0.0  | | | +-- escape-string-regexp@1.0.5  | | | +-- has-ansi@2.0.0  | | | | `-- ansi-regex@2.0.0  | | | +-- strip-ansi@3.0.1  | | | | `-- ansi-regex@2.0.0  | | | `-- supports-color@2.0.0  | | +-- commander@2.9.0  | | | `-- graceful-readlink@1.0.1  | | +-- is-my-json-valid@2.13.1  | | | +-- generate-function@2.0.0  | | | +-- generate-object-property@1.2.0  | | | | `-- is-property@1.0.2  | | | +-- jsonpointer@2.0.0  | | | `-- xtend@4.0.1  | | `-- pinkie-promise@2.0.0  | | `-- pinkie@2.0.4  | +-- hawk@3.1.3  | | +-- boom@2.10.1  | | +-- cryptiles@2.0.5  | | +-- hoek@2.16.3  | | `-- sntp@1.0.9  | +-- http-signature@1.1.1  | | +-- assert-plus@0.2.0  | | +-- jsprim@1.2.2  | | | +-- extsprintf@1.0.2  | | | +-- json-schema@0.2.2  | | | `-- verror@1.3.6  | | `-- sshpk@1.7.4  | | +-- asn1@0.2.3  | | +-- dashdash@1.13.0  | | | `-- assert-plus@1.0.0  | | +-- ecc-jsbn@0.1.1  | | +-- jodid25519@1.0.2  | | +-- jsbn@0.1.0  | | `-- tweetnacl@0.14.1  | +-- is-typedarray@1.0.0  | +-- isstream@0.1.2  | +-- json-stringify-safe@5.0.1  | +-- mime-types@2.1.10  | | `-- mime-db@1.22.0  | +-- node-uuid@1.4.7  | +-- oauth-sign@0.8.1  | +-- qs@6.0.2  | +-- stringstream@0.0.5  | +-- tough-cookie@2.2.2  | `-- tunnel-agent@0.4.2  +-- rimraf@2.5.2  | `-- glob@7.0.3  | +-- inflight@1.0.4  | | `-- wrappy@1.0.1  | +-- inherits@2.0.1  | +-- minimatch@3.0.0  | | `-- brace-expansion@1.1.3  | | +-- balanced-match@0.3.0  | | `-- concat-map@0.0.1  | +-- once@1.3.3  | | `-- wrappy@1.0.1  | `-- path-is-absolute@1.0.0  +-- semver@5.1.0  +-- tar@2.2.1  | +-- block-stream@0.0.8  | +-- fstream@1.0.8  | | `-- graceful-fs@4.1.3  | `-- inherits@2.0.1  `-- tar-pack@3.1.3  +-- debug@2.2.0  | `-- ms@0.7.1  +-- fstream@1.0.8  | +-- graceful-fs@4.1.3  | `-- inherits@2.0.1  +-- fstream-ignore@1.0.3  | +-- inherits@2.0.1  | `-- minimatch@3.0.0  | `-- brace-expansion@1.1.3  | +-- balanced-match@0.3.0  | `-- concat-map@0.0.1  +-- once@1.3.3  | `-- wrappy@1.0.1  +-- readable-stream@2.0.6  | +-- core-util-is@1.0.2  | +-- inherits@2.0.1  | +-- isarray@1.0.0  | +-- process-nextick-args@1.0.6  | +-- string_decoder@0.10.31  | `-- util-deprecate@1.0.2  `-- uid-number@0.0.6   -----> Checking startup method  Found Procfile  -----> Finalizing build  Creating runtime environment  Exporting binary paths  Cleaning npm artifacts  Cleaning previous cache  Caching results for future builds  -----> Build succeeded!   morph-mp-first-speech@1.0.1 /tmp/build  +-- cheerio@0.20.0  +-- queue-async@1.2.1  +-- request@2.69.0  `-- sqlite3@3.1.2   WARNING: Node version not specified in package.json  https://devcenter.heroku.com/articles/nodejs-support#specifying-a-node-js-version  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Fetching Former Senator Judith Adams Fetching Former Senator Mark Bishop Fetching Former Senator the Hon Ronald Boswell Fetching Former Senator the Hon Mark Arbib Fetching Former Senator the Hon Bob Carr Fetching Former Senator Bob Brown Fetching Former Senator Trish Crossin Fetching Former Senator Alan Eggleston Fetching Former Senator the Hon Don Farrell Fetching Former Senator the Hon Christopher Evans Fetching Former Senator the Hon John Faulkner Fetching Hon David Feeney MP Fetching Former Senator Mary Jo Fisher Fetching Former Senator Mark Furner Fetching Former The Hon John Hogg Fetching Former Senator Gary Humphries Fetching Hon Barnaby Joyce MP Fetching Former Senator Helen Kroger Fetching Former Senator the Hon Kate Lundy Fetching Former Senator the Hon Brett Mason Fetching Former Senator Christine Milne Fetching Former Senator Louise Pratt Fetching The Hon Michael Ronaldson MP Fetching Former Senator the Hon Nick Sherry Fetching Former Senator the Hon Ursula Stephens Fetching Hon Matt Thistlethwaite MP Fetching Former Senator the Hon Lin Thorp Fetching Former Senator Mehmet Tillem Fetching Senator Cory Bernardi Fetching Senator Chris Back Fetching Senator Catryna Bilyk Fetching Senator the Hon Eric Abetz Fetching Senator the Hon Simon Birmingham Fetching Senator Carol Brown Fetching Senator Joe Bullock Fetching Senator the Hon George Brandis QC Fetching Senator David Bushby Fetching Senator the Hon Matthew Canavan Fetching Senator the Hon Doug Cameron Fetching Senator the Hon Kim Carr Fetching Senator the Hon Michaelia Cash Fetching Senator the Hon Richard Colbeck Fetching Senator the Hon Mathias Cormann Fetching Senator the Hon Jacinta Collins Fetching Senator the Hon Stephen Conroy Fetching Senator Sam Dastyari Fetching Senator Bob Day AO Fetching Senator Sean Edwards Fetching Senator Richard Di Natale Fetching Senator David Fawcett Fetching Senator the Hon Concetta Fierravanti-Wells Fetching Senator Alex Gallacher Fetching Senator the Hon Mitch Fifield Fetching Senator Sarah Hanson-Young Fetching Senator Katy Gallagher Fetching Senator the Hon Bill Heffernan Fetching Senator the Hon David Johnston Fetching Senator Chris Ketter Fetching Senator David Leyonhjelm Fetching Senator Glenn Lazarus Fetching Senator Jacqui Lambie Fetching Senator Joanna Lindgren Fetching Senator Sue Lines Fetching Senator Scott Ludlam Fetching Senator the Hon Joseph Ludwig Fetching Senator John Madigan Fetching Senator the Hon Ian Macdonald Fetching Senator Gavin Marshall Fetching Senator the Hon James McGrath Fetching Senator Bridget McKenzie Fetching Senator Jenny McAllister Fetching Senator Anne McEwen Fetching Senator Nick McKim Fetching Senator the Hon Jan McLucas Fetching Senator the Hon Fiona Nash Fetching Senator Claire Moore Fetching Senator Ricky Muir Fetching Senator Deborah O'Neill Fetching Senator Barry O'Sullivan Fetching Senator the Hon Stephen Parry Fetching Senator the Hon Marise Payne Fetching Senator Nova Peris OAM Fetching Senator Helen Polley Fetching Senator Linda Reynolds CSC Fetching Senator Janet Rice Fetching Senator the Hon Anne Ruston Fetching Senator Lee Rhiannon Fetching Senator the Hon Scott Ryan Fetching Senator the Hon Nigel Scullion Fetching Senator Rachel Siewert Fetching Senator Zed Seselja Fetching Senator Robert Simms Fetching Senator the Hon Lisa Singh Fetching Senator Dean Smith Fetching Senator Anne Urquhart Fetching Senator the Hon Arthur Sinodinos AO Fetching Senator Zhenya Wang Fetching Senator Glenn Sterle Fetching Senator Nick Xenophon Fetching Senator Larissa Waters Fetching Senator John Williams Fetching Senator Peter Whish-Wilson Fetching Senator the Hon Penny Wong Fetching Mr John Alexander OAM, MP Fetching Hon Tony Abbott MP Fetching Hon Anthony Albanese MP Fetching Hon Karen Andrews MP Fetching Hon Kevin Andrews MP Fetching Hon Bob Baldwin MP Fetching Mr Adam Bandt MP Fetching Hon Bronwyn Bishop MP Fetching Hon Bruce Billson MP Fetching Hon Jamie Briggs MP Fetching Hon Chris Bowen MP Fetching Hon Julie Bishop MP Fetching Hon Sharon Bird MP Fetching Mr Russell Broadbent MP Fetching Mr Andrew Broad MP Fetching Hon Mal Brough MP Fetching Ms Gai Brodtmann MP Fetching Mr Scott Buchholz MP Fetching Hon Mark Butler MP Fetching Hon Tony Burke MP Fetching Ms Anna Burke MP Fetching Dr Jim Chalmers MP Fetching Ms Terri Butler MP Fetching Hon Anthony Byrne MP Fetching Former Senator Penny Wright Fetching Senator David Fawcett Fetching Hon Joe Hockey MP Fetching Mr Nick Champion MP Fetching Hon Darren Chester MP Fetching Mr George Christensen MP Fetching Hon John Cobb MP Fetching Ms Lisa Chesters MP Fetching Ms Sharon Claydon MP Fetching Hon Steven Ciobo MP Fetching Hon Jason Clare MP Fetching Mr Pat Conroy MP Fetching Mr David Coleman MP Fetching Hon Julie Collins MP Fetching Hon Michael Danby MP Fetching Hon Mark Dreyfus QC, MP Fetching Hon Justine Elliot MP Fetching Hon Peter Dutton MP Fetching Hon Kate Ellis MP Fetching Mr Mark Coulton MP Fetching Hon David Feeney MP Fetching Mr Laurie Ferguson MP Fetching Hon Paul Fletcher MP Fetching Hon Josh Frydenberg MP Fetching Hon Warren Entsch MP Fetching Hon Joel Fitzgibbon MP Fetching Hon Teresa Gambaro MP Fetching Dr David Gillespie MP Fetching Mr Andrew Giles MP Fetching Mr Ian Goodenough MP Fetching Hon Gary Gray AO, MP Fetching Hon Luke Hartsuyker MP Fetching Ms Jill Hall MP Fetching Hon Alan Griffin MP Fetching Mrs Natasha Griggs MP Fetching Ms Sarah Henderson MP Fetching Hon Alex Hawke MP Fetching Mr Andrew Hastie MP Fetching Mr Chris Hayes MP Fetching Hon Dr Peter Hendy MP Fetching Mr Kevin Hogan MP Fetching Hon Greg Hunt MP Fetching Hon Ed Husic MP Fetching Mr Eric Hutchinson MP Fetching Mr Luke Howarth MP Fetching Mr Stephen Jones MP Fetching Mr Steve Irons MP Fetching Mr Ewen Jones MP Fetching Dr Dennis Jensen MP Fetching Hon Barnaby Joyce MP Fetching Hon Bob Katter MP Fetching Hon Catherine King MP Fetching Hon Michael Keenan MP Fetching Mr Andrew Laming MP Fetching Mr Craig Kelly MP Fetching Ms Michelle Landry MP Fetching Hon Dr Andrew Leigh MP Fetching Hon Craig Laundy MP Fetching Hon Sussan Ley MP Fetching Hon Ian Macfarlane MP Fetching Hon Jenny Macklin MP Fetching Hon Alannah MacTiernan MP Fetching Ms Nola Marino MP Fetching Mrs Louise Markus MP Fetching Hon Richard Marles MP Fetching Mr Russell Matheson MP Fetching Mrs Karen McNamara MP Fetching Ms Cathy McGowan AO, MP Fetching Hon Michael McCormack MP Fetching Hon Scott Morrison MP Fetching Mr Rob Mitchell MP Fetching Hon Shayne Neumann MP Fetching Mr Andrew Nikolic AM, CSC, MP Fetching Hon Brendan O'Connor MP Fetching Mr Ken O'Dowd MP Fetching Hon Kelly O'Dwyer MP Fetching Ms Clare O'Neil MP Fetching Ms Julie Owens MP Fetching Hon Melissa Parke MP Fetching Mr Tony Pasin MP Fetching Mr Clive Palmer MP Fetching Hon Keith Pitt MP Fetching Hon Christian Porter MP Fetching Mr Graham Perrett MP Fetching Ms Melissa Price MP Fetching Hon Jane Prentice MP Fetching Hon Christopher Pyne MP Fetching Mr Rowan Ramsey MP Fetching Hon Tanya Plibersek MP Fetching Hon Bernie Ripoll MP Fetching Hon Amanda Rishworth MP Fetching Hon Andrew Robb AO, MP Fetching Hon Wyatt Roy MP Fetching Hon Philip Ruddock MP Fetching Hon Stuart Robert MP Fetching Ms Fiona Scott MP Fetching Ms Joanne Ryan MP Fetching Hon Bruce Scott MP Fetching Ms Michelle Rowland MP Fetching Mr Luke Simpkins MP Fetching Hon Bill Shorten MP Fetching The Hon Tony Smith MP Fetching Hon Warren Snowdon MP Fetching Mrs Sophie Mirabella MP Fetching Senator Deborah O'Neill Fetching Mr Don Randall MP Fetching The Hon Michael Ronaldson MP Fetching The Hon Kevin Rudd MP Fetching Mr Mike Symon MP Fetching Dr Andrew Southcott MP Fetching Hon Sharman Stone MP Fetching Mrs Ann Sudmalis MP Fetching Mr Michael Sukkar MP Fetching Hon Angus Taylor MP Fetching Hon Wayne Swan MP Fetching Hon Dan Tehan MP Fetching Hon Matt Thistlethwaite MP Fetching Hon Warren Truss MP Fetching Hon Kelvin Thomson MP Fetching Hon Alan Tudge MP Fetching Mr Bert van Manen MP Fetching Ms Maria Vamvakinou MP Fetching Hon Malcolm Turnbull MP Fetching Mr Nickolas Varvaris MP Fetching Mr Brett Whiteley MP Fetching Mr Tim Watts MP Fetching Mr Ross Vasta MP Fetching Mrs Lucy Wicks MP Fetching Mr Andrew Wilkie MP Fetching Mr Rick Wilson MP Fetching Mr Matt Williams MP Fetching Mr Jason Wood MP Fetching Hon Ken Wyatt AM, MP Fetching Mr Tony Zappia MP

Data

Downloaded 7 times by MikeRalphson dotnetnoobie drzax henare

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.09 MB) Use the API

rows 10 / 491

Statistics

Average successful run time: 13 minutes

Total run time: about 1 hour

Total cpu time used: 2 minutes

Total disk space used: 4.12 MB

History

  • Manually ran revision 4dc869e2 and completed successfully .
    491 records added, 224 records removed in the database
    802 pages scraped
  • Manually ran revision 57881855 and failed .
    nothing changed in the database
    2 pages scraped
  • Manually ran revision 57881855 and completed successfully .
    224 records added, 35 records removed in the database
    474 pages scraped
  • Manually ran revision f8cd66c9 and failed .
    33 records added, 11 records removed in the database
    114 pages scraped
  • Manually ran revision 29336f6c and failed .
    5 records added, 1 record removed in the database
    501 pages scraped
  • Manually ran revision 29336f6c and failed .
    9 records added in the database
    229 pages scraped
  • Manually ran revision 29336f6c and failed .
    nothing changed in the database
    161 pages scraped
  • Manually ran revision a934f4f3 and failed .
    3 records added in the database
    1 page scraped
  • Created on morph.io

Scraper code

Node.js

morph-mp-first-speech / scraper.js