This is a scraper that runs on Morph. To get started see the documentation

Contributors tmtmtmtm

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.3.3 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/.  Fetching https://github.com/everypolitician/scraped.git  Fetching https://github.com/everypolitician/scraped_page_archive.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching rake 12.0.0  Fetching public_suffix 2.0.4  Fetching ast 2.3.0  Installing ast 2.3.0  Installing rake 12.0.0  Installing public_suffix 2.0.4  Using bundler 1.15.2  Fetching coderay 1.1.1  Fetching safe_yaml 1.0.4  Installing coderay 1.1.1  Fetching field_serializer 0.3.0  Installing safe_yaml 1.0.4  Installing field_serializer 0.3.0  Fetching git 1.3.0  Installing git 1.3.0  Fetching hashdiff 0.3.0  Installing hashdiff 0.3.0  Fetching httpclient 2.8.2.4  Fetching method_source 0.8.2  Installing method_source 0.8.2  Fetching mini_portile2 2.1.0  Fetching minitest 5.10.1  Installing mini_portile2 2.1.0  Fetching vcr 3.0.3  Installing httpclient 2.8.2.4  Installing minitest 5.10.1  Fetching open-uri-cached 0.0.5  Installing open-uri-cached 0.0.5  Fetching powerpack 0.1.1  Installing vcr 3.0.3  Installing powerpack 0.1.1  Fetching slop 3.6.0  Fetching rainbow 2.1.0  Installing slop 3.6.0  Installing rainbow 2.1.0  Fetching require_all 1.3.3  Installing require_all 1.3.3  Fetching ruby-progressbar 1.8.1  Installing ruby-progressbar 1.8.1  Fetching unicode-display_width 1.1.2  Installing unicode-display_width 1.1.2  Fetching sqlite3 1.3.12  Fetching parser 2.3.3.1  Fetching addressable 2.5.0  Installing sqlite3 1.3.12 with native extensions  Installing addressable 2.5.0  Fetching crack 0.4.3  Installing parser 2.3.3.1  Installing crack 0.4.3  Fetching nokogiri 1.7.0  Fetching minispec-metadata 2.0.0  Installing minispec-metadata 2.0.0  Fetching pry 0.10.4  Installing pry 0.10.4  Fetching rubocop 0.46.0  Installing rubocop 0.46.0  Fetching webmock 2.0.3  Installing webmock 2.0.3  Fetching minitest-vcr 1.4.0  Installing minitest-vcr 1.4.0  Fetching vcr-archive 0.3.0  Installing vcr-archive 0.3.0  Using scraped_page_archive 0.5.0 from https://github.com/everypolitician/scraped_page_archive.git (at master@28f93d7)  Installing nokogiri 1.7.0 with native extensions  Fetching sqlite_magic 0.0.6  Installing sqlite_magic 0.0.6  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Using scraped 0.1.0 from https://github.com/everypolitician/scraped.git (at master@0b7474c)  Bundle complete! 12 Gemfile dependencies, 36 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from webmock:  WebMock 2.0 has some breaking changes. Please check the CHANGELOG: https://goo.gl/piDGLu  Bundle completed (19.05s)  Cleaning up the bundler cache. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 3819 times by everypolitician tmtmtmtm chrismytton ondenman mhl MikeRalphson octopusinvitro milh0use

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.39 MB) Use the API

rows 10 / 10444

id name image given_name family_name sort_name birth_date gender identifier__bioguide identifier__thomas identifier__lis identifier__govtrack identifier__opensecrets identifier__votesmart identifier__fec identifier__cspan identifier__wikipedia identifier__house_history identifier__ballotpedia identifier__maplight identifier__washington_post identifier__icpsr identifier__wikidata house start_date state district party homepage address phone fax area_id term end_date identifier__bioguide_previous identifier__google_entity_id
G000321
Albert Gore, Jr.
Albert
Gore
Gore, Albert
1948-03-31
male
G000321
00449
S170
404679
Al Gore
13970
14423
Q19673
sen
1993-01-06
TN
Democrat
ocd-division/country:us/state:tn
103
1993-12-31
kg:/m/0d05fv
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
1993-01-06
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
103
1995-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
1995-01-06
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
104
1997-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
1997-01-07
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
105
1999-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
1999-01-06
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
106
2001-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
2001-01-06
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
107
2003-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
2003-01-07
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
108
2005-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
rep
2005-01-06
OH
13
Democrat
ocd-division/country:us/state:oh/cd:13
109
2007-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
sen
2011-01-06
OH
Democrat
713 HART SENATE OFFICE BUILDING WASHINGTON DC 20510
202-224-2315
202-228-6321
ocd-division/country:us/state:oh
112
2013-01-03
kg:/m/034s80
B000944
Sherrod Brown
Sherrod
Brown
Brown, Sherrod
1952-11-09
male
B000944
00136
S307
400050
N00003535
27018
H2OH13033
5051
Sherrod Brown
9996
Sherrod Brown
168
29389
Q381880
sen
2009-01-06
OH
Democrat
713 HART SENATE OFFICE BUILDING WASHINGTON DC 20510
202-224-2315
202-228-6321
ocd-division/country:us/state:oh
111
2011-01-03
kg:/m/034s80

Statistics

Average successful run time: 4 minutes

Total run time: 2 days

Total cpu time used: about 4 hours

Total disk space used: 4.43 MB

History

  • Auto ran revision 8f077ec7 and completed successfully .
    10443 records added, 10443 records removed in the database
  • Auto ran revision 8f077ec7 and completed successfully .
    10443 records added, 10443 records removed in the database
    2 pages scraped
  • Auto ran revision 8f077ec7 and completed successfully .
    10443 records added, 10443 records removed in the database
    2 pages scraped
  • Auto ran revision 8f077ec7 and completed successfully .
    10443 records added, 10443 records removed in the database
    1 page scraped
  • Auto ran revision 8f077ec7 and completed successfully .
    10443 records added, 10443 records removed in the database
    2 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

us-congress-members / scraper.rb