andylolz / italy-camera

README.md

This is a scraper that runs on Morph. To get started see the documentation

Contributors andylolz

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected -----> Installing python-2.7.9  $ pip install -r requirements.txt  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to morph_defaults) to /app/.heroku/src/scraperwiki  Collecting beautifulsoup4==4.4.0 (from -r /tmp/build/requirements.txt (line 8))  Downloading beautifulsoup4-4.4.0-py2-none-any.whl (81kB)  Collecting dumptruck>=0.1.2 (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading dumptruck-0.1.6.tar.gz  Collecting requests (from scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)  Collecting idna<2.7,>=2.5 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading idna-2.6-py2.py3-none-any.whl (56kB)  Collecting urllib3<1.23,>=1.21.1 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)  Collecting certifi>=2017.4.17 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading certifi-2017.7.27.1-py2.py3-none-any.whl (349kB)  Collecting chardet<3.1.0,>=3.0.2 (from requests->scraperwiki->-r /tmp/build/requirements.txt (line 6))  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)  Installing collected packages: dumptruck, idna, urllib3, certifi, chardet, requests, scraperwiki, beautifulsoup4  Running setup.py install for dumptruck: started  Running setup.py install for dumptruck: finished with status 'done'  Running setup.py develop for scraperwiki  Successfully installed beautifulsoup4-4.4.0 certifi-2017.7.27.1 chardet-3.0.4 dumptruck-0.1.6 idna-2.6 requests-2.18.4 scraperwiki urllib3-1.22   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 510 times by everypolitician andylolz chrismytton MikeRalphson tmtmtmtm

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (525 KB) Use the API

rows 10 / 1062

end_date election_list id birth_date start_date name source group area gender term email area_id image
306683
MINNUCCI Emiliano
http://www.camera.it/leg17/29?shadow_deputato=306683&idpersona=306683&idlegislatura=17
male
17
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/306/683/original_d306683.jpg
2013-04-03
FRATELLI D'ITALIA
302757
1963-11-01
CORSARO Massimo Enrico
http://www.camera.it/leg17/29?shadow_deputato=302757&idpersona=302757&idlegislatura=17
MISTO - non iscritto ad alcuna componente politica
LOMBARDIA 1
male
17
CORSARO_M@CAMERA.IT
III
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/302/757/original_d302757.jpg
302946
CUOMO Antonio
http://www.camera.it/leg17/29?shadow_deputato=302946&idpersona=302946&idlegislatura=17
male
17
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/302/946/original_d302946.jpg
306687
CAMANI Vanessa
http://www.camera.it/leg17/29?shadow_deputato=306687&idpersona=306687&idlegislatura=17
female
17
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/306/687/original_d306687.jpg
306681
CIRACI' Nicola
http://www.camera.it/leg17/29?shadow_deputato=306681&idpersona=306681&idlegislatura=17
male
17
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/306/681/original_d306681.jpg
2015-02-05
MOVIMENTO 5 STELLE BEPPEGRILLO.IT
305606
1982-10-23
2015-01-27
ROSTELLATO Gessica
http://www.camera.it/leg17/29?shadow_deputato=305606&idpersona=305606&idlegislatura=17
MISTO - non iscritto ad alcuna componente politica
VENETO 1
female
17
VII
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/305/606/original_d305606.jpg
301565
VICO Ludovico
http://www.camera.it/leg17/29?shadow_deputato=301565&idpersona=301565&idlegislatura=17
male
17
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/301/565/original_d301565.jpg
2016-04-11
LEGA NORD
305632
1963-12-08
2015-03-26
CAON Roberto
http://www.camera.it/leg17/29?shadow_deputato=305632&idpersona=305632&idlegislatura=17
MISTO - non iscritto ad alcuna componente politica
VENETO 1
male
17
VII
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/305/632/original_d305632.jpg
2016-10-27
MOVIMENTO 5 STELLE BEPPEGRILLO.IT
305528
1986-10-17
2016-02-12
CATALANO Ivan
http://www.camera.it/leg17/29?shadow_deputato=305528&idpersona=305528&idlegislatura=17
MISTO - non iscritto ad alcuna componente politica
LOMBARDIA 2
male
17
IV
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/305/528/original_d305528.jpg
2016-04-11
LEGA NORD
305662
1957-03-11
2015-09-09
MARCOLIN Marco
http://www.camera.it/leg17/29?shadow_deputato=305662&idpersona=305662&idlegislatura=17
MISTO - non iscritto ad alcuna componente politica
VENETO 2
male
17
VIII
http://www.camera.it/application/xmanager/projects/leg17/attachments/shadow_deputato/has_fotos/000/305/662/original_d305662.jpg

Statistics

Average successful run time: about 2 hours

Total run time: about 2 months

Total cpu time used: about 8 hours

Total disk space used: 561 KB

History

  • Auto ran revision 747cfd9a and completed successfully .
    nothing changed in the database
  • Auto ran revision 747cfd9a and completed successfully .
    nothing changed in the database
  • Auto ran revision 747cfd9a and completed successfully .
    1052 records added, 1052 records removed in the database
  • Auto ran revision 747cfd9a and completed successfully .
    nothing changed in the database
  • Auto ran revision 747cfd9a and completed successfully .
    1052 records added, 1052 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

italy-camera / scraper.py