This is a scraper that runs on Morph. To get started see the documentation

Contributors tmtmtmtm

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Ruby app detected -----> Compiling Ruby -----> Using Ruby version: ruby-2.4.4 -----> Installing dependencies using bundler 1.15.2  Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.5). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.  Fetching gem metadata from https://rubygems.org/.........  Fetching version metadata from https://rubygems.org/..  Fetching dependency metadata from https://rubygems.org/.  Fetching https://github.com/everypolitician/everypolitician-popolo.git  Fetching https://github.com/everypolitician/everypolitician-ruby.git  Fetching https://github.com/openc/sqlite_magic.git  Fetching https://github.com/openaustralia/scraperwiki-ruby.git  Fetching https://github.com/everypolitician/wikisnakker.git  Fetching https://github.com/everypolitician/wikidata-fetcher.git  Fetching activesupport-inflector 0.1.0  Fetching ast 2.4.0  Using bundler 1.15.2  Fetching coderay 1.1.2  Installing ast 2.4.0  Installing activesupport-inflector 0.1.0  Fetching colorize 0.8.1  Installing coderay 1.1.2  Fetching concurrent-ruby 1.1.4  Fetching diskcached 1.1.3  Installing diskcached 1.1.3  Installing colorize 0.8.1  Fetching unf_ext 0.0.7.5  Fetching require_all 2.0.0  Installing concurrent-ruby 1.1.4  Installing require_all 2.0.0  Fetching excon 0.62.0  Fetching multipart-post 2.0.0  Installing unf_ext 0.0.7.5 with native extensions  Installing multipart-post 2.0.0  Installing excon 0.62.0  Fetching hashie 3.6.0  Installing hashie 3.6.0  Fetching httpclient 2.8.3  Installing httpclient 2.8.3  Fetching jaro_winkler 1.5.2  Installing jaro_winkler 1.5.2 with native extensions  Fetching json 2.1.0  Installing json 2.1.0 with native extensions  Fetching method_source 0.9.2  Installing method_source 0.9.2  Fetching mime-types-data 3.2018.0812  Installing mime-types-data 3.2018.0812  Fetching mini_portile2 2.4.0  Installing mini_portile2 2.4.0  Fetching netrc 0.11.0  Installing netrc 0.11.0  Fetching open-uri-cached 0.0.5  Installing open-uri-cached 0.0.5  Fetching parallel 1.13.0  Installing parallel 1.13.0  Fetching powerpack 0.1.2  Installing powerpack 0.1.2  Fetching rainbow 3.0.0  Installing rainbow 3.0.0  Fetching ruby-progressbar 1.10.0  Installing ruby-progressbar 1.10.0  Fetching unicode-display_width 1.4.1  Installing unicode-display_width 1.4.1  Fetching sqlite3 1.3.13  Installing sqlite3 1.3.13 with native extensions  Fetching yajl-ruby 1.4.1  Installing yajl-ruby 1.4.1 with native extensions  Fetching parser 2.6.0.0  Installing parser 2.6.0.0  Using everypolitician-popolo 0.9.0 from https://github.com/everypolitician/everypolitician-popolo.git (at master@fc35153)  Fetching i18n 1.5.3  Installing i18n 1.5.3  Fetching faraday 0.15.4  Installing faraday 0.15.4  Fetching unf 0.1.4  Installing unf 0.1.4  Fetching pry 0.12.2  Installing pry 0.12.2  Fetching mime-types 3.2.2  Installing mime-types 3.2.2  Fetching nokogiri 1.10.1  Installing nokogiri 1.10.1 with native extensions  Using sqlite_magic 0.0.6 from https://github.com/openc/sqlite_magic.git (at master@4df975e)  Fetching rubocop 0.63.1  Using wikisnakker 0.9.1 from https://github.com/everypolitician/wikisnakker.git (at master@72dbb8b)  Installing rubocop 0.63.1  Fetching faraday_middleware 0.13.0  Fetching domain_name 0.5.20180417  Installing faraday_middleware 0.13.0  Installing domain_name 0.5.20180417  Using everypolitician 0.20.0 from https://github.com/everypolitician/everypolitician-ruby.git (at master@f85e4b6)  Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)  Fetching wikidata-client 0.0.12  Fetching http-cookie 1.0.3  Installing wikidata-client 0.0.12  Installing http-cookie 1.0.3  Fetching faraday-cookie_jar 0.0.6  Installing faraday-cookie_jar 0.0.6  Fetching rest-client 2.0.2  Installing rest-client 2.0.2  Fetching mediawiki_api 0.7.1  Installing mediawiki_api 0.7.1  Using wikidata-fetcher 0.22.0 from https://github.com/everypolitician/wikidata-fetcher.git (at master@e85d178)  Bundle complete! 15 Gemfile dependencies, 48 gems now installed.  Gems in the groups development and test were not installed.  Bundled gems are installed into ./vendor/bundle.  Post-install message from i18n:  HEADS UP! i18n 1.1 changed fallbacks to exclude default locale.  But that may break your application.  Please check your Rails app for 'config.i18n.fallbacks = true'.  If you're using I18n (>= 1.1.0) and Rails (< 5.2.2), this should be  'config.i18n.fallbacks = [I18n.default_locale]'.  If not, fallbacks will be broken in your app by I18n 1.1.x.  For more info see:  https://github.com/svenfuchs/i18n/releases/tag/v1.1.0  Bundle completed (38.73s)  Cleaning up the bundler cache.  Warning: the running version of Bundler (1.15.2) is older than the version that created the lockfile (1.16.5). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. -----> Detecting rake tasks   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Can't find Wikidata IDs for: Zulfan Lindan, Effendy Sianipar, Marsiaman Saragih, Ivan Doly Gultom, Rahmat Nasution Hamka, Jerry Sambuaga, I Putu Sudiartana in id ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P6336 for Q28723595 ‼ Q121998 is_instance_of diplomatic rank & position. Skipping ‼ Q13095079 is_instance_of Wikimedia list article. Skipping ‼ Q2434365 is_instance_of political party. Skipping ‼ Q19725115 is_instance_of Wikimedia list article. Skipping ‼ Q19725113 is_instance_of Wikimedia list article. Skipping ‼ Q3273955 is_instance_of government agency. Skipping ‼ Q2826588 is_instance_of . Skipping ‼ Q12479874 is_instance_of Wikimedia list article. Skipping ‼ Q3000949 is_instance_of constitutional court. Skipping ‼ Q12479911 is_instance_of Wikimedia list article. Skipping ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P1003 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P863 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P2959 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P3762 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P3630 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P3021 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P4629 for Q484141 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P6886 for Q484141 ‼ Q19725118 is_instance_of Wikimedia list article. Skipping ‼ Q198 is_instance_of . Skipping ‼ Q12479867 is_instance_of public office. Skipping ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P597 for Q464583 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P2642 for Q464583 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P2031 for Q464583 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P2121 for Q464583 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P1352 for Q464583 ‼ Q19725119 is_instance_of Wikimedia list article. Skipping ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P2950 for Q151500 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P1543 for Q151500 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P4808 for Q151500 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P5882 for Q151500 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P2962 for Q1350057 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P3314 for Q1350057 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P3940 for Q1350057 ⁇ Unknown property: https://www.wikidata.org/wiki/Property:P3315 for Q1350057 Unknown value for P734 for Q4262592 ‼ Q12479876 is_instance_of Wikimedia list article. Skipping

Data

Downloaded 1874 times by everypolitician MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (375 KB) Use the API

rows 10 / 311

id name__id name wikipedia__id original_wikiname name__it gender given_name birth_date name__en name__nl name__de name__fr wikipedia__en image name__jv name__es wikipedia__jv identifier__VIAF name__ms name__su name__min wikipedia__min wikipedia__ms wikipedia__su name__zh wikipedia__zh name__nb name__da name__nn name__sv identifier__freebase name__ru wikipedia__ru wikipedia__commons name__ar wikipedia__ar wikipedia__es identifier__MusicBrainz name__ja name__pl name__hu name__fa name__pt wikipedia__de wikipedia__fa wikipedia__fr wikipedia__hu wikipedia__it wikipedia__ja wikipedia__nl wikipedia__pl wikipedia__pt identifier__ITF name__map_bms wikipedia__map_bms name__zh_hans name__zh_hant name__zh_hk website name__pa wikipedia__pa name__af name__an name__ast name__bar name__br name__ca name__co name__cs name__cy name__de_at name__de_ch name__en_ca name__en_gb name__eo name__et name__eu name__fi name__frp name__fur name__ga name__gd name__gl name__gsw name__hr name__ia name__ie name__io name__is name__kg name__lb name__li name__lij name__mg name__nap name__nds name__nds_nl name__nrm name__oc name__pcd name__pms name__pt_br name__rm name__ro name__sc name__scn name__sco name__sk name__sl name__sr_el name__sw name__vec name__vi name__vls name__vo name__wa name__wo name__zu name__el death_date name__th wikipedia__th wikipedia__vi identifier__SUDOC identifier__IMDB identifier__NLAtrove twitter identifier__FAST identifier__OLID occupation wikipedia__lv name__lv identifier__LCAuth identifier__quora identifier__ISNI name__ksh identifier__NTA identifier__DBNL identifier__BNF identifier__moviemeter identifier__AlloCine identifier__allmovie identifier__PORT identifier__kinopoisk identifier__snac name__ace name__zh_cn name__be_tarask name__be identifier__GND name__bjn wikipedia__bjn name__ko name__he name__yue wikipedia__cs wikipedia__ko wikipedia__simple wikipedia__zh_yue facebook identifier__wikitree family_name instagram identifier__NLA wikipedia__bn name__bn wikipedia__ml name__ml wikipedia__idwikiquote wikipedia__gor identifier__GTAA name__gor name__lfn name__uz name__ta name__or name__am name__pnb name__tr name__az name__ug name__ks name__yo name__ps name__sa name__hi name__sd name__war name__bg name__te name__ur name__sq name__uk name__ku name__kn name__bs name__gom name__azb name__bpy name__tg name__ckb name__gom_latn name__gom_deva name__mwl name__mr wikipedia__am wikipedia__arwikiquote wikipedia__ast wikipedia__azb wikipedia__az wikipedia__azwikisource wikipedia__be wikipedia__bg wikipedia__bpy wikipedia__bswikiquote wikipedia__ca wikipedia__ckb wikipedia__cswikiquote wikipedia__da wikipedia__enwikiquote wikipedia__enwikisource wikipedia__eo wikipedia__et wikipedia__fawikisource wikipedia__fi wikipedia__gl wikipedia__gom wikipedia__he wikipedia__hi wikipedia__hiwikiquote wikipedia__hr wikipedia__hrwikiquote wikipedia__kn wikipedia__ks wikipedia__ku wikipedia__la wikipedia__lfn wikipedia__mg wikipedia__mwl wikipedia__no wikipedia__or wikipedia__plwikiquote wikipedia__pnb wikipedia__ps wikipedia__ro wikipedia__sa wikipedia__sco wikipedia__sd wikipedia__slwikiquote wikipedia__sq wikipedia__sv wikipedia__ta wikipedia__te wikipedia__tg wikipedia__tr wikipedia__trwikiquote wikipedia__ug wikipedia__uk wikipedia__ur wikipedia__urwikiquote wikipedia__uz wikipedia__war wikipedia__yo identifier__NDL identifier__BIBSYS identifier__Munzinger identifier__SELIBR identifier__N6I identifier__Gutenberg identifier__GNC identifier__NPG identifier__britishmuseum identifier__NLI identifier__BNE identifier__EGAXA identifier__SBN_it identifier__NKC identifier__NUKAT identifier__PTBNP identifier__bigenc_ru identifier__ne_se identifier__Britannica identifier__Oxforddnb identifier__CONOR identifier__LAC name__bm name__frc name__jam name__kab name__lt name__pap name__prg name__rgn name__vmf wikipedia__mr name__zh_sg name__zh_tw name__sh name__hy name__sr_ec name__diq name__mk name__sr name__ka name__tl name__ota name__nan name__kk name__tk wikipedia__af wikipedia__br wikipedia__bs wikipedia__diq wikipedia__el wikipedia__eu wikipedia__hy wikipedia__ka wikipedia__kk wikipedia__mk wikipedia__sh wikipedia__sk wikipedia__sr wikipedia__sw wikipedia__tk wikipedia__trwikisource wikipedia__zh_min_nan noble_title identifier__CERL wikipedia__frr name__frr name__tt name__la wikipedia__tt wikipedia__ha name__ha wikipedia__sl identifier__google_knowledge
Q16188348
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
female
Lucy
1968-02-04
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
Lucy Kurniasari
politician
Q21527411
Hinca Panjaitan
Hinca Panjaitan
Hinca Panjaitan
Hinca Panjaitan
male
1964-09-25
Hinca Panjaitan
Hinca Panjaitan
Hinca Panjaitan
Hinca Panjaitan
Hinca Panjaitan
Hinca Panjaitan
politician
Q12500725
Nico Siahaan
Nico Siahaan
Nico Siahaan
Nico Siahaan
Nico Siahaan
male
Nico
1969-06-13
Nico Siahaan
Nico Siahaan
Nico Siahaan
Nico Siahaan
Nico Siahaan
Nico Siahaan
Nicoshn
actor
Siahaan
Q18421250
Ahmad Basarah
Ahmad Basarah
Ahmad Basarah
Ahmad Basarah
Ahmad Basarah
male
Ahmad
1968-06-16
Ahmad Basarah
Ahmad Basarah
Ahmad Basarah
politician
0000 0004 2319 9665
Q31173938
Abdul Kharis Almasyhari
Abdul Kharis Almasyhari
Abdul Kharis Almasyhari
Abdul Kharis Almasyari
male
1968-08-25
Abdul Kharis Almasyhari
politician
Q28723595
Nurhayati
Nurhayati
Nurhayati
Nurhayati
female
1997-10-18
Nurhayati
Nurhayati
singer
Q12472588
Arzetti Bilbina
Arzetti Bilbina
Arzeti Bilbina
Arzetti Bilbina
female
1976-09-04
Arzetti Bilbina
Arzetti Bilbina
Arzetti Bilbina
Arzetti Bilbina
Arzetti Bilbina
Arzetti Bilbina
Arzetti Bilbina
actor
Q16176436
Evita Nursanty
Evita Nursanty
Evita Nursanty
Evita Nursanty
female
Evita
1960-04-06
Evita Nursanty
Evita Nursanty
Evita Nursanty
Evita Nursanty
Evita Nursanty
Evita Nursanty
politician
Q31173865
Zulfikar Achmad
Zulfikar Achmad
Zulfikar Achmad
Zulfikar Achmad
1946-05-17
Zulfikar Achmad
Q31174016
Mochamad Hasbi Asyidiki Jayabaya
Mochamad Hasbi Asyidiki Jayabaya
Mochamad Hasbi Asyidiki Jayabaya
Mochamad Hasbi Asyidiki Jayabaya
1984-10-10
Mochamad Hasbi Asyidiki Jayabaya
politician

Statistics

Average successful run time: 2 minutes

Total run time: 7 days

Total cpu time used: about 2 hours

Total disk space used: 416 KB

History

  • Auto ran revision 9f37784c and completed successfully .
    310 records updated in the database
  • Auto ran revision 9f37784c and completed successfully .
    310 records updated in the database
  • Auto ran revision 9f37784c and completed successfully .
    311 records updated in the database
  • Auto ran revision 9f37784c and completed successfully .
    310 records updated in the database
  • Auto ran revision 9f37784c and completed successfully .
    311 records updated in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Ruby

indonesian-mps / scraper.rb