MikeRalphson / MetaScraper

Scrapers on Morph.io


MetaScraper

MetaScraper scrapes Morph.io itself to determine the database schemas used by other scrapers.

This data is used to test and validate the Mermade Morph-Proxy database query builder.

This is a scraper that runs on Morph. To get started see the documentation

Contributors MikeRalphson

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... /scrapers/page/210 maxPage = 210 Processing page 1 Processing page 85 Processing page 95 Processing page 94 Processing page 91 Processing page 100 Processing page 102 Processing page 103 Processing page 101 Processing page 104 Processing page 105 Processing page 107 Processing page 108 Processing page 109 Processing page 106 Processing page 110 Processing page 114 Processing page 113 Processing page 117 Processing page 119 Processing page 122 Processing page 124 Processing page 127 Processing page 126 Processing page 132 Processing page 130 Processing page 133 Processing page 134 Processing page 139 Processing page 140 Processing page 144 Processing page 142 Processing page 148 Processing page 141 Processing page 150 Processing page 152 Processing page 156 Processing page 153 Processing page 157 Processing page 158 Processing page 194 Processing page 199 Processing page 197 Processing page 200 Processing page 198 Processing page 201 Processing page 203 Processing page 202 Processing page 121 Processing page 131 Processing page 136 Processing page 146 Processing page 154 Processing page 118 Processing page 125 Processing page 135 Processing page 143 Processing page 149 Processing page 159 Processing page 111 Processing page 120 Processing page 128 Processing page 137 Processing page 145 Processing page 151 Processing page 176 Processing page 174 Processing page 177 Processing page 178 Processing page 179 Processing page 123 Processing page 129 Processing page 138 Processing page 147 Processing page 155 Processing page 163 Processing page 164 Processing page 161 Processing page 165 Processing page 166 Processing page 168 Processing page 172 Processing page 160 Processing page 171 Processing page 175 Processing page 167 Processing page 173 Processing page 170 Processing page 162 Processing page 169 Processing page 181 Processing page 183 Processing page 182 Processing page 184 Processing page 185 Processing page 186 Processing page 180 Processing page 187 Processing page 189 Processing page 191 Processing page 192 Processing page 190 Processing page 193 Processing page 188 Processing page 205 Processing page 207 Processing page 206 Processing page 208 Processing page 209 Processing page 7 Processing page 10 Processing page 2 Processing page 5 Processing page 8 Processing page 9 Processing page 6 Processing page 3 Processing page 4 Processing page 17 Processing page 15 Processing page 12 Processing page 13 Processing page 19 Processing page 11 Processing page 22 Processing page 23 Processing page 21 Processing page 14 Processing page 24 Processing page 25 Processing page 26 Processing page 27 Processing page 30 Processing page 29 Processing page 36 Processing page 32 Processing page 31 Processing page 34 Processing page 35 Processing page 39 Processing page 33 Processing page 18 Processing page 41 Processing page 16 Processing page 38 Processing page 20 Processing page 28 Processing page 46 Processing page 45 Processing page 44 Processing page 48 Processing page 37 Processing page 49 Processing page 53 Processing page 54 Processing page 52 Processing page 51 Processing page 56 Processing page 47 Processing page 50 Processing page 64 Processing page 59 Processing page 58 Processing page 65 Processing page 63 Processing page 61 Processing page 68 Processing page 62 Processing page 40 Processing page 60 Processing page 67 Processing page 57 Processing page 43 Processing page 55 Processing page 42 Processing page 70 Processing page 69 Processing page 71 Processing page 72 Processing page 66 Processing page 73 Processing page 74 Processing page 80 Processing page 81 Processing page 76 Processing page 78 Processing page 79 Processing page 75 Processing page 86 Processing page 88 Processing page 82 Processing page 83 Processing page 92 Processing page 90 Processing page 77 Processing page 84 Processing page 89 Processing page 93 Processing page 87 Processing page 97 Processing page 99 Processing page 96 Processing page 115 Processing page 112 Processing page 116 Processing page 98 Processing page 195 Processing page 196 Processing page 204 { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' } { [Error: socket hang up] code: 'ECONNRESET' }

Data

Downloaded 22 times by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (836 KB) Use the API

rows 10 / 2335

scraper desc lang sql
clasik/test
Python
[]
PatriceParis/Goutte
Scrap
PHP
{"error":"unable to open database file"}
SimKennedy/td_eu_supply_eu
Python
[{"type":"table","name":"data","tbl_name":"data","rootpage":2,"sql":"CREATE TABLE `data` (\n `CPV_CODES` text\n , `country_code` text, `NAME` text, `ADDRESS` text, `URL` text, `DATE_OF_PUBLICATION_clean` text, `CONTACT_PHONE` text, `RESPONSE_DEADLINE_clean` text, `SHORT_DESCRIPTION` text, `RFT_ID` text, `REFERENCE` text, `COUNTRIES` text, `CONTACT_NAME` text, `DATE_OF_PUBLICATION` text, `DESCRIPTION` text, `PROCESS` text, `BUYERS` text, `RESPONSE_DEADLINE` text)"},{"type":"index","name":"data_RFTID","tbl_name":"data","rootpage":3,"sql":"CREATE UNIQUE INDEX data_RFTID ON `data` (`RFT_ID`)"}]
MikeRalphson/MetaScraper
Node.js
[]
randomgitman/td_e_nabavki_mk
randomgitman/td_e_nabavki_mk
Python
[]
CSCG/hide_my_ass_proxy_list_ip
Hide My Ass Proxy List (IP)
Python
[{"type":"table","name":"hidemyass","tbl_name":"hidemyass","rootpage":2,"sql":"CREATE TABLE `hidemyass` (`_type` text, `url` text, `country` text, `anonimity` text, `ipaddress` text, `port` text)"},{"type":"index","name":"hidemyass_index0","tbl_name":"hidemyass","rootpage":3,"sql":"CREATE UNIQUE INDEX `hidemyass_index0` on `hidemyass` (`ipaddress`)"}]
CSCG/musicreviews
Python
[]
ohadnet/stig1
wikipedia swarm int
PHP
{"error":"unable to open database file"}
randomgitman/td_eu_supply_eu
Python
[{"type":"table","name":"data","tbl_name":"data","rootpage":2,"sql":"CREATE TABLE `data` (\n `RESPONSE_DEADLINE` text\n , `ADDRESS` text, `NAME` text, `URL` text, `SHORT_DESCRIPTION` text, `RFT_ID` text, `CPV_CODES` text, `DATE_OF_PUBLICATION` text, `PROCESS` text, `BUYERS` text, `RESPONSE_DEADLINE_clean` text, `DATE_OF_PUBLICATION_clean` text, `CONTACT_NAME` text, `COUNTRIES` text, `CONTACT_PHONE` text, `REFERENCE` text, `DESCRIPTION` text)"},{"type":"index","name":"data_RFTID","tbl_name":"data","rootpage":3,"sql":"CREATE UNIQUE INDEX data_RFTID ON `data` (`RFT_ID`)"}]
CSCG/craigslist_music_instr
Craigslist music instruments for sale in the Waterloo region.
Python
[]

Statistics

Average successful run time: about 1 hour

Total run time: about 1 hour

Total cpu time used: less than 20 seconds

Total disk space used: 861 KB

History

  • Manually ran revision d650d4c9 and completed successfully .
    2335 records added in the database
  • Manually ran revision f8617d74 and failed .
    nothing changed in the database
  • Created on morph.io

Scraper code

Node.js

MetaScraper / scraper.js