MikeRalphson / Nitro9

BBC iPlayer available programmes (Nitro)

Scrapes programmes.api.bbc.com


Nitro9

A tool to index available programmes on the BBC iPlayer / radio-player for offline analysis.

This is a scraper that runs on Morph. Please read the Morph.io documentation.

It can also be run locally.

Set an environment variable called MORPH_API_KEY to your Nitro API key. When initially run on an empty / non-existant database, a full scrape of available programmes is performed. Subsequently, only those programmes broadcast since the last update will be added. Programmes passed their expiry date are removed.

To build a full index again, either delete the database or set the environment variable MORPH_REBUILD to true.

Usage: node scraper.js

Output is an sqlite3 database containing one table, "data":

javascript var fields = [ '#index', 'type', 'name', 'pid', 'available', 'expires', 'episode', 'seriesnum', 'episodenum', 'versions', 'duration', 'desc', 'channel', 'categories', 'thumbnail', 'timeadded', 'guidance', 'web', 'vpids' ];

Also included is a simple command line utility called query.js for performing SQL queries against the remote Morph.io database and returning the rows in JSON format.

Usage: node query {SQL-statement}

Contributors MikeRalphson

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Node.js app detected  -----> Creating runtime environment   NPM_CONFIG_LOGLEVEL=error  NPM_CONFIG_PRODUCTION=true  NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt  NODE_ENV=production  NODE_TLS_REJECT_UNAUTHORIZED=0  NODE_MODULES_CACHE=true  -----> Installing binaries  engines.node (package.json): unspecified  engines.npm (package.json): unspecified (use default)   Resolving node version (latest stable) via semver.io...  Downloading and installing node 6.9.1...  Using default npm version: 3.10.8  -----> Restoring cache  Skipping cache restore (new runtime signature)  -----> Building dependencies  Installing node modules (package.json)   > sqlite3@3.1.8 install /tmp/build/node_modules/sqlite3  > node-pre-gyp install --fallback-to-build   [sqlite3] Success: "/tmp/build/node_modules/sqlite3/lib/binding/node-v48-linux-x64/node_sqlite3.node" is installed via remote  nitro9@1.1.1 /tmp/build  +-- bbcparse@1.2.0  | +-- ajv@4.9.2  | | +-- co@4.6.0  | | `-- json-stable-stringify@1.0.1  | | `-- jsonify@0.0.0  | +-- jgexml@0.3.6  | +-- node-getopt@0.2.3  | +-- openapi2js@1.0.10  | `-- recursive-readdir@2.1.0  | `-- minimatch@3.0.2  | `-- brace-expansion@1.1.6  | +-- balanced-match@0.4.2  | `-- concat-map@0.0.1  `-- sqlite3@3.1.8  +-- nan@2.4.0  `-- node-pre-gyp@0.6.31  +-- mkdirp@0.5.1  | `-- minimist@0.0.8  +-- nopt@3.0.6  | `-- abbrev@1.0.9  +-- npmlog@4.0.0  | +-- are-we-there-yet@1.1.2  | | +-- delegates@1.0.0  | | `-- readable-stream@2.1.5  | | +-- buffer-shims@1.0.0  | | +-- core-util-is@1.0.2  | | +-- inherits@2.0.3  | | +-- isarray@1.0.0  | | +-- process-nextick-args@1.0.7  | | +-- string_decoder@0.10.31  | | `-- util-deprecate@1.0.2  | +-- console-control-strings@1.1.0  | +-- gauge@2.6.0  | | +-- aproba@1.0.4  | | +-- has-color@0.1.7  | | +-- has-unicode@2.0.1  | | +-- object-assign@4.1.0  | | +-- signal-exit@3.0.1  | | +-- string-width@1.0.2  | | | +-- code-point-at@1.0.1  | | | | `-- number-is-nan@1.0.1  | | | `-- is-fullwidth-code-point@1.0.0  | | | `-- number-is-nan@1.0.1  | | +-- strip-ansi@3.0.1  | | | `-- ansi-regex@2.0.0  | | `-- wide-align@1.1.0  | `-- set-blocking@2.0.0  +-- rc@1.1.6  | +-- deep-extend@0.4.1  | +-- ini@1.3.4  | +-- minimist@1.2.0  | `-- strip-json-comments@1.0.4  +-- request@2.76.0  | +-- aws-sign2@0.6.0  | +-- aws4@1.5.0  | +-- caseless@0.11.0  | +-- combined-stream@1.0.5  | | `-- delayed-stream@1.0.0  | +-- extend@3.0.0  | +-- forever-agent@0.6.1  | +-- form-data@2.1.1  | | `-- asynckit@0.4.0  | +-- har-validator@2.0.6  | | +-- chalk@1.1.3  | | | +-- ansi-styles@2.2.1  | | | +-- escape-string-regexp@1.0.5  | | | +-- has-ansi@2.0.0  | | | | `-- ansi-regex@2.0.0  | | | +-- strip-ansi@3.0.1  | | | | `-- ansi-regex@2.0.0  | | | `-- supports-color@2.0.0  | | +-- commander@2.9.0  | | | `-- graceful-readlink@1.0.1  | | +-- is-my-json-valid@2.15.0  | | | +-- generate-function@2.0.0  | | | +-- generate-object-property@1.2.0  | | | | `-- is-property@1.0.2  | | | +-- jsonpointer@4.0.0  | | | `-- xtend@4.0.1  | | `-- pinkie-promise@2.0.1  | | `-- pinkie@2.0.4  | +-- hawk@3.1.3  | | +-- boom@2.10.1  | | +-- cryptiles@2.0.5  | | +-- hoek@2.16.3  | | `-- sntp@1.0.9  | +-- http-signature@1.1.1  | | +-- assert-plus@0.2.0  | | +-- jsprim@1.3.1  | | | +-- extsprintf@1.0.2  | | | +-- json-schema@0.2.3  | | | `-- verror@1.3.6  | | `-- sshpk@1.10.1  | | +-- asn1@0.2.3  | | +-- assert-plus@1.0.0  | | +-- bcrypt-pbkdf@1.0.0  | | +-- dashdash@1.14.0  | | +-- ecc-jsbn@0.1.1  | | +-- getpass@0.1.6  | | +-- jodid25519@1.0.2  | | +-- jsbn@0.1.0  | | `-- tweetnacl@0.14.3  | +-- is-typedarray@1.0.0  | +-- isstream@0.1.2  | +-- json-stringify-safe@5.0.1  | +-- mime-types@2.1.12  | | `-- mime-db@1.24.0  | +-- node-uuid@1.4.7  | +-- oauth-sign@0.8.2  | +-- qs@6.3.0  | +-- stringstream@0.0.5  | +-- tough-cookie@2.3.2  | | `-- punycode@1.4.1  | `-- tunnel-agent@0.4.3  +-- rimraf@2.5.4  | `-- glob@7.1.1  | +-- fs.realpath@1.0.0  | +-- inflight@1.0.6  | | `-- wrappy@1.0.2  | +-- inherits@2.0.3  | +-- minimatch@3.0.3  | | `-- brace-expansion@1.1.6  | | +-- balanced-match@0.4.2  | | `-- concat-map@0.0.1  | +-- once@1.4.0  | | `-- wrappy@1.0.2  | `-- path-is-absolute@1.0.1  +-- semver@5.3.0  +-- tar@2.2.1  | +-- block-stream@0.0.9  | +-- fstream@1.0.10  | | `-- graceful-fs@4.1.9  | `-- inherits@2.0.3  `-- tar-pack@3.3.0  +-- debug@2.2.0  | `-- ms@0.7.1  +-- fstream@1.0.10  | +-- graceful-fs@4.1.9  | `-- inherits@2.0.3  +-- fstream-ignore@1.0.5  | +-- inherits@2.0.3  | `-- minimatch@3.0.3  | `-- brace-expansion@1.1.6  | +-- balanced-match@0.4.2  | `-- concat-map@0.0.1  +-- once@1.3.3  | `-- wrappy@1.0.2  +-- readable-stream@2.1.5  | +-- buffer-shims@1.0.0  | +-- core-util-is@1.0.2  | +-- inherits@2.0.3  | +-- isarray@1.0.0  | +-- process-nextick-args@1.0.7  | +-- string_decoder@0.10.31  | `-- util-deprecate@1.0.2  `-- uid-number@0.0.6   -----> Caching build  Clearing previous node cache  Saving 2 cacheDirectories (default):  - node_modules  - bower_components (nothing to cache)  -----> Build succeeded!  +-- bbcparse@1.2.0  `-- sqlite3@3.1.8   -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Prepping database... Pinging the host... Got a ping ok Removing expired items... 2016-12-08T03:34:00Z to 2016-12-09T06:22:51.918Z Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 0 Rows: 100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 0 Rows: 200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 10 Rows: 300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 10 Rows: 400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 17 Rows: 500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 17 Rows: 600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 32 Rows: 700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 32 Rows: 800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 32 Rows: 900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 44 Rows: 1000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 44 Rows: 1100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 53 Rows: 1200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 53 Rows: 1300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 70 Rows: 1400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 70 Rows: 1500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 70 Rows: 1600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 84 Rows: 1700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 84 Rows: 1800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 94 Rows: 1900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 94 Rows: 2000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 110 Rows: 2100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 110 Rows: 2200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 110 Rows: 2300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 123 Rows: 2400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 133 Rows: 2500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 133 Rows: 2600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Exiting with 2687 rows processed

Data

Downloaded 355 times by MikeRalphson

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (63.8 MB) Use the API

rows 10 / 22636

#index type name pid available expires episode seriesnum episodenum versions duration desc channel categories thumbnail timeadded guidance web vpids
102680
tv
Stacey Dooley in Greece: Migrant Kids in Crisis
p03q8xm4
2016-04-07T09:48:25Z
1492617600
Stacey Dooley in Greece: Migrant Kids in Crisis
iplayerversion
1800
Stacey Dooley travels to Greece to follow children on the migrant trail.
bbc_three
Factual
1461192267
Contains some upsetting scenes.
p03q8xpr
106531
tv
Question Time
b077nv3b
2016-04-15T16:44:43Z
1492209900
signed,default
3600
With Daniel Hannan, Owen Smith, Angus Robertson, Baroness Jones and Dia Chakravarty.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b077s6f1,b077nv2d
106532
tv
Question Time
b076zn7t
2016-04-08T15:31:48Z
1491605100
signed,default
3600
David Dimbleby presents topical debate from Ilford, East London.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b0770crz,b076zn7m
106533
tv
Question Time
b07453vd
2016-03-16T17:13:09Z
1490036400
signed,default
3600
David Dimbleby presents Question Time from Chelmsford.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b0759597,b07453v8
106534
tv
Question Time
b073gmwl
2016-03-09T16:27:08Z
1489431600
signed,default
3600
David Dimbleby presents topical debate from Dundee.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b073qgwz,b073gmwd
106535
tv
Question Time
b072fpn3
2016-03-02T16:28:45Z
1488826800
signed,default
3600
David Dimbleby is joined by Dominic Raab, John McDonnell, Louise Bours and Jermaine Jenas.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b072zjsn,b072fpmj
106536
tv
Question Time
b071yl0c
2016-02-23T17:16:36Z
1488222000
signed,default
3600
David Dimbleby presents topical debate from Poole, Dorset.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b0728pn6,b071ykz2
106537
tv
Question Time
b071g6kj
2016-02-17T11:05:16Z
1487617200
signed,default
3600
Panellists include Justine Greening, Lisa Nandy, John Nicolson and June Sarpong.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b071km8q,b071g6kf
106538
tv
Question Time
b070hsfz
2016-02-12T18:39:51Z
1487012400
signed,default
3600
With Stephen Crabb, Carwyn Jones, Nigel Farage, Leanne Wood and Romesh Ranganathan.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b070rdl7,b070hsfs
106539
tv
Question Time
b07027px
2016-02-05T17:42:18Z
1486407600
signed,default
3600
David Dimbleby presents topical debate from Bradford.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b07027vt,b07027k3

Statistics

Average successful run time: 21 minutes

Total run time: 3 days

Total cpu time used: about 2 hours

Total disk space used: 63.9 MB

History

  • Auto ran revision bb8b7163 and completed successfully .
    948 records added, 921 records removed in the database
    3021 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    491 records added, 529 records removed in the database
    1587 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    663 records added, 570 records removed in the database
    2235 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    779 records added, 794 records removed in the database
    2562 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    869 records added, 774 records removed in the database
    2918 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1515 records added, 1543 records removed in the database
    5135 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    113 records added, 141 records removed in the database
    398 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    4552 records added, 4295 records removed in the database
    15979 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1050 records added, 1052 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    345 records added, 683 records removed in the database
    1235 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1222 records added, 343 records removed in the database
    4062 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1282 records added, 275 records removed in the database
    3996 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1068 records added, 282 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    921 records added, 214 records removed in the database
    2958 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1104 records added, 280 records removed in the database
    3796 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1138 records added, 264 records removed in the database
    3628 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1149 records added, 274 records removed in the database
    3681 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    515 records added, 157 records removed in the database
    1596 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    6239 records added, 771 records removed in the database
    21046 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    908 records added, 219 records removed in the database
    2666 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    927 records added, 149 records removed in the database
    3211 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    983 records added, 310 records removed in the database
    3249 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    687 records added, 211 records removed in the database
    2309 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1408 records added, 286 records removed in the database
    4779 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    747 records added, 150 records removed in the database
    2446 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1060 records added, 134 records removed in the database
    3515 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    58 records added, 35 records removed in the database
    194 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1159 records added, 163 records removed in the database
    3761 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1297 records added, 156 records removed in the database
    4393 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    845 records added, 121 records removed in the database
    2805 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    185 records added, 49 records removed in the database
    761 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    954 records added, 7 records removed in the database
    3324 pages scraped
  • Manually ran revision bb8b7163 and completed successfully .
    335 records added, 17994 records removed, 71 records updated in the database
    8230 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    46 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    82 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    4 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    320 records added, 735 records removed in the database
    858 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    65 records added, 648 records removed in the database
    178 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    3401 records added, 3355 records removed in the database
    12412 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    11099 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    7433 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1670 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    776 records added, 770 records removed in the database
    2375 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    2581 records added, 2379 records removed in the database
    7411 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    9001 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    743 records added, 810 records removed in the database
    2374 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    984 records added, 969 records removed in the database
    3126 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    838 records added, 814 records removed in the database
  • Auto ran revision 24158e26 and completed successfully .
    912 records added, 751 records removed in the database
    2979 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    926 records added, 848 records removed, 2011 records updated in the database
    2939 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    3450 records added, 3013 records removed, 194 records updated in the database
    12405 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    9437 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    7041 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    4888 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    4003 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    474 records added, 1519 records removed in the database
    1383 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    892 records added, 808 records removed in the database
    2814 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    893 records added, 814 records removed in the database
    2952 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1210 records added, 1162 records removed in the database
    4324 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2496 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1401 records added, 872 records removed in the database
    4813 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    2872 records added, 1166 records removed in the database
    10833 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    437 records added, 903 records removed in the database
    1421 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2093 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    2726 records added, 4771 records removed in the database
    10007 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    5968 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    10960 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    10071 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    173 records added, 188 records removed in the database
    541 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1652 records added, 1451 records removed in the database
    5879 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1612 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    776 records added, 830 records removed in the database
    2607 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1578 records added, 1623 records removed in the database
    5578 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    826 records added, 862 records removed in the database
    2660 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1901 records added, 638 records removed, 328 records updated in the database
    7387 pages scraped
  • Manually ran revision 24158e26 and failed .
    400 records added, 1620 records removed in the database
    1116 pages scraped
  • Auto ran revision 8cd6e615 and failed .
    nothing changed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    223 records added, 245 records removed in the database
    1024 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    1433 records added, 1474 records removed in the database
    5104 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    351 records added, 384 records removed in the database
    1372 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    929 records added, 920 records removed in the database
    3516 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    917 records added, 882 records removed in the database
    3243 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    1336 records added, 1285 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1613 records added, 1515 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    445 records added, 554 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1186 records added, 1150 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    832 records added, 814 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    303 records added, 326 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1150 records added, 1169 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    768 records added, 812 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1920 records added, 1935 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    925 records added, 885 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    864 records added, 846 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    104 records added, 124 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1525 records added, 1551 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    626 records added, 593 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1073 records added, 1097 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    731 records added, 845 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    366 records added, 419 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    814 records added, 766 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    934 records added, 969 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1435 records added, 1432 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    811 records added, 794 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    725 records added, 806 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    252 records added, 254 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1553 records added, 1602 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    806 records added, 864 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    905 records added, 864 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    855 records added, 878 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    342 records added, 376 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1045 records added, 1086 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1027 records added, 1119 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    808 records added, 785 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    293 records added, 296 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1276 records added, 1286 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    609 records added, 630 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1333 records added, 1294 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1710 records added, 1795 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1274 records added, 1332 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    703 records added, 780 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1144 records added, 1137 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    108 records added, 127 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1597 records added, 1579 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    606 records added, 756 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    861 records added, 931 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1116 records added, 1157 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    735 records added, 846 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    682 records added, 696 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1126 records added, 1163 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    945 records added, 971 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    64 records added, 72 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1344 records added, 1487 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1038 records added, 1060 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    838 records added, 858 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    117 records added, 131 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1266 records added, 1371 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    447 records added, 492 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1371 records added, 1122 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1064 records added, 1037 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    682 records added, 744 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    993 records added, 988 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    594 records added, 602 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    489 records added, 578 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1608 records added, 1503 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    805 records added, 940 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    795 records added, 323 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    414 records added, 318 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    803 records added, 819 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1387 records added, 1046 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    970 records added, 874 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    655 records added, 762 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    713 records added, 799 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1302 records added, 91587 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1051 records added, 1003 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    198 records added, 194 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1691 records added, 1562 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    921 records added, 813 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    520 records added, 571 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1064 records added, 1100 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    669 records added, 613 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    660 records added, 616 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1384 records added, 1287 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    828 records added, 755 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    550 records added, 552 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    578 records added, 539 records removed in the database
  • Manually ran revision b22b6a60 and completed successfully .
    2187 records added, 1075 records removed in the database
  • Manually ran revision b22b6a60 and failed .
    nothing changed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    274 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    897 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    569 records added, 808 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    1106 records added, 1058 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    1204 records added, 1088 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    220 records added, 241 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    140 records added, 138 records removed, 2253 records updated in the database
  • Manually ran revision b22b6a60 and completed successfully .
    2112 records added, 195 records removed in the database
  • Manually ran revision b22b6a60 and completed successfully .
    209 records added, 776 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    1053 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    388 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    1565 records added, 1402 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    52 records added, 73 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    103 records added, 888 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    1485 records added, 1424 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    377 records added, 837 records removed in the database
  • Auto ran revision cc7562b1 and completed successfully .
    898 records added, 989 records removed in the database
  • Auto ran revision cc7562b1 and completed successfully .
    1073 records added, 1018 records removed in the database
  • Auto ran revision cc7562b1 and completed successfully .
    129 records added, 140 records removed in the database
  • Manually ran revision cc7562b1 and completed successfully .
    816 records added, 755 records removed in the database
  • Manually ran revision 02edb10f and completed successfully .
    68 records added, 102 records removed in the database
  • Manually ran revision d2db138d and completed successfully .
    112075 records added in the database
  • Created on morph.io

Scraper code

Node.js

Nitro9 / scraper.js