MikeRalphson / Nitro9

BBC iPlayer available programmes (Nitro)


Nitro9

A tool to index available programmes on the BBC iPlayer / radio-player for offline analysis.

This is a scraper that runs on Morph. Please read the Morph.io documentation.

It can also be run locally.

Set an environment variable called MORPH_API_KEY to your Nitro API key. When initially run on an empty / non-existant database, a full scrape of available programmes is performed. Subsequently, only those programmes broadcast since the last update will be added. Programmes passed their expiry date are removed.

To build a full index again, either delete the database or set the environment variable MORPH_REBUILD to true.

Usage: node scraper.js

Output is an sqlite3 database containing one table, "data":

javascript var fields = [ '#index', 'type', 'name', 'pid', 'available', 'expires', 'episode', 'seriesnum', 'episodenum', 'versions', 'duration', 'desc', 'channel', 'categories', 'thumbnail', 'timeadded', 'guidance', 'web', 'vpids' ];

Also included is a simple command line utility called query.js for performing SQL queries against the remote Morph.io database and returning the rows in JSON format.

Usage: node query {SQL-statement}

Contributors MikeRalphson

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Prepping database... Pinging the host... Got a ping ok Removing expired items... 2017-03-23T14:39:00Z to 2017-03-24T01:58:16.792Z Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 0 Rows: 100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 0 Rows: 200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 10 Rows: 300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 10 Rows: 400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 16 Rows: 500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 16 Rows: 600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 16 Rows: 700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 18 Rows: 800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 18 Rows: 900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 28 Rows: 1000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 28 Rows: 1100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 34 Rows: 1200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Exiting with 1289 rows processed

Data

Downloaded 357 times by MikeRalphson nroles TJSoler

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (63.8 MB) Use the API

rows 10 / 21311

#index type name pid available expires episode seriesnum episodenum versions duration desc channel categories thumbnail timeadded guidance web vpids
102680
tv
Stacey Dooley in Greece: Migrant Kids in Crisis
p03q8xm4
2016-04-07T09:48:25Z
1492617600
Stacey Dooley in Greece: Migrant Kids in Crisis
iplayerversion
1800
Stacey Dooley travels to Greece to follow children on the migrant trail.
bbc_three
Factual
1461192267
Contains some upsetting scenes.
p03q8xpr
106531
tv
Question Time
b077nv3b
2016-04-15T16:44:43Z
1492209900
signed,default
3600
With Daniel Hannan, Owen Smith, Angus Robertson, Baroness Jones and Dia Chakravarty.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b077s6f1,b077nv2d
106532
tv
Question Time
b076zn7t
2016-04-08T15:31:48Z
1491605100
signed,default
3600
David Dimbleby presents topical debate from Ilford, East London.
bbc_one
Factual,Politics,News,Discussion & Talk
1461192389
b0770crz,b076zn7m
107543
tv
Drugs Map of Britain
p03nyf5k
2016-04-04T15:53:10Z
1491297600
Wolverhampton: Getting Off Mamba
1
editorial
1560
Exploring a legal high ‘epidemic’ in Wolverhampton.
bbc_three
Factual
1461192395
Contains strong language.
p03pmbwh
115827
tv
Cosmic Quantum Ray
p03m4y67
2016-03-24T15:04:29Z
1523710800
Cosmic Quantum... Robbie
3
default
1296
Professor Evil Brainhead switches the minds of Ray and Robbie.
bbc_webonly
Children's,Animation
1461192679
p03m4yhq
115828
tv
Cosmic Quantum Ray
p03m4xg9
2016-03-24T15:01:34Z
1523710800
Alison's New Pet
2
default
1282
A Dadsding eats the wrong food.
bbc_webonly
Children's,Animation
1461192679
p03m4xml
115829
tv
Cosmic Quantum Ray
p03m4w8y
2016-04-15T13:30:52Z
1522479600
Alison Attacks
1
technicalreplacement,default
1296
Kronecker uses Robbie’s friend Allison as his host body.
bbc_webonly
Children's
1461192679
p03r4lm4,p03m4wfl
150796
tv
Frankie Boyle's Election Autopsy
p02qs82x
2015-05-18T09:18:47Z
1589745600
Frankie Boyle's Election Autopsy
default
2700
Frankie Boyle attempts to make sense of the recent general election.
bbc_webonly
Comedy,Satire,Comedy,Standup
1461193659
Contains very strong language.
p02qs85f
154890
tv
Funny Valentines
p02j75pt
2016-02-04T13:19:35Z
1585170000
Nick Helm: Elephant
6
default
582
Two best friends wile away a skint afternoon walking around the backstreets of Brighton.
bbc_webonly
Comedy
1461193784
Contains adult humour.
p02j7771
154893
tv
Funny Valentines
p02j751m
2016-02-04T13:18:25Z
1585170000
Katy Wix: Dear Jean-Pierre
1
default
324
Katy Wix tells the glamorous Frenchman who stole her heart that it's over.
bbc_webonly
Comedy
1461193784
p02j756p

Statistics

Average successful run time: 25 minutes

Total run time: 6 days

Total cpu time used: about 4 hours

Total disk space used: 63.9 MB

History

  • Auto ran revision bb8b7163 and completed successfully .
    427 records added, 485 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    1256 records added, 1230 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    382 records added, 394 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    1430 records added, 763 records removed, 7 records updated in the database
  • Auto ran revision bb8b7163 and failed .
    376 records added, 1291 records removed in the database
    1023 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    582 records added, 671 records removed in the database
    1788 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    570 records added, 567 records removed in the database
    1856 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1169 records added, 1126 records removed, 1 record updated in the database
    3694 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    473 records added, 493 records removed in the database
    1634 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    991 records added, 980 records removed in the database
    3200 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1474 records added, 1464 records removed in the database
    5040 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1818 records added, 1788 records removed in the database
    6558 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1082 records added, 1045 records removed in the database
    3419 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1334 records added, 1355 records removed in the database
    4441 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    38 records added, 29 records removed in the database
    107 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1576 records added, 1491 records removed in the database
    5167 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    4875 records added, 4880 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    14062 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    12932 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    9984 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    5660 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    2977 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1697 records added, 1609 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    15 records added, 38 records removed in the database
    71 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    902 records added, 961 records removed in the database
    2929 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1041 records added, 1140 records removed in the database
    3261 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1254 records added, 635 records removed in the database
    4249 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    490 records added, 426 records removed in the database
    1599 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1550 records added, 1510 records removed in the database
    5364 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    2484 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1346 records added, 1432 records removed in the database
    4458 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    230 records added, 260 records removed in the database
    895 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1424 records added, 1461 records removed in the database
    4894 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1005 records added, 1032 records removed in the database
    3054 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    481 records added, 481 records removed in the database
    1361 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    584 records added, 654 records removed in the database
    3535 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1813 records added, 1720 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    1249 records added, 1327 records removed in the database
    4385 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    546 records added, 671 records removed in the database
    1946 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    957 records added, 912 records removed in the database
    2918 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    756 records added, 259 records removed in the database
    2429 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    2564 records added, 566 records removed in the database
    8814 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    3968 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    1999 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    993 records added, 260 records removed in the database
    3386 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    882 records added, 741 records removed in the database
    3342 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1968 records added, 1849 records removed in the database
    6386 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
    3297 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    5335 records added, 2804 records removed, 4 records updated in the database
    18961 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    81 records added, 983 records removed, 12 records updated in the database
    396 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
    4336 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    3047 records added, 4870 records removed in the database
    10946 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    989 records added, 942 records removed in the database
    3260 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    790 records added, 788 records removed in the database
    2337 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    698 records added, 655 records removed in the database
    2095 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    4953 records added, 994 records removed, 3 records updated in the database
    18128 pages scraped
  • Auto ran revision bb8b7163 and failed .
    225 records added, 2616 records removed in the database
  • Auto ran revision bb8b7163 and failed .
    2673 records added, 7909 records removed in the database
    10540 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    975 records added, 1050 records removed in the database
    3031 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1340 records added, 1511 records removed in the database
    4637 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    581 records added, 626 records removed in the database
    1817 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1593 records added, 1535 records removed in the database
    5119 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    374 records added, 493 records removed in the database
    1321 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1297 records added, 1404 records removed in the database
    4409 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    128 records added, 174 records removed in the database
    535 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    3499 records added, 3586 records removed in the database
    11835 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    1164 records added, 1161 records removed in the database
    3701 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    2337 records added, 2525 records removed in the database
    8071 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    791 records added, 847 records removed in the database
    2628 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1105 records added, 1161 records removed in the database
    3731 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1147 records added, 1247 records removed in the database
    3814 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    866 records added, 950 records removed in the database
    2759 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1235 records added, 1321 records removed in the database
    4117 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    948 records added, 921 records removed in the database
    3021 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    491 records added, 529 records removed in the database
    1587 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    663 records added, 570 records removed in the database
    2235 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    779 records added, 794 records removed in the database
    2562 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    869 records added, 774 records removed in the database
    2918 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1515 records added, 1543 records removed in the database
    5135 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    113 records added, 141 records removed in the database
    398 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    4552 records added, 4295 records removed in the database
    15979 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1050 records added, 1052 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    345 records added, 683 records removed in the database
    1235 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1222 records added, 343 records removed in the database
    4062 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1282 records added, 275 records removed in the database
    3996 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1068 records added, 282 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    921 records added, 214 records removed in the database
    2958 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1104 records added, 280 records removed in the database
    3796 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1138 records added, 264 records removed in the database
    3628 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1149 records added, 274 records removed in the database
    3681 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    515 records added, 157 records removed in the database
    1596 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    6239 records added, 771 records removed in the database
    21046 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    908 records added, 219 records removed in the database
    2666 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    927 records added, 149 records removed in the database
    3211 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    983 records added, 310 records removed in the database
    3249 pages scraped
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    687 records added, 211 records removed in the database
    2309 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1408 records added, 286 records removed in the database
    4779 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    747 records added, 150 records removed in the database
    2446 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1060 records added, 134 records removed in the database
    3515 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    58 records added, 35 records removed in the database
    194 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1159 records added, 163 records removed in the database
    3761 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1297 records added, 156 records removed in the database
    4393 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    845 records added, 121 records removed in the database
    2805 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    185 records added, 49 records removed in the database
    761 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    954 records added, 7 records removed in the database
    3324 pages scraped
  • Manually ran revision bb8b7163 and completed successfully .
    335 records added, 17994 records removed, 71 records updated in the database
    8230 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    46 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    82 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    4 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Auto ran revision 24158e26 and completed successfully .
    320 records added, 735 records removed in the database
    858 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    65 records added, 648 records removed in the database
    178 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    3401 records added, 3355 records removed in the database
    12412 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    11099 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    7433 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1670 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    776 records added, 770 records removed in the database
    2375 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    2581 records added, 2379 records removed in the database
    7411 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    9001 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    743 records added, 810 records removed in the database
    2374 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    984 records added, 969 records removed in the database
    3126 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    838 records added, 814 records removed in the database
  • Auto ran revision 24158e26 and completed successfully .
    912 records added, 751 records removed in the database
    2979 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    926 records added, 848 records removed, 2011 records updated in the database
    2939 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    3450 records added, 3013 records removed, 194 records updated in the database
    12405 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    9437 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    7041 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    4888 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    4003 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    474 records added, 1519 records removed in the database
    1383 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    892 records added, 808 records removed in the database
    2814 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    893 records added, 814 records removed in the database
    2952 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1210 records added, 1162 records removed in the database
    4324 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2496 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1401 records added, 872 records removed in the database
    4813 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    2872 records added, 1166 records removed in the database
    10833 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    437 records added, 903 records removed in the database
    1421 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    2093 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    2726 records added, 4771 records removed in the database
    10007 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    5968 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    10960 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    10071 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    173 records added, 188 records removed in the database
    541 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1652 records added, 1451 records removed in the database
    5879 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
    1612 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    776 records added, 830 records removed in the database
    2607 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1578 records added, 1623 records removed in the database
    5578 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    nothing changed in the database
  • Auto ran revision 24158e26 and completed successfully .
    826 records added, 862 records removed in the database
    2660 pages scraped
  • Auto ran revision 24158e26 and completed successfully .
    1901 records added, 638 records removed, 328 records updated in the database
    7387 pages scraped
  • Manually ran revision 24158e26 and failed .
    400 records added, 1620 records removed in the database
    1116 pages scraped
  • Auto ran revision 8cd6e615 and failed .
    nothing changed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    223 records added, 245 records removed in the database
    1024 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    1433 records added, 1474 records removed in the database
    5104 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    351 records added, 384 records removed in the database
    1372 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    929 records added, 920 records removed in the database
    3516 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    917 records added, 882 records removed in the database
    3243 pages scraped
  • Auto ran revision 8cd6e615 and completed successfully .
    1336 records added, 1285 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1613 records added, 1515 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    445 records added, 554 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1186 records added, 1150 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    832 records added, 814 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    303 records added, 326 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1150 records added, 1169 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    768 records added, 812 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1920 records added, 1935 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    925 records added, 885 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    864 records added, 846 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    104 records added, 124 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1525 records added, 1551 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    626 records added, 593 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1073 records added, 1097 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    731 records added, 845 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    366 records added, 419 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    814 records added, 766 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    934 records added, 969 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1435 records added, 1432 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    811 records added, 794 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    725 records added, 806 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    252 records added, 254 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1553 records added, 1602 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    806 records added, 864 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    905 records added, 864 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    855 records added, 878 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    342 records added, 376 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1045 records added, 1086 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1027 records added, 1119 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    808 records added, 785 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    293 records added, 296 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1276 records added, 1286 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    609 records added, 630 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1333 records added, 1294 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1710 records added, 1795 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1274 records added, 1332 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    703 records added, 780 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1144 records added, 1137 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    108 records added, 127 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1597 records added, 1579 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    606 records added, 756 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    861 records added, 931 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1116 records added, 1157 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    735 records added, 846 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    682 records added, 696 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1126 records added, 1163 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    945 records added, 971 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    64 records added, 72 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1344 records added, 1487 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1038 records added, 1060 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    838 records added, 858 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    117 records added, 131 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1266 records added, 1371 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    447 records added, 492 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1371 records added, 1122 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1064 records added, 1037 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    682 records added, 744 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    993 records added, 988 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    594 records added, 602 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    489 records added, 578 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1608 records added, 1503 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    805 records added, 940 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    795 records added, 323 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    414 records added, 318 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    803 records added, 819 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1387 records added, 1046 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    970 records added, 874 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    655 records added, 762 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    713 records added, 799 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1302 records added, 91587 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1051 records added, 1003 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    198 records added, 194 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1691 records added, 1562 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    921 records added, 813 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    520 records added, 571 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1064 records added, 1100 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    669 records added, 613 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    660 records added, 616 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    1384 records added, 1287 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    828 records added, 755 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    550 records added, 552 records removed in the database
  • Auto ran revision 8cd6e615 and completed successfully .
    578 records added, 539 records removed in the database
  • Manually ran revision b22b6a60 and completed successfully .
    2187 records added, 1075 records removed in the database
  • Manually ran revision b22b6a60 and failed .
    nothing changed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    274 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    897 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    569 records added, 808 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    1106 records added, 1058 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    1204 records added, 1088 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    220 records added, 241 records removed in the database
  • Auto ran revision b22b6a60 and completed successfully .
    140 records added, 138 records removed, 2253 records updated in the database
  • Manually ran revision b22b6a60 and completed successfully .
    2112 records added, 195 records removed in the database
  • Manually ran revision b22b6a60 and completed successfully .
    209 records added, 776 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    1053 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    388 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    1565 records added, 1402 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    52 records added, 73 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    103 records added, 888 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    1485 records added, 1424 records removed in the database
  • Auto ran revision 277aaf61 and completed successfully .
    377 records added, 837 records removed in the database
  • Auto ran revision cc7562b1 and completed successfully .
    898 records added, 989 records removed in the database
  • Auto ran revision cc7562b1 and completed successfully .
    1073 records added, 1018 records removed in the database
  • Auto ran revision cc7562b1 and completed successfully .
    129 records added, 140 records removed in the database
  • Manually ran revision cc7562b1 and completed successfully .
    816 records added, 755 records removed in the database
  • Manually ran revision 02edb10f and completed successfully .
    68 records added, 102 records removed in the database
  • Manually ran revision d2db138d and completed successfully .
    112075 records added in the database
  • Created on morph.io

Scraper code

Node.js

Nitro9 / scraper.js