MikeRalphson / Nitro9

BBC iPlayer available programmes (Nitro)

Scrapes programmes.api.bbc.com


Nitro9

A tool to index available programmes on the BBC iPlayer / radio-player for offline analysis.

This is a scraper that runs on Morph. Please read the Morph.io documentation.

It can also be run locally.

Set an environment variable called MORPH_API_KEY to your Nitro API key. When initially run on an empty / non-existant database, a full scrape of available programmes is performed. Subsequently, only those programmes broadcast since the last update will be added. Programmes passed their expiry date are removed.

To build a full index again, either delete the database or set the environment variable MORPH_REBUILD to true.

Usage: node scraper.js

Output is an sqlite3 database containing one table, "data":

javascript var fields = [ '#index', 'type', 'name', 'pid', 'available', 'expires', 'episode', 'seriesnum', 'episodenum', 'versions', 'duration', 'desc', 'channel', 'categories', 'thumbnail', 'timeadded', 'guidance', 'web', 'vpids' ];

Also included is a simple command line utility called query.js for performing SQL queries against the remote Morph.io database and returning the rows in JSON format.

Usage: node query {SQL-statement}

Contributors MikeRalphson

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Prepping database... Pinging the host... Got a ping ok Removing expired items... 2018-02-16T00:06:00Z to 2018-02-17T07:23:47.292Z Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 0 Rows: 100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 11 Rate-limit: 9 Rows: 200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 9 Rows: 300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 26 Rows: 400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 26 Rows: 500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 39 Rows: 600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 39 Rows: 700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 39 Rows: 800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 60 Rows: 900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 60 Rows: 1000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 12 Rate-limit: 67 Rows: 1100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 67 Rows: 1200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 67 Rows: 1300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 81 Rows: 1400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 81 Rows: 1500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 91 Rows: 1600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 91 Rows: 1700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 108 Rows: 1800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 108 Rows: 1900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 108 Rows: 2000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 118 Rows: 2100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 118 Rows: 2200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 137 Rows: 2300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 137 Rows: 2400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 152 Rows: 2500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 152 Rows: 2600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 165 Rows: 2700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 165 Rows: 2800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 175 Rows: 2900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 175 Rows: 3000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Exiting with 3045 rows processed

Data

Downloaded 365 times by MikeRalphson jbraceg nroles TJSoler

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (63.8 MB) Use the API

rows 10 / 21939

#index type name pid available expires episode seriesnum episodenum versions duration desc channel categories thumbnail timeadded guidance web vpids
115827
tv
Cosmic Quantum Ray
p03m4y67
2016-03-24T15:04:29Z
1523710800
Cosmic Quantum... Robbie
3
default
1296
Professor Evil Brainhead switches the minds of Ray and Robbie.
bbc_webonly
Children's,Animation
1461192679
p03m4yhq
115828
tv
Cosmic Quantum Ray
p03m4xg9
2016-03-24T15:01:34Z
1523710800
Alison's New Pet
2
default
1282
A Dadsding eats the wrong food.
bbc_webonly
Children's,Animation
1461192679
p03m4xml
115829
tv
Cosmic Quantum Ray
p03m4w8y
2016-04-15T13:30:52Z
1522479600
Alison Attacks
1
technicalreplacement,default
1296
Kronecker uses Robbie’s friend Allison as his host body.
bbc_webonly
Children's
1461192679
p03r4lm4,p03m4wfl
150796
tv
Frankie Boyle's Election Autopsy
p02qs82x
2015-05-18T09:18:47Z
1589745600
Frankie Boyle's Election Autopsy
default
2700
Frankie Boyle attempts to make sense of the recent general election.
bbc_webonly
Comedy,Satire,Comedy,Standup
1461193659
Contains very strong language.
p02qs85f
154890
tv
Funny Valentines
p02j75pt
2016-02-04T13:19:35Z
1585170000
Nick Helm: Elephant
6
default
582
Two best friends wile away a skint afternoon walking around the backstreets of Brighton.
bbc_webonly
Comedy
1461193784
Contains adult humour.
p02j7771
154893
tv
Funny Valentines
p02j751m
2016-02-04T13:18:25Z
1585170000
Katy Wix: Dear Jean-Pierre
1
default
324
Katy Wix tells the glamorous Frenchman who stole her heart that it's over.
bbc_webonly
Comedy
1461193784
p02j756p
154896
tv
Funny Valentines
p02j72t3
2016-02-04T13:17:21Z
1585170000
Romesh Ranganathan: Rom Com
2
default
354
Romesh Ranganathan's alter ego offers some dubious advice on his seduction techniques.
bbc_webonly
Comedy,Character
1461193784
Contains adult humour.
p02j74tn
154902
tv
Funny Valentines
p02j5llz
2016-02-04T13:14:53Z
1585170000
Limmy: A Wee Video! For The Lassies!
5
default
372
Limmy plays a nervous romantic filming his clip for a dating website.
bbc_webonly
Comedy,Sketch,Comedy,Character
1461193784
Contains strong language.
p02j5lp8
157181
tv
Frankie Boyle's Referendum Autopsy
p0276z6t
2014-09-26T15:36:51Z
1569700800
Frankie Boyle's Referendum Autopsy
iplayerversion
2520
Frankie Boyle is joined by guests to cast a critical eye over the Scottish Referendum
bbc_webonly
Comedy,Satire,Comedy,Standup
1461193830
Contains very strong language and adult humour
p0276z6y
163678
tv
Funny Valentines
p02j5mqy
2016-02-04T13:16:18Z
1585170000
Bill Bailey: Love Song
8
default
262
Bill Bailey re-imagines one of his most popular songs in this spoof music video.
bbc_webonly
Comedy,Music
1461193968
p02j5myw

Statistics

Average successful run time: 25 minutes

Total run time: 11 days

Total cpu time used: about 12 hours

Total disk space used: 63.9 MB

History

  • Auto ran revision bb8b7163 and completed successfully .
    1102 records added, 1095 records removed in the database
    4314 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    203 records added, 268 records removed in the database
    902 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    1526 records added, 1548 records removed in the database
    5743 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    710 records added, 741 records removed in the database
    2674 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    956 records added, 978 records removed in the database
    3674 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Node.js

Nitro9 / scraper.js