MikeRalphson / Nitro9

BBC iPlayer available programmes (Nitro)

Scrapes programmes.api.bbc.com


Nitro9

A tool to index available programmes on the BBC iPlayer / radio-player for offline analysis.

This is a scraper that runs on Morph. Please read the Morph.io documentation.

It can also be run locally.

Set an environment variable called MORPH_API_KEY to your Nitro API key. When initially run on an empty / non-existant database, a full scrape of available programmes is performed. Subsequently, only those programmes broadcast since the last update will be added. Programmes passed their expiry date are removed.

To build a full index again, either delete the database or set the environment variable MORPH_REBUILD to true.

Usage: node scraper.js

Output is an sqlite3 database containing one table, "data":

javascript var fields = [ '#index', 'type', 'name', 'pid', 'available', 'expires', 'episode', 'seriesnum', 'episodenum', 'versions', 'duration', 'desc', 'channel', 'categories', 'thumbnail', 'timeadded', 'guidance', 'web', 'vpids' ];

Also included is a simple command line utility called query.js for performing SQL queries against the remote Morph.io database and returning the rows in JSON format.

Usage: node query {SQL-statement}

Contributors MikeRalphson

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Node.js app detected  -----> Creating runtime environment   NPM_CONFIG_LOGLEVEL=error  NPM_CONFIG_PRODUCTION=true  NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt  NODE_VERBOSE=false  NODE_ENV=production  NODE_TLS_REJECT_UNAUTHORIZED=0  NODE_MODULES_CACHE=true  -----> Installing binaries  engines.node (package.json): unspecified  engines.npm (package.json): unspecified (use default)   Resolving node version 6.x...  Downloading and installing node 6.14.1...  Using default npm version: 3.10.10  -----> Restoring cache  Skipping cache restore (not-found)  -----> Building dependencies  Installing node modules (package.json)   > sqlite3@4.0.0 install /tmp/build/node_modules/sqlite3  > node-pre-gyp install --fallback-to-build   [sqlite3] Success: "/tmp/build/node_modules/sqlite3/lib/binding/node-v48-linux-x64/node_sqlite3.node" is installed via remote  nitro9@1.1.1 /tmp/build  +-- bbcparse@1.3.1  | +-- ajv@4.11.8  | | +-- co@4.6.0  | | `-- json-stable-stringify@1.0.1  | | `-- jsonify@0.0.0  | +-- jgexml@0.3.8  | +-- node-getopt@0.2.4  | +-- openapi2js@1.1.0  | | `-- js-yaml@3.11.0  | | +-- argparse@1.0.10  | | | `-- sprintf-js@1.0.3  | | `-- esprima@4.0.0  | `-- recursive-readdir@2.2.2  | `-- minimatch@3.0.4  | `-- brace-expansion@1.1.11  | +-- balanced-match@1.0.0  | `-- concat-map@0.0.1  `-- sqlite3@4.0.0  +-- nan@2.9.2  `-- node-pre-gyp@0.9.0  +-- detect-libc@1.0.3  +-- mkdirp@0.5.1  | `-- minimist@0.0.8  +-- needle@2.2.0  | +-- debug@2.6.9  | | `-- ms@2.0.0  | +-- iconv-lite@0.4.19  | `-- sax@1.2.4  +-- nopt@4.0.1  | +-- abbrev@1.1.1  | `-- osenv@0.1.5  | +-- os-homedir@1.0.2  | `-- os-tmpdir@1.0.2  +-- npm-packlist@1.1.10  | +-- ignore-walk@3.0.1  | | `-- minimatch@3.0.4  | | `-- brace-expansion@1.1.11  | | +-- balanced-match@1.0.0  | | `-- concat-map@0.0.1  | `-- npm-bundled@1.0.3  +-- npmlog@4.1.2  | +-- are-we-there-yet@1.1.4  | | +-- delegates@1.0.0  | | `-- readable-stream@2.3.5  | | +-- core-util-is@1.0.2  | | +-- isarray@1.0.0  | | +-- process-nextick-args@2.0.0  | | +-- safe-buffer@5.1.1  | | +-- string_decoder@1.0.3  | | `-- util-deprecate@1.0.2  | +-- console-control-strings@1.1.0  | +-- gauge@2.7.4  | | +-- aproba@1.2.0  | | +-- has-unicode@2.0.1  | | +-- object-assign@4.1.1  | | +-- signal-exit@3.0.2  | | +-- string-width@1.0.2  | | | +-- code-point-at@1.1.0  | | | `-- is-fullwidth-code-point@1.0.0  | | | `-- number-is-nan@1.0.1  | | +-- strip-ansi@3.0.1  | | | `-- ansi-regex@2.1.1  | | `-- wide-align@1.1.2  | `-- set-blocking@2.0.0  +-- rc@1.2.6  | +-- deep-extend@0.4.2  | +-- ini@1.3.5  | +-- minimist@1.2.0  | `-- strip-json-comments@2.0.1  +-- rimraf@2.6.2  | `-- glob@7.1.2  | +-- fs.realpath@1.0.0  | +-- inflight@1.0.6  | | `-- wrappy@1.0.2  | +-- inherits@2.0.3  | +-- once@1.4.0  | `-- path-is-absolute@1.0.1  +-- semver@5.5.0  `-- tar@4.4.0  +-- chownr@1.0.1  +-- fs-minipass@1.2.5  +-- minipass@2.2.1  +-- minizlib@1.1.0  `-- yallist@3.0.2   -----> Caching build  Clearing previous node cache  Saving 2 cacheDirectories (default):  - node_modules  - bower_components (nothing to cache)  -----> Build succeeded!  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Prepping database... Pinging the host... Got a ping ok Removing expired items... 2018-04-23T03:06:00Z to 2018-04-24T14:46:52.732Z Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 0 Rows: 100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 0 Rows: 200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 21 Rows: 300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 21 Rows: 400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 39 Rows: 500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 39 Rows: 600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 53 Rows: 700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 11 Rate-limit: 53 Rows: 800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 53 Rows: 900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 64 Rows: 1000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 64 Rows: 1100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 83 Rows: 1200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 83 Rows: 1300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 96 Rows: 1400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 96 Rows: 1500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 106 Rows: 1600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 13 Rate-limit: 106 Rows: 1700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 106 Rows: 1800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 123 Rows: 1900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 123 Rows: 2000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 137 Rows: 2100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 137 Rows: 2200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 11 Rate-limit: 137 Rows: 2300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 158 Rows: 2400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Prepping database... Pinging the host... Got a ping ok Removing expired items... 2018-04-23T03:06:00Z to 2018-04-24T14:46:52.732Z Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 0 Rows: 100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 0 Rows: 200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 21 Rows: 300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 21 Rows: 400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 39 Rows: 500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 39 Rows: 600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 53 Rows: 700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 11 Rate-limit: 53 Rows: 800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 53 Rows: 900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 64 Rows: 1000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 7 Rate-limit: 64 Rows: 1100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 2 Rate-limit: 83 Rows: 1200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 83 Rows: 1300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 96 Rows: 1400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 96 Rows: 1500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 106 Rows: 1600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 13 Rate-limit: 106 Rows: 1700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 106 Rows: 1800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 123 Rows: 1900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 123 Rows: 2000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 137 Rows: 2100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 10 Rate-limit: 137 Rows: 2200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 11 Rate-limit: 137 Rows: 2300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 158 Rows: 2400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 167 Rows: 2500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 167 Rows: 2600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 167 Rows: 2700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 181 Rows: 2800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 8 Rate-limit: 181 Rows: 2900 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 193 Rows: 3000 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 193 Rows: 3100 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 13 Rate-limit: 193 Rows: 3200 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 193 Rows: 3300 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 4 Rate-limit: 193 Rows: 3400 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 9 Rate-limit: 211 Rows: 3500 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 1 Rate-limit: 211 Rows: 3600 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 3 Rate-limit: 226 Rows: 3700 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 5 Rate-limit: 226 Rows: 3800 Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up In-flight: 6 Rate-limit: 226 Rows: 3900 Cleaning up Cleaning up Exiting with 3907 rows processed

Data

Downloaded 365 times by MikeRalphson jbraceg nroles TJSoler

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (63.8 MB) Use the API

rows 10 / 21689

#index type name pid available expires episode seriesnum episodenum versions duration desc channel categories thumbnail timeadded guidance web vpids
150796
tv
Frankie Boyle's Election Autopsy
p02qs82x
2015-05-18T09:18:47Z
1589745600
Frankie Boyle's Election Autopsy
default
2700
Frankie Boyle attempts to make sense of the recent general election.
bbc_webonly
Comedy,Satire,Comedy,Standup
1461193659
Contains very strong language.
p02qs85f
154890
tv
Funny Valentines
p02j75pt
2016-02-04T13:19:35Z
1585170000
Nick Helm: Elephant
6
default
582
Two best friends wile away a skint afternoon walking around the backstreets of Brighton.
bbc_webonly
Comedy
1461193784
Contains adult humour.
p02j7771
154893
tv
Funny Valentines
p02j751m
2016-02-04T13:18:25Z
1585170000
Katy Wix: Dear Jean-Pierre
1
default
324
Katy Wix tells the glamorous Frenchman who stole her heart that it's over.
bbc_webonly
Comedy
1461193784
p02j756p
154896
tv
Funny Valentines
p02j72t3
2016-02-04T13:17:21Z
1585170000
Romesh Ranganathan: Rom Com
2
default
354
Romesh Ranganathan's alter ego offers some dubious advice on his seduction techniques.
bbc_webonly
Comedy,Character
1461193784
Contains adult humour.
p02j74tn
154902
tv
Funny Valentines
p02j5llz
2016-02-04T13:14:53Z
1585170000
Limmy: A Wee Video! For The Lassies!
5
default
372
Limmy plays a nervous romantic filming his clip for a dating website.
bbc_webonly
Comedy,Sketch,Comedy,Character
1461193784
Contains strong language.
p02j5lp8
157181
tv
Frankie Boyle's Referendum Autopsy
p0276z6t
2014-09-26T15:36:51Z
1569700800
Frankie Boyle's Referendum Autopsy
iplayerversion
2520
Frankie Boyle is joined by guests to cast a critical eye over the Scottish Referendum
bbc_webonly
Comedy,Satire,Comedy,Standup
1461193830
Contains very strong language and adult humour
p0276z6y
163678
tv
Funny Valentines
p02j5mqy
2016-02-04T13:16:18Z
1585170000
Bill Bailey: Love Song
8
default
262
Bill Bailey re-imagines one of his most popular songs in this spoof music video.
bbc_webonly
Comedy,Music
1461193968
p02j5myw
165965
tv
Funny Valentines
p02j5737
2016-02-04T13:08:25Z
1585170000
Modern Toss: Lady and the Fly
4
default
398
A lovestruck fly falls for the woman of his dreams.
bbc_webonly
Comedy
1461194045
p02j57g1
165967
tv
Funny Valentines
p02j53t4
2016-02-04T13:05:16Z
1585170000
Matt Berry: Wild Love
7
default
408
Matt Berry explores the mating rituals of exotic beasts in their pursuit of sex.
bbc_webonly
Comedy,Spoof
1461194045
Contains very strong language.
p02j54f7
170108
radio
You and Yours
b00jbrgm
2012-11-17T18:30:28Z
2147472000
default
3420
Consumer news and issues with Julian Worricker.
bbc_radio_four
Factual,Consumer,Magazines & Reviews
1461194199
b00jbpwf

Statistics

Average successful run time: 24 minutes

Total run time: 12 days

Total cpu time used: about 13 hours

Total disk space used: 63.9 MB

History

  • Auto ran revision bb8b7163 and completed successfully .
    1343 records added, 1280 records removed in the database
    5436 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    750 records added, 803 records removed in the database
    3026 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    204 records added, 250 records removed in the database
  • Auto ran revision bb8b7163 and completed successfully .
    1136 records added, 1203 records removed in the database
    4798 pages scraped
  • Auto ran revision bb8b7163 and completed successfully .
    808 records added, 854 records removed in the database
    3337 pages scraped
  • ...
  • Created on morph.io

Show complete history

Scraper code

Node.js

Nitro9 / scraper.js