MikeRalphson / Nitro9

BBC iPlayer available programmes (Nitro)


Nitro9

A tool to index available programmes on the BBC iPlayer / radio-player for offline analysis.

This is a scraper that runs on Morph. Please read the Morph.io documentation.

It can also be run locally.

Set an environment variable called MORPH_API_KEY to your Nitro API key. When initially run on an empty / non-existant database, a full scrape of available programmes is performed. Subsequently, only those programmes broadcast since the last update will be added. Programmes passed their expiry date are removed.

To build a full index again, either delete the database or set the environment variable MORPH_REBUILD to true.

Usage: node scraper.js

Output is an sqlite3 database containing one table, "data":

javascript var fields = [ '#index', 'type', 'name', 'pid', 'available', 'expires', 'episode', 'seriesnum', 'episodenum', 'versions', 'duration', 'desc', 'channel', 'categories', 'thumbnail', 'timeadded', 'guidance', 'web', 'vpids' ];

Also included is a simple command line utility called query.js for performing SQL queries against the remote Morph.io database and returning the rows in JSON format.

Usage: node query {SQL-statement}

Contributors MikeRalphson

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling...  -----> Node.js app detected  -----> Creating runtime environment   NPM_CONFIG_LOGLEVEL=error  NPM_CONFIG_PRODUCTION=true  NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt  NODE_VERBOSE=false  NODE_ENV=production  NODE_TLS_REJECT_UNAUTHORIZED=0  NODE_MODULES_CACHE=true  -----> Installing binaries  engines.node (package.json): unspecified  engines.npm (package.json): unspecified (use default)   Resolving node version 6.x...  Downloading and installing node 6.17.1...  Using default npm version: 3.10.10  -----> Restoring cache  Skipping cache restore (not-found)  -----> Building dependencies  Installing node modules (package.json)   > sqlite3@5.1.2 install /tmp/build/node_modules/sqlite3  > node-pre-gyp install --fallback-to-build   node-pre-gyp ERR! UNCAUGHT EXCEPTION  node-pre-gyp ERR! stack /tmp/build/node_modules/detect-libc/lib/detect-libc.js:76  node-pre-gyp ERR! stack const family = async () => {  node-pre-gyp ERR! stack ^  node-pre-gyp ERR! stack  node-pre-gyp ERR! stack SyntaxError: Unexpected token (  node-pre-gyp ERR! stack at createScript (vm.js:56:10)  node-pre-gyp ERR! stack at Object.runInThisContext (vm.js:97:10)  node-pre-gyp ERR! stack at Module._compile (module.js:549:28)  node-pre-gyp ERR! stack at Object.Module._extensions..js (module.js:586:10)  node-pre-gyp ERR! stack at Module.load (module.js:494:32)  node-pre-gyp ERR! stack at tryModuleLoad (module.js:453:12)  node-pre-gyp ERR! stack at Function.Module._load (module.js:445:3)  node-pre-gyp ERR! stack at Module.require (module.js:504:17)  node-pre-gyp ERR! stack at require (internal/module.js:20:19)  node-pre-gyp ERR! stack at Object.<anonymous> (/tmp/build/node_modules/@mapbox/node-pre-gyp/lib/util/versioning.js:8:21)  node-pre-gyp ERR! System Linux 5.12.2-x86_64-linode144  node-pre-gyp ERR! command "/tmp/build/.heroku/node/bin/node" "/tmp/build/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"  node-pre-gyp ERR! cwd /tmp/build/node_modules/sqlite3  node-pre-gyp ERR! node -v v6.17.1  node-pre-gyp ERR! node-pre-gyp -v v1.0.10  nitro9@1.1.1 /tmp/build  `-- bbcparse@1.3.1  +-- ajv@4.11.8  | +-- co@4.6.0  | `-- json-stable-stringify@1.0.1  | `-- jsonify@0.0.0  +-- jgexml@0.4.4  +-- node-getopt@0.2.4  +-- openapi2js@1.1.2  | `-- yaml@1.10.2  `-- recursive-readdir@2.2.2  `-- minimatch@3.0.4  `-- brace-expansion@1.1.11  +-- balanced-match@1.0.2  `-- concat-map@0.0.1   npm ERR! Linux 5.12.2-x86_64-linode144  npm ERR! argv "/tmp/build/.heroku/node/bin/node" "/tmp/build/.heroku/node/bin/npm" "install" "--unsafe-perm" "--userconfig" "/tmp/build/.npmrc"  npm ERR! node v6.17.1  npm ERR! npm v3.10.10  npm ERR! code ELIFECYCLE   npm ERR! sqlite3@5.1.2 install: `node-pre-gyp install --fallback-to-build`  npm ERR! Exit status 7  npm ERR!  npm ERR! Failed at the sqlite3@5.1.2 install script 'node-pre-gyp install --fallback-to-build'.  npm ERR! Make sure you have the latest version of node.js and npm installed.  npm ERR! If you do, this is most likely a problem with the sqlite3 package,  npm ERR! not with npm itself.  npm ERR! Tell the author that this fails on your system:  npm ERR! node-pre-gyp install --fallback-to-build  npm ERR! You can get information on how to open an issue for this project with:  npm ERR! npm bugs sqlite3  npm ERR! Or if that isn't available, you can get their info via:  npm ERR! npm owner ls sqlite3  npm ERR! There is likely additional logging output above.   npm ERR! Please include the following file with any support request:  npm ERR! /tmp/build/npm-debug.log  -----> Build failed   We're sorry this build is failing! You can troubleshoot common issues here:  https://devcenter.heroku.com/articles/troubleshooting-node-deploys   Some possible problems:   - Node version not specified in package.json  https://devcenter.heroku.com/articles/nodejs-support#specifying-a-node-js-version   Love,  Heroku 

Data

Downloaded 375 times by MikeRalphson jbraceg AccessSQL nroles TJSoler nvasari lsces imperialdalek

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (63.8 MB) Use the API

rows 10 / 5548

#index type name pid available expires episode seriesnum episodenum versions duration desc channel categories thumbnail timeadded guidance web vpids
170108
radio
You and Yours
b00jbrgm
2012-11-17T18:30:28Z
2147472000
default
3420
Consumer news and issues with Julian Worricker.
bbc_radio_four
Factual,Consumer,Magazines & Reviews
1461194199
b00jbpwf
183376
radio
Home Front - Omnibus
b06l3js2
2016-03-23T00:05:24Z
1774926000
25-29 January 1916
45
podcast,default
3407
Pressures begin to tell at the Bevan hospital.
bbc_radio_four_extra
Drama
1461194613
p03c0kym,b06l3jrp
183379
radio
Home Front - Omnibus
b06kvlgq
2016-01-01T00:04:57Z
1767823080
4-8 January 1916
42
podcast,default
3403
The Bevan hospital becomes a welcome alternative to home for Dorothea.
bbc_radio_four
Drama
1461194613
p03dq1gh,b06kvlbf
183380
radio
Home Front - Omnibus
b06kvl04
2015-12-25T00:06:25Z
1767218280
28 December 1915 - 1 January 1916
41
podcast,default
3404
In the week that the home secretary resigns, the Bevan Hospital celebrates the new year.
bbc_radio_four
Drama
1461194613
p03dpwvq,b06kvkz6
183742
radio
Home Front - Omnibus
b06kvjhx
2015-12-08T04:47:43Z
1766613480
21-25 December 1915
40
podcast,podcast,default
3404
Epic drama series set in Great War Britain a hundred years ago this week.
bbc_radio_four
Drama
1461194616
p03bv8dj,p03bgwzb,b06kvjfj
183786
radio
Home Front - Omnibus
b064g17k
2015-12-03T04:02:45Z
1765771200
5-9 October 1915
36
podcast,default
3398
Individual losses and grief increasingly fuel an interest in spiritualism.
bbc_radio_four_extra
Drama
1461194617
p03045n8,b064g162
183790
radio
Home Front - Omnibus
b06492dw
2015-11-30T12:37:33Z
1765252800
7-11 September 1915
32
podcast,default
3403
The population begins to open up to thoughts of the hereafter.
bbc_radio_four_extra
Drama
1461194617
p031kvbx,b06492dl
183792
radio
Home Front - Omnibus
b0612rj1
2015-08-26T23:25:26Z
1756868400
6-10 July 1915
30
podcast,default
3409
Soldiers on leave find home very different to the one they thought they were fighting for.
bbc_radio_four_extra
Drama
1461194617
p02w69jw,b0612qzs
184184
radio
Home Front - Omnibus
b05xqg4p
2015-08-20T23:29:51Z
1756350000
8-12 June 1915
26
podcast,shortened,default
3395
The epic drama series set in Great War Britain a hundred years ago this week.
bbc_radio_four_extra
Drama
1461194682
p02szt44,b05xw7yb,b05xqfcv
184196
radio
Home Front - Omnibus
b054tnzn
2015-05-22T11:10:33Z
1747710000
9-13 March 1915
21
podcast,default
3400
The Battle of Neuve Chapelle draws to a close while the men at Marshalls spoil for a fight
bbc_radio_four_extra
Drama
1461194682
p02qmjsw,b054tn28

Statistics

Average successful run time: 20 minutes

Total run time: 29 days

Total cpu time used: about 20 hours

Total disk space used: 63.9 MB

History

  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Node.js

Nitro9 / scraper.js