MikeRalphson / Nitro9

BBC iPlayer available programmes (Nitro)


Nitro9

A tool to index available programmes on the BBC iPlayer / radio-player for offline analysis.

This is a scraper that runs on Morph. Please read the Morph.io documentation.

It can also be run locally.

Set an environment variable called MORPH_API_KEY to your Nitro API key. When initially run on an empty / non-existant database, a full scrape of available programmes is performed. Subsequently, only those programmes broadcast since the last update will be added. Programmes passed their expiry date are removed.

To build a full index again, either delete the database or set the environment variable MORPH_REBUILD to true.

Usage: node scraper.js

Output is an sqlite3 database containing one table, "data":

javascript var fields = [ '#index', 'type', 'name', 'pid', 'available', 'expires', 'episode', 'seriesnum', 'episodenum', 'versions', 'duration', 'desc', 'channel', 'categories', 'thumbnail', 'timeadded', 'guidance', 'web', 'vpids' ];

Also included is a simple command line utility called query.js for performing SQL queries against the remote Morph.io database and returning the rows in JSON format.

Usage: node query {SQL-statement}

Contributors MikeRalphson

Last run failed with status code 255.

Console output of last run

Injecting configuration and compiling...  -----> Node.js app detected  -----> Creating runtime environment   NPM_CONFIG_LOGLEVEL=error  NPM_CONFIG_PRODUCTION=true  NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt  NODE_VERBOSE=false  NODE_ENV=production  NODE_TLS_REJECT_UNAUTHORIZED=0  NODE_MODULES_CACHE=true  -----> Installing binaries  engines.node (package.json): unspecified  engines.npm (package.json): unspecified (use default)   Resolving node version 6.x...  Error: Invalid semantic version "6.x"  -----> Build failed   ! Invalid semver requirement   Node, Yarn, and npm adhere to semver, the semantic versioning convention  popularized by GitHub.   http://semver.org/   However you have specified a version requirement that is not a valid  semantic version.   https://kb.heroku.com/why-is-my-node-js-build-failing-because-of-an-invalid-semver-requirement 

Data

Downloaded 375 times by MikeRalphson jbraceg AccessSQL nvasari lsces imperialdalek nroles TJSoler

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (63.8 MB) Use the API

rows 10 / 5548

#index type name pid available expires episode seriesnum episodenum versions duration desc channel categories thumbnail timeadded guidance web vpids
170108
radio
You and Yours
b00jbrgm
2012-11-17T18:30:28Z
2147472000
default
3420
Consumer news and issues with Julian Worricker.
bbc_radio_four
Factual,Consumer,Magazines & Reviews
1461194199
b00jbpwf
183376
radio
Home Front - Omnibus
b06l3js2
2016-03-23T00:05:24Z
1774926000
25-29 January 1916
45
podcast,default
3407
Pressures begin to tell at the Bevan hospital.
bbc_radio_four_extra
Drama
1461194613
p03c0kym,b06l3jrp
183379
radio
Home Front - Omnibus
b06kvlgq
2016-01-01T00:04:57Z
1767823080
4-8 January 1916
42
podcast,default
3403
The Bevan hospital becomes a welcome alternative to home for Dorothea.
bbc_radio_four
Drama
1461194613
p03dq1gh,b06kvlbf
183380
radio
Home Front - Omnibus
b06kvl04
2015-12-25T00:06:25Z
1767218280
28 December 1915 - 1 January 1916
41
podcast,default
3404
In the week that the home secretary resigns, the Bevan Hospital celebrates the new year.
bbc_radio_four
Drama
1461194613
p03dpwvq,b06kvkz6
183742
radio
Home Front - Omnibus
b06kvjhx
2015-12-08T04:47:43Z
1766613480
21-25 December 1915
40
podcast,podcast,default
3404
Epic drama series set in Great War Britain a hundred years ago this week.
bbc_radio_four
Drama
1461194616
p03bv8dj,p03bgwzb,b06kvjfj
183786
radio
Home Front - Omnibus
b064g17k
2015-12-03T04:02:45Z
1765771200
5-9 October 1915
36
podcast,default
3398
Individual losses and grief increasingly fuel an interest in spiritualism.
bbc_radio_four_extra
Drama
1461194617
p03045n8,b064g162
183790
radio
Home Front - Omnibus
b06492dw
2015-11-30T12:37:33Z
1765252800
7-11 September 1915
32
podcast,default
3403
The population begins to open up to thoughts of the hereafter.
bbc_radio_four_extra
Drama
1461194617
p031kvbx,b06492dl
183792
radio
Home Front - Omnibus
b0612rj1
2015-08-26T23:25:26Z
1756868400
6-10 July 1915
30
podcast,default
3409
Soldiers on leave find home very different to the one they thought they were fighting for.
bbc_radio_four_extra
Drama
1461194617
p02w69jw,b0612qzs
184184
radio
Home Front - Omnibus
b05xqg4p
2015-08-20T23:29:51Z
1756350000
8-12 June 1915
26
podcast,shortened,default
3395
The epic drama series set in Great War Britain a hundred years ago this week.
bbc_radio_four_extra
Drama
1461194682
p02szt44,b05xw7yb,b05xqfcv
184196
radio
Home Front - Omnibus
b054tnzn
2015-05-22T11:10:33Z
1747710000
9-13 March 1915
21
podcast,default
3400
The Battle of Neuve Chapelle draws to a close while the men at Marshalls spoil for a fight
bbc_radio_four_extra
Drama
1461194682
p02qmjsw,b054tn28

Statistics

Average successful run time: 20 minutes

Total run time: 29 days

Total cpu time used: about 20 hours

Total disk space used: 63.9 MB

History

  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • Auto ran revision bb8b7163 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Node.js

Nitro9 / scraper.js