UK Government Business Plan data retrieval & analytics

Note that as of November 2015, the government API is no longer maintained and all the resources related to the UK Government Business Plans have been archived. This is a result of the government's decision to build a new system for managing departmental priorities after the 2015 election.

As a result, this tool is obsolete and does not update but the historical data can still be used.

The code in this repository collects and analyses data on the UK Government's progress against its own Structural Reform Plan actions. This was used in several Whitehall Monitor bulletins and Whitehall Monitor 2013 and 2014 annual reports published by the Institute for Government.

Latest data and analysis from the Institute for Government using this dataset: http://www.instituteforgovernment.org.uk/performance

Code by @petrbouchal.

This repository consists of

1. a script that collects data from the UK government's Business Plans API and makes basic calculations, e.g. on overdue and missed actions

2. a script that takes the resulting data and produces overall and departmental analytics reports

Currently, only the basic data collection and per-subaction analytics are done.

Where the data come from

• API base URL: http://transparency.number10.gov.uk/api/
• API documentation: http://transparency.number10.gov.uk/developers
• Number 10 front end: http://transparency.number10.gov.uk/

The data comes in JSON format and is nested as follows:

Department Priority Action Subaction Only subactions consistently have information attached to them on dates, comments, etc.

What it does

It loops through departments, priorities, actions and subactions, then calculates time overdue etc. The key variable that it produces is subaction_[start|end]_status.

The code is hosted on Morph. It is scheduled to run every day but programmed to only scrape & save data every week.

Codebook:

Columns used for analysis in bold.

| Column name | Description | Unprocessed data | | --------------- | ------------ | -------- | | dept_abb | Department abbreviation | N | deptname | Department name | Y | deptid | Department ID in API | Y | depturl | Department URL | Y | prioritybody | Name of priority | Y | priorityid | Priority ID | Y | prioritystrapline | Priority strapline. May contain pointless HTML | Y | actionid | Action ID | Y | actionbody | Name of action | Y | actionnotes | Notes on action | Y | schedulestartdate | Action scheduled start date. Ignore. | N | scheduleenddate | Action scheduled end date. Ignore. | N | actualstartdate | Action actual start date. Ignore. | N | actualenddate | Action actual end date. Ignore. | N | **subactionid** | Subaction ID. | N | subaction_body | Name of subaction. | N | subaction_notes | Typically contains explanations of missed due dates etc. | N | subactionschedulestart_date | Schedule start date for subaction. | N | subactionscheduleend_date | Scheduled end date for subaction | N | actstart | Action actual start date. Ignore. | N | actend | Action actual end date. Ignore. | N | schedstartendmonth | Date at end of month in which subaction is due to start. Created to introduce some leeway into overdue calculations. | N | schedendendmonth | Date at end of month in which subaction is due to end. Created to introduce some leeway into overdue calculations. | N | started | Did subaction start? | N | ended | Did subaction end? | N | start_status | Start status of subaction. | N | end_status | End status of subaction. | N | startearlyby | How long ahead of schedule did subaction start? Treat with caution/ignore. | N | startedlateby | How long behind schedule did subaction start? Treat with caution/ignore. | N | startoverdueby | Treat with caution/ignore. | N | endearlyby | How long ahead of schedule did subaction end? Treat with caution/ignore. | N | endedlateby | How much overdue did subaction end? Treat with caution/ignore. | N | endedoverdueby | Treat with caution/ignore. | N | carriedover | If no start date, start date will be 'Carried over' and this column will=1 | N | subactionschedulestartorig | Unprocessed scheduled start date of subaction. | Y | subactionscheduleendorig | Unprocessed scheduled end date of subaction. | Y | subactionactualstartorig | Unprocessed actual end date of subaction. | Y | subactionactualendorig | Unprocessed actual end date of subaction. | Y | datetime | Date and time of data collection. Use this, not date - to get entries from one run of the scraper rather than whole day | N | date | Date of data collection. | N

How to get the data out

The data can be downloaded from Morph as CSV or SQLite in one piece or as a subset in JSON, XML or CSV via the Morph API, which requires writing a bit of SQL.

To get data out of Morph, you need to login using a Github account.

Caveats

This is a very suboptimally structured piece of code - but it works.

Ideally, it would not loop around every action, but first collect the data (with the option of saving it as a raw text file), then save it into a table, and then, optionally, do some analytics on the whole table at once, ideally saving into another table to preserve the distinction between data collection and data processing.