trazyn / scraper

Scrapes soundcloud.com, en.wikipedia.org, and www.shropshire.gov.uk

SoundCloud is an audio platform that lets you listen to what you love and share the sounds you create.


Contributors trazyn

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Node.js app detected [1G [1G-----> Creating runtime environment [1G [1G NPM_CONFIG_LOGLEVEL=error [1G NPM_CONFIG_PRODUCTION=true [1G NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt [1G NODE_ENV=production [1G NODE_TLS_REJECT_UNAUTHORIZED=0 [1G NODE_MODULES_CACHE=true [1G [1G-----> Installing binaries [1G engines.node (package.json): unspecified [1G engines.npm (package.json): unspecified (use default) [1G [1G Resolving node version (latest stable) via semver.io... [1G Downloading and installing node 7.5.0... [1G Using default npm version: 4.1.2 [1G [1G-----> Restoring cache [1G Skipping cache restore (new runtime signature) [1G [1G-----> Building dependencies [1G Installing node modules (package.json) [1G [1G > sqlite3@3.1.8 install /tmp/build/node_modules/sqlite3 [1G > node-pre-gyp install --fallback-to-build [1G [1G [sqlite3] Success: "/tmp/build/node_modules/sqlite3/lib/binding/node-v51-linux-x64/node_sqlite3.node" is installed via remote [1G /tmp/build [1G +-- request@2.79.0 [1G | +-- aws-sign2@0.6.0 [1G | +-- aws4@1.5.0 [1G | +-- caseless@0.11.0 [1G | +-- combined-stream@1.0.5 [1G | | `-- delayed-stream@1.0.0 [1G | +-- extend@3.0.0 [1G | +-- forever-agent@0.6.1 [1G | +-- form-data@2.1.2 [1G | | `-- asynckit@0.4.0 [1G | +-- har-validator@2.0.6 [1G | | +-- chalk@1.1.3 [1G | | | +-- ansi-styles@2.2.1 [1G | | | +-- escape-string-regexp@1.0.5 [1G | | | +-- has-ansi@2.0.0 [1G | | | | `-- ansi-regex@2.1.1 [1G | | | +-- strip-ansi@3.0.1 [1G | | | `-- supports-color@2.0.0 [1G | | +-- commander@2.9.0 [1G | | | `-- graceful-readlink@1.0.1 [1G | | +-- is-my-json-valid@2.15.0 [1G | | | +-- generate-function@2.0.0 [1G | | | +-- generate-object-property@1.2.0 [1G | | | | `-- is-property@1.0.2 [1G | | | +-- jsonpointer@4.0.1 [1G | | | `-- xtend@4.0.1 [1G | | `-- pinkie-promise@2.0.1 [1G | | `-- pinkie@2.0.4 [1G | +-- hawk@3.1.3 [1G | | +-- boom@2.10.1 [1G | | +-- cryptiles@2.0.5 [1G | | +-- hoek@2.16.3 [1G | | `-- sntp@1.0.9 [1G | +-- http-signature@1.1.1 [1G | | +-- assert-plus@0.2.0 [1G | | +-- jsprim@1.3.1 [1G | | | +-- extsprintf@1.0.2 [1G | | | +-- json-schema@0.2.3 [1G | | | `-- verror@1.3.6 [1G | | `-- sshpk@1.10.2 [1G | | +-- asn1@0.2.3 [1G | | +-- assert-plus@1.0.0 [1G | | +-- bcrypt-pbkdf@1.0.1 [1G | | +-- dashdash@1.14.1 [1G | | | `-- assert-plus@1.0.0 [1G | | +-- ecc-jsbn@0.1.1 [1G | | +-- getpass@0.1.6 [1G | | | `-- assert-plus@1.0.0 [1G | | +-- jodid25519@1.0.2 [1G | | +-- jsbn@0.1.0 [1G | | `-- tweetnacl@0.14.5 [1G | +-- is-typedarray@1.0.0 [1G | +-- isstream@0.1.2 [1G | +-- json-stringify-safe@5.0.1 [1G | +-- mime-types@2.1.14 [1G | | `-- mime-db@1.26.0 [1G | +-- oauth-sign@0.8.2 [1G | +-- qs@6.3.0 [1G | +-- stringstream@0.0.5 [1G | +-- tough-cookie@2.3.2 [1G | | `-- punycode@1.4.1 [1G | +-- tunnel-agent@0.4.3 [1G | `-- uuid@3.0.1 [1G `-- sqlite3@3.1.8 [1G +-- nan@2.4.0 [1G `-- node-pre-gyp@0.6.31 [1G +-- mkdirp@0.5.1 [1G | `-- minimist@0.0.8 [1G +-- nopt@3.0.6 [1G | `-- abbrev@1.0.9 [1G +-- npmlog@4.0.0 [1G | +-- are-we-there-yet@1.1.2 [1G | | +-- delegates@1.0.0 [1G | | `-- readable-stream@2.1.5 [1G | | +-- buffer-shims@1.0.0 [1G | | +-- core-util-is@1.0.2 [1G | | +-- inherits@2.0.3 [1G | | +-- isarray@1.0.0 [1G | | +-- process-nextick-args@1.0.7 [1G | | +-- string_decoder@0.10.31 [1G | | `-- util-deprecate@1.0.2 [1G | +-- console-control-strings@1.1.0 [1G | +-- gauge@2.6.0 [1G | | +-- aproba@1.0.4 [1G | | +-- has-color@0.1.7 [1G | | +-- has-unicode@2.0.1 [1G | | +-- object-assign@4.1.0 [1G | | +-- signal-exit@3.0.1 [1G | | +-- string-width@1.0.2 [1G | | | +-- code-point-at@1.0.1 [1G | | | | `-- number-is-nan@1.0.1 [1G | | | `-- is-fullwidth-code-point@1.0.0 [1G | | | `-- number-is-nan@1.0.1 [1G | | +-- strip-ansi@3.0.1 [1G | | | `-- ansi-regex@2.0.0 [1G | | `-- wide-align@1.1.0 [1G | `-- set-blocking@2.0.0 [1G +-- rc@1.1.6 [1G | +-- deep-extend@0.4.1 [1G | +-- ini@1.3.4 [1G | +-- minimist@1.2.0 [1G | `-- strip-json-comments@1.0.4 [1G +-- request@2.76.0 [1G | +-- aws-sign2@0.6.0 [1G | +-- aws4@1.5.0 [1G | +-- caseless@0.11.0 [1G | +-- combined-stream@1.0.5 [1G | | `-- delayed-stream@1.0.0 [1G | +-- extend@3.0.0 [1G | +-- forever-agent@0.6.1 [1G | +-- form-data@2.1.1 [1G | | `-- asynckit@0.4.0 [1G | +-- har-validator@2.0.6 [1G | | +-- chalk@1.1.3 [1G | | | +-- ansi-styles@2.2.1 [1G | | | +-- escape-string-regexp@1.0.5 [1G | | | +-- has-ansi@2.0.0 [1G | | | | `-- ansi-regex@2.0.0 [1G | | | +-- strip-ansi@3.0.1 [1G | | | | `-- ansi-regex@2.0.0 [1G | | | `-- supports-color@2.0.0 [1G | | +-- commander@2.9.0 [1G | | | `-- graceful-readlink@1.0.1 [1G | | +-- is-my-json-valid@2.15.0 [1G | | | +-- generate-function@2.0.0 [1G | | | +-- generate-object-property@1.2.0 [1G | | | | `-- is-property@1.0.2 [1G | | | +-- jsonpointer@4.0.0 [1G | | | `-- xtend@4.0.1 [1G | | `-- pinkie-promise@2.0.1 [1G | | `-- pinkie@2.0.4 [1G | +-- hawk@3.1.3 [1G | | +-- boom@2.10.1 [1G | | +-- cryptiles@2.0.5 [1G | | +-- hoek@2.16.3 [1G | | `-- sntp@1.0.9 [1G | +-- http-signature@1.1.1 [1G | | +-- assert-plus@0.2.0 [1G | | +-- jsprim@1.3.1 [1G | | | +-- extsprintf@1.0.2 [1G | | | +-- json-schema@0.2.3 [1G | | | `-- verror@1.3.6 [1G | | `-- sshpk@1.10.1 [1G | | +-- asn1@0.2.3 [1G | | +-- assert-plus@1.0.0 [1G | | +-- bcrypt-pbkdf@1.0.0 [1G | | +-- dashdash@1.14.0 [1G | | +-- ecc-jsbn@0.1.1 [1G | | +-- getpass@0.1.6 [1G | | +-- jodid25519@1.0.2 [1G | | +-- jsbn@0.1.0 [1G | | `-- tweetnacl@0.14.3 [1G | +-- is-typedarray@1.0.0 [1G | +-- isstream@0.1.2 [1G | +-- json-stringify-safe@5.0.1 [1G | +-- mime-types@2.1.12 [1G | | `-- mime-db@1.24.0 [1G | +-- node-uuid@1.4.7 [1G | +-- oauth-sign@0.8.2 [1G | +-- qs@6.3.0 [1G | +-- stringstream@0.0.5 [1G | +-- tough-cookie@2.3.2 [1G | | `-- punycode@1.4.1 [1G | `-- tunnel-agent@0.4.3 [1G +-- rimraf@2.5.4 [1G | `-- glob@7.1.1 [1G | +-- fs.realpath@1.0.0 [1G | +-- inflight@1.0.6 [1G | | `-- wrappy@1.0.2 [1G | +-- inherits@2.0.3 [1G | +-- minimatch@3.0.3 [1G | | `-- brace-expansion@1.1.6 [1G | | +-- balanced-match@0.4.2 [1G | | `-- concat-map@0.0.1 [1G | +-- once@1.4.0 [1G | | `-- wrappy@1.0.2 [1G | `-- path-is-absolute@1.0.1 [1G +-- semver@5.3.0 [1G +-- tar@2.2.1 [1G | +-- block-stream@0.0.9 [1G | +-- fstream@1.0.10 [1G | | `-- graceful-fs@4.1.9 [1G | `-- inherits@2.0.3 [1G `-- tar-pack@3.3.0 [1G +-- debug@2.2.0 [1G | `-- ms@0.7.1 [1G +-- fstream@1.0.10 [1G | +-- graceful-fs@4.1.9 [1G | `-- inherits@2.0.3 [1G +-- fstream-ignore@1.0.5 [1G | +-- inherits@2.0.3 [1G | `-- minimatch@3.0.3 [1G | `-- brace-expansion@1.1.6 [1G | +-- balanced-match@0.4.2 [1G | `-- concat-map@0.0.1 [1G +-- once@1.3.3 [1G | `-- wrappy@1.0.2 [1G +-- readable-stream@2.1.5 [1G | +-- buffer-shims@1.0.0 [1G | +-- core-util-is@1.0.2 [1G | +-- inherits@2.0.3 [1G | +-- isarray@1.0.0 [1G | +-- process-nextick-args@1.0.7 [1G | +-- string_decoder@0.10.31 [1G | `-- util-deprecate@1.0.2 [1G `-- uid-number@0.0.6 [1G [1G [1G-----> Caching build [1G Clearing previous node cache [1G Saving 2 cacheDirectories (default): [1G - node_modules [1G - bower_components (nothing to cache) [1G [1G-----> Build succeeded! [1G +-- request@2.79.0 [1G `-- sqlite3@3.1.8 [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... 1: https://i1.sndcdn.com/visuals-000211846129-GwVlC8-original.jpg

Data

Downloaded 2 times by trazyn

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (8 KB) Use the API

rows 1 / 1

Statistics

Average successful run time: half a minute

Total run time: 1 minute

Total cpu time used: less than 5 seconds

Total disk space used: 9.03 MB

History

  • Manually ran revision 222ee8bc and completed successfully .
    2 records removed, 1 record updated in the database
    93 pages scraped
  • Manually ran revision f8558635 and completed successfully .
    3 records added in the database
    103 pages scraped
  • Created on morph.io

Scraper code

scraper