ar73m / dou-scraper

Scrapes dou.ua

Сообщество программистов | DOU

Contributors Yegorov

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... -----> Node.js app detected  -----> Reading application state  package.json...  build directory...  cache directory...  environment variables...   Node engine: unspecified  Npm engine: unspecified  Start mechanism: Procfile  node_modules source: package.json  node_modules cached: false   NPM_CONFIG_PRODUCTION=true  NPM_CONFIG_CAFILE=/etc/ssl/certs/ca-certificates.crt  NODE_MODULES_CACHE=true  -----> Installing binaries  Resolving node version (latest stable) via semver.io...  Downloading and installing node 5.4.0...  Using default npm version: 3.3.12  -----> Building dependencies  No cache available  Installing node modules  dou-scraper@0.0.1 /tmp/build  `-- extractor@0.1.2  `-- jsdom@7.2.2  +-- abab@1.0.3  +-- acorn@2.7.0  +-- acorn-globals@1.0.9  +-- cssom@0.3.0  +-- cssstyle@0.2.30  +-- escodegen@1.8.0  | +-- esprima@2.7.1  | +-- estraverse@1.9.3  | +-- esutils@2.0.2  | +-- optionator@0.8.1  | | +-- deep-is@0.1.3  | | +-- fast-levenshtein@1.1.0  | | +-- levn@0.3.0  | | +-- prelude-ls@1.1.2  | | +-- type-check@0.3.2  | | `-- wordwrap@1.0.0  | `-- source-map@0.2.0  | `-- amdefine@1.0.0  +-- nwmatcher@1.3.7  +-- parse5@1.5.1  +-- request@2.67.0  | +-- aws-sign2@0.6.0  | +-- bl@1.0.0  | | `-- readable-stream@2.0.5  | | +-- core-util-is@1.0.2  | | +-- inherits@2.0.1  | | +-- isarray@0.0.1  | | +-- process-nextick-args@1.0.6  | | +-- string_decoder@0.10.31  | | `-- util-deprecate@1.0.2  | +-- caseless@0.11.0  | +-- combined-stream@1.0.5  | | `-- delayed-stream@1.0.0  | +-- extend@3.0.0  | +-- forever-agent@0.6.1  | +-- form-data@1.0.0-rc3  | | `-- async@1.5.2  | +-- har-validator@2.0.5  | | +-- chalk@1.1.1  | | | +-- ansi-styles@2.1.0  | | | +-- escape-string-regexp@1.0.4  | | | +-- has-ansi@2.0.0  | | | | `-- ansi-regex@2.0.0  | | | +-- strip-ansi@3.0.0  | | | `-- supports-color@2.0.0  | | +-- commander@2.9.0  | | | `-- graceful-readlink@1.0.1  | | +-- is-my-json-valid@2.12.4  | | | +-- generate-function@2.0.0  | | | +-- generate-object-property@1.2.0  | | | | `-- is-property@1.0.2  | | | +-- jsonpointer@2.0.0  | | | `-- xtend@4.0.1  | | `-- pinkie-promise@2.0.0  | | `-- pinkie@2.0.1  | +-- hawk@3.1.2  | | +-- boom@2.10.1  | | +-- cryptiles@2.0.5  | | +-- hoek@2.16.3  | | `-- sntp@1.0.9  | +-- http-signature@1.1.0  | | +-- assert-plus@0.1.5  | | +-- jsprim@1.2.2  | | | +-- extsprintf@1.0.2  | | | +-- json-schema@0.2.2  | | | `-- verror@1.3.6  | | `-- sshpk@1.7.3  | | +-- asn1@0.2.3  | | +-- assert-plus@0.2.0  | | +-- dashdash@1.12.1  | | +-- ecc-jsbn@0.1.1  | | +-- jodid25519@1.0.2  | | +-- jsbn@0.1.0  | | `-- tweetnacl@0.13.3  | +-- is-typedarray@1.0.0  | +-- isstream@0.1.2  | +-- json-stringify-safe@5.0.1  | +-- mime-types@2.1.9  | | `-- mime-db@1.21.0  | +-- node-uuid@1.4.7  | +-- oauth-sign@0.8.0  | +-- qs@5.2.0  | +-- stringstream@0.0.5  | `-- tunnel-agent@0.4.2  +-- sax@1.1.4  +-- symbol-tree@3.1.4  +-- tough-cookie@2.2.1  +-- webidl-conversions@2.0.1  +-- whatwg-url-compat@0.6.5  | `-- tr46@0.0.2  `-- xml-name-validator@2.0.1   -----> Checking startup method  Found Procfile  -----> Finalizing build  Creating runtime environment  Exporting binary paths  Cleaning npm artifacts  Cleaning previous cache  Caching results for future builds  -----> Build succeeded!   dou-scraper@0.0.1 /tmp/build  `-- extractor@0.1.2   WARNING: Node version not specified in package.json  https://devcenter.heroku.com/articles/nodejs-support#specifying-a-node-js-version  -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running... Processed http://dou.ua/ Last modified undefined Page record: {"page_title":[{"innerHTML":"Сообщество программистов | DOU","text":"Сообщество программистов | DOU"}],"li":[{"class":"link","href":"http://dou.ua/lenta/articles/docker/","innerHTML":"Виртуализация процесса разработки, часть 1: Docker"},{"class":"g-comments-round","href":"http://dou.ua/lenta/articles/docker/#comments","innerHTML":"30"},{"class":"link","href":"http://dou.ua/lenta/digests/python-digest-5/","innerHTML":"Python дайджест #5: “We will be moving to GitHub”"},{"class":"g-comments-round","href":"http://dou.ua/lenta/digests/python-digest-5/#comments","innerHTML":"14"},{"class":"link","href":"http://dou.ua/lenta/articles/nearables-wearables-experiment/","innerHTML":"Пошук людей у великому офісі: як ми подружили бікони з розумними годинниками"},{"class":"g-comments-round","href":"http://dou.ua/lenta/articles/nearables-wearables-experiment/#comments","innerHTML":"15"},{"class":"link","href":"http://dou.ua/lenta/articles/dont-move-abroad/","innerHTML":"Как я переехал в Польшу и вернулся назад: несколько причин не эмигрировать за границу"},{"class":"g-comments-round","href":"http://dou.ua/lenta/articles/dont-move-abroad/#comments","innerHTML":"1K"},{"class":"link","href":"http://dou.ua/lenta/interviews/it-career-239/","innerHTML":"Беседа с Викторией Амелиной, ex-it’шницей и писательницей"},{"class":"g-comments-round","href":"http://dou.ua/lenta/interviews/it-career-239/#comments","innerHTML":"25"},{"class":"link","href":"http://dou.ua/lenta/articles/dou-projector-oldstyleracing/","innerHTML":"DOU Проектор: OldStyleRacing — бесплатный timekiller для iOS устройств"},{"class":"g-comments-round","href":"http://dou.ua/lenta/articles/dou-projector-oldstyleracing/#comments","innerHTML":"2"},{"class":"link","href":"http://dou.ua/lenta/digests/java-digest-21/","innerHTML":"Java дайджест #21: язык года 2015"},{"class":"g-comments-round","href":"http://dou.ua/lenta/digests/java-digest-21/#comments","innerHTML":"13"},{"class":"link","href":"http://dou.ua/lenta/digests/qa-digest-14/","innerHTML":"QA дайджест #14: Привет новый год. Статистика-2015, тестовые эвристики, уязвимости Grub Linux, PayPal и Instagram"},{"class":"g-comments-round","href":"http://dou.ua/lenta/digests/qa-digest-14/#comments","innerHTML":"1"},{"class":"link","href":"http://dou.ua/lenta/articles/dou-projector-orna/","innerHTML":"DOU Проектор: Орна — фреймворк та CSS-бібліотека в сукупності"},{"class":"g-comments-round","href":"http://dou.ua/lenta/articles/dou-projector-orna/#comments","innerHTML":"65"}]}

Statistics

Average successful run time: 1 minute

Total run time: 1 minute

Total cpu time used: less than 5 seconds

Total disk space used: 17.8 KB

History

  • Manually ran revision c6002033 and completed successfully .
    nothing changed in the database
    1 page scraped
  • Created on morph.io

Scraper code

Node.js

dou-scraper / scraper.js