Brandhunt / Brandhunt_produpdate_scraper_phantomjs_py_module_1_v2

Collects product information to be updated from product URLs stored at Brandhunt.se - Only products that require usage of headless browser! Module scraper 1 for centrailzed main scraper


This is a scraper that runs on Morph. To get started see the documentation

Contributors Brandhunt

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 33, in <module> exec('mainfunc(' + str(max_prods) + ')', helper.__dict__) File "<string>", line 1, in <module> File "<string>", line 207, in mainfunc File "/app/.heroku/python/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/app/.heroku/python/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/app/.heroku/python/lib/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Data

Downloaded 67058 times by Brandhunt

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.22 MB) Use the API

rows 10 / 1531

productid url domain price salesprice domainmisc prodlogurls prodlogurl finalimgurls validimgurls imgurls notfound notavailable removeon404 soldoutfix soldouthtmlfix catstoaddresult attributes sizetypemapsqls
14957
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
[[{"term_id": 56, "name": "Shirts", "slug": "shirts", "taxonomy": "product_cat", "ancestors": []}, false]]
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 115, "name": "White", "slug": "color-white", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}, {"name": "Sizetype", "options": [[{"term_id": 2527, "name": "Shirts/Suits", "slug": "sizetype-shirtssuits", "taxonomy": "pa_sizetype"}, false]], "position": 4, "visible": 1, "variation": 1}]
["", "", "", ""]
15123
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 456, "name": "Burgundy", "slug": "color-burgundy", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15163
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15225
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15227
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 1330, "name": "Army", "slug": "color-army", "taxonomy": "pa_color"}, false], [{"term_id": 501, "name": "Blue/Army/Black", "slug": "color-bluearmyblack", "taxonomy": "pa_color"}, false], [{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false], [{"term_id": 120, "name": "Blue", "slug": "color-blue", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15229
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15231
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15233
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15235
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 115, "name": "White", "slug": "color-white", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
15237
bergbergstore.com
1
1
""
""
""
""
""
true
false
true
false
false
[[{"term_id": 32, "name": "Dresses", "slug": "dresses", "taxonomy": "product_cat", "ancestors": []}, false], [{"term_id": 55, "name": "Scarves", "slug": "scarves", "taxonomy": "product_cat", "ancestors": [17]}, false], [{"term_id": 17, "name": "Accessories", "slug": "accessories", "taxonomy": "product_cat", "ancestors": []}, false]]
[{"name": "Brand", "options": [[{"term_id": -1, "name": "Berg & Berg", "slug": "berg-berg", "taxonomy": "pa_brand"}, true]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 1183, "name": "Ash", "slug": "color-ash", "taxonomy": "pa_color"}, false], [{"term_id": 503, "name": "Dress Gordon", "slug": "color-dress-gordon", "taxonomy": "pa_color"}, false], [{"term_id": 682, "name": "Tan", "slug": "color-tan", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}, {"name": "Sizetype", "options": [[{"term_id": 2034, "name": "Dresses", "slug": "sizetype-dresses", "taxonomy": "pa_sizetype"}, false], [{"term_id": 2033, "name": "Accessories", "slug": "sizetype-accessories", "taxonomy": "pa_sizetype"}, false]], "position": 4, "visible": 1, "variation": 1}]
["", "", "", ""]

Statistics

Average successful run time: about 2 hours

Total run time: 3 months

Total cpu time used: 7 days

Total disk space used: 4.24 MB

History

  • Auto ran revision f78cd580 and failed .
    nothing changed in the database
  • Auto ran revision f78cd580 and failed .
    nothing changed in the database
  • Auto ran revision f78cd580 and failed .
    nothing changed in the database
  • Auto ran revision f78cd580 and failed .
    nothing changed in the database
  • Auto ran revision f78cd580 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history