Brandhunt / Brandhunt_produpdate_scraper_phantomjs_py_module_2_v2

Collects product information to be updated from product URLs stored at Brandhunt.se - Only products that require usage of headless browser! Module scraper 2 for centrailzed main scraper


This is a scraper that runs on Morph. To get started see the documentation

Contributors Brandhunt

Last run failed with status code 1.

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Traceback (most recent call last): File "scraper.py", line 33, in <module> exec('mainfunc(' + str(max_prods) + ')', helper.__dict__) File "<string>", line 1, in <module> File "<string>", line 207, in mainfunc File "/app/.heroku/python/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/app/.heroku/python/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/app/.heroku/python/lib/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Data

Downloaded 68009 times by Brandhunt

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (4.73 MB) Use the API

rows 10 / 1572

productid url domain price salesprice domainmisc prodlogurls prodlogurl finalimgurls validimgurls imgurls notfound notavailable removeon404 soldoutfix soldouthtmlfix catstoaddresult attributes sizetypemapsqls
27216
pauw.com
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 918, "name": "Pauw", "slug": "brand-pauw", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 2, "visible": 1, "variation": 1}]
["", "", "", ""]
27060
pauw.com
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 918, "name": "Pauw", "slug": "brand-pauw", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 2, "visible": 1, "variation": 1}]
["", "", "", ""]
27276
pauw.com
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 918, "name": "Pauw", "slug": "brand-pauw", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 2, "visible": 1, "variation": 1}]
["", "", "", ""]
27166
pauw.com
1
""
""
""
""
""
true
false
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 918, "name": "Pauw", "slug": "brand-pauw", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false]], "position": 2, "visible": 1, "variation": 1}]
["", "", "", ""]
26046
www.stenstromsstore.com
1
""
""
""
""
""
true
true
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 899, "name": "Stenstr\u00f6ms", "slug": "brand-stenstr%c3%b6ms", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false], [{"term_id": 142, "name": "Female", "slug": "female", "taxonomy": "pa_sex"}, false]], "position": 2, "visible": 1, "variation": 1}]
["", "", "", ""]
26129
www.stenstromsstore.com
1
""
""
""
""
""
true
true
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 899, "name": "Stenstr\u00f6ms", "slug": "brand-stenstr%c3%b6ms", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 115, "name": "White", "slug": "color-white", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false], [{"term_id": 142, "name": "Female", "slug": "female", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
26169
www.stenstromsstore.com
1
""
""
""
""
""
true
true
true
false
false
[[{"term_id": 43, "name": "Jeans", "slug": "jeans", "taxonomy": "product_cat", "ancestors": []}, false], [{"term_id": 94, "name": "Jeans(Trousers)", "slug": "jeans-trousers", "taxonomy": "product_cat", "ancestors": [92, 73]}, false], [{"term_id": 92, "name": "Trousers", "slug": "trousers", "taxonomy": "product_cat", "ancestors": [73]}, false], [{"term_id": 73, "name": "Trousers & Chinos", "slug": "trousers-chinos", "taxonomy": "product_cat", "ancestors": []}, false]]
[{"name": "Brand", "options": [[{"term_id": 899, "name": "Stenstr\u00f6ms", "slug": "brand-stenstr%c3%b6ms", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 119, "name": "Orange", "slug": "color-orange", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false], [{"term_id": 142, "name": "Female", "slug": "female", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}, {"name": "Sizetype", "options": [[{"term_id": 2021, "name": "Bottoms", "slug": "sizetype-bottoms", "taxonomy": "pa_sizetype"}, false]], "position": 4, "visible": 1, "variation": 1}]
["", "", "", ""]
26213
www.stenstromsstore.com
1
""
""
""
""
""
true
true
true
false
false
""
[{"name": "Brand", "options": [[{"term_id": 899, "name": "Stenstr\u00f6ms", "slug": "brand-stenstr%c3%b6ms", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 475, "name": "Green", "slug": "color-green", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false], [{"term_id": 142, "name": "Female", "slug": "female", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}]
["", "", "", ""]
26242
www.stenstromsstore.com
1
""
""
""
""
""
true
true
true
false
false
[[{"term_id": 77, "name": "Underwear", "slug": "underwear", "taxonomy": "product_cat", "ancestors": []}, false]]
[{"name": "Brand", "options": [[{"term_id": 899, "name": "Stenstr\u00f6ms", "slug": "brand-stenstr%c3%b6ms", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 117, "name": "Black", "slug": "color-black", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false], [{"term_id": 142, "name": "Female", "slug": "female", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}, {"name": "Sizetype", "options": [[{"term_id": 2578, "name": "Underwear/Swimwear", "slug": "sizetype-underwearswimwear", "taxonomy": "pa_sizetype"}, false]], "position": 4, "visible": 1, "variation": 1}]
["", "", "", ""]
26388
www.stenstromsstore.com
1
""
""
""
""
""
true
true
true
false
false
[[{"term_id": 61, "name": "Socks", "slug": "socks", "taxonomy": "product_cat", "ancestors": [2584]}, false], [{"term_id": 2584, "name": "Footwear", "slug": "footwear", "taxonomy": "product_cat", "ancestors": []}, false]]
[{"name": "Brand", "options": [[{"term_id": 899, "name": "Stenstr\u00f6ms", "slug": "brand-stenstr%c3%b6ms", "taxonomy": "pa_brand"}, false]], "position": 1, "visible": 1, "variation": 1}, {"name": "Color", "options": [[{"term_id": 493, "name": "Dark Grey", "slug": "color-dark-grey", "taxonomy": "pa_color"}, false], [{"term_id": 116, "name": "Grey", "slug": "color-grey", "taxonomy": "pa_color"}, false]], "position": 2, "visible": 1, "variation": 1}, {"name": "Sex", "options": [[{"term_id": 141, "name": "Male", "slug": "male", "taxonomy": "pa_sex"}, false], [{"term_id": 142, "name": "Female", "slug": "female", "taxonomy": "pa_sex"}, false]], "position": 3, "visible": 1, "variation": 1}, {"name": "Sizetype", "options": [[{"term_id": 2022, "name": "Footwear", "slug": "sizetype-footwear", "taxonomy": "pa_sizetype"}, false]], "position": 4, "visible": 1, "variation": 1}]
["", "", "", ""]

Statistics

Average successful run time: about 2 hours

Total run time: about 1 month

Total cpu time used: 4 days

Total disk space used: 4.75 MB

History

  • Auto ran revision b600ca51 and failed .
    nothing changed in the database
  • Auto ran revision b600ca51 and failed .
    nothing changed in the database
  • Auto ran revision b600ca51 and failed .
    nothing changed in the database
  • Auto ran revision b600ca51 and failed .
    nothing changed in the database
  • Auto ran revision b600ca51 and failed .
    nothing changed in the database
  • ...
  • Created on morph.io

Show complete history