title UK National Storage Mechanism (gb_national_storage_mechanism) draft_failed
description Fetching primary data for notifications about public listed companies in the UK.
current run state not running
last run first run of snapshot draft draft scrape failed on June 25, 2015 16:35
next run n/a
created by xavriley (Xavier Riley)
last reviewed by xavriley
Source gives duplicate records - any way to ignore?
xavriley commented almost 9 years ago

I've fixed up this bot and it is working but it looks like the source has issues with listing the same items twice in some cases. Is there a way to ignore this?
Example
```
[9:04:45] xavierriley:gb_national_storage_mechanism $ ag -Q "368456654547435" output_june_2015.json
342925:{"filed_at":"2015-06-24 17:26","title":"Chemring Group PLC - Holding(s) in Company","filing_id":"368456654547435","url":"http://tools.morningstar.co.uk/tsweu6nqxu/globaldocuments/document/rnsNewsItem.aspx?DocumentId=368456654547435","filing_source":"RNS","company_name":"Schroders PLC","filing_type":"Holding in Company","source_url":"http://tools.morningstar.co.uk/tsweu6nqxu/globaldocuments/list/default.aspx","sample_date":"2015-06-26 09:04:37 +0100"}
342926:{"filed_at":"2015-06-24 17:26","title":"Chemring Group PLC - Holding(s) in Company","filing_id":"368456654547435","url":"http://tools.morningstar.co.uk/tsweu6nqxu/globaldocuments/document/rnsNewsItem.aspx?DocumentId=368456654547435","filing_source":"RNS","company_name":"Chemring Group PLC","filing_type":"Holding in Company","source_url":"http://tools.morningstar.co.uk/tsweu6nqxu/globaldocuments/list/default.aspx","sample_date":"2015-06-26 09:04:37 +0100"}
```

Bot state update
commented almost 9 years ago

A draft run failed

Bot state update
xavriley commented almost 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented almost 9 years ago

A draft run failed

Bot state update
xavriley commented almost 9 years ago

The bot was pushed; scheduling a draft run

(no subject)
phil_smith commented about 9 years ago

I suspect this is failing because the server being scraped is rejecting the requests.

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A run started

Bot state update
commented about 9 years ago

A failed run was resumed by the moderator

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A run started

Bot state update
commented about 9 years ago

An initial run was triggered manually by a moderator

Bot state update
commented over 9 years ago

Automatic bot state update. Was in state scheduled, now in state to_be_scheduled

Run history

event metadata
unknown (1) scrape failed on January 12, 2015 15:22 18712 rows in 5 minutes
incremental run of snapshot 1 scrape failed on January 12, 2015 16:20 176370 rows in about 1 hour
first run of snapshot draft draft scrape failed on June 25, 2015 13:28 0 rows in less than a minute
first run of snapshot draft draft scrape failed on June 25, 2015 16:35 500 rows in 1 minute

Config

{
  "bot_id": "gb_national_storage_mechanism",
  "title": "UK National Storage Mechanism",
  "description": "Fetching primary data for notifications about public listed companies in the UK.",
  "language": "ruby",
  "data_type": "primary data",
  "identifying_fields": [
    "filing_id"
  ],
  "files": [
    "scraper.rb"
  ],
  "publisher": "Morningstar on behalf of the FCA",
  "frequency": "daily",
  "incremental": "true"
}