title Bosnian Financial Licences (ba-financial-licences) storing_data view rate in graphite
description This bot scrapes information about licensed financial market participants from the Securities Commission of Bosnia and Herzegovina
current run state storing may have failed
last run single run snapshot 6 scrape succeeded on April 11, 2016 13:04
next run enqueued for a run, docker status
created by dinotash (Tom Curtis)
last reviewed by peter.evans
Bot state update
commented almost 8 years ago

A run finished; its output is now being processed

Bot state update
commented almost 8 years ago

A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

The bot was accepted; starting run to ingest reviewed data

Bot update
commented over 8 years ago

A manual prescrape using data from /oc/openc/scrapers/output/draft/b/ba-financial-licences/5572 was started

Bot state update
commented over 8 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented over 8 years ago

Hey Tom,
Yep you're right, it's only a small subset, it's getting hard to find issues with the data your bots produce now though & I have to have something to do right?!
I think we're discovering as we go - if there are lots of permissions which look like they are part of 1 licence then I'd say that can be one record, but distinct start_dates definitely indicate distinct licences, if those start dates weren't there though it would be much fuzzier, we'd still get the data either way, this way we just get the start_dates (& I suppose more importantly dont incorrectly assign any).
Cheers!
Peter

Re: (turbot bot [ba-financial-licences])
dinotash commented over 8 years ago

Hi Peter
That makes a lot of sense. I forget sometimes that it's one record per licence, not one per entity.
I don't think it's going to affect too many records though. Only one of the four categories included this level of information on licences and start dates.
Cheers
Tom

(no subject)
peter.evans commented over 8 years ago

Hey Tom,
Reviewing this bot now. The fixup of the ro- bot looks fine, thank you, just dealing with some issues at this end on that one & we can get it up and running again.
So on to this Bosnia bot:
It looks as if some of the start dates on the licences are not identical, so setting the start date at the root level (ie for all permissions) isn't quite right I don't think. For example on this page http://www.komvp.gov.ba/site/index.php/en/investment-funds/depository-banks/299-sparcasse-d-d-sarajevo#license we have 2 start dates for different licences and are setting a start_date of: "start_date": "2005-12-01",
Is that a correct reading of the situation? If so I would actually suggest splitting these different licences off into separate records, specifically 1 record for each actual licence, possibly concatenating the general licence category so that they all make sense. So for this example instead of these 3 permissions:
{
"activity_name": "Depository and custody bank",
"permission_type": "operating"
},
{
"activity_name": "Depository Operations",
"permission_type": "operating"
},
{
"activity_name": "Custody Operations",
"permission_type": "operating"
}
I would split this into 2 distinct records, one of them having:
{
"activity_name": "Depository and custody bank - Custody Operations",
"permission_type": "operating"
}
and the other having:
{
"activity_name": "Depository and custody bank - Depository Operations",
"permission_type": "operating"
},
We could then of course also set the correct start_date for each licence at the root level. What do you think?
Otherwise it's looking great!
Thanks as always,
Peter

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A draft run failed

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape failed on November 08, 2015 22:01 29 rows in 1 minute
single run snapshot draft scrape succeeded on November 08, 2015 22:04 55 rows in 1 minute
single run snapshot draft scrape succeeded on November 09, 2015 07:15 55 rows in 1 minute
single run snapshot draft scrape succeeded on November 09, 2015 21:37 55 rows in 1 minute
single run snapshot final draft scrape succeeded on November 10, 2015 13:45 55 rows in 1 minute
single run snapshot 1 prescrape scrape succeeded on November 10, 2015 13:47 55 rows in less than a minute
single run snapshot 2 scrape succeeded on December 10, 2015 13:48 55 rows in 1 minute
single run snapshot 3 scrape succeeded on January 10, 2016 13:48 55 rows in 1 minute
single run snapshot 4 scrape succeeded on February 10, 2016 13:48 55 rows in 1 minute
single run snapshot 5 scrape succeeded on March 10, 2016 13:48 55 rows in 1 minute
single run snapshot 6 scrape succeeded on April 11, 2016 13:04 55 rows in 5 minutes

Config

{
  "bot_id": "ba-financial-licences",
  "title": "Bosnian Financial Licences",
  "description": "This bot scrapes information about licensed financial market participants from the Securities Commission of Bosnia and Herzegovina",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "name"
  ],
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Securities Commission of Federation of Bosnia and Herzegovina",
    "url": "http://www.komvp.gov.ba/site/index.php/en/",
    "terms": "Copyright",
    "terms_url": "http://www.komvp.gov.ba/site/index.php/en/"
  },
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "licence",
      "identifying_fields": [
        "licence_holder.entity_properties.name",
        "permissions"
      ]
    }
  ]
}