title Austrian financial licences (at-licences) pending_draft_review
description This bot scrapes information on licensed financial market participants in Austria
current run state not running
last run single run snapshot draft scrape succeeded on March 22, 2016 10:14
next run n/a
created by dinotash (Tom Curtis)
last reviewed by peter.evans
Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented about 8 years ago

A failed final draft was sent by the moderator back to draft

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A final draft run failed

Bot state update
peter.evans commented about 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
peter.evans commented about 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented about 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented about 8 years ago

Hey Tom,
No worries - thanks for identifying the problem :) Let me know if I can help at all.
Cheers
Peter

Re: (turbot bot [at-licences])
dinotash commented about 8 years ago

Hi Peter
Thanks for that. Should be an easy fix and entirely my own fault. It's to do with combining Unicode and non-unicode text when generating a log message. It's just for my diagnostics so I can easily change it.
Cheers
Tom

(no subject)
peter.evans commented about 8 years ago

Hi Tom,
Adjusting the scraper's identifying_fields to match the change to the scraper (permission -> perms) has fixed that issue, if you could update it at your end before you next push to this bot that would be great.
The bot has now run into an encoding error:
Traceback (most recent call last):
File "scraper.py", line 47, in <module>
turbotlib.log("Loading category " + category_index + " / " + category_count + ": " + category['name'])
File "/usr/local/lib/python2.7/dist-packages/turbotlib/__init__.py", line 24, in log
print(message, file=sys.stderr)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xd6' in position 37: ordinal not in range(128)
Is this something that makes sense at your end or shall I pass this on to Peter?
Thanks & Hope you're well,
Peter

Bot state update
commented about 8 years ago

A final draft run failed

Bot state update
peter.evans commented about 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
peter.evans commented about 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented about 8 years ago

A failed final draft was sent by the moderator back to draft

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A final draft run failed

Bot state update
peter.evans commented about 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
peter.evans commented about 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented about 8 years ago

A failed final draft was sent by the moderator back to draft

Saved vars cleared
dinotash commented about 8 years ago

(no subject)
peter.evans commented about 8 years ago

Hi Tom,
Thanks for that! Working getting the bot accepted now. It fell over a few times because the website was being a bit unstable but we seem to have it now.
The identifying_fields for the scraper needs to reflect the change from `permissions` to `perms` when you get a moment - We've already changed this on the server to see if we can get the bot accepted today but you'll need to make that change at your end before you push anything new to this bot, so might be worth doing so the issue doesn't get re-introduced later.
Cheers!
Peter

Bot state update
commented about 8 years ago

A final draft run failed

Bot state update
peter.evans commented about 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
peter.evans commented about 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
peter commented about 8 years ago

A failed draft run was restarted by the moderator

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A draft run failed

Bot state update
peter.evans commented about 8 years ago

A failed final draft was sent by the moderator back to draft

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A final draft run failed

Bot state update
peter.evans commented about 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
peter.evans commented about 8 years ago

A moderator has started reviewing the draft bot

Re: (turbot bot [at-licences])
dinotash commented about 8 years ago

Hi Peter
That seems a quick fix, so I’ve just pushed a version where I’ve done just that. The raw data should now have a field called “perms” instead, and the licence record has one called “permissions”. Hopefully that will work.
I know I’ve got a whole stack of bot review points sitting in my inbox, but they’ll probably have to wait a bit longer. We are so close to the move date that everything is manic. This one was doable because it took two minutes.
Hope you’re well
Tom

Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented about 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented about 8 years ago

Hi Tom,
Hope all is well with you. This bot has been hitting an error on our side, Peter Inglesby has investigated and has this to say:
For primary data, the permissions attribute is an array of strings, while for transformed data, it's an array of hashes. Elasticsearch tries to store them both in the same shape hole, but can't, because they're different shapes.
A proper fix for this is going to take some time, so the only thing I can suggest is that Tom changes the name of the permissions attribute in the primary data (maybe to "perms"?).
---
So I think by the sounds of it just changing the primary data attribute name should do it for now, these can be taken from the data source or just semantically.
Thanks for another great looking scraper.
Best wishes,
Peter

Bot state update
commented about 8 years ago

Run succeeded; sending for draft review

Bot state update
peter commented about 8 years ago

A failed draft run was restarted by the moderator

Saved vars cleared
dinotash commented about 8 years ago

Bot update
commented over 8 years ago

Bot triggered error in framework

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented over 8 years ago

A failed draft run was restarted by the moderator

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape succeeded on November 06, 2015 21:42 2014 rows in 33 minutes
single run snapshot draft scrape succeeded on January 13, 2016 16:49 2002 rows in 3 minutes
single run snapshot draft scrape succeeded on February 26, 2016 10:16 2015 rows in 5 minutes
single run snapshot draft scrape succeeded on February 26, 2016 20:00 2015 rows in 17 minutes
single run snapshot final draft scrape failed on March 02, 2016 16:25 0 rows in less than a minute
single run snapshot draft scrape failed on March 02, 2016 16:28 0 rows in less than a minute
single run snapshot draft scrape succeeded on March 02, 2016 16:41 2015 rows in 4 minutes
single run snapshot final draft scrape failed on March 02, 2016 16:50 4403 rows in 7 minutes
single run snapshot draft scrape succeeded on March 03, 2016 14:05 2015 rows in 10 minutes
single run snapshot final draft scrape failed on March 03, 2016 14:35 4008 rows in 25 minutes
single run snapshot draft scrape succeeded on March 03, 2016 15:11 2015 rows in 9 minutes
single run snapshot final draft scrape failed on March 03, 2016 16:14 7225 rows in 25 minutes
single run snapshot draft scrape succeeded on March 19, 2016 03:17 2012 rows in 4 minutes
single run snapshot final draft scrape failed on March 21, 2016 10:44 3116 rows in 7 minutes
single run snapshot draft scrape succeeded on March 22, 2016 10:14 2015 rows in 5 minutes

Config

{
  "bot_id": "at-licences",
  "title": "Austrian financial licences",
  "description": "This bot scrapes information on licensed financial market participants in Austria",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "name",
    "perms"
  ],
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Austrian Financial Market Authority",
    "url": "https://www.fma.gv.at/en",
    "terms": "The FMA grants permission to distribute and reproduce material on which the FMA holds copyright and which has been downloaded or copied from its website, and to create links to this website, provided the source is acknowledged at all times. Moreover, any person who gains direct or indirect access to material from the FMA’s website must be explicitly informed about the conditions of use. The commercial use of material presented on the FMA’s website requires the FMA’s express permission. Any material from the FMA’s website which the FMA deems to have been used unlawfully or improperly must be deleted or destroyed upon the FMA’s request.",
    "terms_url": "https://www.fma.gv.at/en/footer/imprint.html"
  },
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "licence",
      "identifying_fields": [
        "licence_holder.entity_properties.name",
        "permissions"
      ]
    }
  ]
}