title Azerbaijani Securities Licences (az-licences) scheduled
description This is a simple bot which scrapes data from the State Committee for Securities of the Republic of Azerbaijan relating to securities market participants. Data is scraped in English.
current run state not running
last run single run snapshot 2 scrape failed on December 04, 2015 11:49
next run n/a
created by dinotash (Tom Curtis)
last reviewed by peter.evans
Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

The bot was accepted; starting run to ingest reviewed data

Bot update
commented over 8 years ago

A manual prescrape using data from /oc/openc/scrapers/output/draft/a/az-licences/5500 was started

Bot state update
commented over 8 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented over 8 years ago

Yes that makes sense... The problem is, early on, that as you have all of Turbot to get used to even simple Missions can take a long time to get right - Our long term goal is to cut out that time - we're thinking maybe a codecademy.com type of approach with a worked example that takes you all of the way, that won't happen in the short-term though.
Which bit of code do you tend to copy between bots, could you paste in an example? I'll probably be thinking about how to explain the new licence schema better soon / explain it more easily to people who haven't dealt with data models before.
Anyway I digress, thanks for the reply & I think we'll push this one into OpenC after the adjustment.
Thanks!
Peter

Re: (turbot bot [az-licences])
dinotash commented over 8 years ago

Hi Peter
Yep, that change makes sense.
This one was super quick because the data source is so simple. I tend to copy the top part from previous bots, but the main reason I've sped up is practice.

(no subject)
peter.evans commented over 8 years ago

Hey Tom,
Just checking this scraper now - thanks for submitting yet another, you're getting very fast at writing them now! Are you using a local template or something like this? (I'm always thinking about how we can make it easier for people, thinking about what we can add to the docs etc)
So re the bot... we're currently transforming the addresses into mailing address:
"mailing_address": "Baku, Nasimi district, Lev Tolstoy 170",
I'm pretty sure that this can actually be registered_address, which is the legal address of the company. This is based on the registration instructions including "Application to receive a license. The application should include the following: name, organizational – legal form, legal address, settlement account number and bank name, activity type;" - So from this I think the address they have must be the legal address. What do you think?
Otherwise the bot looks solid, thanks as always for your precise work!
Cheers!
Peter

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape succeeded on November 02, 2015 21:33 37 rows in less than a minute
single run snapshot draft scrape succeeded on November 03, 2015 20:48 37 rows in less than a minute
single run snapshot final draft scrape succeeded on November 04, 2015 11:48 37 rows in less than a minute
single run snapshot 1 prescrape scrape succeeded on November 04, 2015 11:48 37 rows in less than a minute
single run snapshot 2 scrape failed on December 04, 2015 11:49 0 rows in 1 minute

Config

{
  "bot_id": "az-licences",
  "title": "Azerbaijani Securities Licences",
  "description": "This is a simple bot which scrapes data from the State Committee for Securities of the Republic of Azerbaijan relating to securities market participants. Data is scraped in English.",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "name",
    "category"
  ],
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "State Committee for Securities of the Republic of Azerbaijan",
    "url": "http://scs.gov.az",
    "terms": "Copyright, All rights reserved",
    "terms_url": "http://scs.gov.az"
  },
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "licence",
      "identifying_fields": [
        "licence_holder.entity_properties.name",
        "permissions"
      ]
    }
  ]
}