TURBOT
title Banks of the Dominican Republic (do-banks) draft_transformer_failed
description This bot retrieves information about licensed banks in the Dominican Republic
current run state not running
last run single run snapshot draft scrape transformer failed on September 09, 2016 06:53
next run n/a
created by dinotash (Tom Curtis)
last reviewed by peter.evans
State changed to draft_running for run #8807, snapshot draft
dinotash commented over 7 years ago

The bot was pushed; scheduling a draft run

State changed to draft_running for run #8802, snapshot draft
dinotash commented over 7 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented about 8 years ago

Hey Tom,
Thanks for pushing a fix to this bot, looks like this one also ran into an error while it was doing its draft run, here's the traceback:
File "licence.py", line 59
'type': "Person"
^
SyntaxError: invpalid syntax
Hopefully something simpler than the dire use of HTML by the data source for the previous one!
Thanks
Peter

Re: (turbot bot [do-banks])
dinotash commented over 8 years ago

I think the “Y” licence is something which changed on the site between scraping and reviewing. I just re-ran the bot locally and it got the right result.
Just pushed a fix which should take care of the phone/fax issue. The problem was that the site didn’t close all of its tags properly, so the bot went wrong when trying to work through them. I changed the way it iterates through so it seems to be working now.

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented over 8 years ago

Hey Tom,
Thank you for pushing this bot - I'm reviewing it currently, a couple of things to note:
In the example of "AGENTE DE CAMBIO PRIMIUM, S. A." (http://www.sb.gob.do/entidades/956) for licence_holder we have:
{"entity_properties"=>{"mailing_address"=>"PLAZA LAS AMERICAS, 2DO. NIVEL, LOCAL 9-B, AV. WINSTON CHURCHILL,", "name"=>"AGENTE DE CAMBIO PRIMIUM, S. A.", "fax_number"=>"8095493022"}, "entity_type"=>"unknown"}
From the source it looks as if what we have recorded as the fax_number is a telephone number & we have not recoreded the fax. I expect this varies across different records based on the availability of the telephone number?
There seem to be some licence numbers not getting captured properly, for example we seem to be capturing the licence_number for http://www.sb.gob.do/entidades/735 as just "Y" whereas on the website it is currently "Z1-004-1-00-0101" (could this just have been updated since the bot last ran?)
Otherwise the bot is looking good!
Thanks
Peter

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
peter.evans commented over 8 years ago

A failed draft run was restarted by the moderator

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape transformer failed on August 29, 2015 11:22 317 rows in 6 minutes
single run snapshot draft scrape transformer failed on August 29, 2015 11:31 317 rows in 6 minutes
single run snapshot draft scrape transformer failed on August 29, 2015 12:08 317 rows in 7 minutes
single run snapshot draft scrape transformer failed on September 03, 2015 12:12 317 rows in 7 minutes
single run snapshot draft scrape transformer failed on September 03, 2015 20:42 317 rows in 6 minutes
single run snapshot draft scrape transformer failed on September 03, 2015 20:58 317 rows in 7 minutes
single run snapshot draft scrape transformer failed on September 03, 2015 21:11 317 rows in 6 minutes
single run snapshot draft scrape succeeded on September 03, 2015 21:28 317 rows in 7 minutes
single run snapshot draft scrape transformer failed on January 18, 2016 14:52 318 rows in 5 minutes
single run snapshot draft scrape transformer failed on September 09, 2016 05:09 322 rows in 5 minutes
single run snapshot draft scrape transformer failed on September 09, 2016 06:53 320 rows in 5 minutes

Config

{
  "bot_id": "do-banks",
  "title": "Banks of the Dominican Republic",
  "description": "This bot retrieves information about licensed banks in the Dominican Republic",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "Nombre"
  ],
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Superintendencia de Bancos de la República Dominicana",
    "url": "http://www.sb.gob.do",
    "terms": "Copyright, all rights reserved",
    "terms_url": "http://www.sb.gob.do"
  },
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "licence",
      "identifying_fields": [
        "licence_holder.entity_properties.name"
      ]
    }
  ]
}