title FINANCE LICENCES (INSURANCE) - GENERAL DIRECTORATE OF INSURANCE (SPAIN) (dgsfp-insurance-licences) pending_draft_review
description List of all finance licences from the GENERAL DIRECTORATE OF INSURANCE (SPAIN)
current run state not running
last run single run snapshot draft scrape succeeded on March 28, 2015 10:04
next run n/a
created by bertocq (berto cq)
last reviewed by peter.evans
(no subject)
peter.evans commented almost 9 years ago

Hi bertocq,
David inquired about your bot in Slack so I responded saying this:
"""
This is a great bot, it was really thorough, lots of information available there. I responded via our turbot mail system (Hope that got through?) outlining the basic idea of a licence transformer.
Scraping the primary data is brilliant but in order to normalise data from different sources and import it into opencorporates we need to apply licence transformers. We are only using the simple-licence transformer for MTB at the moment, we have a much better rich-licence transformer ready but we need to document it before people can apply it easily.
So my answer is that there are no issues with the bot (it is great!) but with a licence transformer it could be even better
"""
I tried to send you some mail with Turbot previously but perhaps there some problem with our system, I'll paste what I said below:
"""
Hey bertocq,
As I said this looks like a really tricky dataset to scrape, well done! I'll give this scraper a proper review soon but it looks like the kind of dataset that merits a very thorough review. Was this one that was started at the Madrid FlashHacks?
In the meantime can I suggest that you have a look at our tutorial examples for how to apply a basic transformer to the dataset? A transformer outputs a second set of data which is formatted such that the data may be ingested into the main OpenCorporates database (& thus made open).
You can find the examples for how to construct a transformer here: http://turbot.opencorporates.com/docs/examples#structured-bots
and the description of the simple licence schema here: http://turbot.opencorporates.com/docs/supported_data_types
If you have any questions do feel free to be in touch by responding to this email - I've also invited you to our Slack group.
All the best & thank you for writing this scraper!
Peter
"""
So in summary the bot is in great shape, how do you feel about writing a licence transformer? :)
All the best,
Peter

(no subject)
peter.evans commented almost 9 years ago

Hey bertocq,
As I said this looks like a really tricky dataset to scrape, well done! I'll give this scraper a proper review soon but it looks like the kind of dataset that merits a very thorough review. Was this one that was started at the Madrid FlashHacks?
In the meantime can I suggest that you have a look at our tutorial examples for how to apply a basic transformer to the dataset? A transformer outputs a second set of data which is formatted such that the data may be ingested into the main OpenCorporates database (& thus made open).
You can find the examples for how to construct a transformer here: http://turbot.opencorporates.com/docs/examples#structured-bots
and the description of the simple licence schema here: http://turbot.opencorporates.com/docs/supported_data_types
If you have any questions do feel free to be in touch by responding to this email - I've also invited you to our Slack group.
All the best & thank you for writing this scraper!
Peter

Bot state update
commented about 9 years ago

A draft run succeeded; sending for review

Bot state update
commented about 9 years ago

A draft run started

Bot state update
bertocq commented about 9 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape succeeded on March 28, 2015 10:04 2002 rows in about 1 hour

Config

{
  "bot_id": "dgsfp-insurance-licences",
  "title": "FINANCE LICENCES (INSURANCE) - GENERAL DIRECTORATE OF INSURANCE (SPAIN)",
  "description": "List of all finance licences from the GENERAL DIRECTORATE OF INSURANCE (SPAIN)",
  "language": "ruby",
  "data_type": "primary data",
  "identifying_fields": [
    "company_id"
  ],
  "files": [
    "scraper.rb"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Direccion General de Seguros y Fondos de Pensiones",
    "url": "http://www.dgsfp.mineco.es/RegistrosPublicos/AseguradorasReaseguradoras/AseguradorasReaseguradoras.aspx",
    "terms": "Open Government License",
    "terms_url": "http://www.mineco.gob.es/portal/site/mineco/menuitem.b6a9121612a3362cafb0240e026041a0/?vgnextoid=53c9a5539d915310VgnVCM1000001d04140aRCRD"
  },
  "public_repo_url": "https://github.com/bertocq/flashhacks-dgsfp-scraper",
  "tags": [
    "insurance",
    "financial",
    "licences"
  ]
}