TurbotServer

title	Isle of Man - Financial Supervision (iom-financial-supervision) failed
description	Bot for opencorporates mission 564
current run state	not running
last run	single run snapshot 8 scrape failed on November 06, 2015 15:46
next run	n/a
created by	k-nut (Knut Hühne)
last reviewed by	sebbacon

Show audit notes

Currently failing

sebbacon commented about 8 years ago

This can't be ingested at the moment; not only does the bot not run, but the existing data includes an invalid field "licence_jursidiction" (note spelling, but also not a supported field).

Bot state update

commented over 8 years ago

A run failed

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

k-nut commented over 8 years ago

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update

commented over 8 years ago

A run started

Bot state update

commented almost 9 years ago

A run succeeded; scheduling the next run

Bot state update

commented almost 9 years ago

A run started

(no subject)

peter.evans commented almost 9 years ago

Hi Knut,

Thank you for the fixes, this is back up and running smoothly again :) Good work!

Thanks,
Peter

Bot state update

commented almost 9 years ago

A run succeeded; scheduling the next run

Bot state update

peter.evans commented almost 9 years ago

The bot was accepted; starting run to ingest reviewed data

Bot state update

commented almost 9 years ago

A draft run succeeded; sending for final review

Bot state update

peter.evans commented almost 9 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update

peter.evans commented almost 9 years ago

A moderator has started reviewing the draft bot

Bot state update

commented almost 9 years ago

Run succeeded; sending for draft review

Re: (turbot bot [iom-financial-supervision])

k-nut commented almost 9 years ago

OK, I did just that. Does it look good now or does it still need
improvement?
Best wishes,
Knut

Bot state update

k-nut commented almost 9 years ago

The bot was pushed; scheduling a draft run

(no subject)

peter.evans commented almost 9 years ago

Hi Knut,

Thanks for the response - You're right that there are no applicable terms, I just meant that the field was missing from the manifest E.g.

"publisher": {
"name": "Publisher of the data",
"url": "Publisher's website",
"terms": "Copyright terms (e.g. Open Government License, n/a, etc)",
"terms_url": "A place where these terms can be checked or verified"
}

If we can't find any terms then we can just map this to:

"publisher": {
"name": "Financial Supervision Commission - Isle of Man Government",
"url": "http://www.fsc.gov.im/licenceholders/SearchLicenceHolders.aspx?type=Current&id=1&searchtext=&selection=All&submit1=Submit",
"terms": "",
"terms_url": "http://www.fsc.gov.im/licenceholders/SearchLicenceHolders.aspx?type=Current&id=1&searchtext=&selection=All&submit1=Submit"
}

Does this make sense?

Thanks!
Peter

Re: (turbot bot [iom-financial-supervision])

k-nut commented almost 9 years ago

Hm, the thing is that I set terms to n/a since I could not find any
information about the terms. I find that very hard for most of the bots
I write. The closest thing that I could find was [1] but that seems to
refer to the website itself more then the data. But maybe you can find
something that applies to the data as well?

[1] http://www.fsc.gov.im/terms.gov?parentid=36&sectionid=36

(no subject)

peter.evans commented almost 9 years ago

Hi Knut,

Thank you for applying this fix, this bot should be back up and running now.

I've just noticed that the "terms_url" is missing from your publisher fields, do you think we could add that in to all of your bots?

E.g.

"publisher": { # <- essential so we can be sure it's open data
"name": "Publisher of the data",
"url": "Publisher's website",
"terms": "Copyright terms (e.g. Open Government License, n/a, etc)",
"terms_url": "A place where these terms can be checked or verified"
}

If you can't find a URL for terms then we can just use the source_url / Mission URL

Thanks!
Peter

Bot state update

commented almost 9 years ago

Run succeeded; sending for draft review

Re: (turbot bot [iom-financial-supervision])

k-nut commented almost 9 years ago

OK, I also added the publisher information to the manifest. I will get
to the other bots later :)

Bot state update

k-nut commented almost 9 years ago

The bot was pushed; scheduling a draft run

(no subject)

peter.evans commented almost 9 years ago

Hi Knut,

Looking at some more of the bots that you have submitted. Thank you for all of your work on these as ever! This one has previously run and generated data however we have recently improved the validation of produced data & this one tripped over as it is currently outputting a field called "license_jurisdiction" this should actually be "licence_jurisdiction". Hopefully this should be a very easy fix & push to get this bot back up and running.

All the best & Thank you
Peter

Bot state update

commented about 9 years ago

A run failed

Bot state update

commented about 9 years ago

A run started

Bot state update

commented about 9 years ago

A run succeeded; scheduling the next run

Bot state update

commented about 9 years ago

A run started

Bot state update

commented over 9 years ago

A run succeeded; scheduling the next run

Bot state update

commented over 9 years ago

A run started

Bot state update

commented over 9 years ago

An initial run was triggered manually by a moderator

Bot state update

commented over 9 years ago

Automatic bot state update. Was in state running, now in state to_be_scheduled

Run history

event	metadata
single run snapshot 1 scrape succeeded on January 12, 2015 21:38	244 rows in 2 minutes
single run snapshot 2 scrape succeeded on February 12, 2015 21:38	242 rows in 2 minutes
single run snapshot 3 scrape failed on March 12, 2015 21:38	243 rows in 1 minute
single run snapshot draft scrape succeeded on May 25, 2015 10:33	241 rows in 2 minutes
single run snapshot draft scrape succeeded on June 06, 2015 08:40	241 rows in 2 minutes
single run snapshot final draft scrape succeeded on June 06, 2015 15:22	241 rows in 2 minutes
single run snapshot 3 prescrape scrape succeeded on June 06, 2015 15:46	241 rows in less than a minute
single run snapshot 4 scrape succeeded on July 06, 2015 15:47	240 rows in 2 minutes
single run snapshot 5 scrape succeeded on August 06, 2015 15:48	239 rows in 2 minutes
single run snapshot 6 scrape succeeded on September 06, 2015 15:47	238 rows in 2 minutes
single run snapshot 7 scrape succeeded on October 06, 2015 15:48	238 rows in 3 minutes
single run snapshot 8 scrape failed on November 06, 2015 15:46	0 rows in less than a minute

Config

{
  "bot_id": "IoM-Financial-Supervision",
  "title": "Isle of Man - Financial Supervision",
  "description": "Bot for opencorporates mission 564",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "Reference_Number"
  ],
  "transformers": [
    {
      "file": "licence_transformer.py",
      "data_type": "simple-licence",
      "identifying_fields": [
        "licence_number"
      ]
    }
  ],
  "files": [
    "scraper.py",
    "licence_transformer.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Financial Supervision Commission - Isle of Man Government",
    "url": "http://www.fsc.gov.im/licenceholders/SearchLicenceHolders.aspx?type=Current&id=1&searchtext=&selection=All&submit1=Submit",
    "terms": "",
    "terms_url": "http://www.fsc.gov.im/licenceholders/SearchLicenceHolders.aspx?type=Current&id=1&searchtext=&selection=All&submit1=Submit"
  }
}

TURBOT

OpenCorporates
The Open Database Of The Corporate World

Run history

Config

TURBOT

OpenCorporates The Open Database Of The Corporate World

Run history

Config

OpenCorporates
The Open Database Of The Corporate World