TURBOT
title Dutch Financial Market Authority (nl-afm-licences) pending_draft_review
description This bot returns data on auditors, investment funds, investment firms, financial service providers, clearing houses and trading platforms regulated by the Dutch Financial Market Authority (AFM)
current run state not running
last run single run snapshot draft scrape succeeded on September 09, 2016 06:42
next run n/a
created by dinotash (Tom Curtis)
last reviewed by
State changed to pending_draft_review for run #8804, snapshot draft
commented over 7 years ago

Run succeeded; sending for draft review

State changed to draft_running for run #8804, snapshot draft
dinotash commented over 7 years ago

The bot was pushed; scheduling a draft run

State changed to failed for run #7682, snapshot 8
commented almost 8 years ago

A run failed

State changed to running for run #7682, snapshot 8
commented almost 8 years ago

A run started

State changed to scheduled for run #7682, snapshot 8
commented almost 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented almost 8 years ago

State changed to ingesting_data for run #7681, snapshot 7
commented almost 8 years ago

The run's output is being ingested

State changed to storing_data for run #7681, snapshot 7
commented almost 8 years ago

The run's output is being stored

State changed to running for run #7681, snapshot 7
Alex Skene commented almost 8 years ago

The run has been restarted as a prescrape

State changed to docker_step_errored
commented almost 8 years ago

For run #6785:
General error in dockerised part of pipeline.
Error reported from docker:
Exception (#<Docker::Error::ServerError: engine is shutdown
>) raised. Backtrace:
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/docker-api-1.22.4/lib/docker/connection.rb:50:in `rescue in request'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/docker-api-1.22.4/lib/docker/connection.rb:38:in `request'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/docker-api-1.22.4/lib/docker/connection.rb:65:in `block (2 levels) in <class:Connection>'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/docker-api-1.22.4/lib/docker/container.rb:20:in `wait'
/home/openc/sites/morph/releases/20160427141934/lib/turbot_docker_runner.rb:186:in `run_in_container'
/home/openc/sites/morph/releases/20160427141934/lib/turbot_docker_runner.rb:35:in `run'
/home/openc/sites/morph/releases/20160427141934/lib/turbot_docker_runner.rb:6:in `perform'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/job.rb:227:in `perform'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:250:in `perform'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:189:in `block in work'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:166:in `loop'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/resque-1.25.2/lib/resque/worker.rb:166:in `work'
/home/openc/sites/morph/releases/20160427141934/lib/tasks/resque.rake:54:in `block (2 levels) in <top (required)>'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:248:in `call'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:248:in `block in execute'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:243:in `each'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:243:in `execute'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:187:in `block in invoke_with_call_chain'
/home/openc/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:180:in `invoke_with_call_chain'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/task.rb:173:in `invoke'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:150:in `invoke_task'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:106:in `each'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:106:in `block in top_level'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:115:in `run_with_threads'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:100:in `top_level'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:78:in `block in run'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:176:in `standard_exception_handling'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/lib/rake/application.rb:75:in `run'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/gems/rake-11.1.1/bin/rake:33:in `<top (required)>'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/bin/rake:23:in `load'
/home/openc/sites/morph/shared/bundle/ruby/2.0.0/bin/rake:23:in `<main>'

State changed to running
commented almost 8 years ago

For run #6785:
A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run finished; its output is now being processed

Bot state update
commented about 8 years ago

A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

The bot was accepted; starting run to ingest reviewed data

Bot update
commented over 8 years ago

A manual prescrape using data from /oc/openc/scrapers/output/draft/n/nl-afm-licences/5457 was started

Bot state update
commented over 8 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A final draft run failed

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A final draft run failed

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A final draft run failed

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented over 8 years ago

A failed draft run was restarted by the moderator

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A failed final draft was sent by the moderator back to draft

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A final draft run failed

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A final draft run failed

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
peter.evans commented over 8 years ago

A failed draft run was restarted by the moderator

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A draft run failed

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
peter.evans commented over 8 years ago

The bot needs more work

Bot state update
commented over 8 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A final draft run failed

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
peter.evans commented over 8 years ago

The bot needs more work

Bot state update
commented over 8 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented over 8 years ago

A failed draft run was restarted by the moderator

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A draft run failed

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A draft run failed

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented over 8 years ago

A draft run failed

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape failed on August 30, 2015 22:51 92 rows in 3 minutes
single run snapshot draft scrape failed on August 31, 2015 17:26 0 rows in less than a minute
single run snapshot draft scrape failed on August 31, 2015 17:30 92 rows in 4 minutes
single run snapshot draft scrape succeeded on August 31, 2015 18:59 2002 rows in about 1 hour
single run snapshot draft scrape interrupted on September 01, 2015 22:23 23769 rows in 1 day
single run snapshot draft scrape succeeded on September 08, 2015 18:41 1366 rows in about 4 hours
single run snapshot final draft scrape succeeded on September 09, 2015 18:36 1366 rows in about 4 hours
single run snapshot draft scrape succeeded on September 15, 2015 21:33 2002 rows in about 1 hour
single run snapshot final draft scrape failed on September 16, 2015 13:38 2562 rows in about 2 hours
single run snapshot draft scrape succeeded on September 22, 2015 22:38 2037 rows in about 2 hours
single run snapshot final draft scrape succeeded on September 24, 2015 04:19 20273 rows in about 20 hours
single run snapshot draft scrape failed on September 26, 2015 21:23 1025 rows in 1 day
single run snapshot draft scrape succeeded on September 27, 2015 03:14 2038 rows in about 2 hours
single run snapshot draft scrape transformer failed on September 27, 2015 20:45 2002 rows in about 1 hour
single run snapshot draft scrape transformer failed on September 28, 2015 07:33 2002 rows in about 1 hour
single run snapshot draft scrape transformer failed on September 28, 2015 22:45 2002 rows in about 1 hour
single run snapshot draft scrape transformer failed on September 30, 2015 10:25 2002 rows in about 2 hours
single run snapshot draft scrape transformer failed on October 02, 2015 07:50 2002 rows in about 1 hour
single run snapshot draft scrape transformer failed on October 02, 2015 20:17 2002 rows in about 1 hour
single run snapshot draft scrape transformer failed on October 03, 2015 22:14 2002 rows in about 1 hour
single run snapshot draft scrape succeeded on October 05, 2015 22:50 2002 rows in about 1 hour
single run snapshot final draft scrape failed on October 07, 2015 17:40 24615 rows in 1 day
single run snapshot draft scrape succeeded on October 12, 2015 13:45 2002 rows in 2 days
single run snapshot final draft scrape failed on October 17, 2015 05:47 5581 rows in 4 days
single run snapshot draft scrape transformer failed on October 17, 2015 19:08 2002 rows in about 2 hours
single run snapshot draft scrape succeeded on October 18, 2015 16:49 2002 rows in about 2 hours
single run snapshot final draft scrape errored on October 21, 2015 10:54 0 rows in less than a minute
single run snapshot draft scrape errored on October 21, 2015 12:00 0 rows in less than a minute
single run snapshot draft scrape succeeded on October 21, 2015 14:59 2002 rows in about 1 hour
single run snapshot final draft scrape failed on October 22, 2015 16:37 24389 rows in 1 day
single run snapshot draft scrape succeeded on October 22, 2015 22:55 2002 rows in about 1 hour
single run snapshot draft scrape transformer failed on October 23, 2015 07:19 2002 rows in about 1 hour
single run snapshot draft scrape succeeded on October 23, 2015 21:03 2002 rows in about 1 hour
single run snapshot final draft scrape failed on October 27, 2015 10:11 24625 rows in 1 day
single run snapshot draft scrape succeeded on October 27, 2015 20:28 2002 rows in about 2 hours
single run snapshot final draft scrape failed on October 29, 2015 11:43 24626 rows in 1 day
single run snapshot draft scrape succeeded on October 30, 2015 00:12 2002 rows in about 4 hours
single run snapshot final draft scrape succeeded on October 31, 2015 10:49 24700 rows in 1 day
single run snapshot 1 prescrape scrape succeeded on November 02, 2015 09:37 24700 rows in 28 minutes
single run snapshot 2 scrape succeeded on December 03, 2015 11:52 24765 rows in 1 day
single run snapshot 3 scrape succeeded on January 03, 2016 14:32 24865 rows in 1 day
single run snapshot 4 scrape succeeded on February 03, 2016 11:46 24942 rows in 1 day
single run snapshot 5 scrape succeeded on March 03, 2016 15:09 24962 rows in 1 day
single run snapshot 6 scrape succeeded on April 03, 2016 13:06 24838 rows in 1 day
single run snapshot 7 scrape docker step errored on May 02, 2016 15:03 0 rows in about 6 hours
single run snapshot 7 prescrape scrape succeeded on June 07, 2016 12:38 0 rows in less than a minute
single run snapshot 8 scrape failed on July 07, 2016 12:39 0 rows in less than a minute
single run snapshot draft scrape succeeded on September 09, 2016 06:42 2002 rows in about 2 hours

Config

{
  "bot_id": "nl-afm-licences",
  "title": "Dutch Financial Market Authority",
  "description": "This bot returns data on auditors, investment funds, investment firms, financial service providers, clearing houses and trading platforms regulated by the Dutch Financial Market Authority (AFM)",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "name",
    "category",
    "licence_number",
    "country",
    "source_url"
  ],
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Netherlands Authority for the Financial Markets (AFM)",
    "url": "http://www.afm.nl",
    "terms": "The AFM reserves copyright to this website. The information on this website may not be taken over and/or disseminated, in whole or in part, and in any way, unless with the explicit indication of the AFM as reference.",
    "terms_url": "http://www.afm.nl/en/professionals/disclaimer"
  },
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "licence",
      "identifying_fields": [
        "licence_holder.entity_properties.name",
        "licence_issuer.jurisdiction",
        "licence_issuer.name",
        "licence_number",
        "licence_holder.entity_properties.jurisdiction",
        "permissions"
      ]
    }
  ],
  "duplicates_allowed": "true"
}