title Malaysian Licensed Intermediaries (my-seccom) running
description This bot extracts data relating to licensed financial companies in Malaysia
current run state scraping or awaiting scrape, docker status
last run single run snapshot 18 scrape succeeded on March 01, 2017 20:35
next run enqueued for a run, docker status
created by dinotash (Tom Curtis)
last reviewed by peter.evans
State changed to running for run #10123, snapshot 19
commented about 7 years ago

A run started

State changed to scheduled for run #10123, snapshot 19
commented about 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 7 years ago

State changed to ingesting_data for run #10038, snapshot 18
commented about 7 years ago

The run's output is being ingested

State changed to storing_data for run #10038, snapshot 18
commented about 7 years ago

The run's output is being stored

State changed to running for run #10038, snapshot 18
commented about 7 years ago

A run started

State changed to scheduled for run #10038, snapshot 18
commented about 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 7 years ago

State changed to ingesting_data for run #9803, snapshot 17
commented about 7 years ago

The run's output is being ingested

State changed to storing_data for run #9803, snapshot 17
commented about 7 years ago

The run's output is being stored

State changed to running for run #9803, snapshot 17
commented about 7 years ago

A run started

State changed to scheduled for run #9803, snapshot 17
commented over 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 7 years ago

State changed to ingesting_data for run #9522, snapshot 16
commented over 7 years ago

The run's output is being ingested

State changed to storing_data for run #9522, snapshot 16
commented over 7 years ago

The run's output is being stored

State changed to running for run #9522, snapshot 16
commented over 7 years ago

A run started

State changed to scheduled for run #9522, snapshot 16
commented over 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 7 years ago

State changed to ingesting_data for run #9275, snapshot 15
commented over 7 years ago

The run's output is being ingested

State changed to storing_data for run #9275, snapshot 15
commented over 7 years ago

The run's output is being stored

State changed to running for run #9275, snapshot 15
commented over 7 years ago

A run started

State changed to scheduled for run #9275, snapshot 15
commented over 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 7 years ago

State changed to ingesting_data for run #9013, snapshot 14
commented over 7 years ago

The run's output is being ingested

State changed to storing_data for run #9013, snapshot 14
commented over 7 years ago

The run's output is being stored

State changed to running for run #9013, snapshot 14
commented over 7 years ago

A run started

State changed to scheduled for run #9013, snapshot 14
commented over 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 7 years ago

State changed to ingesting_data for run #8720, snapshot 13
commented over 7 years ago

The run's output is being ingested

State changed to storing_data for run #8720, snapshot 13
commented over 7 years ago

The run's output is being stored

State changed to running for run #8720, snapshot 13
commented over 7 years ago

A run started

State changed to scheduled for run #8720, snapshot 13
commented over 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 7 years ago

State changed to ingesting_data for run #8414, snapshot 12
commented over 7 years ago

The run's output is being ingested

State changed to storing_data for run #8414, snapshot 12
commented over 7 years ago

The run's output is being stored

State changed to running for run #8414, snapshot 12
commented over 7 years ago

A run started

State changed to scheduled for run #8414, snapshot 12
commented over 7 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 7 years ago

State changed to ingesting_data for run #8039, snapshot 11
commented over 7 years ago

The run's output is being ingested

State changed to storing_data for run #8039, snapshot 11
commented over 7 years ago

The run's output is being stored

State changed to running for run #8039, snapshot 11
commented over 7 years ago

A run started

State changed to scheduled for run #8039, snapshot 11
commented almost 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented almost 8 years ago

State changed to ingesting_data for run #7626, snapshot 10
commented almost 8 years ago

The run's output is being ingested

State changed to storing_data for run #7626, snapshot 10
commented almost 8 years ago

The run's output is being stored

State changed to running for run #7626, snapshot 10
commented almost 8 years ago

A run started

State changed to scheduled for run #7626, snapshot 10
commented almost 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented almost 8 years ago

State changed to ingesting_data for run #7349, snapshot 9
commented almost 8 years ago

The run's output is being ingested

State changed to storing_data for run #7349, snapshot 9
commented almost 8 years ago

The run's output is being stored

State changed to running
commented almost 8 years ago

For run #7349:
A run started

State changed to scheduled
commented almost 8 years ago

For run #7349:
A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented almost 8 years ago

State changed to ingesting_data
commented almost 8 years ago

For run #6776:
The run's output is being ingested

State changed to storing_data
commented almost 8 years ago

For run #6776:
The run's output is being stored

State changed to running
commented almost 8 years ago

For run #6776:
A run started

Bot state update
commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run finished; its output is now being processed

Bot state update
Alex Skene commented about 8 years ago

A failed run was restarted by the moderator

Saved vars cleared
dinotash commented about 8 years ago

Bot state update
commented about 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
dinotash commented over 8 years ago

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update
commented over 8 years ago

A run started

Bot state update
commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update
peter.evans commented over 8 years ago

The bot was accepted; starting run to ingest reviewed data

Bot state update
commented over 8 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update
peter.evans commented over 8 years ago

A moderator has started reviewing the draft bot

Bot state update
commented over 8 years ago

Run succeeded; sending for draft review

(no subject)
peter.evans commented over 8 years ago

Great! Thanks Tom.
The run takes a few hours so I'll let you know how it gets on.
Thanks,
Peter

Re: (turbot bot [my-seccom])
dinotash commented over 8 years ago

Thanks Peter
Just re-pushed another update which should fix this. Looks like I was missing a comma at the end of the row above.
Thanks
Tom

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented over 8 years ago

Hi Tom,
Thank you for working on this scraper again, I'm attaching the traceback that I can see:
File "licence.py", line 31
"end_date": raw_record['anniversary_date']
^
SyntaxError: invalid syntax
Hope that's helpful - if not I'll get PeterI involved, if so we should definitely make the tracebacks visibile to you!
All the best,
Peter

Re: (turbot bot [my-seccom])
dinotash commented over 8 years ago

Hi Peter
I’ve pushed an update which should take care of the date issue in the scraper.out - converting everything to yyyy-mm-dd format.
However, it’s now running into a different error: "draft run transformer failed”. I’m not sure why, and the turbot page isn’t showing me any of the useful debugging output to see where it failed. Any chance you could let me know?
Thanks
Tom

Bot state update
dinotash commented over 8 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented over 8 years ago

Hi Tom,
Hope all is well with you, sorry to have taken a while with this one - I think we've got to the bottom of it though... apparently the issue is caused by some date formatting in the scraper.out E.g. 'Invalid format: \"28 Sep 2007\"'.
Do you think you could try altering the format of these dates & we'll see if it runs okay? I'd be very interested to see the results of this actually because I'm surprised we're running date validation on the scraper.out.
All the best & thank you as always,
Peter

(no subject)
peter.evans commented almost 9 years ago

Hi Tom,
Just fyi - I haven't forgotten about your bot for Malaysia, we're having an issue at our end where only the licence data is appearing - When Peter has looked at this I'll be able to review this bot properly.
Best,
Peter

Bot state update
commented almost 9 years ago

Run succeeded; sending for draft review

Bot state update
peter.evans commented almost 9 years ago

A failed draft run was restarted by the moderator

Bot state update
commented about 9 years ago

A draft run succeeded; sending for review

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
sebbacon commented about 9 years ago

A failed draft run was restarted by the moderator

Bot state update
commented about 9 years ago

A draft_errored run encountered an internal system error

Bot state update
commented about 9 years ago

A draft run started

Bot state update
peter.evans commented about 9 years ago

A failed draft run was restarted by the moderator

Bot state update
commented about 9 years ago

A draft_errored run encountered an internal system error

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update
commented about 9 years ago

A run failed

Bot state update
commented about 9 years ago

A draft run started

Bot state update
dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape failed on March 12, 2015 13:32 0 rows in about 1 hour
single run snapshot draft scrape failed on March 17, 2015 07:37 0 rows in about 1 hour
single run snapshot draft scrape errored on March 21, 2015 10:30 0 rows in less than a minute
single run snapshot draft scrape errored on March 25, 2015 19:45 0 rows in less than a minute
single run snapshot draft scrape failed on March 26, 2015 10:00 0 rows in about 1 hour
single run snapshot draft scrape failed on March 26, 2015 21:20 0 rows in about 1 hour
single run snapshot draft scrape failed on March 28, 2015 00:47 327 rows in about 5 hours
single run snapshot draft scrape failed on March 28, 2015 14:24 327 rows in about 5 hours
single run snapshot draft scrape failed on March 28, 2015 22:45 327 rows in about 5 hours
single run snapshot draft scrape failed on March 29, 2015 04:01 327 rows in about 5 hours
single run snapshot draft scrape succeeded on March 29, 2015 12:28 327 rows in about 5 hours
single run snapshot draft scrape succeeded on July 14, 2015 19:36 537 rows in about 6 hours
single run snapshot draft scrape transformer failed on August 17, 2015 00:33 537 rows in about 6 hours
single run snapshot draft scrape succeeded on August 17, 2015 14:04 537 rows in about 6 hours
single run snapshot final draft scrape succeeded on August 17, 2015 20:48 537 rows in about 6 hours
single run snapshot 1 prescrape scrape succeeded on August 18, 2015 07:46 537 rows in less than a minute
single run snapshot 2 scrape succeeded on September 18, 2015 13:30 537 rows in about 6 hours
single run snapshot 3 scrape succeeded on October 18, 2015 13:30 538 rows in about 6 hours
single run snapshot 4 scrape succeeded on November 18, 2015 13:32 537 rows in about 6 hours
single run snapshot 5 scrape succeeded on December 18, 2015 13:30 540 rows in about 6 hours
single run snapshot 6 scrape succeeded on January 18, 2016 13:30 540 rows in about 6 hours
single run snapshot 7 scrape errored on February 18, 2016 13:41 0 rows in about 6 hours
single run snapshot 7 scrape succeeded on April 01, 2016 20:36 540 rows in about 6 hours
single run snapshot 8 scrape succeeded on May 01, 2016 20:44 539 rows in about 6 hours
single run snapshot 9 scrape succeeded on June 01, 2016 20:40 540 rows in about 6 hours
single run snapshot 10 scrape succeeded on July 01, 2016 20:41 0 rows in about 6 hours
single run snapshot 11 scrape succeeded on August 01, 2016 20:41 0 rows in about 6 hours
single run snapshot 12 scrape succeeded on September 01, 2016 20:40 0 rows in about 6 hours
single run snapshot 13 scrape succeeded on October 01, 2016 20:40 0 rows in about 6 hours
single run snapshot 14 scrape succeeded on November 01, 2016 20:39 0 rows in about 6 hours
single run snapshot 15 scrape succeeded on December 01, 2016 20:41 0 rows in about 6 hours
single run snapshot 16 scrape succeeded on January 01, 2017 20:40 0 rows in about 6 hours
single run snapshot 17 scrape succeeded on February 01, 2017 20:39 0 rows in about 6 hours
single run snapshot 18 scrape succeeded on March 01, 2017 20:35 0 rows in about 6 hours
single run snapshot 19 scrape scheduled on April 01, 2017 14:59 0 rows

Config

{
  "bot_id": "my-seccom",
  "title": "Malaysian Licensed Intermediaries",
  "description": "This bot extracts data relating to licensed financial companies in Malaysia",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "name",
    "licence_number"
  ],
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Securities Commission, Malaysia",
    "url": "http://www.sc.com.my",
    "terms": "No part of this Site may be reproduced or stored in any other website or be included in any public or private electronic retrieval system or services without the prior written permission of the SC.",
    "terms_url": "http://www.sc.com.my/home/disclaimer/"
  },
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "simple-licence",
      "identifying_fields": [
        "company_name",
        "licence_number",
        "end_date"
      ]
    }
  ]
}