title Capital Markets Authority (Saudi Arabia) (saudiarabia-cma) running
description Finance Licences in Saudi Arabia
current run state scraping or awaiting scrape, docker status
last run single run snapshot 22 scrape succeeded on February 15, 2017 08:18
next run enqueued for a run, docker status
created by s (Salman Aljammaz)
last reviewed by peter.evans
State changed to running for run #10103, snapshot 23
commented over 3 years ago

A run started

State changed to scheduled for run #10103, snapshot 23
commented over 3 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 3 years ago

State changed to ingesting_data for run #9898, snapshot 22
commented over 3 years ago

The run's output is being ingested

State changed to storing_data for run #9898, snapshot 22
commented over 3 years ago

The run's output is being stored

State changed to running for run #9898, snapshot 22
commented over 3 years ago

A run started

State changed to scheduled for run #9898, snapshot 22
commented over 3 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 3 years ago

State changed to ingesting_data for run #9631, snapshot 21
commented over 3 years ago

The run's output is being ingested

State changed to storing_data for run #9631, snapshot 21
commented over 3 years ago

The run's output is being stored

State changed to running for run #9631, snapshot 21
commented over 3 years ago

A run started

State changed to scheduled for run #9631, snapshot 21
commented over 3 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 3 years ago

State changed to ingesting_data for run #9363, snapshot 20
commented over 3 years ago

The run's output is being ingested

State changed to storing_data for run #9363, snapshot 20
commented over 3 years ago

The run's output is being stored

State changed to running for run #9363, snapshot 20
commented over 3 years ago

A run started

State changed to scheduled for run #9363, snapshot 20
commented over 3 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 3 years ago

State changed to ingesting_data for run #9129, snapshot 19
commented over 3 years ago

The run's output is being ingested

State changed to storing_data for run #9129, snapshot 19
commented over 3 years ago

The run's output is being stored

State changed to running for run #9129, snapshot 19
commented over 3 years ago

A run started

State changed to scheduled for run #9129, snapshot 19
commented almost 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented almost 4 years ago

State changed to ingesting_data for run #8865, snapshot 18
commented almost 4 years ago

The run's output is being ingested

State changed to storing_data for run #8865, snapshot 18
commented almost 4 years ago

The run's output is being stored

State changed to running for run #8865, snapshot 18
commented almost 4 years ago

A run started

State changed to scheduled for run #8865, snapshot 18
commented almost 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented almost 4 years ago

State changed to ingesting_data for run #8534, snapshot 17
commented almost 4 years ago

The run's output is being ingested

State changed to storing_data for run #8534, snapshot 17
commented almost 4 years ago

The run's output is being stored

State changed to running for run #8534, snapshot 17
commented almost 4 years ago

A run started

State changed to scheduled for run #8534, snapshot 17
commented almost 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented almost 4 years ago

State changed to ingesting_data for run #8184, snapshot 16
commented almost 4 years ago

The run's output is being ingested

State changed to storing_data for run #8184, snapshot 16
commented almost 4 years ago

The run's output is being stored

State changed to running for run #8184, snapshot 16
commented almost 4 years ago

A run started

State changed to scheduled for run #8184, snapshot 16
commented about 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented about 4 years ago

State changed to ingesting_data for run #7836, snapshot 15
commented about 4 years ago

The run's output is being ingested

State changed to storing_data for run #7836, snapshot 15
commented about 4 years ago

The run's output is being stored

State changed to running for run #7836, snapshot 15
commented about 4 years ago

A run started

State changed to scheduled for run #7836, snapshot 15
commented about 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented about 4 years ago

State changed to ingesting_data for run #7460, snapshot 14
commented about 4 years ago

The run's output is being ingested

State changed to storing_data for run #7460, snapshot 14
commented about 4 years ago

The run's output is being stored

State changed to running for run #7460, snapshot 14
commented about 4 years ago

A run started

State changed to scheduled
commented about 4 years ago

For run #7460:
A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented about 4 years ago

State changed to ingesting_data
commented about 4 years ago

For run #6928:
The run's output is being ingested

State changed to storing_data
commented about 4 years ago

For run #6928:
The run's output is being stored

State changed to running
commented about 4 years ago

For run #6928:
A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented over 4 years ago

A run finished; its output is now being processed

Bot state update
commented over 4 years ago

A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented over 4 years ago

A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented over 4 years ago

A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented over 4 years ago

A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented over 4 years ago

A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented over 4 years ago

A run started

Bot state update
commented over 4 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared
s commented over 4 years ago

Bot state update
commented almost 5 years ago

A run started

Bot state update
commented almost 5 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update
commented almost 5 years ago

A run started

Bot state update
commented almost 5 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update
commented almost 5 years ago

A run started

Bot state update
commented about 5 years ago

A run succeeded; scheduling the next run

Bot state update
commented about 5 years ago

A run started

Bot state update
commented about 5 years ago

A run succeeded; scheduling the next run

Bot state update
peter.evans commented about 5 years ago

The bot was accepted; starting run to ingest reviewed data

Bot state update
commented about 5 years ago

A draft run succeeded; sending for final review

Bot state update
peter.evans commented about 5 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update
peter.evans commented about 5 years ago

A moderator has started reviewing the draft bot

(no subject)
peter.evans commented about 5 years ago

Great, thank you - re-accepting it now :)

Bot state update
commented about 5 years ago

Run succeeded; sending for draft review

Bot state update
s commented about 5 years ago

The bot was pushed; scheduling a draft run

Re: (turbot bot [saudiarabia-cma])
s commented about 5 years ago

Done. :)

(no subject)
peter.evans commented about 5 years ago

Hi Salman,
Thank you for submitting this bot, we accepted it & it's been running fine but it looks like there's the same "licence_jursidiction" issue, it would be great if we could make a quick fix to the transformer, should be very easy :)
Thanks!
Peter

Bot state update
commented over 5 years ago

A run failed

Bot state update
commented over 5 years ago

A run started

(no subject)
peter.evans commented over 5 years ago

Hi Salman,
Thank you for responding, hope all is well with you, don't worry about there being gaps between getting in touch - it's very busy here at the moment as well!
Glad you spotted the SAMA Mission! :) will look forward to having a look at how you tackle it.
The super-contributors group should become more active soon & we'll be looking for one or two people to get on board and have some responsibility in helping to manage turbot administration as well.
All the best
Peter

Re: (turbot bot [saudiarabia-cma])
s commented over 5 years ago

Hi Peter
Sorry, I've been super busy the past couple of weeks! I'm looking at the
SAMA one next :) I'll do my best to translate it properly.
Pages that use .aspx are the ones built using Microsoft's .NET framework
(in the same way .php gives us a hint that the website is written in PHP
for example). Unfortunately their content can be anything so there's no
secret recipe to parse them. The best bet is to just look at the HTML and
try to figure out how it's structured.
Also, I've noticed the new group. Looks pretty good! I'll post an
introduction soon.
All the best,
Salman

Bot state update
commented over 5 years ago

A run succeeded; scheduling the next run

Bot state update
commented over 5 years ago

A run started

Bot state update
peter.evans commented over 5 years ago

The bot was accepted; scheduling immediately

Bot state update
peter.evans commented over 5 years ago

A moderator has started reviewing the bot

(no subject)
peter.evans commented over 5 years ago

Hi Salman,
That's brilliant, I'll accept this now. Thanks for spotting the missing closing bracket in"Licensed Agent (Dealing", too.
Careful what you wish for as I may well take you up on explaining some code! Also did you spot the monster of a Mission for Saudi Arabian Monetary Agency? (http://www.sama.gov.sa/BankingControl/Pages/LicensedBanks.aspx) How to get around scraping from pages that use .aspx would be something I'd be interested in hearing!
All the best,
Peter

Re: (turbot bot [saudiarabia-cma])
s commented over 5 years ago

Hi Peter!
Those names look good, thanks! Done & done.
Also, feel free to ask me anything if you want me to explain a bit of code!
s

Bot state update
commented over 5 years ago

A draft run succeeded; sending for review

Bot state update
commented over 5 years ago

A draft run started

Bot state update
s commented over 5 years ago

The bot was pushed; scheduling a draft run

(no subject)
peter.evans commented over 5 years ago

Hi Salman,
Thanks for working on the Saudi Arabia securities mission, the scraper is looking great. Particularly I liked the thorough use of comments, I'm learning to code, as you know, so being able to understand what you've done to scrape the data is brilliant.
As you've noted (# TODO perhaps find better names to these.) the names for the jurisdiction_classification field in the simple licence transformer could be improved - they're a bit non standard though... What do you think of something like this:
Licensed Principal (Dealing)
Licensed Agent (Dealing
Licensed Underwriter (Dealing)
Investment Fund Management (Managing)
Discretionary Portfolio Management (Managing)
Licensed for Arranging
Licensed for Advising
Licensed for Custody
Also for the simple licence we could add a regulator field as they're all regulated by the publisher. So something like "regulator": "Saudi Arabia Capital Market Authority (CMA)"
Other than that I can't find any issues, I'll give it another once over before we submit it but it looks great.
Best,
Peter

Bot state update
commented over 5 years ago

A draft run succeeded; sending for review

Bot state update
commented over 5 years ago

A draft run started

Bot state update
commented over 5 years ago

A failed draft run was restarted by the moderator

Bot state update
commented over 5 years ago

A draft run started

Bot state update
commented over 5 years ago

A failed draft run was restarted by the moderator

Bot state update
commented over 5 years ago

A run encountered an internal system error

Bot state update
commented over 5 years ago

A draft run started

Bot state update
commented over 5 years ago

The bot was pushed; scheduling a draft run

Run history

event metadata
single run snapshot draft scrape errored on January 23, 2015 08:36 0 rows in about 17 hours
single run snapshot draft scrape succeeded on January 26, 2015 09:24 88 rows in less than a minute
single run snapshot draft scrape succeeded on January 30, 2015 11:11 88 rows in less than a minute
single run snapshot 1 scrape succeeded on January 30, 2015 20:00 88 rows in less than a minute
single run snapshot 2 scrape failed on February 28, 2015 20:00 88 rows in less than a minute
single run snapshot draft scrape succeeded on June 11, 2015 08:30 88 rows in less than a minute
single run snapshot final draft scrape succeeded on June 12, 2015 08:16 88 rows in 1 minute
single run snapshot 2 prescrape scrape succeeded on June 15, 2015 08:18 88 rows in less than a minute
single run snapshot 3 scrape succeeded on July 15, 2015 08:18 88 rows in 1 minute
single run snapshot 4 scrape succeeded on August 15, 2015 08:18 88 rows in less than a minute
single run snapshot 5 scrape succeeded on September 15, 2015 08:18 88 rows in less than a minute
single run snapshot 6 scrape succeeded on October 16, 2015 01:02 88 rows in about 17 hours
single run snapshot 7 scrape succeeded on November 15, 2015 08:18 88 rows in less than a minute
single run snapshot 8 scrape succeeded on December 15, 2015 08:18 88 rows in less than a minute
single run snapshot 9 scrape succeeded on January 15, 2016 08:18 88 rows in 1 minute
single run snapshot 10 scrape succeeded on February 15, 2016 08:18 88 rows in 1 minute
single run snapshot 11 scrape succeeded on March 15, 2016 08:18 88 rows in less than a minute
single run snapshot 12 scrape succeeded on April 15, 2016 08:19 88 rows in less than a minute
single run snapshot 13 scrape succeeded on May 15, 2016 08:18 88 rows in less than a minute
single run snapshot 14 scrape succeeded on June 15, 2016 08:18 84 rows in 1 minute
single run snapshot 15 scrape succeeded on July 15, 2016 08:18 84 rows in less than a minute
single run snapshot 16 scrape succeeded on August 15, 2016 08:18 83 rows in less than a minute
single run snapshot 17 scrape succeeded on September 16, 2016 12:21 83 rows in about 2 hours
single run snapshot 18 scrape succeeded on October 15, 2016 08:18 83 rows in less than a minute
single run snapshot 19 scrape succeeded on November 15, 2016 08:18 83 rows in less than a minute
single run snapshot 20 scrape succeeded on December 15, 2016 08:18 83 rows in less than a minute
single run snapshot 21 scrape succeeded on January 15, 2017 08:18 83 rows in less than a minute
single run snapshot 22 scrape succeeded on February 15, 2017 08:18 84 rows in less than a minute
single run snapshot 23 scrape scheduled on March 15, 2017 08:18 0 rows

Config

{
  "bot_id": "saudiarabia-cma",
  "title": "Capital Markets Authority (Saudi Arabia)",
  "description": "Finance Licences in Saudi Arabia",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "authorised_person"
  ],
  "company_fields": {
    "name": "authorised_person"
  },
  "files": [
    "scraper.py",
    "licence_transformer.py"
  ],
  "transformers": [
    {
      "file": "licence_transformer.py",
      "data_type": "simple-licence",
      "identifying_fields": [
        "company_name"
      ]
    }
  ],
  "frequency": "monthly",
  "tags": [
    "world-finance-licences",
    "saudiarabia",
    "mapthebanks"
  ],
  "publisher": {
    "name": "Capital Markets Authority",
    "terms": "© Capital Market Authority All rights reserved 2014",
    "terms_url": "http://www.cma.org.sa/en/pages/disclaimer.aspx",
    "url": "http://www.cma.org.sa/En/Pages/AuthorisedPersons.aspx"
  }
}