TurbotServer

title	Nigerian Finance licences (securities) (ng-sec-licences) storing_data view rate in graphite
description	This bot scrapes information on capital market participants and approved fund managers from the Nigerian Securities and Exchange Commission
current run state	storing may have failed
last run	single run snapshot 12 scrape succeeded on April 11, 2016 13:02
next run	enqueued for a run, docker status
created by	dinotash (Tom Curtis)
last reviewed by	peter.evans

Show audit notes

Bot state update

commented about 8 years ago

A run finished; its output is now being processed

Bot state update

commented about 8 years ago

A run started

Bot state update

commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

dinotash commented about 8 years ago

Bot state update

commented about 8 years ago

A run started

Bot state update

commented about 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

dinotash commented about 8 years ago

Bot state update

commented about 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

dinotash commented over 8 years ago

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

dinotash commented over 8 years ago

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

dinotash commented over 8 years ago

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Saved vars cleared

dinotash commented over 8 years ago

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update

commented over 8 years ago

A run started

Bot state update

commented over 8 years ago

A snapshot completed; scheduling the first run of the next snapshot

Bot state update

commented over 8 years ago

A run started

Bot state update

commented almost 9 years ago

A run succeeded; scheduling the next run

Bot state update

commented almost 9 years ago

A run started

Bot state update

commented almost 9 years ago

A run succeeded; scheduling the next run

Bot state update

commented almost 9 years ago

A run started

Bot state update

commented almost 9 years ago

A run succeeded; scheduling the next run

Bot state update

peter.evans commented almost 9 years ago

The bot was accepted; starting run to ingest reviewed data

Bot state update

commented almost 9 years ago

A draft run succeeded; sending for final review

Bot state update

peter.evans commented almost 9 years ago

A moderator has approved the draft bot; running a full draft for final review

Bot state update

peter.evans commented almost 9 years ago

A moderator has started reviewing the draft bot

(no subject)

peter.evans commented almost 9 years ago

Hi Tom,

That was quick! Thank you for the fix, accepting this into openc now.

Thanks & hope you're having a good day.
Peter

Bot state update

commented almost 9 years ago

Run succeeded; sending for draft review

Re: (turbot bot [ng-sec-licences])

dinotash commented almost 9 years ago

Hi Peter

Just took a look at that and think I’ve fixed it. Updated bot submitted.

Thanks
Tom

Bot state update

dinotash commented almost 9 years ago

The bot was pushed; scheduling a draft run

(no subject)

peter.evans commented almost 9 years ago

Hi Tom,

Thank you for making those changes, that all looks sorted now - I think this one is very close to being ready. I did spot something else which which we could improve, this one is down to the structure of their website. At the moment there are newline characters within the company names E.g.

"FSDH\n Asset Management Ltd"
"Kakawa Asset Mgt. Co.Limited"
"Stanbic\n IBTC Asset Mgt. Limited"
"Stanbic\n IBTC Asset Mgt. Limited"
"Sterling\n Capital Market Limited"
"Zenith Capital Limited"

It would be good to strip these out if it's not difficult to do.

Thanks,
Peter

Bot state update

commented almost 9 years ago

Run succeeded; sending for draft review

Re: (turbot bot [ng-sec-licences])

dinotash commented almost 9 years ago

Hi Peter

Updated bot submitted.

Thanks
Tom

Bot state update

dinotash commented almost 9 years ago

The bot was pushed; scheduling a draft run

Bot state update

commented almost 9 years ago

Run succeeded; sending for draft review

Bot state update

dinotash commented almost 9 years ago

The bot was pushed; scheduling a draft run

(no subject)

peter.evans commented almost 9 years ago

Hi Tom,

Thanks for the updates - I'm just giving ng-sec-licences a more thorough review before accepting. Looks like there are a couple of thing we might want to edit, firstly one of the jurisdiction_classifications is "illegal operator" I think we should not transform these as they do not sound like licences (still good to have in the primary data though) The same is also possibly true of "Capital market operator with incomplete registration" - What do you think? (not quite a financial licence yet?)

Also spotted a few records getting scraped with the name "FUND MANAGER" which I think was originally a header. I'll paste an example record below:

{
"licence_jurisdiction": "Nigeria",
"category": "Financial",
"company_name": "FUND MANAGER",
"regulator": "Securities and Exchange Commission, Nigeria",
"sample_date": "2015-03-02",
"company_jurisdiction": "Nigeria",
"jurisdiction_classification": "Approved fund manager",
"source_url": "http://www.sec.gov.ng/files/CMO%20DATA/FUND%20MANAGERS.htm"
}

I think it would also be a good idea to set the identifying fields to include jurisdiction_classification.

There are also some duplicate records in the fund managers set, but these are duplicated at source as well, so I think this is another case where we should allow duplicates in the manifest. We could also possibly make the jurisdiction_classification more specific for the fund managers page based on those sub-headings, but I don't think that is necessary.a

Thanks,
Peter

Bot state update

commented about 9 years ago

A draft run succeeded; sending for review

Bot state update

commented about 9 years ago

A draft run started

Bot state update

dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update

commented about 9 years ago

A draft run succeeded; sending for review

Bot state update

commented about 9 years ago

A draft run started

Bot state update

dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Bot state update

commented about 9 years ago

A draft run succeeded; sending for review

Bot state update

commented about 9 years ago

A draft run started

Bot state update

dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

(no subject)

peter.evans commented about 9 years ago

Hi Tom,

Thanks for writing these scrapers, they look to be in very good shape indeed. I noticed that they are missing company_fields property which I think is required and in this one could be just:

"company_fields": {"name": "name"},

Otherwise this one is ready to accept, will aim to have a look at the other tomorrow.

Best wishes,
Peter

Bot state update

commented about 9 years ago

A draft run succeeded; sending for review

Bot state update

commented about 9 years ago

A draft run started

Bot state update

dinotash commented about 9 years ago

The bot was pushed; scheduling a draft run

Run history

event	metadata
single run snapshot draft scrape succeeded on March 01, 2015 20:08	1169 rows in 1 minute
single run snapshot draft scrape succeeded on March 02, 2015 06:13	1169 rows in 1 minute
single run snapshot draft scrape succeeded on March 02, 2015 06:20	1169 rows in 1 minute
single run snapshot draft scrape succeeded on March 02, 2015 21:49	1169 rows in 1 minute
single run snapshot draft scrape succeeded on May 04, 2015 19:57	1142 rows in 2 minutes
single run snapshot draft scrape succeeded on May 04, 2015 20:00	1142 rows in 2 minutes
single run snapshot draft scrape succeeded on May 05, 2015 08:09	1142 rows in 2 minutes
single run snapshot final draft scrape succeeded on May 05, 2015 15:05	1142 rows in 2 minutes
single run snapshot 1 prescrape scrape succeeded on May 05, 2015 15:09	1142 rows in 1 minute
single run snapshot 2 scrape succeeded on June 05, 2015 15:09	1143 rows in 2 minutes
single run snapshot 3 scrape succeeded on July 05, 2015 15:08	1145 rows in 1 minute
single run snapshot 4 scrape succeeded on August 05, 2015 15:08	1146 rows in 1 minute
single run snapshot 5 scrape succeeded on September 05, 2015 15:08	1145 rows in 1 minute
single run snapshot 6 scrape succeeded on October 05, 2015 15:08	1149 rows in 1 minute
single run snapshot 7 scrape succeeded on November 05, 2015 15:08	71 rows in less than a minute
single run snapshot 8 scrape succeeded on December 05, 2015 15:08	71 rows in less than a minute
single run snapshot 9 scrape succeeded on January 05, 2016 15:08	71 rows in less than a minute
single run snapshot 10 scrape succeeded on February 05, 2016 15:09	71 rows in 2 minutes
single run snapshot 11 scrape succeeded on March 05, 2016 15:08	71 rows in less than a minute
single run snapshot 12 scrape succeeded on April 11, 2016 13:02	71 rows in 3 minutes

Config

{
  "bot_id": "ng-sec-licences",
  "title": "Nigerian Finance licences (securities)",
  "description": "This bot scrapes information on capital market participants and approved fund managers from the Nigerian Securities and Exchange Commission",
  "language": "python",
  "data_type": "primary data",
  "identifying_fields": [
    "name",
    "category"
  ],
  "company_fields": {
    "name": "name"
  },
  "files": [
    "scraper.py",
    "licence.py"
  ],
  "transformers": [
    {
      "file": "licence.py",
      "data_type": "simple-licence",
      "identifying_fields": [
        "company_name",
        "jurisdiction_classification"
      ]
    }
  ],
  "frequency": "monthly",
  "publisher": {
    "name": "Securities and Exchange Commission, Nigeria",
    "url": "http://www.sec.gov.ng",
    "terms": "Copyright - no explanation given",
    "terms_url": "http://www.sec.gov.ng"
  },
  "duplicates_allowed": "true"
}

TURBOT

OpenCorporates
The Open Database Of The Corporate World

Run history

Config

TURBOT

OpenCorporates The Open Database Of The Corporate World

Run history

Config

OpenCorporates
The Open Database Of The Corporate World