Hi Peter

I think that’s because that company was licensed as both an investment fund manager and AIFM with the same license number. Because the bot works through each category, it scraped it twice.

I’ve just pushed a bot with permissions added as an identifying field for licenses. Hopefully that will stop it. Otherwise, I’ll need to tell it to keep track of what it’s already scraped and not to scrape it a second time if an entity comes up in more than one category.

Cheers
Tom

Hey Tom,

Thanks for pushing a fix here - It looks like that has cleared up the error. When running the bot to completion it's run into a duplicate licence_number (and so has thrown this error)

The following record is invalid:
{"Institute type":"AIFM","Company Registration Number":"01-10-044261","Activity":"Article 5 (3) a) of Directive 85/611/EEC","Website address":"www.aegonalapkezelo.hu","Broker field":"Capital market","Adress":"1091 Budapest, Üllői út 1.","Function":"Securities issuer","Activities":[{"Active":"Active","Activity name":"contract settlements (including certificate dispatch)","Law reference":"Kbftv 6. § (1) bh) és 7.§ (2) ah)","Activity content":"n.a."},{"Active":"Active","Activity name":"distribution of income","Law reference":"Kbftv. 7. § (2) bek. a) pont af) alpont","Activity content":"n.a."},{"Active":"Active","Activity name":"intermediation of functions of administration connected to the marketing and distribution of collective investment instruments","Law reference":"Kbftv 7. § (4) alapján Kbftv 7. § (2) ag)","Activity content":"n.a."},{"Active":"Active","Activity name":"investment advice","Law reference":"Kbftv 7.§ (3) b)","Activity content":"n.a."},{"Active":"Active","Activity name":"investment management (decisions relating to investment strategies and strategic asset allocation in connection with the investment policy, including the implementation thereof)","Law reference":"Kbftv. 7. § (1) bek. a) pont","Activity content":"n.a."},{"Active":"Active","Activity name":"legal and fund management accounting services","Law reference":"Kbftv. 7. § (2) bek. a) pont aa) alpont","Activity content":"n.a."},{"Active":"Active","Activity name":"maintenance of registers relating to investors","Law reference":"Kbftv. 7. § (2) bek. a) pont ae) alpont","Activity content":"n.a."},{"Active":"Active","Activity name":"management of portfolios of investments, including those owned by pension funds and institutions for occupational retirement provision in accordance with Article 19 (1) of Directive 2003/41/EC","Law reference":"Kbftv 7.§ (3) a)","Activity content":"n.a."},{"Active":"Active","Activity name":"marketing and distribution of collective investment instruments managed by AIFM","Law reference":"Kbftv 7.§ (2) b)","Activity content":"n.a."},{"Active":"Active","Activity name":"providing information to investors","Law reference":"Kbftv 6. § (1) bb) és 7.§ (2) ab)","Activity content":"n.a."},{"Active":"Active","Activity name":"record keeping","Law reference":"Kbftv 6. § (1) bi) és 7.§ (2) ai)","Activity content":"n.a."},{"Active":"Active","Activity name":"regulatory compliance monitoring","Law reference":"Kbftv. 7. § (2) bek. a) pont ad) alpont","Activity content":"n.a."},{"Active":"Active","Activity name":"risk management","Law reference":"Kbftv. 7. § (1) bek. b) pont","Activity content":"n.a."},{"Active":"Active","Activity name":"safekeeping and management of collective investment instruments, including the related administration services, etc.","Law reference":"Kbftv 7.§ (3) c)","Activity content":"n.a."},{"Active":"Active","Activity name":"valuation and pricing of assets, including tax returns","Law reference":"Kbftv. 7. § (2) bek. a) pont ac) alpont","Activity content":"n.a."},{"Active":"Active","Activity name":"Issuance and trading of collective investment securities","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Securities safekeeping, custodianship, administrative services, securities account and client account management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Ancillary activity for investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Administrative activities in relation to the issuance and trading of collective investment securities","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Asset valuation, pricing and administration of taxation matters as administrative services completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Bookkeeping and legal activities as administrative services completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Information provision to investors as administrative service completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Investment management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Maintenance of investor records as administrative services completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Record keeping as administrative service completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Settlement of concluded transactions (including document submission)","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Training of sales staff by investment fund manager, provision of written/verbal information","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Verification of compliant conduct as administrative services completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Yield payout as administrative service completing collective portfolio management","Law reference":"Batv. 4. §","InvestmentInstrument":"n.a.","Activity content":"Investment fund management/collective portfolio management"},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"a) transferable securities","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"b) money-market instruments","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"c) securities issued by collective investment trusts","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"e) options, futures, swaps, forward rate agreements and any other derivative contracts and instruments relating to commodities that must be settled in cash or may be settled in cash at the option of one of the parties (otherwise than by reason of a default or other termination event","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"f) options, futures, swaps, and any other derivative contract relating to commodities that can be physically settled provided that they are traded on a regulated market and/or on multilateral trading facilities","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"h) derivative instruments for the transfer of credit risk","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"i) financial contracts for differences","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"j) options, futures, swaps, forward rate agreements and any other derivative contracts and instruments relating to climatic variables, freight rates, greenhouse gas emission allowance units and other rights of emission of air polluting substances, inflation rates or other official economic statistics that must be settled in cash or may be settled in cash at the option of one of the parties (otherwise than by reason of a default","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"k) any other derivative contracts and instruments relating to assets, rights, obligations, indices and measures not otherwise mentioned under Paragraphs a)-j) of Section 6 of the IRA, which have the characteristics of other derivative financial instruments, having regard to whether, inter alia, they are traded on a regulated market or multilateral trading facilities, are cleared and settled through recognized clearing houses or are subject to regular margin calls, furthermore, the derivative contracts referred to in Article 39 of Commission Regulation (EC) No. 1287/2006","Business branch limit":"n.a."},{"Active":"Active","Activity name":"Investment consulting","Law reference":"Bszt. 5. §","InvestmentInstrument":"options, futures, swaps, forwards (carried out on an OTC basis or exchange-traded) and any other derivative contracts relating to commodities, that can be physically settled not otherwise mentioned in Paragraph f) of Section 6 of the IRA and not being for commercial purposes, which have the characteristics of other derivative financial instruments, having regard to whether, inter alia, they are cleared and settled through recognized clearing houses or are subject to regular margin calls","Business branch limit":"n.a."},{"Activity content":"Portfolio management","InvestmentInstrument":"a) transferable securities","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"b) money-market instruments","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"c) securities issued by collective investment trusts","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"e) options, futures, swaps, forward rate agreements and any other derivative contracts and instruments relating to commodities that must be settled in cash or may be settled in cash at the option of one of the parties (otherwise than by reason of a default or other termination event","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"f) options, futures, swaps, and any other derivative contract relating to commodities that can be physically settled provided that they are traded on a regulated market and/or on multilateral trading facilities","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"h) derivative instruments for the transfer of credit risk","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"i) financial contracts for differences","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"j) options, futures, swaps, forward rate agreements and any other derivative contracts and instruments relating to climatic variables, freight rates, greenhouse gas emission allowance units and other rights of emission of air polluting substances, inflation rates or other official economic statistics that must be settled in cash or may be settled in cash at the option of one of the parties (otherwise than by reason of a default","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"k) any other derivative contracts and instruments relating to assets, rights, obligations, indices and measures not otherwise mentioned under Paragraphs a)-j) of Section 6 of the IRA, which have the characteristics of other derivative financial instruments, having regard to whether, inter alia, they are traded on a regulated market or multilateral trading facilities, are cleared and settled through recognized clearing houses or are subject to regular margin calls, furthermore, the derivative contracts referred to in Article 39 of Commission Regulation (EC) No. 1287/2006","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Activity content":"Portfolio management","InvestmentInstrument":"options, futures, swaps, forwards (carried out on an OTC basis or exchange-traded) and any other derivative contracts relating to commodities, that can be physically settled not otherwise mentioned in Paragraph f) of Section 6 of the IRA and not being for commercial purposes, which have the characteristics of other derivative financial instruments, having regard to whether, inter alia, they are cleared and settled through recognized clearing houses or are subject to regular margin calls","Activity name":"Portfolio management (personal)","Business branch limit":"n.a.","Active":"Active","Law reference":"Bszt. 5. §"},{"Active":"Active","Activity name":"Securities lending","Law reference":"Bszt. 8. §","InvestmentInstrument":"n.a."}],"Date of foundation":"2000.02.21","source":"Magyar Nemezeti Bank","Predecessor/Successor":[{"Date":"31/07/2013","Predecessor/Successor institute":"AEGON Magyarország Befektetési Jegy Forgalmazó Zrt.","Type":"Predecessor"},{"Date":"06/03/2015","Predecessor/Successor institute":"AEGON Magyarország Befektetési Jegy Forgalmazó Zrt.","Type":"Predecessor"}],"Managed invesmtent funds":[{"Fund name":"Aegon Panoráma Származtatott Befektetési Alap","Fund registration number":"1111-586"},{"Fund name":"AEGON Smart Money Befektetési Alapok Alapja","Fund registration number":"1111-349"},{"Fund name":"Aegon Prémium Dynamic Alapokba Fektető Részalap","Fund registration number":"1111-675-2"},{"Fund name":"Aegon Prémium Esernyőalap","Fund registration number":"1111-675"},{"Fund name":"Aegon Prémium Everest Alapokba Fektető Részalap","Fund registration number":"1111-675-3"},{"Fund name":"Aegon Prémium Expert Alapokba Fektető Részalap","Fund registration number":"1111-675-1"},{"Fund name":"AEGON Climate Change Részvény Befektetési Alap","Fund registration number":"1111-233"},{"Fund name":"AEGON Pénzpiaci Befektetési Alap","Fund registration number":"1111-121"},{"Fund name":"AEGON Alfa Származtatott Befektetési Alap","Fund registration number":"1111-171"},{"Fund name":"AEGON MoneyMaxx Expressz Abszolút Hozamú Befektetési alap","Fund registration number":"1111-129"},{"Fund name":"Aegon Maraton Aktív Vegyes Befektetési Alap","Fund registration number":"1111-616"},{"Fund name":"Aegon Tempó Allegro 10 Alapokba Fektető Részalap","Fund registration number":"1111-580-10"},{"Fund name":"Aegon Tempó Allegro 8 Alapokba Fektető Részalap","Fund registration number":"1111-580-8"},{"Fund name":"Aegon Tempó Allegro 9 Alapokba Fektető Részalap","Fund registration number":"1111-580-9"},{"Fund name":"Aegon Tempó Andante 1 Alapokba Fektető Részalap","Fund registration number":"1111-580-1"},{"Fund name":"Aegon Tempó Andante 2 Alapokba Fektető Részalap","Fund registration number":"1111-580-2"},{"Fund name":"Aegon Tempó Andante 3 Alapokba Fektető Részalap","Fund registration number":"1111-580-3"},{"Fund name":"Aegon Tempó Esernyőalap","Fund registration number":"1111-580"},{"Fund name":"Aegon Tempó Moderato 4 Alapokba Fektető Részalap","Fund registration number":"1111-580-4"},{"Fund name":"Aegon Tempó Moderato 5 Alapokba Fektető Részalap","Fund registration number":"1111-580-5"},{"Fund name":"Aegon Tempó Moderato 6 Alapokba Fektető Részalap","Fund registration number":"1111-580-6"},{"Fund name":"Aegon Tempó Moderato 7 Alapokba Fektető Részalap","Fund registration number":"1111-580-7"},{"Fund name":"AEGON ÓZON Éves Tőkevédett Alap","Fund registration number":"1111-212"},{"Fund name":"AEGON Ázsia Részvény Befektetési Alapok Alapja","Fund registration number":"1111-221"},{"Fund name":"AEGON Belföldi Kötvény Befektetési Alap","Fund registration number":"1111-58"},{"Fund name":"AEGON Bessa Származatott Befektetési Alap","Fund registration number":"1111-241"},{"Fund name":"AEGON EuroExpress Befektetési Alap","Fund registration number":"1111-254"},{"Fund name":"AEGON IstanBull Részvény Befektetési Alap","Fund registration number":"1111-306"},{"Fund name":"AEGON Közép-Európai Részvény Befektetési Alap","Fund registration number":"1111-57"},{"Fund name":"AEGON Közép-Európai Vállalati Kötvény Befektetési Alap","Fund registration number":"1111-431"},{"Fund name":"AEGON Lengyel Kötvény Befektetési Alap","Fund registration number":"1111-220"},{"Fund name":"AEGON Lengyel Pénzpiaci Befektetési Alap","Fund registration number":"1111-496"},{"Fund name":"AEGON Lengyel Részvény Befektetési Alap","Fund registration number":"1111-479"},{"Fund name":"AEGON Nemzetközi Kötvény Befektetési Alap","Fund registration number":"1111-80"},{"Fund name":"AEGON Nemzetközi Részvény Befektetési Alap","Fund registration number":"1111-81"},{"Fund name":"AEGON Russia Részvény Befektetési Alap","Fund registration number":"1111-305"},{"Fund name":"AEGON Value Származtatott Befektetési Alap","Fund registration number":"1111-172"}],"categories":["Capital market","Fund manager","AIFM"],"Address":"1091 Budapest, Üllői út 1.","sample_date":"2016-03-21","Which country":"Lengyelország, Csehország, Szlovákia","Nyilvántartott egyed neve":"AEGON Magyarország Befektetési Alapkezelő Zrt.","Típus":"Befektetési alapkezelő","Name":"Aegon Magyarország Befektetési Alapkezelő Zártkörűen Működő Részvénytársaság","Broker type":"Batv. szerinti közvetítő","source_url":"https://alk.mnb.hu/en/left_menu/market_participants/kereso/kereses/talalatok/?ri=2637&portalstate=ktadataview","Törzsszám":"11951766","Executive manager":[{"Position name":"Chairman of the Board","Name":"Kadocsa Péter"},{"Position name":"Managing director","Name":"Kocsis Bálint"},{"Position name":"Supervisory Board chairman","Name":"Kepecs Gábor"},{"Position name":"Managing director","Name":"Kadocsa Péter"},{"Position name":"Board member","Name":"Loncsák András"},{"Position name":"Supervisory Board member","Name":"Hindrik Eggens"},{"Position name":"Board member","Name":"Kocsis Bálint"},{"Position name":"Supervisory Board member","Name":"Gáti György"}],"Mandator":"AEGON Magyarország Általános Biztosító Zártkörűen Működő Részvénytársaság","Persons (institute) with influence":[{"Ownership rate":"200.0000 %","Influence rate":"100.0000 %","Name":"AEGON Magyarország Általános Biztosító Zártkörűen Működő Részvénytársaság","Influence type":"Ownership based"},{"Ownership rate":"100.0000 %","Influence rate":"50.0000 %","Name":"Aegon Hungary Holding B.V.","Influence type":"Ownership based"},{"Ownership rate":"100.0000 %","Influence rate":"50.0000 %","Name":"AEGON Hungary Holding II B.V.","Influence type":"Ownership based"}],"Identifier":"11951766","Registered Number":"11951766","Place of publication":"n.a.","Registered capital":"1000000000","Broker service":"n.a.","Date of licensing":"n.a.","Company type":"Joint stock company"}
* Already seen record with these identifying fields: {"Identifier"=>"11951766"}

The following record is invalid:
{"category":["Financial"],"confidence":"HIGH","jurisdiction_of_licence":"Hungary","licence_holder":{"entity_properties":{"website":"www.aegonalapkezelo.hu","mailing_address":"1091 Budapest, Üllői út 1.","jurisdiction":"Hungary","name":"Aegon Magyarország Befektetési Alapkezelő Zártkörűen Működő Részvénytársaság","officers":[{"position":"Chairman of the Board","name":"Kadocsa Péter"},{"position":"Managing director","name":"Kocsis Bálint"},{"position":"Supervisory Board chairman","name":"Kepecs Gábor"},{"position":"Managing director","name":"Kadocsa Péter"},{"position":"Board member","name":"Loncsák András"},{"position":"Supervisory Board member","name":"Hindrik Eggens"},{"position":"Board member","name":"Kocsis Bálint"},{"position":"Supervisory Board member","name":"Gáti György"}]},"entity_type":"unknown"},"licence_number":"11951766","sample_date":"2016-03-21","licence_issuer":{"jurisdiction":"Hungary","name":"Magyar Nemezeti Bank"},"permissions":[{"activity_name":"Capital market - Fund manager - AIFM","permission_type":"operating"}],"source_url":"https://alk.mnb.hu/en/left_menu/market_participants/kereso/kereses/talalatok/?ri=2637&portalstate=ktadataview"}
* Already seen record with these identifying fields: {"licence_number"=>"11951766"}

Is this expected? Thanks,

Peter

Hey Tom,

Thanks for pushing the fix to this bot - It now seems to be bailing on seemingly different points in the run - Does this traceback make sense to you?

Loading details for entity 45166642 (56 / 159)
Loading intermediate search page for entity 20720511 (57 / 159)
Traceback (most recent call last):
File "scraper.py", line 285, in <module>
parse_index(page_doc, source['categories'], start_num, results_count)
File "scraper.py", line 130, in parse_index
if (row_search_table.tbody.tr is not None):
AttributeError: 'NoneType' object has no attribute 'tbody'

All the best & Hope your move is going well!

Peter

Sounds sensible and easily done.

Hey Tom,

Thanks for the great description, I understand perfectly now.

I think we *definitely* want to capture the hierarchy that you describe because the bottom level doesn't always identify the type of licence we're dealing with, for example "Insurer" where it is a child of "foreign institution providing cross-border services" would lose a lot of its meaning if we only captured "Insurer". I don't think we can have an individual permission for each of these though as I think that would insinuate separate licences.

What do you think of appending each level? E.g.

"Foreign institution providing cross-border services - Insurance sector - Insurer"

It's a bit verbose and we could probably cut out "insurance sector" in that instance, but I think that would be micro-managing too much and just appending the hierarchy from top to bottom would be fine.

What do you think?

Thanks
Peter

Hi Peter

This one is because the Hungarian Central Bank classified permissions in a hierarchy, and I thought it made sense to catch all the different levels of specificity. What the bot is trying to say is that this entity is a fee paying multiple agent, which is a type of multiple agent, which is a type of independent money market broker, which is a type of non-financial institution, which is a kind of money market participant.

I don’t think this shows up clearly when you go directly to the source_url because of the way the website uses frames. However, it is clear if you go into the website through the more typical human-facing way in, as below:

1) This is the page which leads into the market participant data: http://www.mnb.hu/en/supervision/licensing-and-institution-oversight/market-participants/search-of-market-participants.

2) That page links through to a search page: https://alk.mnb.hu/en/left_menu/market_participants/kereso/kereses

3) If you go to the “List based on primary type” tab on that page, you can click through and expand out the hierarchy, which is what I followed.

Let me know if my approach makes a bit more sense now, or if you’d rather keep just the most detailed level of specificity on each licence.

Thanks
Tom

Hey Tom,

This bot is looking good, the only thing I'm unsure on is the setting of permissions. If we take the company of licence number 10242869 as an example we have the below permissions:

[{"activity_name"=>"Money market", "permission_type"=>"operating"}, {"activity_name"=>"Non-financial institution", "permission_type"=>"operating"}, {"activity_name"=>"Independent money market broker", "permission_type"=>"operating"}, {"activity_name"=>"Multiple agent (D)", "permission_type"=>"operating"}, {"activity_name"=>"Fee paying multiple agent", "permission_type"=>"operating"}]

from looking at the website at the source_url it seems to me that the financial licence being expressed here is for "Insurance brokerage", or something similar to this. Probably "activity_name"=>"Independent money market broker" is the closest in the permissions we have at the moment? I'm not sure what the other permissions are expressing in this instance. Am I missing something here? Can you give me some indication of what the extra permissions fields are expressing?

Thanks
Peter