Banco de Portugal

Background

• Regulatory and supervisory powers. Banco de Portugal is a banking supervisory authority with prudential and market conduct mandates over financial services providers (FSPs), including the banking sector and institutions providing transaction accounts, such as payment services providers.
• Consumer protection supervision role. The Banking Conduct Supervision Department is responsible for all activities related to market conduct supervision. Market monitoring makes up a large portion of the work and is an essential element of a risk-based supervisory approach. BCSD annually publishes a report summarizing its supervisory activities and achievements. BCSD has staff specialized in market conduct supervision that use several tools to carry out its systematic market monitoring.
• Reporting obligation of draft consumer credit contracts. To comply with draft contracts monitoring, FSPs are required to report to BCSD all draft contracts of new credit products and changes to draft contracts of existing products. They must also notify BCSD when a draft contract is no longer in use. Reporting must be completed 15 days prior to a contract change or product launch, although BCSD is not required to approve draft contracts before they are implemented. Banco de Portugal maintains a database of all reported draft contracts, which are stored in the format in which they were reported (readable PDFs, Word documents, and, in some cases, scanned images).
• Team responsible for analyzing draft consumer credit contracts. BCSD has a team of lawyers responsible for, among other tasks, analyzing draft contracts from a regulatory compliance perspective. Analysis is not carried out immediately after a draft contract is reported. Rather, the activity follows the BCSD supervision plan, which also includes other activities. The BCSD team specializes in consumer protection and market conduct in banking and other regulated sectors, with deep expertise in consumer protection regulation and legislation.  

Current manual process for analyzing draft credit contracts

• Time spent on draft contract analysis. BCSD estimates that manual analysis of draft contracts takes between 24–48 months of staff time, with approximately two-thirds of that time (16–32 months) spent validating rules. The effort is the equivalent of having more than one staff member solely dedicate all their annual business hours to the analysis of average-sized draft contracts.
• The process. The current process follows these steps:
  • Reporting. An FSP sends BCSD a copy of a new draft credit contract or amendments to an existing contract.
  • Analysis. BCSD verifies compliance with contractual provisions via specific requirements set out in laws and regulations. Approximately 60 regulatory provisions are considered in each analysis.
  • Action. If a breach in regulations is identified, BCSD may issue specific orders demanding (i) amendment or removal of the relevant terms and conditions and (ii) the reporting of substitute draft contracts.
  • Follow-up. BCSD monitors implementation of any orders it issues, namely whether provisions have been amended or removed.
  • Enforcement. If BCSD orders are not implemented, the FSP is once again invited to send a new version of the contract. Failure to do so allows BCSD to initiate administrative proceedings against the FSP, if necessary.

Purpose and incentives

• What is the tool used for? In 2019, BCSD partnered with Banco de Portugal’s recently created Innovation Lab to pilot a suptech tool that would automate the analysis of draft consumer credit contracts. The pilot led to the successful development of a dashboard that allows BCSD staff to not only analyze whether draft contracts comply with a subset of regulatory requirements, but also to identify new types of products, product features, and contract terms. The pilot’s success resulted in the decision to invest in a full rollout. The production phase started in January 2021.
• Incentives for tool development. The main incentive for piloting a suptech tool was to prove the concept and gain internal approval for investment at scale of a tool that would reduce staff time dedicated to this activity which, despite requiring a high level of regulatory expertise, is seen as repetitive and not rewarding from an analyst’s point of view. The goal for automated contract analysis was to free up intellectual capacity for complex conduct supervision activities that cannot be done by machines. Another objective was to reduce response time (i.e., the time between reporting a draft contract and its analysis), allowing for timely action to avoid consumer harm. These outcomes would increase the efficiency and effectiveness of conduct supervision.
• From pilot phase to production phase. Within its operating model, the Innovation Lab followed a four-step flow for each initiative: plan, ideate, execute, and adopt. In the first step, the BCSD team and the Innovation Lab defined the initiative’s utility, and also identified who would be involved, resources needed, and the initiative’s timeline. In the second step, “design thinking” sessions were conducted to refine the concept, its benefits and challenges, and project goals. With the pilot’s goals agreed upon, the project moved to the third step: execution. (If successful, the pilot phase would be followed by a production phase to roll out the tool at scale). Following the proof of value demonstrated during execution, the final step—adoption—consisted of defining the strategy to promote the idea and convey its worth to the central bank.

Technical methodology and data ecosystem

• The vendor. For the pilot, BCSD entered into a mutual agreement with a large technology vendor with recognized expertise in artificial intelligence (name undisclosed). The vendor agreed to provide BCSD with cost-free access to its NLP platform for the pilot.
  • Production phase. BCSD will rely on the vendor’s programming capabilities for this phase. However, the vendor was not hired for this project alone. It provides Banco de Portugal with a range of technology and IT services focused on artificial intelligence and machine learning, which support the bank’s program to modernize its data capabilities. The BCSD project was inserted into the vendor’s deliverables.
• Overview of the pilot. BCSD ran a three-month pilot in 2019 to evaluate the feasibility and effectiveness of automating draft contract analysis. Its main purpose was to evaluate and compare the accuracy of compliance assessments performed by a suptech solution against human-performed assessments. The pilot covered 18 regulatory requirements (about 20 percent of all manually checked requirements) with a relatively low level of complexity and one type of draft contract (personal loans). It took a month for the vendor and BCSD to develop the tool together.
  • Production phase. Based on experience with the pilot, BCSD gained approval to develop a similar tool at scale. The project began in January 2021; the first version of the solution is expected in the second half of 2021. A different vendor is being used for this phase. The plan is for the tool to include almost all regulatory requirements for all contract types. However, the first version will only cover an initial set of requirements. The tool will gradually incorporate other requirements.
• Technology. The pilot was based on natural language processing (NLP), an artificial intelligence field that gives computers the ability to “understand” human language, including text analysis. To run the pilot, BCSD installed a technology vendor’s platform on its own servers. In addition to the vendor platform’s dashboard, it used internal APIs to “attach” an internally-developed customized dashboard to improve visualization.
  • Production phase. The production phase will use the same paradigms and processes as the pilot. However, the plan is to incorporate a machine learning application that will add analytical layers on top of the rules repository. If enough tool analyses are checked against past human analyses, the rules repository will organically evolve to incorporate other types of analyses.
• Pilot components. The central element of the pilot was to translate regulatory requirements—already organized in a repository of rules that staff use for contract analyses—into a set of rules created in a format that the vendor’s platform could utilize. The vendor’s platform would then check each contract in the contracts database against these rules and return a result. Figure 1 illustrates the project’s four components:

1. Draft Contracts Repository. This repository is the database of draft contracts that sits at Banco de Portugal.
2. Rule Management. The rule creation and management tool is a repository of requirements on contracts used by the vendor’s platform to produce analytical results on the level of compliance found in contracts.
3. Data Processing. This is the software that runs the rules pertaining to draft contracts.
4. Visualization. These dashboards and applications enable extended analytics by BCSD staff on all documents in the database. The applications include an exploration of the rules themselves. To improve visualization options, BCSD used APIs to plug internal visualization tools into the vendor’s platform.

• Source data. BCSD used its rules repository and contract database to access the vendor’s NLP application. Only contracts in machine-readable format could be used and images were excluded. (The capability of transposing images to text was discussed but ultimately not included in the tool’s scope). All source data sit at Banco de Portugal.
• Reports produced. The pilot tool offered a simple dashboard that BCSD analysts could use to search any contract or regulatory requirement and produce simple reports by contract, product, regulatory requirement, date, FSP type, average number of submissions, and top nonconformities.

Staff, expertise, and other requirements 

• Technical requirements. There were no additional technical requirements for the pilot as BCSD used the vendor’s platform for free and Banco de Portugal hosted it. It is important to note that draft contracts must be in computer-readable format in order to use this type of tool. While some standardization in contract format would facilitate the use of the tool, it is not a requirement.
  • Production phase. The production phase also will not require new BCSD technical capabilities
• Staff requirements. The pilot had no new staffing requirements. The pilot and production phases used or will use existing BCSD capacity and the existing vendor contract. Approximately four BCSD staff members will be involved in both phases, along with an assigned IT department team. The production phase implies significant effort by different IT experts—from architecture and infrastructure to project management and data analysis.
• Expertise and skills. Staff members involved in piloting the tool were consumer protection supervision specialists—subject matter experts who were responsible for identifying the regulatory requirements to be included in the automated checks and for verifying results of the tool against results of analysts. These specialists are essential to providing input into customization of the tool. Staff did not require training in order to use the tool.
  • Production phase. The extended depth of machine learning capabilities will require specialists to provide input that trains models and constant monitoring of the tool to ensure acceptable levels of confidence in its output. This phase will also require rigorous governance processes to continuously evaluate model performance, incorporate specialist feedback, and empower data scientists to adapt or fine tune models as needed. Banco de Portugal has internal capacity in other departments to conduct this work (in the short term). However, regardless of this project, it has broader plans to increase its data science capabilities as part of an enterprise-wide program to modernize data architecture. Developing machine learning capabilities, for instance, is part of the broader plan. Training needs for BCSD staff will be defined when the new tool and its new capabilities are in place. Banco de Portugal also has a Data Science Academy program that focuses on improving its data science expertise.

Vendor selection and cost

• Pilot phase. BCSD chose a globally renowned vendor that was willing to provide no-cost access to its NLP platform.
• Production phase. Beginning in January 2021, BCSD began using a vendor Banco de Portugal had already engaged for a larger IT and technology services contract. This vendor is also a renowned company, chosen following government procurement rules for cost-benefit analyses. The cost of this larger contract and the specific funds allocated to the BCSD project have not been disclosed.

Benefits and impact

• Pilot phase. The pilot enabled development of a framework for rules creation, plus document processing and navigation. It helped identify visualization needs and developed an intuitive, web-based navigation interface with dashboards that allow for extended analytics capacities compared to the manual systems currently in use. It offered a high level of confidence that a suptech tool would provide reliable compliance analyses—at least for regulatory requirements with relatively low levels of complexity.
• Production phase. High efficiency gains are expected with the tool. The analysis of draft contracts (and potential analyses of a great volume of similarly unstructured data) will be greatly accelerated, with a potential savings of 8–16 months of staff time. (This follows a conservative 50 percent estimate of rule automation, which may reduce approximately a third of estimated staff time spent manually analyzing draft contracts). Human brain power will be shifted to more complex analyses, decision-making tasks, and process review, and the timeframe for supervisory analyses will be dramatically reduced almost to real time. An almost instantaneous first-level compliance analysis that is impossible today will be made available. Errors due to manual tasks are expected to be greatly reduced. If NLP is combined with machine learning in the future, the tool will allow supervision to evolve from analysis of a fixed set of rules into the advanced discovery of data patterns.

Limitations and implementation challenges

• Pilot phase. Since draft contracts are not standardized and there are several ways to comply with the same rule, automating draft contract analysis faced certain challenges:
  • Transposing text rules into concrete programable rules (codes) is not a straightforward process.
  • Setting parameters for analyzing how well a document complies with rules is an iterative process that requires constant monitoring and feedback from multiple experts.
  • Adding more complex legal and compliance checks requires higher levels of investment in time and budget to appropriately define and refine NLP algorithms.
  • Hardly any tool has the capability to assess compliance with each highly complex regulatory requirement. Some human analysis will always be necessary. 
• Production phase
  • The main challenge with the production (rollout) phase will be the expanded implementation timeline since it was necessary to engage another vendor to develop the tool. However, BCSD staff has entered the production phase already armed with knowledge accumulated in the pilot, in particular, rules and visualization needs.
  • Knowing that the suptech tool is limited in its ability to process highly complex regulatory requirements or to cover all regulatory requirements from the start, BCSD will have to prioritize which requirements the tool includes. Prioritization considers factors such as the importance of regulatory requirements on consumer outcomes, the time staff needs to manually analyze requirements, and the level of confidence the tool will return for each requirement.
  • A further challenge will be the development of a machine learning tool, which will greatly depend on the volume of training data needed to achieve high levels of confidence in its results and to remain at such levels. BCSD will specifically need to provide a great number of draft credit contracts and information about past analyses so machine learning analytics can adjust itself. In the tool’s first stages, the level of confidence may vary on a daily basis, depending on changes in the volume and types of data the tool digests. For these reasons, the new tool will require constant monitoring and evaluation to ensure confidence and quality.

Future plans for the tool

• Based on experience with the pilot, BCSD has entered the production stage and is using an external vendor for rollout. The tool will be expanded beyond the pilot to cover a higher number of regulatory requirements and most types of consumer contracts.
• BCSD plans to incorporate machine learning into the tool under production so analyses expand from simple checks against a fixed set of rules to discovery of patterns in the data (i.e., text from contracts held in the database). This will include contrasting the suptech tool’s results against past analyses of the same draft contracts by staff members. In this manner, BCSD will calculate and calibrate the level of confidence achieved with the suptech tool.
• Machine learning could potentially expand the types of analyses BCSD performs using the same sources of information. The tool could be used to analyze other types of documents BCSD currently collects and processes manually, such as marketing materials. This would require further refinement and advanced programming, based on a large volume of past draft contracts and past analyses feeding the machine learning tool. BCSD has a large database of past compliance analyses, mostly compiled in Excel files. It is difficult to manually extract knowledge from this history of analyses. A machine learning tool could help BCSD optimize the use of such information.

Learnings

• It is important to internally make the case to invest in suptech. The pilot was crucial in moving the process along.
• Managing expectations about what a suptech tool can achieve is important, particularly the machine learning component. It takes effort, data, and time to arrive at the level of confidence supervisors require. 
• Adding machine learning capabilities to the NLP tool opens a host of possibilities but also a range of challenges and requirements. Constant tool monitoring (algorithm governance) is needed and, along with it, staff with the technical expertise and availability to evolve the tool.
• Suptech can greatly enhance supervisory efficiency and efficacy. However, like any tool, it has limitations and will never completely be a substitute for brain power. The tool allows BCSD to allocate months of saved staff time to more complex tasks. It may also contribute to staff motivation and satisfaction by reducing time dedicated to repetitive tasks.
• Since machine learning solutions are complex and highly dependent on representative datasets, it is crucial to manage expectations and prioritize requirements. It is important to consider the weight regulatory requirement analysis has on financial consumer protection, as well as expected gains in operational efficiency and efforts required to develop the tool.

Annex: Banco de Portugal’s Systematic Market Monitoring

• Remote oversight of financial advertisements. The tool enhances observation of which products are being promoted in the market, which product features are highlighted in promotions, marketing strategies, and the intensity of marketing efforts. A data vendor gathers and reports information to BCSD daily, including details on TV ads, newspapers, and outdoor advertising. FSPs are additionally required to submit any remaining marketing materials to BCSD. Analysis is done once information is manually extracted from the Banco de Portugal database. The process has yet to employ automated analytical tools.
• Analysis of draft consumer contracts. The tool helps BCSD ensure compliance with regulatory requirements (mostly disclosure requirements) on terms and conditions, and also on contractual formats and content. It helps BCSD remain up to date on types and models of products in the market, monitor new product launches, identify product development trends, etc. Draft contract analysis plays an important role in conduct supervision as it simplifies assessment of regulatory compliance and can prevent illegal or abusive contractual provisions that harm consumers. Once identified, any such provisions must be removed.
• Analysis of actual consumer credit contracts. On a monthly basis, FSPs must report basic information about actual consumer credit contracts signed by customers. The information is sent in encrypted computer-readable HMTL format to the Banco de Portugal’s database. Analysis is focused on the cost of credit to consumers and compliance with applicable rate caps. In cases where infringement is detected, BCSD may ask the institution to send a copy of the complete actual contract. Currently, such information is not disaggregated by customer demographic characteristics, including gender.
• Monitoring of structured deposit products. Monitoring of each product is based on an analysis of key information documents (KIDs). KIDs are reported to BCSD two days prior to the start of product commercialization and made available on Banco de Portugal’s Bank Customer Website. BCSD also approves any marketing materials related to structured deposits. The focus of the analysis is on disclosures, which are based on national legislation that transposes the European Regulation on key information documents for packaged retail and insurance-based investment products.
• Fee monitoring. Monitoring is based on price lists that FSPs report to BCSD, which contain fees charged for retail banking services, including fees on payment accounts, credit, checks, cards, and other transaction accounts. This tool also monitors other fee disclosures and information sources, such as consumer complaints.

Back to top ↑