When does it end? Monte Carlo Simulation Applied to Risk Management in Defense Logistics’ Procurement Processes
Abel de Castro Laudares
abel.lau@hotmail.com
University of the Air Force – UNIFA, Rio de Janeiro, Rio de Janeiro, Brazil
Maria Filomena Fontes Ricco
filricco@gmail.com
University of the Air Force – UNIFA, Rio de Janeiro, Rio de Janeiro, Brazil
Rodrigo Antônio Silveira dos Santos
rsilveira01@gmail.com
University of the Air Force – UNIFA, Rio de Janeiro, Rio de Janeiro, Brazil
ABSTRACT
Goal: The main objective of this research was to evaluate the application of the Monte Carlo simulation, in comparison with other estimation methodology, in order to verify if it would be possible to apply this tool to improve the risk perception of deadlines during a procurement process for logistical support of defense projects.
Methodology: By means of collected data from 2015 to 2017, a comparison was conducted among deadlines of real procurement processes versus simulated processes with the use of the Monte Carlo simulation.
Results: The performance analysis of the Monte Carlo simulation suggests that it could be possible to apply it to the first phase of risk management in order to improve the risk perception of deadlines during a procurement process for logistical support of defense projects.
Limitations of the investigation: The main limitation of the research relates to the database analyzed, which involved three different modalities of Bidding Processes.
Practical implications: The data from this research reveals that the Monte Carlo simulation can be used as an effective tool for advising the decision maker and, mainly, the managers of internal control about aspects of corporate risk management in public organizations.
Originality/Value: This research presents an original contribution to corporate governance related to risk management, which is not commonly seen in the Brazilian public sector.
Keywords: Logistics; Risk Management; Monte Carlo Simulation.
1. INTRODUCTION
When the quantity of projects managed by the Brazilian Air Force (FAB, acronym in Portuguese) increases, the number of procurement processes increases as well. Moreover, the amount of time involved in these procurement processes has also increased in recent years, causing a slowdown from the formalization of a need (object request) to the receipt of this acquisition. This slowdown causes a negative impact on the public administration and, consequently, an increase of the uncertainty throughout the process of public procurement (Araújo et Jesus, 2013).
In this context, achieving organizational agility and control is very important in the processes of stock replenishment for logistical support, since any delay would incur, for example, a lack of items to the Aeronautics Material Park and Logistical Groups. That would consequently lead to unavailability of aircrafts and equipment and, thus, compromise the fulfillment of the Brazilian Air Force (FAB) mission.
Corporate Risk Management faces a lack of specific approaches and tools that enable its implementation, since the standards and models of governance processes do not give the same focus to the Management of Corporate Risks that the risk management applied to Projects Management provides (Etges, 2015).
For this reason, some questions about the possibility of using the Monte Carlo simulation in the government bidding processes arise, since this simulation has already been used as a tool for the quantitative risk analysis involving deadlines in the project management area. Similarly, bidding processes, according to Brazilian Law 8.666/93 (Brasil, 1993), have their steps well defined with deadlines that are likely to be estimated.
Then, would it be possible to apply a methodology for quantitative risk analysis in order to improve risk perception involving deadlines during a process of acquiring items focused on logistical support? How is an approach supposed to be formulated in order to forecast deadlines and measure risks involving procurement processes focused on the logistical support of defense projects by means of the Monte Carlo simulation?
For this purpose, three modalities of bidding were selected (Ineligibility for Tender, Electronic Reverse Auction and Request for Quotes) for the simulation. The results obtained in the applicability of the Monte Carlo simulation, through the @RISK software, were compared to the real procurement data for logistical support items, managed by one of the main logistics acquisition centers of the COMAER. Data collection involved procurement processes that occurred between 2015 and 2017.
2. LITERATURE REVIEW
The theoretical framework that provides a grounding base to this article is related to impacts of procurement on governance and risk management, Monte Carlo simulation and public procurements by means of bidding processes.
2.1 Risk management
Risk is linked to the exposure to change. It is the probability that some future event or set of events will occur. It differs from uncertainty, which is related to a set of known variables about an event. Colloquially it could be said that the risk is a known uncertainty and that uncertainty is an unknown risk. Therefore, the risk analysis involves the identification of adverse potential change and, as a result, the expected impact on the organization (Damodaran, 2009).
Risk refers to a threat that, in the event it becomes a reality, may adversely impact a project, either in terms of cost, term, quality or another dimension (Terribini Filho, 2011). A risk is characterized by its likelihood of occurrence and its impact on the objectives of the project. Risk, from a project management perspective, is an event or an uncertain condition that can influence at least one project goal if it occurs (Project Management Institute, 2013).
Many risks can be classified in different contexts. In an external context, it includes all the external environmental parameters and factors that influence how it manages risk and how it tries to achieve its objectives. Internal context includes all the internal environmental parameters and factors such as internal stakeholders, approach to governance and others that can be reduced with the adoption of mitigating measures (Instituto Brasileiro de Governança Corporativa, 2007; Cleland and Ireland, 2007; Fortes, 2011).
2.2 Monte Carlo simulation
Simulation is the representation of a real system by means of utilizing relevant information about it in order to analyze its behavior. It allows an understanding of what would happen through certain changes to minimize cost and time without the need of modifying for construction or modification of the actual system (Beneti, 2016).
According to Prado (2014), simulation is the technique that makes it possible to imitate the functioning of a real system through the use of a computer by means of simulation software. Thus, it has several applications in the present day on a variety of areas, ranging from a toll, the functioning of a bank, a port, the production in a manufacturer up to the movement of papers in an office.
Monte Carlo simulation specifically uses random number generation to assign values to the system variables one wants to investigate. Currently, random numbers can be generated from several computational methods, such as a spreadsheet or a specific random number generating software. Simulation is then replicated many times, until there is safety over the characteristic behavior of the decision-making variable on which to decide (Palisade, 2015).
It is an efficient tool in the generation of estimates, and therefore it is highly recommended for the evaluation of uncertainty. The reason is that it can provide a series of valuable information generated from various combinations of events established as input variables in the simulation, as well as their respective results within a project. (Abreu et Amorim, 2017).
2.3 Public procurements by means of bidding processes
Brazilian Law No. 8,666, of June 21, 1993, regulates the art. 37, section XXI, of the Federal Constitution, and establishes rules for bidding and procurement of public administration and other matters related to administrative works and services, including advertising, purchasing, sales and leases under the authority of Federal, State, Federal District and Municipalities. All contracts with third parties will necessarily be preceded by bidding, except for the assumptions provided in this law.
The acquisition process has a strategic role for any institution or organization to reach its goals. Purchases directly interfere with the quality and capacity of delivering essential products or services, which will meet the demand of those who enjoy profit from the public service (Cavalcanti et Martens, 2013).
Depending on how the procurement process was conducted in the private sector, acquisition of raw materials, supplies and components can generate cost reduction and considerable improvements in profits. Public corporations have the same objective, although they do not seek profit as a goal; the proper public expenditure management is fundamental to reach the public interest (Tashima et Gomes, 2016).
The bidding process is intended to ensure compliance with the constitutional principle of isonomy or equality of rights (impossibility of unequal treatment) and to select the most advantageous proposal for the administration, that is, the one that best meets the service interest in an objective way (Tribunal de Contas da União, 2010; Fonseca, 2013).
Brazilian Law No. 8.666/93 also states that every bidding process must follow certain procedures, according to the contract object, that are termed as bidding modalities. Bidding modality is the specific way of conducting the bidding procedure, from criteria defined in law. It is the estimated value for hiring the main factor for choosing the bidding modality, except when it comes to electronic bidding, which is not limited by values (Tribunal de Contas da União, 2006).
Making a contract with third parties in the public administration must necessarily be preceded by bidding, except for the possibility of dismissal and non-requirement of bidding (Tribunal de Contas da União, 2006). These are exceptional situations that must be fundamentally justified considering the limits imposed on such discretion to the public administrator, which may be penalized in the form of the Brazilian Law (Oliveira et al., 2017).
3. METHODOLOGICAL ASSUMPTIONS
This research initiated through the collection of data in one of the main acquisition centers of the COMAER, which was responsible, until the year 2017, for the logistical activities of demand, purchase, receipt, distribution of material and hiring of services necessary to the logistical support of several FAB’s projects.
Three different bidding modalities were selected, and a future scenario simulation was carried out for each one. A future scenario simulation was carried out related to the deadline for the process to be completed. A beginning and an end for analysis of the bidding process were assumed, which was from the phase of elaboration of requirements (shopping list) to the phase of contracting.
Subsequently, the results were compared to the data of processes already completed considering the same class of material acquired to verify the performance of the simulation regarding the probability of the process time, as well as to identify and classify the risks of this same process.
4. SELECTING ITEMS FOR ANALYSIS
From data obtained in the procurements department, the procurement of an item was simulated by means of the following modalities: ineligibility for tender, Electronic Reverse Auction and request for quotations, according to Brazilian Law No. 8.666/93 (Brasil, 1993) and Law No. 10.520/02 (Brasil, 2002).
For each bidding modalities analyzed, the analysis was evaluated with the help of the managers responsible for the procurements by means of an unstructured interview—all the phases prescribed by law for the realization process. Additionally, in order to enter the uncertainties of each phase of the processes, an estimate was obtained with these managers based on their experiences: optimistic, pessimistic and a more probable for each phase of the process in order to feed the simulation.
4.1 Using the @RISK software
The computational simulation, through @RISK software, was developed following three steps executed separately for each of the bidding modalities.
The @RISK software used is available on the website's URL http://www.palisade-br.com/risk/website, in its cheaper student version, with a subscription of one year; however, it maintains the same features of the Standard, Professional and Industrial versions.
In addition, the @RISK software includes a Top Rank analysis for spreadsheet that is a tool for performing quantitative risk assessments that evaluates the relationships between process parameters to compare the relative importance of variables, allowing the test of sensitivity/risk associated to uncertainty/variable.
4.2 Step 1: Definition of Uncertainty
Initially, it was necessary to define where there was uncertainty throughout each process' step. In the analyzed cases, uncertainty was considered in all bidding process’ steps. Thus, in order to estimate these uncertainties and to have an estimate beyond the perception of managers, a distribution of PERT probability was carried out using three weighted estimates based on the data, which will behave within a curve of Odds.
According to Santos (2014), the application of the Project Evaluation and Review Technique (PERT) is the most commonly used method in project timeline estimates, as it is a model based on mathematical analysis that uses the well-known expected value (or weighted average) to define the duration of the project.
Equation 1 - PERT.
Expected Time = P+4R+O / 6
In which,
O: Optimistic Time, R: Realistic Time, and P: Pessimistic Time.
4.2 Step 2: Adding Output and Configuring Simulation
Once you have defined the uncertainty throughout the process and entered the probability distribution scores for each activity, you must configure the simulation parameters in the @RISK software. A simulation was performed running 10,000 rotated iterations and then the possibility for the software to add outputs was chosen.
4.3 Step 3: Adding Output and Configuring Simulation
A simulation was performed for each of the three selected bidding modalities. For the electronic reverse auction mode (Chart 1) it was possible to observe that the deadline considered most likely by the managers of 261 days was short, but very close, to the estimate based on the results of the PERT probability distribution—265 days.
Chart 1. Definition of estimates for electronic reverse auction.
Source: The authors.
Regarding the Monte Carlo simulation applied to each bidding modalities, in the electronic reverse auction, 90% probability of occurrence was obtained, with an estimate of 306 days and standard deviation of 9 days (Figure 1).
Figure 1. Cumulative upward simulation score for electronic reverse auction.
Source: The authors.
Concerning the modality of ineligibility for tender, it was possible to observe an estimate based on the managers’ experience of 234 days and an estimate based on the results of the PERT probability distribution of 233 days. Regarding the Monte Carlo simulation, with a 90% probability of occurrence, the estimate obtained was 279 days with a standard deviation of 11 days.
For the request for quotes modality the estimates obtained were 246 days based on the experience of the managers, of 253 days based on the results of the distribution of PERT probability and 303 days, with a standard deviation of 11 days, based on the Monte Carlo simulation, with a 90% probability of occurrence.
5. NUMERICAL COMPARISON
In order to respond to research issues and verify whether it is possible to apply a methodology for quantitative risk analysis, typically used in simulation of future scenarios involving project schedule on procurement processes for support Logistics of defense projects, a sample of 21 procurements was used. All of them already finished and thus separated: already completed, ten of the Electronic Reverse Auction, seven of ineligibility for tender and four of request for quotes in order to compare the results obtained in the simulation with the real duration of the processes.
The 21 real processes used were chosen for convenience among the 60 processes that had items with direct applicability in defense projects as object of contract items with direct applicability in defense projects. In case of delay in the completion of these processes, disorder and operational impact would possibly occur. The processes analyzed were initiated and completed between 2015 and 2017. Finally, those that, according to managers, were considered points outside the curve, exceptions to the normal management standards, were excluded.
The results obtained are shown in Chart 2, in which, concerning the three modalities of procurement, one can observe the most probable duration estimate made by the managers, the PERT estimates, and the Monte Carlo simulation, as well as the real duration of the 21 cases.
Chart 2. Comparing Simulation Results Versus Real Cases
Source: The authors.
The estimated number of days of real cases was measured in the bidding process starting by the phase “Drafting requirements” to the phase “Agreement formalization”. As regards the sequence of the actual cases, as listed in Table 2, they were placed in order from the oldest case to the most recent case.
Comparison between estimates and actual duration of the real cases can be best viewed from figures 2 to 4. It is possible to observe a better approximation between the Monte Carlo simulation, considering the standard deviation calculated, versus average cases, which corroborates what the literature says about the efficiency of the Monte Carlo simulation in representing a real system.
Figure 2. Comparison between estimates and real cases of Electronic Reverse Auction.
Source: The authors.
Figure 3. Comparison between estimates and real cases of Ineligibility for tender.
Source: The authors.
Figure 4. Comparison between estimates and real cases of Request for quotes.
Source: The authors.
6. CONCLUSION
With respect to the first phase of risk management, in the sense of risk identification to enable the following phases of analysis and mitigation, this study intended to be a first approach to the subject, applied to the procurement processes for the logistical support of defense projects.
With the purpose of investigating the forecast of deadlines that allow the measurement of risks and its intended mitigation, this research made use of the Monte Carlo simulation, which is widely used in other areas for the same purpose.
While the Monte Carlo simulation is used for risk management in other areas, especially in project management, there does not seem to be many studies addressing the application of this simulation to the effects of procurement processes in Corporate Governance.
According to the context in which the data was obtained, it was found that the Monte Carlo simulation, carried out by the @RISK software, behaved as an effective tool for advising the decision maker and, mainly, the managers of internal control about aspects of corporate risk management in public organizations.
Furthermore, according to the present results, it seems to be possible to apply a methodology for quantitative analysis of risks, typically used in simulation of future scenarios involving project schedule in order to improve the perception of risk involving deadlines during a process of acquiring items focused on logistical support of defense items.
REFERENCES
Abreu, P. H. C.; F. R. Amorim. (2017), Gerenciamento dos Riscos em Projetos de Software, Revista Interface Tecnológica, Vol. 14, No. 1, pp. 19.
Araújo, P. M. C.; Jesus, R. G. (2013), Processo licitatório tipo menor preço e eficiência em compras públicas: um estudo de caso. In: Encontro da ANPAD, 37, Rio de Janeiro. Anais... Rio de Janeiro: ANPAD, 2013. p. 1-16.
Beneti, J. L. (2016), Análise de risco em prazos e custos aplicado à indústria de confecção utilizando simulação de Monte Carlo. Dissertation, Pontifícia Universidade Católica de Goiás, Goiânia.
Brasil. Lei nº 10.520, de 17 de julho de (2002), Institui a modalidade de licitação pregão no âmbito da União, Estados, Distrito Federal e Municípios, nos termos do artigo 37 inciso XXI da CF. Brasília: Presidência da República, 2002.
Brasil. Lei nº 8.666, de 21 de junho de (1993), Regulamenta o art. 37, inciso XXI, da Constituição Federal, Diário Oficial [da] República Federativa do Brasil, Poder Executivo, Brasília, DF, June 22, 1993. Seção 1, p. 8269.
Cavalcanti, A. C. F.; Martens, C. D. P. (2013), Gerenciamento de Risco nas aquisições públicas: uma análise empírica de um hospital da rede sentinela. In: Encontro da ANPAD, 37, 2013, Rio de Janeiro. Anais... Rio de Janeiro: ANPAD, p. 1-16.
Cleland, D. I.; Ireland, L. R. (2007), Gerência de Projetos, 2. ed., LTC, Rio de Janeiro.
Damodaran, A. (2009), Gestão estratégica do risco: uma referência para a tomada de riscos empresariais, Bookman, Porto Alegre, RS.
Etges, A. P. B. S. (2015), Análise do impacto corporativo de riscos a partir de um modelo de gestão de riscos orientado a ambientes inovadores. Dissertation, Universidade Federal do Rio Grande do Sul, Porto Alegre.
Fonseca, R. A. (2013), O regime diferenciado de contratação e a governança pública no Brasil. Dissertation, Universidade Federal de Lavras, Lavras.
Fortes, F. S. D. (2011), Influência do gerenciamento de risco no processo decisório: análise de casos. Dissertation, Universidade de São Paulo, São Paulo.
Instituto Brasileiro de Governança Corporativa (IBGC) (2007), Guia de orientação para gerenciamento de riscos corporativos, IBGC, São Paulo.
Oliveira, C. F. M., From, D. A.; Selow, M. L. C. (2017), Contratação direta na administração pública, através dos processos de dispensa e inexigibilidade de licitação. Vitrine de Produção Acadêmica, Vol. 4, No. 1, pp. 122-133.
Palisade. Simulação de Monte Carlo, (2015), Available at http://www.palisade-br.com/risk/monte_carlo_ simulation.asp. (accessed date March 10, 2018).
Prado, D. S. (2014), Teoria das filas e da simulação, 5. ed., Falconi, Nova Lima.
Project Management Institute (PMI). (2013), A guide to the project management body of knowledge (PMBOK guide), 5. ed., PMI, Newton Square.
Santos, R. R. (2014), A utilização da rede PERT do projeto para elaboração do cronograma e da rede de atividades, Revista Especialize On-line IPOG, Vol. 1, No. 9, pp. 1-14. Available at https://www.ipog.edu.br/revista-especialize-online/edicao-n8-2014/?setarParametros=true&pagingPage=20& (accessed date March 13, 2018).
Tashima, L. C. N.; Gomes, J. C. (2016), Gestão de compras na administração pública, REPAE, Vol. 2, No. 1, pp. 92-108.
Terribini Filho, A. (2011), Gerenciamento de projetos em 7 passos: uma abordagem prática, M. Books, São Paulo.
Tribunal de Contas da União (TCU) (2006), Licitações e Contratos: Orientações básicas. 3. ed. Brasília: TCU, Secretaria de Fiscalização de Obras e Patrimônio da União.
Tribunal de Contas da União (TCU) (2010), Licitações e contratos: orientações básicas. 4. ed., Brasília: TCU, Senado Federal.
Received: 16 Dec 2018
Approved: 28 Dec 2018
DOI: 10.14488/BJOPM.2019.v16.n1.a14
How to cite: Laudares, A. C.; Ricco, M. F. F.; Silveira dos Santos, R. A. (2019), “When does it end? Monte Carlo Simulation Applied to Risk Management in Defense Logistics’ Procurement Processes”, Brazilian Journal of Operations & Production Management, Vol. 16, No. 1, pp. 149-156, available from: https://bjopm.emnuvens.com.br/bjopm/article/view/739 (access year month day).