Introduction
In this fourth episode of the Backtesting Series, we will get into Portfolio Construction & Portfolio Optimization processes with a deep focus on MVO. If you haven’t already, we recommend reading the first three episodes before proceeding with this article. Please note that the series will include the following parts:
- Part 1: Introduction
- Part 2: Cross-Validation techniques
- Part 3: Factor Models
- Part 4: Portfolio Construction & Portfolio Optimization
- Part 5: Backtest Analysis
- Part 6: Introducing TCs, assessing capacity, preparing for production
Naïve MVO
Let’s take an objective function V of the portfolio weights and of returns. This is usually interpreted as the utility function of an investor taking different values under different realizations of the future. The expected value of the utility function gives the ex-ante value of the bet he would be taking by investing in that portfolio. Assuming the investor has an initial wealth and we know the distribution of returns than the investor needs to solve this problem to get to its optimal portfolio:
The choice of V is not so obvious. Usually, V is set to be monotonically increasing and concave. One approach followed by Markovitz (1959) is to consider a polynomial local approximation of the objective function:
Taking the expectation, we obtain:
We maximize a quadratic objective function which is the weighted sum of expected returns and variance; hence here is where mean-variance portfolio optimization comes from.
Also, when the utility function is quadratic, we can reduce the associated expected utility maximization problem to:
Where is the risk aversion parameter.
The converse is less obvious but still true: if an investor selects investment on the basis of mean and variance only, his utility function is necessarily quadratic [Baron 1977, Johnstone and Lindley 2011].
Mean-Variance Optimal Portfolio
The simplest optimisation problem is to maximise expected PnL subject to a constraint on the maximum tolerable volatility (), stated as:
While another maximizes the Sharpe ratio given by:
To make the problem well-defined, we fix the denominator by setting the volatility constraint:
When simplified, this confirms that maximising the Sharpe ratio is equivalent to solving MVO with a variance constraint. When solving the problem we get:
With the following expected return and Sharpe ratio:
We see that the optimal portfolio is proportional to the investor’s volatility budget : the higher the budget, the bigger the portfolio. Despite this dependency on risk tolerance, the portfolio is independent of the absolute scale of expected returns; replacing
with
gives the same solution. The parameter
is the shadow price of the volatility constraint and can be interpreted as the derivative of the objective function with respect to the variance.
There are multiple equivalent formulations of this same problem, for example we derive the same solution when solving an unconstrained problem with an objective function including a penalty term:
Indeed, when solved reveals that the larger the volatility budget, the smaller the penalty coefficient. This penalty value is the same as the shadow price before – we obtain the same solution when we price out the constraint and we give the variance a unit price equal to the shadow price of that constraint. In another vein, the optimal dollar volatilities are proportional to the Sharpe Ratios, multiplied by the inverse of the correlation matrix.
Shortcoming of Naïve MVO
An investor starts with their estimate for expected returns and covariance matrix and gets their portfolio weights proportional to the two as previously shown. Although their realized Sharpe Ratio will not be a function of neither of those, however it is a function of true expected returns and covariance matrix, ,
. We compare the realized Sharpe Ratio to the best Sharpe ratio:
We call this the Sharpe Ratio Efficiency (SRE). It takes values less or equal to 1 and it is equal to 1 if and only if my estimates are exactly correct [Paleologo 2024].
This is why reducing bias in the estimation of covariance matrix and expected returns is crucial when dealing with mean variance optimization if you are not willing to lose a great deal of performance from inaccurate parameter estimation.
To make you better understand these limitations we bring you the simplest of the examples. Let’s take just two assets with non-negative true Sharpe Ratios and we set their return to have true correlation
. Their true covariance matrix will be:
We can compute the weights of our portfolio as , where
and
are estimates of the covariance matrix and the vector of Sharpe Ratios.
Impact of errors on forecasted Sharpe Ratio
We assume the error between the true and the forecasted Sharpe Ratio is bounded such that: and we solve the problem:
The relative reduction in PnL turns out to be:
This is also the relative loss in Sharpe, since the volatility of the portfolio is unaffected by return forecast error. The figure below shows numerical results for two assets assuming an error and varying level of correlation.
From this it is possible to notice that high correlation increases the loss in Sharpe Ratio for all the values of and
. Of course, the effect is less pronounced for higher Sharpe Ratios as the relative forecasting error gets smaller. Still even in the simplest of the environments, a relatively small
has an impact on realized Sharpe that never gets below 10% and can get well above 50%.
Impact of errors in correlation among assets
We repeat the same analysis as before this time allowing for error in the correlation among assets. We assume the estimation error is bounded: . The error in estimated correlation affects volatility, while the expected returns remain unaffected, and the Sharpe Ratio is minimized when the volatility is maximized. So, to understand the magnitude of errors in correlation among assets on realized Sharpe Ratio we need to solve the problem:
The associated relative loss in Sharpe ratio turns out to be:
The figure below shows numerical results for two assets assuming an error and varying level of correlation.
Again, the magnitude of errors for higher values of correlation even though it always has a smaller impact on realized Sharpe than errors on forecasted Sharpe Ratio. This is because if your estimate is wrong on how good an asset is (the Sharpe ratio), you might over- or under-allocate to it, leading to large deviations in performance. But if you misestimate how assets correlate, the portfolio weights do not change as drastically.
Constraints
The formula we previously introduced is the general starting point for more complex optimisation problems; in practice optimisation formulations differ based on the constraints they must satisfy. These can be investor preferences, implementation/tactical/fiduciary considerations or regulatory challenges. There are many types of constraint, with linear being the most common and taking the form:
(inequality constraints) or
(equality constraints)
These capture a wide range of restrictions because a lot can be rewritten with auxiliary variables, with their final form being linear.
Alternatively, we can model constraints non-linearly, for example as quadratic functions which appear naturally when controlling for risk at a finer resolution than that on total variance. A much less common type of constraint is non-convex which makes solving the problem and finding a global optimal NP-hard, for example having a constraint on the maximum number of assets in a portfolio. However, it is usually preferable to model those directly or not include a constraint at all.
Constraints can also be viewed as penalties on the objective function given a problem:
With optimal solution , there is a
has the same optimal solution. In this formulation, problems can be solved even if strict constraints would make the problem infeasible.
In either formulation, constraints, when implemented into a problem with correct covariance matrix and expected returns estimates, worsen performance. Intuitively, if we shrink the feasible region of our optimisation problem by adding a constraint, a better optimum will not be achieved. If we take estimation error into account, however, constraints may help.
The importance of covariance specifications within MVO
As discussed in the previous sections, reducing the estimation error in the variance-covariance matrix (VCV) is crucial for investors. Using the simple VCV sample estimator for portfolio optimization is especially problematic in the case of large asset universes as these imply high concentration ratios (that is, the number of assets over the number of observations). This often leads to (near) singularity of the sample estimator which in turn then makes those optimization methods infeasible which require the inversion of the VCV matrix to determine their allocations. Even in cases where the sample VCV matrix is non-singular, the estimator is known for producing highly unstable portfolios with poor out-of-sample performance and extreme portfolio weights [9]. One of the main reasons for this can be explained by transforming the problem into the principal component space. For this, note that the VCV matrix can be decomposed into a correlation matrix and the diagonal matrix of asset volatilities
[12]. By conducting an eigen-decomposition of the correlation matrix, the principal components (PC) can be obtained being scaled such that the sum of the square weights is 1. With each PC representing a portfolio, its expected and realized returns as well as variance can be determined. By definition, the first PC represents the riskiest portfolio whilst the last PC is the portfolio with the smallest possible variance among all portfolios (although not necessarily the one with the smallest expected return). When applying MVO now to the universe of the PC portfolios, given that the PCs are by definition uncorrelated, the portfolio allocation to each PC is proportional to the PC’s Sharpe ratio [12]. Any error in the estimation of the initial variance-covariance matrix will most likely lead to an underestimation of the real risk of the PCs with the lowest variances, whilst any error in the estimation of the expected return of these portfolios will be relatively large compared to their estimated risk levels leading to overstated expected Sharpe ratios for the PCs with the lowest eigenvalues [12]. As a consequence, MVO tends to allocate inflated weights to these portfolios leading to poor ex-post results. To mitigate this issue, one can increase the estimated volatility of the problematic PC portfolios by shrinking the estimated correlation matrix of the underlying assets towards the identity matrix via
where
is the parameter determining the shrinkage extent [12]. Plugging the new correlation matrix back into the VCV decomposition results in a shrunk variance-covariance matrix
which can then be used as the input for portfolio optimization leading to more robust ex-post outcomes.
Empirical benchmarking of different covariance specifications
Whilst the optimization results of an unconstrained MVO can be improved using shrinkage methods and other alternatives to the sample estimate when modelling covariance dynamics, the vast majority of portfolio managers does not trade in an unconstrained setting. Instead, most face leverage, shorting and concentration constraints as well as significant transaction costs which make too frequent rebalancing not worthwhile. Empirical studies looking at constrained minimum-variance portfolios (the portfolios at the vertex of the optimal frontier in MVO) of 1000 US stocks from a practical point of view find a significantly reduced opportunity for achieving outperformance using alternative VCV estimators instead of the simple sample estimator [2]. This phenomenon can be mainly explained by concentration constraints and transaction costs introducing penalties to the optimization which have very similar effects to that of common shrinking techniques (constraint implicit shrinkage). Nevertheless, portfolio volatility can still be significantly reduced by accounting for the time-series dynamics in asset returns [2]. Covariance approaches benchmarked include the linear and nonlinear shrinkage estimators by Ledoit and Wolf [7][8] as well as RiskMetrics’ exponentially weighted moving average smooting [13]. Additionally, dynamic conditional correlation models which allow for time-varying correlations between assets and are thus able to capture time-series dependence of returns were also considered [3][4]. Leverage ratios and turnover rates appear to be especially sensitive to covariance specifications with gross exposures varying by up to 1100 ppt (percentage points) between different estimators in an unconstrained base case and turnover rates by up to 655 ppt [2]. As one introduces more and more constraints to the optimization model, resulting portfolio leverage levels and turnover rates greatly converge between estimators. Looking at ex-post portfolio volatility as a measure of estimator performance, a similar behaviour can be detected with the range of resulting optimised ex-post annualized portfolio volatilities dropping from 7.4 ppt in the unconstrained case to 0.5 ppt after weight- and long-only constraints as well as transaction cost penalties where imposed [2]. Using ex-post portfolio volatility as a measure for assessing the performance of a variance-covariance estimator follows logically given the model’s ex-ante goal of minimizing volatility when considering the minimum-variance portfolio in MVO to simplify comparison. A similar behaviour can be noted when examining the resulting net Sharpe ratios of the portfolios [2]. Therefore, whilst in the unconstrained case alternative VCV dynamics do indeed outperform (especially DCC and nonlinear shrinkage), there is clear evidence, that the level of improvement due to shrinkage estimators significantly declines under more realistic circumstances as the sample estimator benefits greatly from the implicit constraint driven shrinkage. However, even in the constrained scenario, accounting for time-series dependencies between asset returns still remains essential as nonlinear, dynamic covariance estimators consistently outperform their peers. The empirical findings suggest that practical, regulatory constraints in portfolio optimisation in the real world naturally mitigate some of the deficiencies of naive covariance estimation in MVO, thereby partially reducing the need for complex estimation techniques. Furthermore, whilst there are more sophisticated models for the VCV dynamics which outperform the simple sample estimator, their usage should be informed by benchmarking the models in realistic portfolio settings resembling those of the specific investment mandate in place.
The following graphs summarize the results of the empirical estimator benchmarking conducted by Dom et al. 2025, focusing on an asset universe of 1000 US stocks with an out-of-sample period of 1995 to 2021[2]:
The three optimization settings are GMV UNC (unconstrained global minimum-variance portfolio), GMV LO (global minimum variance portfolio with long-only constraints) and GMV CON (global minimum-variance portfolio with long-only and maximum weight constraints as well as asset specific transaction cost penalties)[2]. The examined estimators include: Sample (sample VCV estimator), LS (linear shrinkage towards a scalar multiple of the identity matrix), NLS (nonlinear, quadratic shrinkage of the inverse eigenvalues)[8], RM (RiskMetrics smoothed estimator)[13], RM-NLS (RiskMetrics estimator with nonlinear shrinkage) and DCC-NLS (dynamic conditional correlation model with nonlinear shrinkage)[4][2]. As described previously, the findings show a clear convergence of estimator performance as additional constraints are imposed with the DCC-NLS estimator consistently performing well in terms of ex-post portfolio volatility and net Sharpe ratio.
Multi-Period Optimization (MPO)
A natural yet complex extension to the models seen so far tackles the issue that they all present: investment decisions are optimized period by period, without accounting for the dynamic nature of the investment process. As Munk [10] explains, single-period optimization treats portfolio selection as a static decision made at a single point in time failing to account for the sequential nature of investment decisions. This can still be optimal in cases with limited market frictions but becomes an issue in more realistic scenarios. Isichenko [6], for instance, explains how SPO accounts for the costs of entering an illiquid position but not the cost of moving out of it.
The following sections will explain the theoretical framework of the problem, a non-comprehensive list of solutions to the optimization problem and some practical considerations on the implementation of the model with possible limitations.
The Framework
Following the notation of Boyd et al. [1], we define a planning horizon . Clearly, at time
, many quantities in the future are unknown. For any quantity
, we mark it’s forecast at future period
done at time
as
. Our objective is the maximization of the following risk adjusted return:
:
where:
is our estimate at time t of the return vector at time τ
represents the portfolio weights at time τ
represents the trades at time τ
is the risk penalty
is the estimated holding cost
is the estimated transaction cost
is the set of allowable trades and
is the set of allowable porfolios.
Mathematically, this is equivalent to the discrete version of a Hamilton-Jacobi-Bellman (HJB) equation that characterizes optimal investment strategies in continuous time.
If we examine the problem from an investor’s wealth perspective, we have that the change in wealth from to
is:
With being the amount invested in the risk-free asset,
the risk-free rate and
the vector of amounts in the risky assets on their associated returns, while
representing income and consumption respectively. Isichenko [6] provides a useful insight on expressing multi-period utility as:
where represents the forecast of returns as a function of time horizon,
captures the market impact, and
represents the risk penalty.
Solving the Optimization problem
Theoretically, one can find a solution to the multi-period problem using dynamic programming:
where is the indirect utility function at time
,
is the utility from consumption, and
is the subjective discount rate. In continuous time (i.e., as
) this becomes the HJB equation:
Yet, this solution is generally infeasible practically, due to the “curse of dimensionality” [1]. A more practical approach is described by Boyd as model predictive control (MPC) which involves planning trades for multiple periods and then executing only the first period’s, recalculating next ones with the new available info. In Math terms:
subject to:
Practically we only implement and then repeat the process for the next time step.
An alternative approach, described by Isichenko [6], presents a solution for when trading happens more slowly than impact decay:
With being the ratio of impact decay. The optimality condition then becomes:
Whose solution is: Intuitively, the formula shows that the optimal trading rate combines a mean-reversion term and an exponentially weighted average of future forecasted returns. More liquid assets (lower μ) can be traded more aggressively on shorter-term signals, while less liquid assets should only be traded on longer-term signals.
Practical implementation and issues
Complexity: Although the complexity of the MPO problem can be programmed to scale in O(H)[5], back testing can still be computationally expensive for more than a few tens of periods at a time when combined with other models. Overall, it can also be infeasible to be used in high-frequency trading scenarios, given the overall complexity of the optimization problem being solved.
Forecast Revision: Another issue [6] is that of forecast revision. Indeed, every period new information becomes available, possibly changing the forecast of returns for the assets and, consequently, the optimal positioning. Isichenko proposes 3 possible ways to tackle the problem:
- Open-loop optimization creates the initial positions but doesn’t account for future forecast revisions
- Semi-open-loop optimization dynamically incorporates revised forecasts through a predefined “plan”
- Closed-loop optimization explicitly incorporates new revision in the optimization problem
Closed-loop can be formulated as:
subject to:
With being the system state (i.e., portfolio positions, forecasts, impact etc…) and
being the controls (i.e., trades).
Factor Risk: To account for factor risks, it’s possible to use an iterative approach [6] where we use risk adjusted forecasts, alternating between optimizing each asset independently and updating risk adjustments of the overall portfolio. Mathematically:
Model Uncertainty: As the forecast horizon extends, prediction accuracy typically degrades, potentially undermining the benefits of the multi-period approach. The quality of these longer-term forecasts becomes a critical factor in determining whether MPO outperforms simpler methods in practice. In particular, as Paleologo [11] notes, estimation errors can compound over multiple periods, significantly impacting overall performance.
Conclusions
In this fourth episode of the Backtesting Series we analysed the complexity of Portfolio Construction and Portfolio Optimization starting from Naïve MVO highlighting the key role of expected returns and the covariance matrix in determining optimal portfolio weights. Indeed, we showed how crucial it is to carefully estimate this inputs to do not lose performance. Shrinkage methods and dynamic conditional correlation models emerged as valuable tools for improving covariance matrix estimation, especially in high-dimensional settings where sample estimators tend to underperform. Finally, to face also a more realistic framework we proposed also a framework for Multi-Period Optimization for dynamic portfolio construction, acknowledging the challenges of forecast revision, transaction costs, and factor risks in a multi-period setting.
Stay tuned for the next episode of the series that will be about Backtest Analysis.
References
[1] Boyd, S. et al. “Multi-Period Trading via Convex Optimization”, 2017
[2] Dom, M.S. et al. “Beyond GMV: the relevance of covariance matrix estimation for risk-based portfolio construction”, Quantitative Finance, March 2025, pp.1-17, 2025
[3] Engle, R. “Dynamic conditional correlation: A simple class of multivariate generalized autoregressive conditional heteroskedasticity models”, Journal of Business & Economic Statistics, 20(3), pp.339-350, 2002
[4] Engle, R. et al. “Large dynamic covariance matrices”, Journal of Business & Economic Statistics, 37(2), pp.363-375, 2019
[5] Grinold, Khan, “Active Portfolio Management”, 2023
[6] Isichenko, Michael, “Quantitative Portfolio Management”, 2021
[7] Ledoit, O. and Wolf, M. “A well-conditioned estimator for large-dimensional covariance matrices”, Journal of Multivariate Analysis, 88(2), pp.365-411, 2004
[8] Ledoit, O. and Wolf, M. “Markowitz portfolios under transaction costs”, Working Paper Series, 2024
[9] Michaud, R.O. “The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal?”, Financial Analysts Journal, 45(1), pp.31-42, 1989
[10] Munk, Claus, “Dynamic Asset Allocation”, 2017
[11] Paleologo, Giuseppe, “Elements of Quantitative Investing”, 2024
[12] Pedersen, L.H. et al. “Enhanced Portfolio Optimization”, Financial Analysts Journal, 77(2), pp.124-151, 2021
[13] RiskMetrics, RiskMetrics – Technical Document, 1996
0 Comments