In this post, you will read:
An introduction to key concepts in portfolio optimization, including assets, returns, weights, and covariance
A discussion of different portfolio optimization strategies, such as Mean-Variance Optimization (MVO), Global Minimum Variance Portfolio (GMVP), Equal-Weighted Portfolio (EWP), Inverse-Volatility Portfolio (IVP), and Most Diversified Portfolio (MDP)
An explanation of the importance of accurate covariance matrix estimation and the role of shrinkage methods in improving portfolio optimization outcomes
Please note all code and complete QuantJourney Framework is available to Paid subscribers. Please subscribe to support my work.
Introduction
Portfolio optimization is a quantitative technique that aims to determine the optimal allocation of assets within a portfolio. The goal is to find the asset distribution that maximizes the portfolio's expected return while minimizing its risk. This process involves analyzing the performance, correlations, and risk contributions of individual assets to construct a portfolio that aligns with our objectives and risk tolerance. Mathematically, portfolio optimization can be formulated as an optimization problem, where the objective is to maximize a desirable factor, such as expected return, while minimizing an undesirable factor, such as portfolio risk or volatility. This is subject to various constraints, such as budget limitations, asset allocation bounds, and investment policies. So, the optimization process considers the trade-off between expected returns and risk, seeking to find the most efficient allocation that balances the two.
Introduction to key concepts before we jump into the code.
Assets and Their Returns:
Each asset in a portfolio has an expected return, denoted as $R_i$ for asset $i$.
The portfolio contains $n$ different assets.
Asset Weights in the Portfolio:
Each asset has a weight, $\omega_i$ representing the proportion of the portfolio invested in asset $i$
The sum of all weights in the portfolio must equal 1, ensuring the entire budget is allocated (not always a case)
Covariance Between Asset Returns:
Covariance, $σ_{ij}$, measures how the returns of two assets, $i$ and $j$, move together. It's a key component in understanding portfolio risk.
So, we denote it all as:
So, the portfolio's expected return is a weighted sum of the individual expected returns:
and portfolio risk is assessed by its variance, denoted as $\sigma_p^2$. It's calculated using the weights and covariances of all asset pairs:
where $w_i$ and $w_j$ are the weights of assets $i$ and $j$ in the portfolio, and $\sigma_{ij}$ is the covariance between the returns of assets $i$ and $j$.
Formulating the Optimization Problem:
The goal is to find a set of weights that minimize the portfolio's risk. So, the optimization problem can be formulated as:
So, the solution to this optimization problem will provide the set of weights $\{ w_1, w_2, .. , w_n \}$ of the assets in the portfolio that minimize the portfolio risk $σ^2_p$ while satisfying the given constraints.
Let’s write the same using matrix notation for simplicity and efficiency. We define the weight vector $\omega$ as $\omega = [ w_1, w_2, .. , w_n]^T$, where $T$ denotes the transpose of the vector. This vector represents the distribution of weights across the different assets in the portfolio. We also have a covariance matrix $\Sigma$ for the portfolio. And portfolio variance as the expression $\omega^T \Sigma \omega$.
The objective is to minimise variance subject to the constraints that $\omega^T\mu = p$ (target return) and the sum of $\omega_i$ equals to 1. We usually denote this as:
With Lagrange multiplier, the objective function becomes:
where $\mu$ is the return of each portfolio component.
And unknown parameters such as $\Sigma$ (covariance matrix) and $\mu$ (mean of component stocks) are unknown and need to be estimated to optimize $\omega$.
Quadratic utility function
While the Lagrange multiplier approach focuses on minimizing variance subject to a return constraint, for more active investors who have specific return targets, we approach that from utility theory and use a quadratic utility function incorporating risk aversion.
So, we have the quadratic utility function as:
whereλis the investor’s risk aversion coefficient, and$w$is the portfolio weights vector, andμand Σ are the mean return vector and the covariance matrix, respectively.
We aim to maximize $UU$ with respect to $ω$. This meansdU/dw=0 (1st order condition) and$d^2U/dw^2<0$ (2nd order condition). These yields optimized weights:
Both approaches require estimates of the mean and covariance matrix of returns, but they use these estimates differently in the optimization process. In the utility theory approach, the investor's risk aversion plays a central role, whereas in the Lagrange multiplier approach, it is more about balancing risk (variance) and return.
In this post we explore different strategies of portfolio optimization, including Mean-Variance Optimization (MVO), Global Minimum Variance Portfolio (GMVP), Equal-Weighted Portfolio (EWP), Inverse-Volatility Portfolio (IVP), and Most Diversified Portfolio (MDP):
Mean-Variance Optimization (MVO):
MVO aims to find the optimal asset weights that maximize the portfolio's expected return for a given level of risk.
It relies on the estimation of expected returns and the covariance matrix of asset returns.
MVO assumes that investors are risk-averse and prefer portfolios with higher expected returns and lower volatility.
Global Minimum Variance Portfolio (GMVP):
GMVP seeks to minimize the overall portfolio variance, regardless of the expected returns.
It focuses solely on reducing portfolio risk without explicitly considering returns.
GMVP is useful when reliable estimates of expected returns are challenging to obtain or when the primary objective is risk minimization.
Equal-Weighted Portfolio (EWP):
EWP assigns equal weights to all assets in the portfolio, resulting in a 1/N allocation.
It is a simple and intuitive approach that does not rely on optimization or estimation of parameters.
EWP provides a benchmark for comparing the performance of more sophisticated optimization strategies.
Inverse-Volatility Portfolio (IVP):
IVP assigns weights to assets inversely proportional to their volatilities.
Assets with lower volatilities receive higher weights, while those with higher volatilities receive lower weights.
IVP aims to allocate more capital to less volatile assets, potentially reducing overall portfolio risk.
Most Diversified Portfolio (MDP):
MDP seeks to maximize the diversification ratio, which measures the ratio of the portfolio's weighted average volatility to its overall volatility.
It aims to create a portfolio that is as diversified as possible, with weights allocated to minimize the impact of correlated risks.
MDP is particularly useful when dealing with a large number of assets and complex correlation structures.
Please see the results we will create later with the code:
As you can see, we have returns per portfolio of below stocks in 2020-2022, with daily rebalance. In fact in code, you can state ‘monthly’ rebalance or any frequency rebalance to see which works for you the best (it’s a parameter).
tickers = ['AAPL', 'MSFT', 'GOOG', 'AMZN', 'META', 'ORCL', 'IBM', 'TSLA', 'NVDA', 'INTC']rebalance_frequency = 'daily'So below is the code for Markowitz Maximum Sharpe-Ratio (MSR) which using MSRP Solver function:
def MSR_portfolio(self, data: np.ndarray) -> np.ndarray: """ Markowitz Maximum Sharpe-Ratio Portfolio (MSRP) Returns the optimal asset weights for the MSRP portfolio, given historical price data of assets. Original equation to solve: max (w^T mu) / sqrt(w^T Sigma w) subject to sum(w) = 1 Quadratic Programming (QP) reformulation: min (1/2) * w^T Sigma w subject to w^T mu = 1 sum(w) = 1 Args: data (np.ndarray): Historical price data of assets Returns: w (np.ndarray): Optimal asset weights """ X = np.diff(np.log(data), axis=0) # Calculate log returns from historical price data mu = np.mean(X, axis=0) # Calculate the mean returns of the assets Sigma = np.cov(X, rowvar=False) # Calculate the covariance matrix of the returns w = self.MSRP_solver(mu, Sigma) # Use the MSRP solver to get the optimal weights return w # Return the optimal weights def MSRP_solver(self, mu: np.ndarray, Sigma: np.ndarray) -> np.ndarray: """ Method for solving Markowitz Maximum Sharpe-Ratio Portfolio (MSRP) Returns the optimal asset weights for the MSRP portfolio, given mean returns and covariance matrix. Original equation to solve: max (w^T mu) / sqrt(w^T Sigma w) subject to sum(w) = 1 Quadratic Programming (QP) reformulation: min (1/2) * w^T Sigma w subject to w^T mu = 1 sum(w) = 1 Args: mu (np.ndarray): Mean returns Sigma (np.ndarray): Covariance matrix Returns: w (np.ndarray): Optimal asset weights """ N = Sigma.shape[0] # Number of assets (stocks) if np.all(mu <= 1e-8): # Check if mean returns are close to zero return np.zeros(N) # Return zero weights if no returns Dmat = 2 * Sigma # Quadratic term for the optimizer Amat = np.vstack((mu, np.ones(N))) # Combine mean returns and sum constraint for constraints bvec = np.array([1, 1]) # Right-hand side of constraints (1 for mean returns and sum) dvec = np.zeros(N) # Linear term (zero for this problem) # Call the QP solver w = self.solve_QP(Dmat, dvec, Amat, bvec, meq=2) return w / np.sum(abs(w)) # Normalize weights to sum to 1We also use solve_QP function, which is a quadratic programming solver that finds the optimal asset weights for a given portfolio optimization problem. As stated above quadratic programming is a mathematical optimization technique used to solve problems with a quadratic objective function subject to linear constraints.
The solve_QP function uses the minimize function from scipy.optimize to numerically solve the constrained quadratic programming problem and find the optimal weights that maximize the utility function while satisfying the specified constraints.
def solve_QP(self, Dmat: np.ndarray, dvec: np.ndarray, Amat: np.ndarray, bvec: np.ndarray, meq: int = 0) -> np.ndarray: """ Quadratic programming solver. Returns the optimal asset weights for the QP problem. Args: Dmat (np.ndarray): Matrix of quadratic coefficients dvec (np.ndarray): Vector of linear coefficients Amat (np.ndarray): Matrix of linear constraints bvec (np.ndarray): Vector of linear constraints meq (int): Number of equality constraints Returns: x (np.ndarray): Optimal asset weights """ def portfolio_obj(x): """ Objective function for the QP problem. Minimize 0.5 * x^T * Dmat * x + dvec^T * x. """ return 0.5 * np.dot(x, np.dot(Dmat, x)) + np.dot(dvec, x) def portfolio_constr_eq(x): """ Equality constraints for the QP problem. """ return np.dot(Amat[:meq], x) - bvec[:meq] def portfolio_constr_ineq(x): """ Inequality constraints for the QP problem. """ if Amat.shape[0] - meq == 0: return np.array([]) else: return np.dot(Amat[meq:], x) - bvec[meq:] # Define constraints for the optimizer cons = [{'type': 'eq', 'fun': portfolio_constr_eq}] if meq < len(bvec): cons.append({'type': 'ineq', 'fun': portfolio_constr_ineq}) # Initial guess for the weights initial_guess = np.ones(Dmat.shape[0]) / Dmat.shape[0] # Use the 'SLSQP' method to minimize the objective function subject to constraints res = minimize(portfolio_obj, initial_guess, constraints=cons, method='SLSQP') # Check if the optimization was successful if not res.success: raise ValueError('Quadratic programming failed to find a solution.') # Return the optimal weights return res.xWe also produce weights for each portfolio strategy and for some it looks as:
The full QuantJourney Framework code is available on private GitHub for paid subscribers. Please subscribe to support my work and further development of the complete Trading Framework for investors.
The Global Minimum Variance Portfolio and Shrinkage
The Global Minimum Variance Portfolio (GMVP) aims to find the portfolio weights that minimize the overall portfolio variance, without considering the expected returns of the assets. In the process of constructing the GMVP, the covariance matrix of asset returns plays a crucial role. However, estimating the covariance matrix from historical data can be challenging, especially when the number of assets is large compared to the number of observations. In such cases, the sample covariance matrix may be poorly conditioned or even singular, leading to suboptimal portfolio weights.
To address this issue, various shrinkage methods are employed in the GMVP implementation to improve the estimation of the covariance matrix. Shrinkage methods aim to find a balance between the sample covariance matrix and a more structured estimator, such as the identity matrix or a constant correlation matrix. By "shrinking" the sample covariance matrix towards a well-conditioned target matrix, shrinkage methods reduce estimation error and enhance the stability of the covariance estimates.
def GMV_portfolio(self, data: np.ndarray, shrinkage: bool =False, shrinkage_type = 'ledoit', shortselling: bool =True, leverage: int =None) -> np.ndarray: """ Global Minimum Variance Portfolio Returns the optimal asset weights for the GMVP, given historical price data of assets. Args: data (np.ndarray): Historical price data of assets shrinkage (bool): Flag to use Ledoit-Wolf shrinkage estimator shortselling (bool): Flag to allow short-selling leverage (int): Leverage factor Returns: w (np.ndarray): Optimal asset weights """ X = np.diff(np.log(data), axis=0) X = X[~np.isnan(X).any(axis=1)] # Remove rows with NaN values # Calculate covariance matrix if shrinkage: if shrinkage_type == 'ledoit': # Use Ledoit-Wolf shrinkage estimator Sigma = self.ledoit_wolf_shrinkage(X) elif shrinkage_type == 'ledoit_cc': # Use Ledoit-Wolf shrinkage estimator with custom covariance Sigma = self.ledoitwolf_cc(X) elif shrinkage_type == 'oas': # Use Oracle Approximating Shrinkage (OAS) estimator Sigma = self.oas_shrinkage(X) elif shrinkage_type == 'graphical_lasso': # Use Graphical Lasso estimator Sigma = self.graphical_lasso_shrinkage(X) elif shrinkage_type == 'mcd': # Use Minimum Covariance Determinant (MCD) estimator Sigma = self.mcd_shrinkage(X) else: raise ValueError('Invalid shrinkage type. Choose from: ledoit, ledoit_cc, oas, graphical_lasso, mcd') else: Sigma = np.cov(X, rowvar=False) if not shortselling: N = Sigma.shape[0] Dmat = 2 * Sigma Amat = np.vstack((np.ones(N), np.eye(N))) bvec = np.array([1] + [0] * (N+1)) # Update bvec to have length N+1 dvec = np.zeros(N) w = self.solve_QP(Dmat, dvec, Amat, bvec, meq=1) else: ones = np.ones(Sigma.shape[0]) w = np.linalg.solve(Sigma, ones) w /= np.sum(w) if leverage is not None and leverage < np.inf: w = leverage * w / np.sum(np.abs(w)) return wIn the provided code for the GMV_portfolio function, several shrinkage methods are available to estimate the covariance matrix:
Ledoit-Wolf Shrinkage (ledoit):
This method shrinks the sample covariance matrix towards a scaled identity matrix.
It aims to minimize the expected mean squared error between the estimated and the true covariance matrix.
The shrinkage intensity is determined analytically based on the data.
def ledoit_wolf_shrinkage(self, returns: np.ndarray) -> np.ndarray: """ Define Ledoit-Wolf shrinkage function using sklearn. Returns the Ledoit-Wolf shrinkage covariance matrix. Args: returns (np.ndarray): Historical returns data Returns: np.ndarray: Ledoit-Wolf shrinkage covariance matrix """ lw = LedoitWolf() lw_cov = lw.fit(returns).covariance_ return lw_covLedoit-Wolf Shrinkage with Custom Covariance (ledoit_cc):
This variant of Ledoit-Wolf shrinkage allows for a custom target covariance matrix instead of the default scaled identity matrix.
The custom target matrix can be designed to incorporate prior knowledge or assumptions about the asset correlations.
def ledoitwolf_cc(self, returns: np.ndarray) -> np.ndarray: """ Implementation of Ledoit-Wolf shrinkage estimator with custom covariance. The Ledoit-Wolf shrinkage method is a technique used to improve the estimation of the covariance matrix by shrinking the sample covariance matrix towards a target matrix. Returns the Ledoit-Wolf shrinkage covariance matrix. Args: returns (np.ndarray): Historical returns data Returns: np.ndarray: Ledoit-Wolf shrinkage covariance matrix """ T, N = returns.shape returns = returns - np.mean(returns, axis=0, keepdims=True) # Subtract mean using numpy mean df = pd.DataFrame(returns) Sigma = df.cov().values Cor = df.corr().values diagonals = np.diag(Sigma) var = diagonals.reshape(len(Sigma), 1) vols = var ** 0.5 rbar = np.mean((Cor.sum(1) - 1) / (Cor.shape[1] - 1)) cc_cor = np.matrix([[rbar] * N for _ in range(N)]) np.fill_diagonal(cc_cor, 1) F = np.diag((diagonals ** 0.5)) @ cc_cor @ np.diag((diagonals ** 0.5)) # vol-cor decomposition y = returns ** 2 mat1 = (y.transpose() @ y) / T - Sigma ** 2 # y is centered, cross term cancels pihat = mat1.sum() mat2 = ((returns ** 3).transpose() @ returns) / T - var * Sigma # cross term cancels np.fill_diagonal(mat2, 0) rhohat = np.diag(mat1).sum() + rbar * ((1 / vols) @ vols.transpose() * mat2).sum() gammahat = np.linalg.norm(Sigma - F, "fro") ** 2 kappahat = (pihat - rhohat) / gammahat delta = max(0, min(1, kappahat / T)) return delta * F + (1 - delta) * SigmaOracle Approximating Shrinkage (OAS) (oas):
OAS is another shrinkage method that aims to minimize the expected mean squared error between the estimated and the true covariance matrix.
It shrinks the sample covariance matrix towards a scaled identity matrix, similar to Ledoit-Wolf shrinkage, but uses a different formula for determining the shrinkage intensity.
Graphical Lasso (graphical_lasso):
Graphical Lasso is a shrinkage method that estimates a sparse inverse covariance matrix by imposing an L1 penalty on the matrix entries.
It is particularly useful when the true covariance matrix is believed to have a sparse structure, meaning that many of the pairwise correlations are zero.
Minimum Covariance Determinant (MCD) (mcd):
MCD is a robust estimator of the covariance matrix that is designed to handle outliers or observations that deviate significantly from the majority of the data.
It identifies a subset of the data that has the smallest covariance determinant and uses that subset to estimate the covariance matrix.
The choice of shrinkage method depends on the specific characteristics of the data and the assumptions about the underlying covariance structure. By incorporating these shrinkage methods into the GMVP implementation, the resulting covariance matrix estimates are more reliable and stable, leading to improved portfolio optimization outcomes.
The GMV_portfolio function takes the historical price data of assets (data) as input, along with optional parameters to control the use of shrinkage methods (shrinkage and shrinkage_type), allow short selling (shortselling), and specify a leverage factor (leverage). Based on the selected shrinkage method, the function estimates the covariance matrix (Sigma) using the corresponding shrinkage estimator. It then solves the quadratic programming problem using the solve_QP function to obtain the optimal asset weights that minimize the portfolio variance while satisfying the constraints.
By employing shrinkage methods in the GMVP, the impact of estimation errors in the covariance matrix is mitigated, resulting in more robust and reliable portfolio weights. This approach enhances the stability and performance of the optimized portfolios, particularly when dealing with a large number of assets or limited historical data.
In summary, portfolio optimization involves the selection of the best portfolio from a set of portfolios to maximize expected returns and minimize risk. Using methods like the Lagrange multiplier and quadratic utility functions, we can derive optimal portfolio weights under different constraints and assumptions. Both approaches require accurate estimation of mean returns and the covariance matrix, which are crucial for effective optimization.
These portfolio optimization strategies and methods will be integrated into our QuantJourney Backtesting framework. I'm looking forward to further enhancement on these topics and exploring framework applications in more detail.
