^{1}

^{2}

^{*}

Highly Active Antiretroviral Therapy (HAART) has changed the course of human immunodeficiency virus (HIV) treatments since its introduction. However, for many patients, long term continuous HAART is expensive and can include problems with drug toxicity and side effects, as well as increased drug resistance. Because of these reasons, some HIV infected patients will voluntarily terminate HAART. Some of these patients will also interrupt the continuous prescribed therapies for short or long periods. After discontinuing HAART, patients will usually experience a rapid increase in viral load coupled with an immediate decline in CD4+ counts. The canonical example of a patient undergoing unsupervised breaks in HAART is that of the “Berlin patient”. In this case, the patient was able to control viral load in the absence of treatment by cycling HAART on and off due to non-related infections. Due to this patient, interest in the use of structured treatment interruptions (STI) as a mechanism to regulate an HIV infection piqued. This paper describes an optimal control approach to determine STI regimen for HIV patients. The optimal STI was implemented in the context of the receding horizon control (RHC) using a mathematical model for the in-vivo dynamics of an HIV type 1 infection. Using available clinical data, we calibrate the model by estimating on a patient specific basis, a best estimable set of parameters using sensitivity analysis and subset selection. We demonstrate how customized STI protocols can be designed through the variation of control parameters on a patient specific basis.

There are few diseases in our society today that are more polarizing, politicized, and personal than human immunodeficiency virus/acquired immune deficiency syndrome, or HIV/AIDS. According to the Joint United Nations Programme on HIV and AIDS (UNAIDS), there were 34 million people living with HIV/AIDS in 2010, 22.9 million of which live in sub-Saharan Africa. Despite the prevalence and availability of effective treatment of HIV, even as late as 2010 the best practices for management of acute HIV infection are still unknown [

Throughout the last few decades, mathematical models have been developed to further the understanding of the progression of the virus as it replicates in-host. These models focus on the interaction between the virus and the CD4^{+} cells of the immune system, one of the virus’ primary targets [

The models of in-vivo dynamics are most often formulated as a system of nonlinear ordinary differential equations (ODE), though recently several size structured partial differential equation models have been developed. These models allow for variations in viral production rates as a function of time, since it is thought that as infected cells age their overall fitness decreases [

A primary goal of using modern models of HIV is to influence treatment decisions and construct better treatment protocols for infected patients. The construction of optimal control methodologies has been investigated by many authors, [

The investigation into parameter identifiability is not unique to the mathematical sciences, and many papers in engineering, statistics, and biomedical engineering have shed light into this area [^{+} count and viral load. We utilize both sensitivity analysis and identifiability analysis methods to compute patient specific best-estimatable subsets of parameters. In [

Finally, the paper investigates the construction of optimal control and treatment therapies in a patient specific context, i.e. in the setting of personalized medicine or precision medicine. We develop treatments known as structured treatment interruptions (STI), wherein the effective treatment is terminated at specified times and then restarted in an attempt to increase an in-host immune response and provide a drug holiday for the patient. These treatment methods have been studied in detail, though a lack of double-blind, randomized clinical trials has lead to a lack of consensus on the effectiveness of these types of treatments [

The organization of the paper is as follows. Section 2 describes the mathematical model of the HIV dynamics, including the control mechanisms. This section also describes the a-priori identifiability analysis, the available data and its incorporation into the parameter estimation problem, and the results of the model validation. In particular, we show how the model can well-fit both the CD4^{+} count and viral load clinical data. The optimal control formulation to compute optimal treatment strategies of an STI type is considered in Section 3. In this section we also provide computed optimal control simulations along side the clinical data and discuss the relevance.

There have been numerous advances in modeling HIV viral dynamics in recent years. An excellent general overview of the modeling challenges of HIV, especially with regards to treatment protocols and stability analyses is given in [

The dynamics of the HIV model are described by the system of nonlinear ordinary differential equations

T ˙ 1 = λ 1 − d 1 T 1 − ( 1 − ε 1 ) k 1 V I T 1 , T ˙ 2 = λ 2 − d 2 T 2 − ( 1 − f ε 1 ) k 2 V I T 2 T ˙ 1 ∗ = ( 1 − ε 1 ) k 1 V I T 1 − δ T 1 ∗ − m 1 T 1 * E , T ˙ 2 ∗ = ( 1 − f ε 1 ) k 2 V I T 2 − δ T 2 ∗ − m 2 T 2 * E V ˙ I = ( 1 − ε 2 ) N T δ ( T 1 ∗ + T 2 ∗ ) − ( c + ( 1 − ε 1 ) ρ 1 k 1 T 1 + ( 1 − f ε 1 ) ρ 2 k 2 T 2 ) V I V ˙ N I = ε 2 N T δ ( T 1 ∗ + T 2 ∗ ) − c V N I , E ˙ = λ E + b E T 1 ∗ + T 2 ∗ T 1 ∗ + T 2 ∗ + K b E − d E T 1 ∗ + T 2 ∗ T 1 ∗ + T 2 ∗ + K d E − δ E E , (1)

with an initial condition vector

[ T 1 ( 0 ) T 2 ( 0 ) T 1 ∗ ( 0 ) T 2 ∗ ( 0 ) V I ( 0 ) V N I ( 0 ) E ( 0 ) ] Τ .

The model compartments are T 1 (type 1 target cells, e.g. CD4^{+} T-cells, cells/ ml blood), T 2 (type 2 target cells, e.g. macrophages, cells/ml blood), V I (infectious free viron particles, RNA copies/ml blood), V N I (non-infectious free viron particles, RNA copies/ml blood) and E (cytotoxic T-lymphocytes (CTL), cells/ml blood). A superscript asterisk denotes infected cells. This model has two important features. The presence of a second target cell type, T 2 satisfies a modeling requirement suggested in [

External mechanisms by which the virus is controlled are incorporated into the model. We allow for therapeutic treatment by reverse transcriptase inhibitors (RTIs) and protease inhibitors (PIs). RTIs interfere with the viral RNA to DNA synthesis; when reverse transcriptase is inhibited HIV can enter a cell but the host cell will not become infected. PIs cause infected cells to produce non- infectious virions. Thus when considering treatment that includes PIs, we consider both infectious and noninfectious viral particles. Noninfectious viral particles can remain present even in the absence of PIs. The prevailing paradigm of HIV treatment is to prescribe a highly active antiretroviral therapy (HAART) consisting of both PIs and RTIs. The treatment factors are given as ε 1 ( t ) = ε ¯ 1 u ( t ) and ε 2 ( t ) = ε ¯ 2 u ( t ) , consisting of efficacy ε ¯ 1 ∈ [ 0,1 ] corresponding to a reverse transcriptase inhibitor and ε ¯ 2 ∈ [ 0,1 ] modeling the effectiveness of a protease inhibitor with a time dependent function u ( t ) ,0 ≤ u ( t ) ≤ 1. We assume that the treatment has a reduced efficacy for the population of T 2 , where the efficacy is f ε 1 ( t ) , f ∈ ( 0,1 ) . Treatment protocols are always implemented as a combination of both PIs and RTIs, so monotherapy is not considered.

The model (1) contains 22 biologically relevant parameters that must be specified before numerical simulations can be performed. We evaluate the model at a nominal set of parameter values that is justified through literature and past clinical studies; these values are given in

Index | Parameter | Value | Units | Description |
---|---|---|---|---|

1 | 10 | cells/mm^{3} day | Target cell type 1 source rate | |

2 | 0.03198 | cells/mm^{3} day | Target cell type 2 source rate | |

3 | 0.01 | 1/day | Target cell type 1 death rate | |

4 | 0.01 | 1/day | Target cell type 2 death rate | |

5 | mm^{3}/virions day | Population 1 infection rate | ||

6 | 0.1 | mm^{3}/virions day | Population 2 infection rate | |

7 | 0.01 | mm^{3}/cells day | Immune-induced clearance rate for population 1 | |

8 | 0.01 | mm^{3}/cells day | Immune-induced clearance rate for population 2 | |

9 | 1 | virions/cell | Average number virions infecting a type 1 cell | |

10 | 1 | virions/cell | Average number virions infecting a type 2 cell | |

11 | 0.7 | 1/day | Infected cell death rate | |

12 | 13 | 1/day | Virus natural death rate | |

13 | Treatment efficacy reduction in population 2 | |||

14 | 100 | virions/cell | Virions produced per infected cell | |

15 | .001 | cells/mm^{3} day | Immune effector production (source) rate | |

16 | 0.1 | 1/day | Natural death rate for immune effectors | |

17 | 0.3 | 1/day | Maximum birth rate for immune effectors | |

18 | 0.25 | 1/day | Maximum death rate for immune effectors | |

19 | 100 | cells/ml | Saturation constant for immune effector birth | |

20 | 500 | cells/ml | Saturation constant for immune effector death | |

21 | RTI efficacy | |||

22 | PI efficacy |

chosen so that several conditions on viral load and target cell equilibria are satisfied. These conditions are actually not reflected in our model since we have omitted the chronically infected state from the Callaway-Perelson model, though it is believed that small adjustments to the parameter values will obtain the same qualitative behavior [

The parameters used in the immune response state are not well known, and are chosen primarily to exhibit expected model behavior in the simulations. The parameters m 1 , m 2 represent the effectiveness that the immune response E clears the infected cells of the target populations. The commonly used values are taken from [

Because the range of both the state variables and parameter values can vary over several orders of magnitude, we use a l o g 10 transformed system for the state variables. We define the new state variable x i ( t , q ) = log 10 x ¯ i ( t , q ) , where x ¯ i represents the original, non-log transformed state. We also log-transform the parameters, q → log 10 q , as they also varying widely in magnitude (see

x ˙ i ( t , q ) = 10 − x i ln 10 f ( 10 x i ( q ) , 10 q ) (2)

where f is the right hand side of (1). Performing this substitution puts both the states and parameters within an order of (log) magnitude of each other, allowing for a more robust estimation procedure. This transformation also resolves issues of physically unrealistic negative solutions that occur due to round- off error.

A first step in the a-priori identifiability analysis of a system is to determine which parameters contribute greatest to the output of the system. For our model, the output of the system is y 1 = T 1 + T 1 ∗ representing the CD4^{+} count and y 2 = V I + V N I representing the viral load. Sensitivity analysis has traditionally primarily been used in the analysis of the forward problem when one needed to understand the effects of parameter perturbation on the output. In recent years sensitivity analysis has become an important component of inverse problem analysis, partly because it directly aids in uncertainty analysis. When physiologically based mathematical models are developed, the parameters typically have biological relevance, and thus one may wish to estimate these parameters from data. If these parameters are insensitive then they could be difficult to estimate despite their relevance. Sensitivity analysis determines the degree to which parameters affect system output. Sensitivity analysis can also aid in the optimal design of experiments and data collection [

Sensitivity analysis constitutes a subset of techniques within identifiability analysis. An excellent survey of methods for identifiability in ODE models is given in [

The data available have come from a clinical study at Massachusetts General Hospital of over 100 adults with acute HIV infection between 1996 and 2004. The data collected consists of CD4^{+} T-lymphocyte count (cells/ml) and RNA viral load (RNA copies/ml) represented by y 1 = T 1 + T 1 ∗ and y 2 = V I + V N I , respectively. The viral load is quantified through different assay techniques; the standard assay can detect viral loads in the range of 400 to 750,000 copies/ml, while an ultra-sensitive assay has a detection range of 50 to 100,000 copies/ml. The ultra-sensitive assay is used when a measurement is below the 400 copies/ml lower detection limit for the standard assay. This can often be the case when patients are successfully suppressing the virus due to HAART.

Due to the range sensitivity limits of the assays, the clinical viral load effectively have upper and lower limits of quantification. The upper limit of quantification is addressed by repeatedly diluting the sample until it falls within the detection range. The lower limit, or left censoring point, directly affects the collected data and how it is used within the inverse problem. When a datum is returned at a censor point, L 1 = 400 copies/ml or L 2 = 50 copies/ml, the only available knowledge is that the true measurement is between zero and the left quantification limit. These limits are illustrated in the longitudinal data set from patient identified by the number 1, shown in

In addition to the data being censored at certain time values, the data are also collected at different time intervals. These intervals also differ between patients. This is visible in ^{+} may not have been made at the same time points. In general for patient j , we have CD4^{+} data pairs ( t 1 i j , y 1 i j ) , i = 1 , ⋯ , N 1 j and possibly different viral RNA pairs ( t 2 i j , y 2 i j ) , i = 1 , ⋯ , N 2 j .

Before we use the HIV model given in (1) to design the optimal treatment for

HIV patients in the context of personalized medicine, we first calibrate the model parameters to the clinical data through the parameter estimation formulation.

The parameter estimation problem is patient specific; we use data from a single patient j in order to estimate the patient specific parameter value q j . A common approach for conducting parameter estimation in ODE models is through minimization of an ordinary least squares cost function [

J ( q ) = ∑ s = 1 2 1 N s j ∑ i = 1 N s j | z s ( t s i j ; q ) − y s i j | 2 , (3)

over an admissible parameter space Ω ⊆ ℝ l , where l is the number of parameters being identified by the practical identifiability analysis. The parameter estimation problem is to compute the unique minimizer of J, q ⋆ j = a r g m i n J ( q ) , if it exists. Note that uniqueness of the minimizing parameter vector is an issue addressed by the identifiability analysis. Assuming the data are uncensored and produced by the model with additive and independent Gaussian noise, the ordinary least squares estimate is also the maximum likelihood estimate. However, in our case there is censoring in the data as discussed in section 2.2, and least squares does not suffice as a statistically rigorous methodology. Instead we compute maximum likelihood parameter estimates using an expectation maximization (EM) algorithm [

We begin the parameter estimation problem by performing for each patient an a-priori sensitivity analysis at the nominal parameter values q ∗ given in

Before a subset specific estimate is formed, we first attempt to estimate all 22 parameters using the global optimization algorithm DIRECT with the EM algorithm. DIRECT is a bounded, sampling method based on dividing hyper-rec- tangles [

The number of identifiable parameters range from k = 8 to k = 10 . In addition to these parameters, all seven initial conditions were estimated for each patient. Each patient subset is given in

Representative of the parameter estimation results are depicted in

Patient | Identifiable Subset |
---|---|

1 | |

2 | |

3 | |

4 | |

5 | |

6 | |

7 | |

8 | |

9 | |

10 | |

11 | |

12 | |

13 | |

14 |

data are given in blue dots as well as the adjusted data (black circles) calculated through the expectation maximization algorithm. The calibrated model prediction is the solid red line. Near the bottom of the figures are structured treatment

interruption as prescribed by the medical doctors for each patient (green for on-treatment and red for off-treatment). In general, the model fits the clinical data reasonably well.

Highly Active Antiretroviral Therapy (HAART) has changed the course of HIV treatments since its introduction. Under HAART, viral load decreases exponentially in the weeks after the start of treatment and is maintained at low levels so long as treatment is continued. Despite this exponential decline in viral load, these treatments do not completely clear the virus from the patient, and a life long therapy is necessary to maintain low viral load. Even with this kind of success, the best way to treat acutely infected HIV patients is still an open question with many approaches under investigation [^{+} counts [

The canonical example of a patient undergoing unsupervised breaks in HAART is that of the “Berlin patient” [

One of the key variables in the design of any STI based protocol is the criteria in which patients are removed or placed back on treatment. In some studies the viral load was the primary indicator as to when treatment would resume, whereas in other studies off periods were of a fixed length, followed by a fixed length on period. The former approach is simpler, but harder to administer as it requires a higher level of patience scrutiny with more expensive and inconvenient blood tests. The latter strategy is easier to implement in larger patient groups, but the on/off periods are still up for debate and the subject of many clinical trials. Currently the period durations are determined either in an ad-hoc style, or by trying to adapt the methodologies of other trials.

The use of mathematical modeling to determine optimal STI regimes has flourished in recent years. The survey paper [

Although several control paradigms exist for the control and simulation of nonlinear systems, not many nonlinear controllers can be applied to the discrete sample-and-hold methodology of STI treatment. One method that is highly promising and is well suited to finding an optimal treatment amenable to HIV treatment is the receding horizon control (RHC) [

Receding horizon, or model predictive control seeks to gain the benefits of a feedback control while maintaining the overall existing control methodology (i.e. STI) and model structure. The basic structure of the RHC methodology is shown in

current control from solving a finite time open loop control problem, using the current state of the system as the initial state. This implicit optimization then yields a control for future time, and the first control in the sequence is applied to the system for a specified control window. This is the primary difference with the open loop methodologies, where the controls are computed for the entire simulation interval before simulation.

This framework is uniquely suited for the computation of controls that fall within the STI paradigm under HAART. Since the number of elements in the control sequence is finite due to the on-off nature of the STI control, the existence of an optimal control on each window is guaranteed. Moreover, this approach is model invariant, in that the HIV model can be updated or swapped out with a model with minimal modifications to the control setup. Lastly the RHC parameters such as horizon and window length, discussed later, can be easily modified based on clinical needs and observations.

There are several basic elements to the RHC system, outlined in [

1) Model equations which govern the system dynamics.

2) The calculation of a sequence of optimal control laws, subject to the cost function.

3) A “receding horizon” strategy, so that on each interval or horizon, that the control is computed is shifted forward in time. The control that is computed is then only used for a portion of the horizon length.

To solidify the RHC methodology, we present the following control problem formulation. Together with the model equations (1), we consider a control problem defined by the objective function,

J ( u ( t ) ) = ∫ 0 t f [ Q V ( t ) + R 1 ε 1 ( t ) 2 + R 2 ε 2 ( t ) 2 − S E ( t ) ] d t , (4)

where ε 1 ( t ) = ε 1 u ( t ) , an RTI, and ε 2 ( t ) = ε 2 u ( t ) , a PI. The values ε 1 and ε 2 are the patient specific values estimated in Section 2. The parameters Q , R 1 , R 2 and S represent the control weights for the virus, control input, and immune response, respectively. By varying these weights, the optimal control can be customized in a patient specific sense, and exact treatments can be designed. By including the control terms in (4) we attempt to minimize the systematic cost of the drugs both in the sense of physiologic side effects as well as the monetary treatment cost.

For the RHC formulation, we denote [ t i t i + 1 ] a sequence of time intervals. Let t h o r , i such that t h o r , i ≥ t i + 1 − t i be the control horizon. We denote t i + 1 − t i to be t w i n , the current control window. Consider a sequence of control problems P i ,

u * ∈ U s .t . u * = arg min J ( u ) = ∫ t i t i + t h o r , i [ Q V ( t ) + R 1 ε 1 ( t ) 2 + R 2 ε 2 ( t ) 2 − S E ( t ) ] d t , (5)

on the interval [ t i t i + 1 ] subject to the model equations given by (1). The solution to each control problem P i is computed by the direct search routine as outlined in [

Mathematically, we represent the control numerically with a pair of vectors describing the control function u ( t ) and the corresponding time intervals the control applies to. For example, for a treatment regimen with a treatment interval of m = 15 days, a possible control could be u ( t ) = [ 1 1 0 0 1 ] . This corresponds to an overall treatment length of 75 days with on treatment for the first 30 days, off for the next 30, and then back on for the final 15 days of treatment. For any treatment interval with n switchings of treatment, there exists a possible 2 n ways to define a treatment. Because this is a finite set, there exists an optimal control that minimizes (5). We denote the set of all control vectors U , with | U | = 2 n . We then seek the optimal control,

u * ∈ U = arg min J ( ε 1 ( t ) , ε 2 ( t ) ) . (6)

On average, the data sets contain four years of longitudinal patient data,

therefore n ≈ 1400 days. Because n is relatively large, modeling daily changes in treatment is both impractical from a computational and clinical stand point. In practice, an interval of two to three weeks is clinically appropriate and is what we simulate.

If we assume a 30 day treatment interval, modeling the entire control interval could involve the calculation of nearly 50 control values, or switches. As the problem is defined, m = 30 and n ≈ 1400 would involve testing upwards of 2 50 possible treatment protocols; clearly this is not computationally tractable. To mitigate this issue, we seek to reduce the number of iterations necessary to find the optimal control. One strategy is to break the treatment interval into bins and calculate independent controls on these intervals before combining all of these controls for the entire interval. For instance, for 30 day segments we first compute an optimal control over the set [ 0,30,60,90,120,150,180 ] , a calculation that involves the testing of 2 7 = 128 distinct possible controls. Once this control is computed, the resulting trajectories are integrated to t = 180 , and a control is computed over the set [ 180,210,240, ⋯ ,360 ] . This control is then appended to the prior control vector, and this sequence is repeated until the final time t f is reached.

Because we divide the treatment interval into several separate bins, the resulting control is only suboptimal; however in [

This process is extended by RHC, outlined as follows:

1) Given an initial condition x ( t i ) solve the optimal control problem P i on the horizon interval [ t i t h o r , i ] .

2) Use the control calculated in the prior step to compute the trajectory over the current control window [ t i t i + 1 ] .

3) Repeat this process by extending to the next control horizon to compute the next control. Terminate when the next t i > t f .

A schematic overview of this process is given in

Receding control depends on having full state knowledge in the observations of the system. Our control functions are computed after the completion of the inverse problem, so we assume our model to be accurate and predictive for each patient, therefore yielding full state knowledge at each time step.

For problems where there exists only a reduced observation of the system, a (typically) nonlinear estimation of the true state values is needed for the RHC to operate successfully. In cases such as this, the estimator is installed between the system model and the optimizer, as shown in

In this section we present optimal controls and the resulting state solutions for

patient 3 under utilizing differing control weights and treatment intervals. We wish to illustrate that treatment protocols can be swayed and adjusted on a patient specific basis using the control weights. We also show cases with an “unexpected” perturbation to the optimal treatment, e.g. forced off treatment to demonstrate the feedback capability of the RHC. The control weights are presented as the vector [ Q S R 1 R 2 ] . In all of the following plots, the green and red bars at the bottom of the plot indicate clinical adherence to control, whereas the cyan and magenta at the top of the plot indicate the optimally computed control of on treatment and off treatment, respectively.

We first investigate the impact of different treatment intervals has on viral load.

When the interval is increased to 30 days in

40 off with m = 20 in

Compare these results with

One advantage of the control methodology presented is being able to customize a treatment for a given patient. This can be done by varying the control weights to emphasize or deemphasize specific terms in the goal function (4). Between

In

In this work we have examined an in-vivo model describing the acute and chronic infection regimes of HIV. With this model, we first have performed a

patient specific model calibration to clinically collected data sets consisting of CD4^{+} count as well as viral load, which are censored. We then take the first steps in investigating the construction of optimal treatment strategies, noting the lack of consensus in the best standards of treatment for acutely infected patients. To construct the treatment protocols, we use receding horizon control which computes several successive controls over a shifting interval. The broader controls are then used on short time intervals, allowing for the control to adjust to unexpected physical perturbations to the system. These controls mimic the form of structured treatment interruptions, and there are several ongoing clinical studies to assess their effectiveness against standard HAART therapy. In our control implementation, we assumed full state knowledge of the dynamic system. This is an unrealistic assumption and the use of nonlinear estimators would be beneficial. In addition, calibration and validation of more advanced HIV models could lead to greater insight into the replication processes during acute infection. For example, the development of new immune response models, an area that is currently poorly understood, could shed light into the complicated immune pro- cesses that occur post-infection. There are CD8 immune response data available, but we do not use them in this work, as the data do not match the type of immune response that we have modeled.

We thank the Editor and the referee for their comments. Research of H. Tran is funded in part by the National Science Foundation grant DMS 1022688 and by the National Institute of Allergy and Infectious Diseases grant NIAID 9R01AI071915.

Attarian, A. and Tran, H. (2017) An Optimal Control Approach to Structured Treatment Interruptions for HIV Patients: A Personalized Me- dicine Perspective. Applied Mathematics, 8, 934-955. https://doi.org/10.4236/am.2017.87074