# Solving the Rosenbrock Problem in >10 Ways

This example is a demonstration of many different solvers to demonstrate the
flexibility of Optimization.jl. This is a gauntlet of many solvers to get a feel
for common workflows of the package and give copy-pastable starting points.

!!! note

    This example uses many different solvers of Optimization.jl. Each solver
    subpackage needs to be installed separate. For example, for the details on
    the installation and usage of OptimizationOptimJL.jl package, see the
    [Optim.jl page](@ref optim).

The objective of this exercise is to determine the $(x, y)$ value pair that minimizes the result of a Rosenbrock function $f$ with some parameter values $a$ and $b$. The Rosenbrock function is useful for testing because it is known *a priori* to have a global minimum at $(a, a^2)$.
```math
f(x,\,y;\,a,\,b) = \left(a - x\right)^2 + b \left(y - x^2\right)^2
```

The Optimization.jl interface expects functions to be defined with a vector of optimization arguments $\bar{x}$ and a vector of parameters $\bar{p}$, i.e.:
```math
f(\bar{x},\,\bar{p}) = \left(p_1 - x_1\right)^2 + p_2 \left(x_2 - x_1^2\right)^2
```

Parameters $a$ and $b$ are captured in a vector $\bar{p}$ and assigned some arbitrary values to produce a particular Rosenbrock function to be minimized.
```math
\bar{p} = \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 1 \\ 100 \end{bmatrix}
```

The original $x$ and $y$ domains are captured in a vector $\bar{x}$.
```math
\bar{x} = \begin{bmatrix} x \\ y \end{bmatrix}
```

An initial estimate $\bar{x}_0$ of the minima location is required to initialize the optimizer.
```math
\bar{x}_0 = \begin{bmatrix} x_0 \\ y_0 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}
```


An optimization problem can now be defined and solved to estimate the values for $\bar{x}$ that minimize the output of this function.

```@example rosenbrock
# Define the problem to solve
using SciMLBase, OptimizationBase
using ADTypes, ForwardDiff, Zygote

rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
_p = [1.0, 100.0]

f = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff())
l1 = rosenbrock(x0, _p)
prob = SciMLBase.OptimizationProblem(f, x0, _p)
```

## Optim.jl Solvers

### Start with some derivative-free optimizers

```@example rosenbrock
using OptimizationOptimJL
sol = solve(prob, SimulatedAnnealing())
prob = SciMLBase.OptimizationProblem(f, x0, _p, lb = [-1.0, -1.0], ub = [0.8, 0.8])
sol = solve(prob, SAMIN())

l1 = rosenbrock(x0, _p)
prob = SciMLBase.OptimizationProblem(rosenbrock, x0, _p)
sol = solve(prob, NelderMead())
```

### Now a gradient-based optimizer with forward-mode automatic differentiation

```@example rosenbrock
optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff())
prob = SciMLBase.OptimizationProblem(optf, x0, _p)
sol = solve(prob, BFGS())
```

### Now a second order optimizer using Hessians generated by forward-mode automatic differentiation

```@example rosenbrock
sol = solve(prob, Newton())
```

### Now a second order Hessian-free optimizer

```@example rosenbrock
sol = solve(prob, Optim.KrylovTrustRegion())
```

### Now derivative-based optimizers with various constraints

```@example rosenbrock
cons = (res, x, p) -> res .= [x[1]^2 + x[2]^2]
optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff(); cons = cons)

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-Inf], ucons = [Inf])
sol = solve(prob, IPNewton()) # Note that -Inf < x[1]^2 + x[2]^2 < Inf is always true

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-5.0], ucons = [10.0])
sol = solve(prob, IPNewton()) # Again, -5.0 < x[1]^2 + x[2]^2 < 10.0

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-Inf], ucons = [Inf],
    lb = [-500.0, -500.0], ub = [50.0, 50.0])
sol = solve(prob, IPNewton())

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [0.5], ucons = [0.5],
    lb = [-500.0, -500.0], ub = [50.0, 50.0])
sol = solve(prob, IPNewton())

# Notice now that x[1]^2 + x[2]^2 ≈ 0.5:
res = zeros(1)
cons(res, sol.u, _p)
println(res)
```

```@example rosenbrock
function con_c(res, x, p)
    res .= [x[1]^2 + x[2]^2]
end

optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff(); cons = con_c)
prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-Inf], ucons = [0.25^2])
sol = solve(prob, IPNewton()) # -Inf < cons_circ(sol.u, _p) = 0.25^2
```

## Evolutionary.jl Solvers

```@example rosenbrock
using OptimizationEvolutionary
sol = solve(prob, CMAES(μ = 40, λ = 100), abstol = 1e-15) # -Inf < cons_circ(sol.u, _p) = 0.25^2
```

## IPOPT through OptimizationMOI

```@example rosenbrock
using OptimizationMOI, Ipopt

function con2_c(res, x, p)
    res .= [x[1]^2 + x[2]^2, x[2] * sin(x[1]) - x[1]]
end

optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoZygote(); cons = con2_c)
prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-Inf, -Inf], ucons = [100.0, 100.0])
sol = solve(prob, Ipopt.Optimizer())
```

## Now let's switch over to OptimizationOptimisers with reverse-mode AD

```@example rosenbrock
import OptimizationOptimisers
optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoZygote())
prob = SciMLBase.OptimizationProblem(optf, x0, _p)
sol = solve(prob, OptimizationOptimisers.Adam(0.05), maxiters = 1000, progress = false)
```

## Try out CMAEvolutionStrategy.jl's evolutionary methods

```@example rosenbrock
using OptimizationCMAEvolutionStrategy
sol = solve(prob, CMAEvolutionStrategyOpt())
```

## Now try a few NLopt.jl solvers with symbolic differentiation via ModelingToolkit.jl

```@example rosenbrock
using OptimizationNLopt, ModelingToolkit
optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoSymbolics())
prob = SciMLBase.OptimizationProblem(optf, x0, _p)

sol = solve(prob, Opt(:LN_BOBYQA, 2))
sol = solve(prob, Opt(:LD_LBFGS, 2))
```

### Add some box constraints and solve with a few NLopt.jl methods

```@example rosenbrock
prob = SciMLBase.OptimizationProblem(optf, x0, _p, lb = [-1.0, -1.0], ub = [0.8, 0.8])
sol = solve(prob, Opt(:LD_LBFGS, 2))
sol = solve(prob, Opt(:G_MLSL_LDS, 2), local_method = Opt(:LD_LBFGS, 2), maxiters = 10000) #a global optimizer with random starts of local optimization
```

## BlackBoxOptim.jl Solvers

```@example rosenbrock
using OptimizationBBO
prob = SciMLBase.OptimizationProblem(rosenbrock, [0.0, 0.3], _p, lb = [-1.0, 0.2],
    ub = [0.8, 0.43])
sol = solve(prob, BBO_adaptive_de_rand_1_bin()) # -1.0 ≤ x[1] ≤ 0.8, 0.2 ≤ x[2] ≤ 0.43
```

And this is only a small subset of what Optimization.jl has to offer!
