# System construction

!!! warning "Internal API"
    The content on this page describes internal implementation details of ModelingToolkit.
    These APIs may change without notice.

A `System` is the fundamental data structure in MTK. It is a hierarchical data structure which
contains all symbolic information about a particular numerical problem. A `System` in general
does not constrain what sorts of information can and cannot be provided about the system. It
merely enforces invariants required for the construction of a valid system. Different problem
constructors validate that the system they receive is valid for generating code for that type
of numerical problem.

## Time-dependent and independent systems

The most broad categorization of a system is time-dependence. A system may represent dynamics
that vary over time (or any independent variable in general) or a time-independent problem.
Time-dependent systems include those for ODEs, SDEs, Jump processes, etc. Time-independent
ones include nonlinear systems, optimization systems, and so on. Programmatically, this distinction
is made via the `iv::Union{SymbolicT, Nothing}` field of the system. Union-splitting or type-asserting
this field access is highly recommended. For example, if a function knows it operates on time-dependent
systems, it should use [`get_iv`](@ref)`(sys)::SymbolicT` when accessing the field to improve type-stability
and the quality of the code generated by the Julia compiler. In general, these sorts of type-assertions
on unstable fields are very useful for performant symbolic-numeric code.

## Contracts of the type

The inner constructor for `System` takes the fields in order. While the order of fields doesn't
really matter, it is useful to group them along with similar fields in the struct. All fields
should be given as concrete of a type as possible without making `System` parametric.

The system should _never_ store wrapped types. Any symbolic expression anywhere in the system
should always be of type `SymbolicT`. Storing `Num`, `Arr{T, N}`, etc. is invalid. In general,
`SymbolicT` is a useful type-alias. Due to the `Const` variant of `BasicSymbolic`, it is often
useful to store values of various different types and sizes as a `SymbolicT`.

All fields should have a docstring. Fields which are only intended for internal access should
be annotated with `$INTERNAL_FIELD_WARNING` at the top of the docstring. Note that "internal"
here doesn't mean only ModelingToolkitBase. Internal fields can also be used by
ModelingToolkitTearing and ModelingToolkit.

## Immutability and updating fields

!!! danger "Immutability of `System`"
  `System` should only ever be an immutable type with no type-parameters. Introducing type-parameters makes
  ensuring type-stability of most symbolic passes very difficult. In the future, if the struct
  grows large enough that passing it to functions is a significant slowdown (evidenced by benchmarks or
  Julia's lowered IR) it can be made a `mutable struct` with all `const` fields. This is _imperative_ for
  ensuring correct behavior.

Almost all fields of `System` should be considered immutable. Internal mutation is also disallowed, with
the specific exceptions of [`get_initial_conditions`](@ref)(sys) and [`get_guesses`](@ref)(sys). Outside of these two
fields, no other field may be mutated. To change fields of a system, use `Setfield.@set` or `Setfield.@set!`.
This may be changed in the future to `Accessors.@set` and `Accessors.@reset` respectively, without breaking
backward compatibility. For example, to add a parameter `p` to a system, do

```julia
sys # The system
ps = copy(get_ps(sys)) # note the `copy`
push!(ps, p)
@set! sys.ps = ps
```

In case multiple such fields need to be set in quick succession, using `ConstructionBase.setproperties`
directly generates better code after compilation. This function is implemented for `AbstractSystem` as
an `@generated` function which performs additional validation/updates and is what `@set`/`@set!` eventually
call.

## Constructors

The inner constructor of `System` optionally performs some basic validation fundamental to the type, and
builds the struct. The `checks` keyword is used to control what validation is performed. The checks here
are not allowed to mutate the fields, and should not be unduly expensive to perform. This constructor
is only ever called internally, and is not user-facing.

The main user-facing constructor is `System(eqs::Vector{Equation}, iv, dvs, ps, brownians = SymbolicT[]; kws...)`.
Most fields of `System` are provided as keyword arguments. This constructor performs significant additional
validation.

!!! note `__legacy_defaults__`
  This keyword arguments only exists for the now-deprecated `@mtkmodel`. It should not be used by anything
  else.

This constructor is responsible for:

- Unwrapping all wrapped symbolic variables/expressions passed to it.
- Coercing all containers to the appropriate type.
- Populating `var_to_name`.
- Validating bindings.
- Parsing bindings/initial conditions/guesses from variable metadata.
- Converting symbolic event shorthand into `SymbolicContinuousCallback`/`SymbolicDiscreteCallback`.

!!! note "Parsing metadata"
  It is often useful to disable parsing bindings/initial conditions/guesses from variable metadata during
  system construction. This is usually when a new system is built from an existing one. Passing
  `discover_from_metadata = false` disables the parsing.

Other constructors are shorthand that call the main constructor. Notably, `System(eqs, t)` automatically
discovers variables and parameters from the equations (and other information) passed to the constructor.
Similar behavior exists for the time-independent constructors. The variable discovery process is very
carefully crafted. Delays, BVP constraints, etc. all reference variables in non-standard ways which
require precise handling.

## System equality

`System` uses the default fallback of `isequal`, which is to compare referential equality. This is
because checking if two expressions are equivalent is undecidable in general. In addition, it is
very expensive to check equality in a way that is agnostic to variable/equation ordering.
[`Base.isapprox`](@ref Base.isapprox(::System, ::System)) is provided to check this sort of equality, in case it is
necessary.

```@docs
Base.isapprox(::System, ::System)
```

## Metadata API

Systems allow storing and accessing metadata using the same API as symbolic expressions.
`SymbolicUtils.setmetadata`, `SymbolicUtils.getmetadata`, `SymbolicUtils.hasmetadata` are
implemented appropriately. Note that `setmetadata` returns a new system with the added metadata,
since `System`s are immutable (similar to the behavior for expressions).

In addition, `System` provides a caching API. This cache is mutable, and storing values here does
not return a new system. This cache is currently only for internal use. It is highly useful for
caching expensive intermediate results, such as symbolic jacobians.

```@docs
ModelingToolkitBase.check_mutable_cache
ModelingToolkitBase.store_to_mutable_cache!
ModelingToolkitBase.should_invalidate_mutable_cache_entry
```

These functions are only intended to be used on flattened systems. Behavior is undefined when used on
non-flattened systems.

## Preparation for codegen

Systems have additional constraints before codegen can be done. For differential-equation-based systems
(ODEs, DAEs, SDEs, etc.) the equations must either be differential equations of the form

```julia
D(x) ~ rhs
```

or algebraic equations of the form

```julia
0 ~ rhs
```

Here, `rhs` is an expression that cannot involve operator applications such as `D(x)`. `rhs` may involve
observed variables, in which case the required observed equations will be present during code generation.

In addition, all systems must be flattened and marked as complete via [`complete`](@ref). The function will not
validate the above structure of equations. It performs the following operations:

- Discover all `GlobalScope` variables in the system.
- Expand all `connect` equations.
- Flatten the hierarchical system.
- Convert discrete variables (`@discretes`) to parameters.
- Construct `ParameterBindingsGraph`.
- Remove bound parameters from the list of parameters
- Validate the absence of any equations involving only parameters.
- Add `Initial` parameters to the system.
- Appropriately prepare all symbolic callbacks.
- If `split = true` (recommended) creates the `IndexCache` which contains indexing and
  structure information for parameters. Also reorders all parameters as required by
  the data structure.
- Disabling namespacing for the top-level system.

Note that this list may change between versions. In general, systems are allowed to perform any
transformation and caching during [`complete`](@ref) which does not modify the intent of the system.

### `Initial` parameters

ModelingToolkit inserts special `Initial` parameters during [`complete`](@ref). These are an important
utility for correctly building the initialization system during codegen. They are almost always
invisible to the user. They do not show up in [`parameters`](@ref)(sys), but can be obtained via
[`get_ps`](@ref)(sys). They are stored in the `.initials` field of `MTKParameters`.

`Initial` parameters exist for every unknown and observed. For time dependent systems, `Initial`
parameters are also added for the first derivative of each unknown and observed variable. All
floating-point discrete variables ([`ModelingToolkitBase.is_variable_floatingpoint`](@ref)) also
have an `Initial` parameter, as do floating-point bound parameters.

```@docs
ModelingToolkitBase.is_variable_floatingpoint
```

Observed equations must be topologically sorted. This is also required for [`full_equations`](@ref) to
work correctly.

SDEs must be phrased in terms of `noise_eqs` and not `brownians`. No brownian variables should
be present in the equations.

The order and number of unknowns should correspond to that of equations. For example, if the
`i`th equation is

```julia
D(x) ~ rhs
```

Then `x` should be the `i`th unknown. Algebraic variables often have a weaker correspondence.

## `AtomicArrayDict`

ModelingToolkit stores variable-value mappings using `AtomicArrayDict`. It is a custom
`AbstractDict` wrapping any other `AbstractDict`, and disallows scalarized/indexed arrays from
being keys. Initial conditions, guesses and bindings are stored as this type (or wrappers thereof).

Several functions exist for manipulating `AtomicArrayDict`.

```@docs
ModelingToolkitBase.AtomicArrayDict
ModelingToolkitBase.as_atomic_dict_with_defaults
ModelingToolkitBase.write_possibly_indexed_array!
ModelingToolkitBase.has_possibly_indexed_key
ModelingToolkitBase.get_possibly_indexed
```

In addition, `AtomicArraySet` wraps `AtomicArrayDict` to expose an `AbstractSet` which cannot
contain scalarized/indexed arrays.

```@docs
ModelingToolkitBase.AtomicArraySet
ModelingToolkitBase.as_atomic_array_set
ModelingToolkitBase.push_as_atomic_array!
ModelingToolkitBase.contains_possibly_indexed_element
```

When using an `AtomicArrayDict` as a list of substitution rules, it is typically necessary to
wrap it in `AtomicArrayDictSubstitutionWrapper`.

```@docs
ModelingToolkitBase.AtomicArrayDictSubstitutionWrapper
```

## Variable utilities

To get all the discrete variables of a system, use [`ModelingToolkitBase.get_all_discretes`](@ref). If the system
has been marked as [`complete`](@ref), use [`ModelingToolkitBase.get_all_discretes_fast`](@ref) instead.

```@docs
ModelingToolkitBase.get_all_discretes
ModelingToolkitBase.get_all_discretes_fast
```

For array variables, it is often necessary to obtain the parent array from a scalarized element. This
should be done using [`ModelingToolkitBase.split_indexed_var`](@ref). To obtain a
`SymbolicUtils.StableIndex{Int}` representing the index the scalarized variable accesses the array at,
use [`ModelingToolkitBase.get_stable_index`](@ref).

```@docs
ModelingToolkitBase.split_indexed_var
ModelingToolkitBase.get_stable_index
```

## Best practices

`struct Schedule` is defined alongside `System` for convenience, since it is stored in the system by
ModelingToolkitTearing when running `mtkcompile` and contains useful information about the simplified
system.

Always use `@unpack` to obtain variables/parameters/subsystems from a system you intend to `extend`.

Always use `@named` to call component constructors. All component constructors should use the
`@component` or `@connector` macros as appropriate.

In case of conflicts, initial conditions and guesses from parent systems take priority over those in
children.

Any container of symbolic variables/expressions should use `SymbolicT` as the corresponding type.
Functions should be as concretely typed as possible, even to the extent of restricting inputs
to exact types such as `Vector{SymbolicT}`. This helps improve type-stability, reduce invalidations,
and improve precompilation. This is all the more true of internal functions. Such practice also helps
identify bugs when unsupported types are passed into functions, or the impacts of changing a
container type when it is passed downstream.
