Commit f8c1176c authored by Le Galiot Agathe's avatar Le Galiot Agathe
Browse files

initial commit

.. _user_guide:
.. toctree::
:maxdepth: 2
:caption: Contents:
User guide
test working example
understanding Adjustin
Adjustin structure
Adjustin application structure
Adding your model
Try out algorithms
Known bugs
* When using Atom variables are registered, error will be thrown when you override those variables. To fix it just close and re-open Atom
.. Adjustin documentation master file, created by
sphinx-quickstart on Tue May 4 16:09:07 2021.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Adjustin's documentation!
.. toctree::
:maxdepth: 4
.. topic:: Installation guide
:ref:`To installation_guide <installation_guide>`
.. topic:: User guide
:ref:`To user guide <user_guide>`
.. topic:: Developper guide
:ref:`To developer guide <developer_guide>`
.. _developer_guide :
.. toctree::
:maxdepth: 4
:caption: Contents:
Developper guide
Right before getting to work on your model, let's just see quickly the workflow you could implement so that everything works smoothly.
* Modifying Adjustin package only locally
When adding your model it is possible that you will end up wanting to add/modify files in the Adjustin package.
Because you can't directly change this package, a workaround is to ``pkg> dev --local``
This replace the command ``pkg> add`` as seen in the installation guide.
Tip : If you want to make your workflow faster, you can download the package Revise by doing ``pkg> add Revise``.
You then only need to add ``using Revise`` before your ``using Adjustin``. This will allow you to avoid closing Atom and compiling again the whole Adjustin package everytime you make a change to it.
* Modifying Adjustin package on gitlab
The steps are the same as in the previous section, however if you want to afterwards push your change to gitlab, there's a few more steps that need to be done.
**Before modifying any files** of the package,
1. Create a new branch for this project
2. Open a command line (git bash) and go to the root directory of the local package of Adjustin that you were using up until now. It contains the folder .git (.julia/environment/[julia_version]/dev/Adjustin)
3. Indicate the branch you want to work on `` git checkout your_branch_name``. If branch name is not recognised try ``git fetch`` and then checkout again
You are now free to make any changes you like.
And finally, when you want to push back the changes on git, simply do :
1. ``git add .``
2. ``git commit -m "message"``
3. ``git push origin your_branch_name``
Add probability fucntion
Add noises
Add algorithms
Add models
Add simulations
.. _installation_guide :
.. toctree::
:maxdepth: 4
:caption: Contents:
Installation guide
* Download julia
* Download an editor (Atom for example) and the package associated to this editor to work on julia code (uber-juno if working with Atom -> file/settings/install/uber-juno)
If you chose Atom as your editor, you can now access julia REPL by opening Atom.
Note that you can configure what you see when you open Atom, including the REPL if it's not automaticaly opened (file/settings/Packages/julia-client/settings/UI option/Layout options)
Note also that when you access the julia REPL, you should see by default ``julia>``, press key "]" (i.e "alt gr" + ")" ) to see ``pkg>``
Remember that if you want to check the installed package, you can do so by writting ``pkg> status``.
You can use it at any time during installation to check if everything was done properly.
IMPORTANT : This git depository must not be cloned on your computer. It is a julia package it thus need to be added like you would do for a julia package (cf. below).
Installation for use only
1. master branch
Write and execute the command below :
.. code-block:: julia
pkg> add
2. specific branch
Write and execute the command below :
.. code-block:: julia
Installation for use and developement
1. master branch
Write and execute the command below :
.. code-block:: julia
pkg> dev --local
Note that you ``dev`` a package and not ``add`` it so to be able to change it.
If, in the past, you have already added the package Adjustin, you might end up with different Adjustin package on your computer which is not a problem.
You can check the package currently used by going in a julia REPL and executing the command :
.. code-block:: julia
pkg> status
The first line will indicate from where the packages are recovered (Status C:/Users/MICS/.julia/environments/v1.3/Project.toml).
* When Adjustin is imported with ``dev``:
The Adjustin package will be in C:/Users/MICS/.julia/environments/v1.3/dev/Adjustin.
When doing ``pkg>status`` you will see *dev/Adjustin* next to the Adjustin pkg.
* When it is imported with ``add`` :
It will be in C:/Users/MICS/.julia/packages/Adjustin.
This can of course be slightly different depending on julia's version, computer.
If the packages are not recovered from where you want them to be, then from the julia REPL go to the path where the correct file Project.toml is located by using ``cd("path")`` and then execute the command below :
.. code-block:: julia
pkg> activate .
2. specific branch
Follow exactly the same steps as for the master branch.
However if you want to use the code of another branch, continue with the following steps.
In git bash terminal go to path to Adjustin but remember that they might be multiple path to Adjustin and that the one you want is the one in *[PATH_TO_JULIA]/.julia/environments/[JULIA_VERSION]/dev/Adjustin*
.. code-block:: julia
cd path_to_Adjustin
You can then execute the commands :
.. code-block:: julia
git fetch origin galiota_branch:galiota_branch
git checkout galiota_branch
Adjustin application
1. master branch
In git bash terminal, in the directory where you want to store your applications
.. code-block:: julia
git clone
2. specific branch
In git bash terminal, in the directory where you want to store your applications
.. code-block:: julia
git clone --branch galiota_branch
.. _user_guide :
.. toctree::
:maxdepth: 4
:caption: Contents:
User guide
testing working example
Let's try out one of the working examples.
In an editor, open the file adjustinapplications>applications>specific>gls-lnas
Select the whole code and press ctrl + enter
If everything is working this should execute the selected code otherwise an error will be thrown.
understanding Adjustin
Adjustin structure
algorithms #store all the algorithms
|filters.jl #handles use of particles and indices
|input_output.jl #functions to import files from database, write files in results folder...
Adjustin application structure
|generic #generic examples of applications using various algorithms
|specific #specific examples of applications using various algorithms
|your_app_here #where you will write your own application
database #store model variables that will be loaded (cf. applications code) and then use in your code
results #store the results of every execution, files inside can be deleted once out of use
Mathematical framework
.. math::
x_{n+1} &= f(x_{n}, u_{n}, \theta_{n}, mn_{n}) \\
y_{n} &= g(x_{n}, \theta_{n}, on_{n})
* :math:`x_{n}` represents the state variables
* :math:`y_{n}` represents the observation on the system
* :math:`u_{n}` represents the external variables (the environment) influencing the system
* :math:`\theta` represents the parameters of our model
* :math:`mn_{n}` represents the modelling/process noise - accounts for either possible model limitations or imperfections
* :math:`on_{n}` represents the observation noise - accounts for the measurement errors
The objective is to understand statistical systems thanks to the estimation of model parameters using experimental data.
We want to create a model which modelised the best way possible what we want to study in order to analyse it.
In other terms the objective is to find f, :math:`\theta`, mn, on, so that the experimental observations fit the best way possible y for each time step.
Note that x can either be hidden state variables or observed data. y is thus x filtered in order to keep only the state variable that were actually measured experimentally (which exclude the hidden state variables)
Adjustin structures & functions
1. structures
* x, u, p (the mathematical :math:`\theta`) are mutable structs with fields of your choice
* mn, on are mutable structs with field of type Noise. The type noise is itselft a mutable struct with the fields :
.. code-block:: julia
mutable struct Noise
ll_det::LabelList #the name of the deterministic variable
ll_sto::LabelList #the name of the stochastic variable
kind::Label #distribution type
vl::Tuple #parameters of this distribution like (mean, standard deviation) or (alpha, beta)...
* so_* stands for system of observation, they are mutable struct of the form :
.. code-block:: julia
mutable struct SystemObservation
oml::LabelList #list of names of the observed states : ["om1","om2","om3"]
otll::Array{ObservationTupleList} #list of values (vl) at each time step for each observed state :
# [(t1,[x_t1_om2]),(t2,[x_t2_om2]),...,(tn,[x_tn_om2])],
# [(t1,[x_t1_om3]),(t2,[x_t2_om3]),...,(tn,[x_tn_om3])]]
set::Function #function to set a new tuple in otll : so.set(, t, vl)
* A list of Observer represents in Adjustin the mathematical function g in the sense that it is a struct that contains the necessary informations to recover the observed states. One Observer is a mutable struct of the form :
.. code-block:: julia
struct Observer
om::Label #name of one field of the state x or name of an observation_noise
f::Function #function to retrieve the value of "om"
tml::Timeline #every times on which om is going to be computed
nbr_args::Int #number of arguments that f takes as input
* The observer model is used to create the Observer. It's a mutable struct of the form :
.. code-block:: julia
struct ObserverModel
2. functions
* create_observation_function(ol) creates a list of Observer (i.e what stand for the mathematical g function) with ol, a list of ObserverModel, as input
* f, the transition function, is as decribed mathematically
* simulate(f, g, x0, u, p; mn=nothing, on=nothing) is a function that will return a SystemObservation with the values of only the observed state variables on the timeline defined in g. To do so it will launch the function f at every timestep (timestep being 1), computing every field of the state, but will then keep in memory in the SystemObservation only the values on the desired timeline for the desired field of state x.
Adding your model : Detailed template
We can now finally start adding our model.
Note that in the whole section we will use the notation *TC_* to indicate code that needs to be change.
For example "TC_model_name" can become "greenlab-arabidopsis"
* main file .jl template in adjustin application
.. code-block:: julia
ENV["ADJUSTIN_PATH_DATABASE"] = string(@__DIR__, "/database/")
ENV["ADJUSTIN_PATH_RESULTS"] = string(@__DIR__, "/results/")
using Adjustin
str_m must be the same name as folder database/control/TC_model_name and database/observations/TC_model_name
and database/parameters/TC_model_name and database/state/TC_model_name. Add folder TC_model_name manually if it doesn't already exist.
If you are using a preexisting package in Adjustin it must also be the same name, as the file in the package Adjustin located at src/models/TC_model_name.jl.
If you are writing a new model it must be the same name as the file in Adjustin application at location database/models/TC_model_name/TC_model_name_version.jl
.. code-block:: julia
str_m = "TC_model_name"
str_c must be the same as file located in database/control/TC_model_name/TC_controle_file_name.jl
str_x must be the same as file located in database/state/TC_model_name/TC_state_file_name.jl
str_p must be the same as file located in database/parameters/TC_model_name/TC_parameters_file_name.jl
str_d can be anything
.. code-block:: julia
str_c = "TC_controle_file_name"
str_x = "TC_state_file_name"
str_p = "TC_parameters_file_name"
str_d = "TC_results_part_of_file_name"
str_dir = create_results_direTCory(; title = str_d)
load_local_model(str_m) #use load_model(str_m) instead if you are using a preexistant model defined in Adjustin package
load_state(str_m, str_x)
load_control(str_m, str_c)
load_parameters(str_m, str_p)
The observation noise, on, is optionnal
TC_field_name_observed must be the same as one of the field in the struct ObservationNoise in Adjustin:src/models/TC_model_name.jl
which is also the same as one of the field in the struct State in Adjustin:src/models/TC_model_name.jl
obs in the beginning of obs_TC_field_name_observed must be there.
TC_kind can be : additive-normal/multiplicative-normal/multiplicative-lognormal/additive-uniform/multiplicative-uniform
How TC_int_param_noise_1 and 2 are used is detailed in Adjustin src/core/noise.jl in function noise!()
.. code-block:: julia
on = ObservationNoise(Noise(
(TC_int_param_noise_1, TC_int_param_noise_2),
The modelling noise mn is optionnal
TC_field_name_det (det for deterministic) must be the same as one of the field in the struct State in Adjustin:src/models/TC_model_name.jl
TC_field_name_sto (sto for stochastic) must be the same as one of the field in the struct State in Adjustin:src/models/TC_model_name.jl
.. code-block:: julia
mn = ModellingNoise(Noise(
(TC_int_param_noise_1, TC_int_param_noise_1)),
g will be a list of Observer. '\r\n'
As seen above, an observer is a structure containing the name of the object we want to observe (usually fields in state and/or observation noise), a function to retrieve the value of this object, the timeline on witch we are observing it, and the number of arguments that the function needs.
`all` (see below) is optional
it is added (usually more in simulations than estimations) if you want to observed all fields in the state
and not only the stochastic ones
.. code-block:: julia
tmax = TC_int
tml = 1:tmax
g = create_observation_function([ObserverModel("all,obs_TC_field_name_observed", tml)])
another way to compute g, if observation data are imported (for estimation), is :
.. code-block:: julia
so_exp = load_from_obs("database/observations/TC_model_name/TC_observation_file_name.obs")
g = create_observation_function(eval(Meta.parse(get_observation_function(so_exp))))
We can then launch the simulation
.. code-block:: julia
so = simulate(f, g, x0, u, p; mn = mn, on = on)
OR the estimation algorithm
see cfg parameters in Adjustin : src/algorithms/estimation/TC_algo_estimation.jl
.. code-block:: julia
cfg = ConfigurationTC_algo_estimation(...)
so = TC_algo_estimation(f, g, x0, u, p, so_exp, cfg; mn = mn, on = on,)
OR the sensitivity_analysis
see cfg parameters in Adjustin : src/algorithms/sensitivity_analysis/TC_algo_sensitivity_analysis.jl
.. code-block:: julia
cfg = ConfigurationTC_algo_sensitivity_analysis(...)
so = TC_algo_sensitivity_analysis(f, g, x0, u, p, cfg; mn = mn, on = on)
We can then explicitly save the results but by activating flag_plot or flag_write in the algorithme configuration, there is automtically saved results
.. code-block:: julia
save_to_obs(TC_what_you_want_to_save, str_dir * "TC_what_you_want_to_save.obs")
* model.jl template - located in Adjustin : src/models/model.jl
First all the struct must be defined
.. code-block:: julia
mutable struct State
TC_fields1 :: TC_type_fields1
TC_field_name_observed :: TC_type_field_name_observed
TC_field_name_det :: TC_type_field_name_sto #cf code *.jl above (lign starting with : "mn = ModellingNoise")
TC_field_name_sto :: TC_type_field_name_sto #cf code *.jl above (lign starting with : "mn = ModellingNoise")
mutable struct Control
TC_fieldc1 :: TC_type_fieldc1
TC_fieldc2 :: TC_type_fieldc2
mutable struct Parameters
TC_fieldp1 :: TC_type_fieldp1
TC_fieldp2 :: TC_type_fieldp2
mutable struct ModellingNoise
TC_fieldm1 :: TC_type_fieldm1
TC_fieldm2 :: TC_type_fieldm2
mutable struct ObservationNoise
#cf main file .jl above (line starting with : "on = ObservationNoise")
TC_field_name_observed :: TC_type_field_name_observed #same field name as in one field of state
Then the function returning the state at the next time step must be defined
.. code-block:: julia
function f(n, xn, u, p)
return xnplus1
If there is a modelling noise in the model then it must be explicitly put as a parameter of f
.. code-block:: julia
function f(n, xn, u, p, mn)
return xnplus1
Finally this next part must be there if we have observation noises on a state variable