From 579b790cce721e7333514a41f5bb164d3c28ef06 Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 8 Oct 2020 19:08:02 -0400 Subject: [PATCH 1/4] Initial attempt at documenting the gp functions (Issue #185) --- .../deprecated_functions.Rmd | 64 +++- src/functions-reference/matrix_operations.Rmd | 294 +++++++++++++++--- 2 files changed, 309 insertions(+), 49 deletions(-) diff --git a/src/functions-reference/deprecated_functions.Rmd b/src/functions-reference/deprecated_functions.Rmd index 77cc11749..954f7ee7c 100644 --- a/src/functions-reference/deprecated_functions.Rmd +++ b/src/functions-reference/deprecated_functions.Rmd @@ -9,6 +9,12 @@ if (knitr::is_html_output()) { } ``` +```{r results='asis', echo=FALSE} +if (knitr::is_html_output()) { + cat(' * cov_exp_quad\n') +} +``` + ## integrate_ode_rk45, integrate_ode_adams, integrate_ode_bdf ODE Integrators {#functions-old-ode-solver} These ODE integrator functions have been replaced by those described in: @@ -24,7 +30,7 @@ if (knitr::is_html_output()) { A system of ODEs is specified as an ordinary function in Stan within the functions block. The ODE system function must have this function signature: - + ``` real[] ode(real time, real[] state, real[] theta, real[] x_r, int[] x_i) @@ -103,7 +109,7 @@ are as follows. * *`ode`*: function literal referring to a function specifying the system of differential equations with signature: - + ``` (real, real[], real[], data real[], data int[]):real[] ``` @@ -126,7 +132,7 @@ derivatives with respect to time of the state, For more fine-grained control of the ODE solvers, these parameters can also be provided: - + * `data` *`rel_tol`*: relative tolerance for the ODE solver, type `real`, data only, @@ -145,7 +151,7 @@ with values consisting of solutions at the specified times. The sizes must match, and in particular, the following groups are of the same size: - + * state variables passed into the system function, derivatives returned by the system function, initial state passed into the solver, and rows of the return value of the solver, @@ -155,3 +161,53 @@ solver, * parameters, real data and integer data passed to the solver will be passed to the system function + + +## Exponentiated quadratic covariance functions {#cov_exp_quad} + +These covariance functions have been replaced by those described in: + +```{r results='asis', echo=FALSE} +if (knitr::is_html_output()) { + cat(' * Gaussian Process Covariance Functions\n') +} +``` + +With scale $\alpha$ and length scale $l$, the exponentiated quadratic kernel is: + +$$ +k(x_i, x_j) = \alpha^2 \exp \left(-\dfrac{1}{2\rho^2} \sum_{d=1}^D (x_{i,d} - x_{j,d})^2 \right) +$$ + +\index{{\tt \bfseries cov\_exp\_quad }!{\tt (row\_vectors x, real alpha, real rho): matrix}|hyperpage} + +`matrix` **`cov_exp_quad`**`(row_vectors x, real alpha, real rho)`
\newline +The covariance matrix with an exponentiated quadratic kernel of x. + +\index{{\tt \bfseries cov\_exp\_quad }!{\tt (vectors x, real alpha, real rho): matrix}|hyperpage} + +`matrix` **`cov_exp_quad`**`(vectors x, real alpha, real rho)`
\newline +The covariance matrix with an exponentiated quadratic kernel of x. + +\index{{\tt \bfseries cov\_exp\_quad }!{\tt (real[] x, real alpha, real rho): matrix}|hyperpage} + +`matrix` **`cov_exp_quad`**`(real[] x, real alpha, real rho)`
\newline +The covariance matrix with an exponentiated quadratic kernel of x. + +\index{{\tt \bfseries cov\_exp\_quad }!{\tt (row\_vectors x1, row\_vectors x2, real alpha, real rho): matrix}|hyperpage} + +`matrix` **`cov_exp_quad`**`(row_vectors x1, row_vectors x2, real alpha, real rho)`
\newline +The covariance matrix with an exponentiated quadratic kernel of x1 and +x2. + +\index{{\tt \bfseries cov\_exp\_quad }!{\tt (vectors x1, vectors x2, real alpha, real rho): matrix}|hyperpage} + +`matrix` **`cov_exp_quad`**`(vectors x1, vectors x2, real alpha, real rho)`
\newline +The covariance matrix with an exponentiated quadratic kernel of x1 and +x2. + +\index{{\tt \bfseries cov\_exp\_quad }!{\tt (real[] x1, real[] x2, real alpha, real rho): matrix}|hyperpage} + +`matrix` **`cov_exp_quad`**`(real[] x1, real[] x2, real alpha, real rho)`
\newline +The covariance matrix with an exponentiated quadratic kernel of x1 and +x2. diff --git a/src/functions-reference/matrix_operations.Rmd b/src/functions-reference/matrix_operations.Rmd index 65dcd32bd..8aeda05c0 100644 --- a/src/functions-reference/matrix_operations.Rmd +++ b/src/functions-reference/matrix_operations.Rmd @@ -1237,66 +1237,270 @@ The cumulative sum of v `row_vector` **`cumulative_sum`**`(row_vector rv)`
\newline The cumulative sum of rv -## Covariance Functions {#covariance} +## Gaussian Process Covariance Functions {#gaussian-process-covariance-functions} -### Exponentiated quadratic covariance function +The Gaussian process covariance functions compute the covariance between a +set of points and itself, $K_{ij} = k(x_i, x_j)$, or the covariance between +two different sets of points, $K_{ij} = k(x_i, x_j)$. $x$ can be an array of +scalars for a one dimensional kernel or an array of vectors for a +multidimensional kernel. -The exponentiated quadratic kernel defines the covariance between -$f(x_i)$ and $f(x_j)$ where $f\colon \mathbb{R}^D \mapsto \mathbb{R}$ -as a function of the squared Euclidian distance between $x_i \in -\mathbb{R}^D$ and $x_j \in \mathbb{R}^D$: \[ \text{cov}(f(x_i), -f(x_j)) = k(x_i, x_j) = \alpha^2 \exp \left( - -\dfrac{1}{2\rho^2} \sum_{d=1}^D (x_{i,d} - x_{j,d})^2 \right) \] with -$\alpha$ and $\rho$ constrained to be positive. +### Squared exponential kernel -There are two variants of the exponentiated quadratic covariance -function in Stan. One builds a covariance matrix, $K \in \mathbb{R}^{N -\times N}$ for $x_1, \dots, x_N$, where $K_{i,j} = k(x_i, x_j)$, which -is necessarily symmetric and positive semidefinite by construction. -There is a second variant of the exponentiated quadratic covariance -function that builds a $K \in \mathbb{R}^{N \times M}$ covariance -matrix for $x_1, \dots, x_N$ and $x^\prime_1, \dots, x^\prime_M$, -where $x_i \in \mathbb{R}^D$ and $x^\prime_i \in \mathbb{R}^D$ and -$K_{i,j} = k(x_i, x^\prime_j)$. +With scale $\sigma$ and length scale $l$, the squared exponential kernel is: - -\index{{\tt \bfseries cov\_exp\_quad }!{\tt (row\_vectors x, real alpha, real rho): matrix}|hyperpage} +$$ +k(x_i, x_j) = \alpha^2 \exp \left( -\dfrac{1}{2\rho^2} |x_i - x_j|^2 \right) +$$ -`matrix` **`cov_exp_quad`**`(row_vectors x, real alpha, real rho)`
\newline -The covariance matrix with an exponentiated quadratic kernel of x. +\index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} - -\index{{\tt \bfseries cov\_exp\_quad }!{\tt (vectors x, real alpha, real rho): matrix}|hyperpage} +`matrix` **`gp_exp_quad_cov`**`(real[] x, real sigma, real length_scale)`
\newline -`matrix` **`cov_exp_quad`**`(vectors x, real alpha, real rho)`
\newline -The covariance matrix with an exponentiated quadratic kernel of x. +Gaussian process with squared exponential kernel in one dimension. - -\index{{\tt \bfseries cov\_exp\_quad }!{\tt (real[] x, real alpha, real rho): matrix}|hyperpage} +\index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} -`matrix` **`cov_exp_quad`**`(real[] x, real alpha, real rho)`
\newline -The covariance matrix with an exponentiated quadratic kernel of x. +`matrix` **`gp_exp_quad_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline - -\index{{\tt \bfseries cov\_exp\_quad }!{\tt (row\_vectors x1, row\_vectors x2, real alpha, real rho): matrix}|hyperpage} +Gaussian process with squared exponential kernel in one dimension between `x1` +and `x2`. -`matrix` **`cov_exp_quad`**`(row_vectors x1, row_vectors x2, real alpha, real rho)`
\newline -The covariance matrix with an exponentiated quadratic kernel of x1 and -x2. +\index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} - -\index{{\tt \bfseries cov\_exp\_quad }!{\tt (vectors x1, vectors x2, real alpha, real rho): matrix}|hyperpage} +`matrix` **`gp_exp_quad_cov`**`(vector[] x, real sigma, real length_scale)`
\newline -`matrix` **`cov_exp_quad`**`(vectors x1, vectors x2, real alpha, real rho)`
\newline -The covariance matrix with an exponentiated quadratic kernel of x1 and -x2. +Gaussian process with squared exponential kernel in multiple dimensions. - -\index{{\tt \bfseries cov\_exp\_quad }!{\tt (real[] x1, real[] x2, real alpha, real rho): matrix}|hyperpage} +\index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} -`matrix` **`cov_exp_quad`**`(real[] x1, real[] x2, real alpha, real rho)`
\newline -The covariance matrix with an exponentiated quadratic kernel of x1 and -x2. +`matrix` **`gp_exp_quad_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline + +Gaussian process with squared exponential kernel in multiple dimensions with a +per-dimensional length scale. + +\index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_exp_quad_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with squared exponential kernel in multiple dimensions between +`x1` and `x2`. + +\index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_exp_quad_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline + +Gaussian process with squared exponential kernel in multiple dimensions between +`x1` and `x2` with a per-dimensional length scale. + +### Dot product kernel + +With scale $\sigma$ the dot product kernel is: + +$$ +k(x_i, x_j) = \sigma^2 + x_i^T x_j +$$ + +\index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (real[] x, real sigma): matrix}|hyperpage} + +`matrix` **`gp_dot_prod_cov`**`(real[] x, real sigma)`
\newline + +Gaussian process with squared exponential kernel in one dimension. + +\index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (real[] x1, real[] x2, real sigma): matrix}|hyperpage} + +`matrix` **`gp_dot_prod_cov`**`(real[] x1, real[] x2, real sigma)`
\newline + +Gaussian process with squared exponential kernel in one dimension between `x1` and `x2`. + +\index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (vector[] x, real sigma): matrix}|hyperpage} + +`matrix` **`gp_dot_prod_cov`**`(vector[] x, real sigma)`
\newline + +Gaussian process with squared exponential kernel in multiple dimensions. + +\index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (vector[] x1, vector[] x2, real sigma): matrix}|hyperpage} + +`matrix` **`gp_dot_prod_cov`**`(vector[] x1, vector[] x2, real sigma)`
\newline + +Gaussian process with squared exponential kernel in multiple dimensions between +`x1` and `x2`. + +### Exponential kernel + +With scale $\sigma$ and length scale $l$, the exponential kernel is: + +$$ +k(x_i, x_j) = \sigma^2 exp(-\frac{|x_i - x_j|}{2 l^2}) +$$ + +\index{{\tt \bfseries gp\_exponential\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_exponential_cov`**`(real[] x, real sigma, real length_scale)`
\newline + +Gaussian process with exponential kernel in one dimension. + +\index{{\tt \bfseries gp\_exponential\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_exponential_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with exponential kernel in one dimension between `x1` and `x2`. + +\index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_exponential_cov`**`(vector[] x, real sigma, real length_scale)`
\newline + +Gaussian process with exponential kernel in multiple dimensions. + +\index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_exponential_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline + +Gaussian process with exponential kernel in multiple dimensions with a +per-dimensional length scale. + +\index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_exponential_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with exponential kernel in multiple dimensions between `x1` +and `x2`. + +\index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_exponential_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline + +Gaussian process with exponential kernel in multiple dimensions between `x1` +and `x2` with a per-dimensional length scale. + +### Matern 3/2 kernel + +With scale $\sigma$ and length scale $l$, the Matern 3/2 kernel is: + +$$ +k(x_i, x_j) = \sigma^2(1 + \frac{\sqrt{3}|x_i - x_j|}{l}) \exp(-\frac{\sqrt{3}|x_i - x_j|)}{l}) +$$ + +\index{{\tt \bfseries gp\_matern32\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern32_cov`**`(real[] x, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 3/2 kernel in one dimension. + +\index{{\tt \bfseries gp\_matern32\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern32_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 3/2 kernel in one dimension between `x1` and `x2`. + +\index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern32_cov`**`(vector[] x, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 3/2c kernel in multiple dimensions. + +\index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern32_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline + +Gaussian process with Matern 3/2 kernel in multiple dimensions with a +per-dimensional length scale. + +\index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern32_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 3/2 kernel in multiple dimensions between `x1` +and `x2`. + +\index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern32_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline + +Gaussian process with Matern 3/2 kernel in multiple dimensions between `x1` +and `x2` with a per-dimensional length scale. + +### Matern 5/2 kernel + +With scale $\sigma$ and length scale $l$, the Matern 5/2 kernel is: + +$$ +k(x_i, x_j) = \sigma^2 \left(1 + + \frac{\sqrt{5}|x_i - x_j|}{l} + \frac{5 |x_i - x_j|^2}{3l^2} \right) + \exp \left(-\frac{5 |x_i - x_j|}{l} \right) +$$ + +\index{{\tt \bfseries gp\_matern52\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern52_cov`**`(real[] x, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 5/2 kernel in one dimension. + +\index{{\tt \bfseries gp\_matern52\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern52_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 5/2 kernel in one dimension between `x1` and `x2`. + +\index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern52_cov`**`(vector[] x, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 5/2 kernel in multiple dimensions. + +\index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern52_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline + +Gaussian process with Matern 5/2 kernel in multiple dimensions with a +per-dimensional length scale. + +\index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern52_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline + +Gaussian process with Matern 5/2 kernel in multiple dimensions between `x1` +and `x2`. + +\index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} + +`matrix` **`gp_matern52_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline + +Gaussian process with Matern 5/2 kernel in multiple dimensions between `x1` +and `x2` with a per-dimensional length scale. + +### Periodic kernel + +With scale $\sigma$, length scale $l$, and period $p$, the periodic kernel is: + +$$ +k(x_i, x_j) = \sigma^2 \exp \left(-\frac{2 \sin^2(\pi \frac{|x_i - x_j|}{p}) }{l^2} \right) +$$ + +\index{{\tt \bfseries gp\_periodic\_cov }!{\tt (real[] x, real sigma, real length_scale, real period): matrix}|hyperpage} + +`matrix` **`gp_periodic_cov`**`(real[] x, real sigma, real length_scale, real period)`
\newline + +Gaussian process with periodic kernel in one dimension. + +\index{{\tt \bfseries gp\_periodic\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale, real period): matrix}|hyperpage} + +`matrix` **`gp_periodic_cov`**`(real[] x1, real[] x2, real sigma, real length_scale, real period)`
\newline + +Gaussian process with periodic kernel in one dimension between `x1` and `x2`. + +\index{{\tt \bfseries gp\_periodic\_cov }!{\tt (vector[] x, real sigma, real length_scale, real period): matrix}|hyperpage} + +`matrix` **`gp_periodic_cov`**`(vector[] x, real sigma, real length_scale, real period)`
\newline + +Gaussian process with periodic kernel in multiple dimensions. + +\index{{\tt \bfseries gp\_periodic\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale, real period): matrix}|hyperpage} + +`matrix` **`gp_periodic_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale, real period)`
\newline + +Gaussian process with periodic kernel in multiple dimensions between `x1` and +`x2`. ## Linear Algebra Functions and Solvers From 7f1c2f3f0437bf2f229f0c3bd7e892326bdf256f Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 8 Oct 2020 19:14:38 -0400 Subject: [PATCH 2/4] Updated link to cov_exp_quad docs (Issue #185) --- src/functions-reference/deprecated_functions.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/functions-reference/deprecated_functions.Rmd b/src/functions-reference/deprecated_functions.Rmd index 954f7ee7c..7a8c9ceb5 100644 --- a/src/functions-reference/deprecated_functions.Rmd +++ b/src/functions-reference/deprecated_functions.Rmd @@ -11,7 +11,7 @@ if (knitr::is_html_output()) { ```{r results='asis', echo=FALSE} if (knitr::is_html_output()) { - cat(' * cov_exp_quad\n') + cat(' * cov_exp_quad\n') } ``` From 9bf43409d61f07777eae4c62fd5601c2515a8564 Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 18 Oct 2020 17:19:33 -0400 Subject: [PATCH 3/4] Updated gp section for review comments (#185) --- .../deprecated_functions.Rmd | 2 +- src/functions-reference/matrix_operations.Rmd | 144 ++++++++++-------- 2 files changed, 79 insertions(+), 67 deletions(-) diff --git a/src/functions-reference/deprecated_functions.Rmd b/src/functions-reference/deprecated_functions.Rmd index 7a8c9ceb5..8bd6029ea 100644 --- a/src/functions-reference/deprecated_functions.Rmd +++ b/src/functions-reference/deprecated_functions.Rmd @@ -173,7 +173,7 @@ if (knitr::is_html_output()) { } ``` -With scale $\alpha$ and length scale $l$, the exponentiated quadratic kernel is: +With magnitude $\alpha$ and length scale $l$, the exponentiated quadratic kernel is: $$ k(x_i, x_j) = \alpha^2 \exp \left(-\dfrac{1}{2\rho^2} \sum_{d=1}^D (x_{i,d} - x_{j,d})^2 \right) diff --git a/src/functions-reference/matrix_operations.Rmd b/src/functions-reference/matrix_operations.Rmd index 8aeda05c0..a6513ad99 100644 --- a/src/functions-reference/matrix_operations.Rmd +++ b/src/functions-reference/matrix_operations.Rmd @@ -1239,268 +1239,280 @@ The cumulative sum of rv ## Gaussian Process Covariance Functions {#gaussian-process-covariance-functions} -The Gaussian process covariance functions compute the covariance between a -set of points and itself, $K_{ij} = k(x_i, x_j)$, or the covariance between -two different sets of points, $K_{ij} = k(x_i, x_j)$. $x$ can be an array of -scalars for a one dimensional kernel or an array of vectors for a -multidimensional kernel. +The Gaussian process covariance functions compute the covariance between +observations in an input data set or the cross-covariance between two input +data sets. + +For one dimensional GPs, the input data sets are arrays of scalars. The +covariance matrix is given by $K_{ij} = k(x_i, x_j)$ (where $x_i$ is the +$i^{th}$ element of the array $x$) and the cross-covariance is given by +$K_{ij} = k(x_i, y_j)$. + +For multi-dimensional GPs, the input data sets are arrays of vectors. The +covariance matrix is given by $K_{ij} = k(\mathbf{x}_i, \mathbf{x}_j)$ (where +$\mathbf{x}_i$ is the $i^{th}$ vector in the array $x$) and the cross-covariance +is given by $K_{ij} = k(\mathbf{x}_i, \mathbf{y}_j)$. ### Squared exponential kernel -With scale $\sigma$ and length scale $l$, the squared exponential kernel is: +With magnitude $\sigma$ and length scale $l$, the squared exponential kernel is: $$ -k(x_i, x_j) = \alpha^2 \exp \left( -\dfrac{1}{2\rho^2} |x_i - x_j|^2 \right) +k(\mathbf{x}_i, \mathbf{x}_j) = \sigma^2 \exp \left( -\frac{|\mathbf{x}_i - \mathbf{x}_j|^2}{2l^2} \right) $$ \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(real[] x, real sigma, real length_scale)`
\newline -Gaussian process with squared exponential kernel in one dimension. +Gaussian process covariance with squared exponential kernel in one dimension. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline -Gaussian process with squared exponential kernel in one dimension between `x1` -and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with squared exponential +kernel in one dimension. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x, real sigma, real length_scale)`
\newline -Gaussian process with squared exponential kernel in multiple dimensions. +Gaussian process covariance with squared exponential kernel in multiple dimensions. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline -Gaussian process with squared exponential kernel in multiple dimensions with a -per-dimensional length scale. +Gaussian process covariance with squared exponential kernel in multiple +dimensions with a length scale for each dimension. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline -Gaussian process with squared exponential kernel in multiple dimensions between -`x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with squared exponential +kernel in multiple dimensions. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline -Gaussian process with squared exponential kernel in multiple dimensions between -`x1` and `x2` with a per-dimensional length scale. +Gaussian process cross-covariance of `x1` and `x2` with squared exponential +kernel in multiple dimensions with a length scale for each dimension. ### Dot product kernel -With scale $\sigma$ the dot product kernel is: +With bias $\sigma_0$ the dot product kernel is: $$ -k(x_i, x_j) = \sigma^2 + x_i^T x_j +k(\mathbf{x}_i, \mathbf{x}_j) = \sigma_0^2 + \mathbf{x}_i^T \mathbf{x}_j $$ \index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (real[] x, real sigma): matrix}|hyperpage} `matrix` **`gp_dot_prod_cov`**`(real[] x, real sigma)`
\newline -Gaussian process with squared exponential kernel in one dimension. +Gaussian process covariance with dot product kernel in one dimension. \index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (real[] x1, real[] x2, real sigma): matrix}|hyperpage} `matrix` **`gp_dot_prod_cov`**`(real[] x1, real[] x2, real sigma)`
\newline -Gaussian process with squared exponential kernel in one dimension between `x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with dot product +kernel in one dimension. \index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (vector[] x, real sigma): matrix}|hyperpage} `matrix` **`gp_dot_prod_cov`**`(vector[] x, real sigma)`
\newline -Gaussian process with squared exponential kernel in multiple dimensions. +Gaussian process covariance with dot product kernel in multiple dimensions. \index{{\tt \bfseries gp\_dot\_prod\_cov }!{\tt (vector[] x1, vector[] x2, real sigma): matrix}|hyperpage} `matrix` **`gp_dot_prod_cov`**`(vector[] x1, vector[] x2, real sigma)`
\newline -Gaussian process with squared exponential kernel in multiple dimensions between -`x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with dot product +kernel in multiple dimensions. ### Exponential kernel -With scale $\sigma$ and length scale $l$, the exponential kernel is: +With magnitude $\sigma$ and length scale $l$, the exponential kernel is: $$ -k(x_i, x_j) = \sigma^2 exp(-\frac{|x_i - x_j|}{2 l^2}) +k(\mathbf{x}_i, \mathbf{x}_j) = \sigma^2 \exp \left( -\frac{|\mathbf{x}_i - \mathbf{x}_j|}{l} \right) $$ \index{{\tt \bfseries gp\_exponential\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exponential_cov`**`(real[] x, real sigma, real length_scale)`
\newline -Gaussian process with exponential kernel in one dimension. +Gaussian process covariance with exponential kernel in one dimension. \index{{\tt \bfseries gp\_exponential\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exponential_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline -Gaussian process with exponential kernel in one dimension between `x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with exponential +kernel in one dimension. \index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exponential_cov`**`(vector[] x, real sigma, real length_scale)`
\newline -Gaussian process with exponential kernel in multiple dimensions. +Gaussian process covariance with exponential kernel in multiple dimensions. \index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_exponential_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline -Gaussian process with exponential kernel in multiple dimensions with a -per-dimensional length scale. +Gaussian process covariance with exponential kernel in multiple +dimensions with a length scale for each dimension. \index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exponential_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline -Gaussian process with exponential kernel in multiple dimensions between `x1` -and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with exponential +kernel in multiple dimensions. \index{{\tt \bfseries gp\_exponential\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_exponential_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline -Gaussian process with exponential kernel in multiple dimensions between `x1` -and `x2` with a per-dimensional length scale. +Gaussian process cross-covariance of `x1` and `x2` with exponential +kernel in multiple dimensions with a length scale for each dimension. ### Matern 3/2 kernel -With scale $\sigma$ and length scale $l$, the Matern 3/2 kernel is: +With magnitude $\sigma$ and length scale $l$, the Matern 3/2 kernel is: $$ -k(x_i, x_j) = \sigma^2(1 + \frac{\sqrt{3}|x_i - x_j|}{l}) \exp(-\frac{\sqrt{3}|x_i - x_j|)}{l}) +k(\mathbf{x}_i, \mathbf{x}_j) = \sigma^2 \left( 1 + \frac{\sqrt{3}|\mathbf{x}_i - \mathbf{x}_j|}{l} \right) \exp \left( -\frac{\sqrt{3}|\mathbf{x}_i - \mathbf{x}_j|}{l} \right) $$ \index{{\tt \bfseries gp\_matern32\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern32_cov`**`(real[] x, real sigma, real length_scale)`
\newline -Gaussian process with Matern 3/2 kernel in one dimension. +Gaussian process covariance with Matern 3/2 kernel in one dimension. \index{{\tt \bfseries gp\_matern32\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern32_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline -Gaussian process with Matern 3/2 kernel in one dimension between `x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with Matern 3/2 +kernel in one dimension. \index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern32_cov`**`(vector[] x, real sigma, real length_scale)`
\newline -Gaussian process with Matern 3/2c kernel in multiple dimensions. +Gaussian process covariance with Matern 3/2 kernel in multiple dimensions. \index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_matern32_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline -Gaussian process with Matern 3/2 kernel in multiple dimensions with a -per-dimensional length scale. +Gaussian process covariance with Matern 3/2 kernel in multiple +dimensions with a length scale for each dimension. \index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern32_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline -Gaussian process with Matern 3/2 kernel in multiple dimensions between `x1` -and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with Matern 3/2 +kernel in multiple dimensions. \index{{\tt \bfseries gp\_matern32\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_matern32_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline -Gaussian process with Matern 3/2 kernel in multiple dimensions between `x1` -and `x2` with a per-dimensional length scale. +Gaussian process cross-covariance of `x1` and `x2` with Matern 3/2 +kernel in multiple dimensions with a length scale for each dimension. ### Matern 5/2 kernel -With scale $\sigma$ and length scale $l$, the Matern 5/2 kernel is: +With magnitude $\sigma$ and length scale $l$, the Matern 5/2 kernel is: $$ -k(x_i, x_j) = \sigma^2 \left(1 + - \frac{\sqrt{5}|x_i - x_j|}{l} + \frac{5 |x_i - x_j|^2}{3l^2} \right) - \exp \left(-\frac{5 |x_i - x_j|}{l} \right) +k(\mathbf{x}_i, \mathbf{x}_j) = \sigma^2 \left( 1 + \frac{\sqrt{5}|\mathbf{x}_i - \mathbf{x}_j|}{l} + \frac{5 |\mathbf{x}_i - \mathbf{x}_j|^2}{3l^2} \right) + \exp \left( -\frac{\sqrt{5} |\mathbf{x}_i - \mathbf{x}_j|}{l} \right) $$ \index{{\tt \bfseries gp\_matern52\_cov }!{\tt (real[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern52_cov`**`(real[] x, real sigma, real length_scale)`
\newline -Gaussian process with Matern 5/2 kernel in one dimension. +Gaussian process covariance with Matern 5/2 kernel in one dimension. \index{{\tt \bfseries gp\_matern52\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern52_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline -Gaussian process with Matern 5/2 kernel in one dimension between `x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with Matern 5/2 +kernel in one dimension. \index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern52_cov`**`(vector[] x, real sigma, real length_scale)`
\newline -Gaussian process with Matern 5/2 kernel in multiple dimensions. +Gaussian process covariance with Matern 5/2 kernel in multiple dimensions. \index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_matern52_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline -Gaussian process with Matern 5/2 kernel in multiple dimensions with a -per-dimensional length scale. +Gaussian process covariance with Matern 5/2 kernel in multiple +dimensions with a length scale for each dimension. \index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_matern52_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline -Gaussian process with Matern 5/2 kernel in multiple dimensions between `x1` -and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with Matern 5/2 +kernel in multiple dimensions. \index{{\tt \bfseries gp\_matern52\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_matern52_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline -Gaussian process with Matern 5/2 kernel in multiple dimensions between `x1` -and `x2` with a per-dimensional length scale. +Gaussian process cross-covariance of `x1` and `x2` with Matern 5/2 +kernel in multiple dimensions with a length scale for each dimension. ### Periodic kernel -With scale $\sigma$, length scale $l$, and period $p$, the periodic kernel is: +With magnitude $\sigma$, length scale $l$, and period $p$, the periodic kernel is: $$ -k(x_i, x_j) = \sigma^2 \exp \left(-\frac{2 \sin^2(\pi \frac{|x_i - x_j|}{p}) }{l^2} \right) +k(\mathbf{x}_i, \mathbf{x}_j) = \sigma^2 \exp \left(-\frac{2 \sin^2 \left( \pi \frac{|\mathbf{x}_i - \mathbf{x}_j|}{p} \right) }{l^2} \right) $$ \index{{\tt \bfseries gp\_periodic\_cov }!{\tt (real[] x, real sigma, real length_scale, real period): matrix}|hyperpage} `matrix` **`gp_periodic_cov`**`(real[] x, real sigma, real length_scale, real period)`
\newline -Gaussian process with periodic kernel in one dimension. +Gaussian process covariance with periodic kernel in one dimension. \index{{\tt \bfseries gp\_periodic\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale, real period): matrix}|hyperpage} `matrix` **`gp_periodic_cov`**`(real[] x1, real[] x2, real sigma, real length_scale, real period)`
\newline -Gaussian process with periodic kernel in one dimension between `x1` and `x2`. +Gaussian process cross-covariance of `x1` and `x2` with periodic +kernel in one dimension. \index{{\tt \bfseries gp\_periodic\_cov }!{\tt (vector[] x, real sigma, real length_scale, real period): matrix}|hyperpage} `matrix` **`gp_periodic_cov`**`(vector[] x, real sigma, real length_scale, real period)`
\newline -Gaussian process with periodic kernel in multiple dimensions. +Gaussian process covariance with periodic kernel in multiple dimensions. \index{{\tt \bfseries gp\_periodic\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale, real period): matrix}|hyperpage} `matrix` **`gp_periodic_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale, real period)`
\newline -Gaussian process with periodic kernel in multiple dimensions between `x1` and -`x2`. +Gaussian process cross-covariance of `x1` and `x2` with periodic +kernel in multiple dimensions with a length scale for each dimension. ## Linear Algebra Functions and Solvers From 49d0d15c9ea56aa494cd280f04050842e6e4713c Mon Sep 17 00:00:00 2001 From: Ben Date: Mon, 19 Oct 2020 09:39:46 -0400 Subject: [PATCH 4/4] Changed squared exponential to exponentiated quadratic (Issue #185) --- src/functions-reference/matrix_operations.Rmd | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/functions-reference/matrix_operations.Rmd b/src/functions-reference/matrix_operations.Rmd index a6513ad99..c23d98fd6 100644 --- a/src/functions-reference/matrix_operations.Rmd +++ b/src/functions-reference/matrix_operations.Rmd @@ -1253,9 +1253,9 @@ covariance matrix is given by $K_{ij} = k(\mathbf{x}_i, \mathbf{x}_j)$ (where $\mathbf{x}_i$ is the $i^{th}$ vector in the array $x$) and the cross-covariance is given by $K_{ij} = k(\mathbf{x}_i, \mathbf{y}_j)$. -### Squared exponential kernel +### Exponentiated quadratic kernel -With magnitude $\sigma$ and length scale $l$, the squared exponential kernel is: +With magnitude $\sigma$ and length scale $l$, the exponentiated quadratic kernel is: $$ k(\mathbf{x}_i, \mathbf{x}_j) = \sigma^2 \exp \left( -\frac{|\mathbf{x}_i - \mathbf{x}_j|^2}{2l^2} \right) @@ -1265,40 +1265,40 @@ $$ `matrix` **`gp_exp_quad_cov`**`(real[] x, real sigma, real length_scale)`
\newline -Gaussian process covariance with squared exponential kernel in one dimension. +Gaussian process covariance with exponentiated quadratic kernel in one dimension. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (real[] x1, real[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(real[] x1, real[] x2, real sigma, real length_scale)`
\newline -Gaussian process cross-covariance of `x1` and `x2` with squared exponential +Gaussian process cross-covariance of `x1` and `x2` with exponentiated quadratic kernel in one dimension. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x, real sigma, real length_scale)`
\newline -Gaussian process covariance with squared exponential kernel in multiple dimensions. +Gaussian process covariance with exponentiated quadratic kernel in multiple dimensions. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x, real sigma, real[] length_scale)`
\newline -Gaussian process covariance with squared exponential kernel in multiple +Gaussian process covariance with exponentiated quadratic kernel in multiple dimensions with a length scale for each dimension. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x1, vector[] x2, real sigma, real length_scale)`
\newline -Gaussian process cross-covariance of `x1` and `x2` with squared exponential +Gaussian process cross-covariance of `x1` and `x2` with exponentiated quadratic kernel in multiple dimensions. \index{{\tt \bfseries gp\_exp\_quad\_cov }!{\tt (vector[] x1, vector[] x2, real sigma, real[] length_scale): matrix}|hyperpage} `matrix` **`gp_exp_quad_cov`**`(vector[] x1, vector[] x2, real sigma, real[] length_scale)`
\newline -Gaussian process cross-covariance of `x1` and `x2` with squared exponential +Gaussian process cross-covariance of `x1` and `x2` with exponentiated quadratic kernel in multiple dimensions with a length scale for each dimension. ### Dot product kernel