LeastSquares{T,S} <: AbstractNLSModel{T,S}

Implements the data structure for defining a least squares problem.

Objective Function

\[\min_{x} 0.5\Vert F(x) \Vert_2^2,\]

where

\[F(x) = A * x - b.\]

A is the coefficient matrix. b is the constant vector.

Fields

• meta::NLPModelMeta{T, S}, data structure for nonlinear programming models
• nls_meta::NLSMeta{T, S}, data structure for nonlinear least squares models
• counters::NLSCounters, counters for nonlinear least squares models
• coef::Matrix{T}, coefficient matrix, A, for least squares problem
• cons::Vector{T}, constant vector, b, for least squares problem

Constructors

LeastSquares(::Type{T}; nequ=1000, nvar=50) where {T}

Constructs a least squares problems with 1000 equations and 50 unknowns, where the entries of the matrix and constant vector are independent standard Gaussian variables.

Arguments

• T::DataType, specific data type of the optimization parameter

Optional Keyword Arguments

• nequ::Int64=1000, the number of equations in the system
• nvar::Int64=50, the number of parameters in the system

LeastSquares(design::Matrix{T}, response::Vector{T}; 
    x0 = ones(T, size(design, 2))) where {T}

Constructs a least squares problem using the design as the coef matrix and the response as the cons vector.

Arguments

• design::Matrix{T}, coefficient matrix for least squares.
• response::Vector{T}, constant vector for least squares.

Optional Keyword Arguments

• x0::Vector{T}=ones(T, size(design, 2)), default starting point for optimization algorithms.

LogisticRegression{T,S} <: AbstractNLPModel{T,S}

Implements logistic regression problem with canonical link function. If the covariate (i.e., design) matrix and response vector are not supplied, then these are simulated.

Objective Function

Let $A$ be the covariate matrix and $b$ denote the response vector. The rows of $A$ and corresponding entry of $b$ correspond to the predictors and response for the same experimental unit. Let $A_i$ be the vector that is row i of $A$, and let $b_i$ be the i entry of $b$. Note, $b_i$ is either 0 or 1.

Let $\mu(x)$ denote a vector-valued function whose ith entry is

\[\mu_i(x) = \frac{1}{1 + \exp(-A_i^\intercal x)}.\]

If $A$ has $n$ rows (i.e., $n$ is the number of observations), then the objective function is negative log-likelihood function given by

\[F(x) = -\sum_{i=1}^n b_i \log( \mu_i(x) ) + (1 - b_i) \log(1 - \mu_i(x)).\]

Fields

• meta::NLPModelMeta{T,S}, data structure for nonlinear programming models
• counters::Counters, counters for a nonlinear programming model
• design::Matrix{T}, the design matrix, $A$, of the logistic regression problem
• response::Vector{Bool}, the response vector, $b$, of the logistic regression problem

Constructors

LogisticRegression(::Type{T}; nobs::Int64 = 1000, nvar::Int64 = 50) where T

Constructs a simulated problem where the number of observations is nobs and the dimension of the parameter is nvar. The generated design matrix's first column is all 1s. The remaining columns are independent random normal entries such that each row (excluding the first entry) has unit variance. The design matrix is stored as type Matrix{T}.

LogisticRegression(design::Matrix{T}, response::Vector{Bool};
    x0::Vector{T} = ones(T, size(design, 2)) ./ sqrt(size(design, 2))
    ) where T

Constructs a LogisticRegression problem with design matrix design and response vector response. The default initial iterate, x0 is a scaling of the vector of ones. x0 is optional.

PoissonRegression{T, S} <: AbstractNLPModel{T, S}

Implements Poisson regression with the canonical link function. If the design matrix (i.e., the covariates) and responses are not supplied, they are randomly generated.

Objective Function

Let $A$ be the design matrix, and $b$ be the responses. Each row of $A$ and corresponding entry in $b$ are the predictor and observation from one unit. The entries in $b$ must be integer valued and non-negative.

Let $A_i$ be row $i$ of $A$ and $b_i$ entry $i$ of $b$. Let

\[\mu_i(x) = \exp(A_i^\intercal x).\]

Let $n$ be the number of rows of $A$ (i.e., number of observations), then the negative log-likelihood of the model is

\[\sum_{i=1}^n \mu_i(x) - b_i (A_i^\intercal x) + C(b),\]

where $C(b)$ is a constant depending on the data. We implement the objective function to be the negative log-likelihood up to the constant term $C(b)$. That is,

\[F(x) = \sum_{i=1}^n \mu_i(x) - b_i (A_i^\intercal x).\]

Fields

• meta::NLPModelMeta{T, S}, NLPModel struct for storing meta information for the problem
• counters::Counters, NLPModel Counter struct that provides evaluations tracking.
• design::Matrix{T}, covariate matrix for the problem/experiment ($A$).
• response::Vector{T}, observations for the problem/experiment ($b$).

Constructors

PoissonRegression(::Type{T}; nobs::Int64 = 1000, nvar::Int64 = 50) where {T}

Construct the struct for Poisson Regression when simulated data is needed. The design matrix ($A$) and response vector $b$ are randomly generated as follows. For the design matrix, the first column is all ones, and the rest are generated according to a normal distribution where each row has been scaled to have unit variance (excluding the first column). For the response vector, let $\beta$ be the "true" relationship between the covariates and response vector for the poisson regression model, then the $i$th entry of the response vector is generated from a Poisson Distribution with rate parameter $\exp(A_i^\intercal \beta)$.

PoissonRegression(design::Matrix{T}, response::Vector{T}; 
    x0::Vector{T} = zeros(T, size(design)[2])) where {T}

Constructs the struct for Poisson Regression when the design matrix and response vector are known. The initial guess, x0 is a keyword argument that is set to all zeros by default.

!!! Remark When using this constructor, the number of rows of design must be equal to the size of response. When providing x0, the number of entries must be the same as the number of columns in design.

QLLogisticSin{T, S} <: AbstractDefaultQL{T, S}

Implements a Quasi-likelihood objective with a logistic link function and linear_plus_sin variance funtion. If the design matrix and the responses are not supplied, they are randomly generated.

Objective Function

Let $A$ be the design matrix, and $b$ be the responses. Each row of $A$ and corresponding entry in $b$ are the predictor and observations from one unit. The statistical model for this objective function assums $b$ are between $0$ and $1$.

Let $A_i$ be row $i$ of $A$ and $b_i$ entry $i$ of $b$. Let

\[ \mu_i(x) = \mathrm{logistic}(A_i^\intercal x)\]

and

\[ v_i(\mu) = 1 + \mu + \sin(2 \pi \mu). \]

Let $n$ be the number of rows in $A$, then the quasi-likelihood objective is

\[ F(x) = -\sum_{i=1}^n \int_0^{\mu_i(x)} \frac{b_i - m}{v_i(m)} dm.\]

Fields

• meta::NLPModelMeta{T, S}, NLPModel struct for storing meta information for the problem
• counters::Counters, NLPModel Counter struct that provides evaluations tracking.
• design::Matrix{T}, covariate matrix for the problem/experiment ($A$).
• response::Vector{T}, observations for the problem/experiment ($b$).
• mean::Function, component-wise function of the linear predictor $Ax$ to predict expected value of the response.
• mean_first_derivative::Function, function for the first derivative of the mean function.
• mean_second_derivative::Function, function for the second derivative of the mean funciton.
• variance::Function, variance function that estimates the variance of each response.
• variance_first_derivative::Function, function that returns the first derivative of the variance function.
• weighted_residual::Function, computes the weighted residual of the model.

Constructors

Inner Constructors

QLLogisticSin{T, S}(meta::NLPModelMeta{T, S}, counters::Counters,
    design::Matrix{T}, response::Vector{T})

Initializes the data structure for a quasi-likelihood estimation problem with a logistic link function and a linear plus sine variance function.

Outer Constructors

QLLogisticSin(::Type{T}; nobs::Int64 = 1000,
    nvar::Int64 = 50) where {T}

Construct a quasi-likelihood estimation problem with a logistic link function; a linear plus sine variance function; and a randomly generated design matrix and response vector that is consistent with the quasi-likelihood model. The design matrix has a column of all ones and the rest generated from a normal distribution with its entries divided by sqrt(nvar - 1). The response vector has entries generated by mean.(design * x) + variance.(mean.(design * x)) .^ (.5) * ϵ, where ϵ is a noise vector generated from the Arcsine distribution with default parameters.

QLLogisticSin(design::Matrix{T}, response::Vector{T}; 
    x0::Vector{T} = zeros(T, size(design)[2])) where {T}

Constructs a quasi-likelihood estimation problem with a logistic link function; a linear plus sine variance function; and user-supplied design matrix and response vector.

QLLogisticCenteredExp{T, S} <: AbstractDefaultQL{T, S}

Implements a Quasi-likelihood objective with a logistic link function and centered_exp variance funtion. If the design matrix and the responses are not supplied, they are randomly generated.

Objective Function

Let $A$ be the design matrix, and $b$ be the responses. Each row of $A$ and corresponding entry in $b$ are the predictor and observations from one unit. The statistical model for this objective function assums $b$ are between $0$ and $1$.

Let $A_i$ be row $i$ of $A$ and $b_i$ entry $i$ of $b$. Let

\[ \mu_i(x) = \mathrm{logistic}(A_i^\intercal x)\]

and

\[ v_i(\mu) = \exp\left( -|\mu - c|^{2p} \right). \]

where $p \in \mathbb{R}$ and $c \in \mathbb{R}$.

Let $n$ be the number of rows in $A$, then the quasi-likelihood objective is

\[ F(x) = -\sum_{i=1}^n \int_0^{\mu_i(x)} \frac{b_i - m}{v_i(m)} dm.\]

Fields

• meta::NLPModelMeta{T, S}, NLPModel struct for storing meta information for the problem
• counters::Counters, NLPModel Counter struct that provides evaluations tracking.
• design::Matrix{T}, covariate matrix for the problem/experiment ($A$).
• response::Vector{T}, observations for the problem/experiment ($b$).
• mean::Function, component-wise function of the linear predictor $Ax$ to predict expected value of the response.
• mean_first_derivative::Function, function for the first derivative of the mean function.
• mean_second_derivative::Function, function for the second derivative of the mean funciton.
• variance::Function, variance function that estimates the variance of each response.
• variance_first_derivative::Function, function that returns the first derivative of the variance function.
• weighted_residual::Function, computes the weighted residual of the model.
• p::T, parameter of the variance function saved for testing
• c::T, parameter of the variance function saved for testing

Constructors

Inner Constructors

QLLogisticCenteredExp{T, S}(meta::NLPModelMeta{T, S}, counters::Counters,
    design::Matrix{T}, response::Vector{T}, p::T, c::T)

Initializes the data structure for a quasi-likelihood estimation problem with a logistic link function and a centered exponential variance function.

Outer Constructors

QLLogisticCenteredExp(::Type{T}; nobs::Int64 = 1000, nvar::Int64 = 50,
    p::T = T(1), c::T = T(1)) where {T}

Construct a quasi-likelihood estimation problem with a logistic link function; a centered exponential variance function; and a randomly generated design matrix and response vector that is consistent with the quasi-likelihood model. The design matrix has a column of all ones and the rest generated from a normal distribution with its entries divided by sqrt(nvar - 1). The response vector has entries generated by mean.(design * x) + variance.(mean.(design * x)) .^ (.5) * ϵ, where ϵ is a noise vector generated from the Arcsine distribution with default parameters.

QLLogisticCenteredExp(design::Matrix{T}, response::Vector{T};
    x0::Vector{T} = zeros(T, size(design)[2]), p::T = T(1), c::T = T(1)
    ) where {T}

Constructs a quasi-likelihood estimation problem with a logistic link function; a centered exponential variance function; and user-supplied design matrix and response vector.

QLLogisticCenteredLog{T, S} <: AbstractDefaultQL{T, S}

Implements a Quasi-likelihood objective with a logistic link function and centered_shifted_log variance funtion. If the design matrix and the responses are not supplied, they are randomly generated.

Objective Function

Let $A$ be the design matrix, and $b$ be the responses. Each row of $A$ and corresponding entry in $b$ are the predictor and observations from one unit. The statistical model for this objective function assums $b$ are between $0$ and $1$.

Let $A_i$ be row $i$ of $A$ and $b_i$ entry $i$ of $b$. Let

\[ \mu_i(x) = \mathrm{logistic}(A_i^\intercal x)\]

and

\[ v_i(\mu) = \log(|\mu-c|^{2p} + 1) + d,\]

where $p \in \mathbb{R}$, $c \in \mathbb{R}$, $d \in \mathbb{R}$.

Let $n$ be the number of rows in $A$, then the quasi-likelihood objective is

\[ F(x) = -\sum_{i=1}^n \int_0^{\mu_i(x)} \frac{b_i - m}{v_i(m)} dm.\]

Fields

• meta::NLPModelMeta{T, S}, NLPModel struct for storing meta information for the problem
• counters::Counters, NLPModel Counter struct that provides evaluations tracking.
• design::Matrix{T}, covariate matrix for the problem/experiment ($A$).
• response::Vector{T}, observations for the problem/experiment ($b$).
• mean::Function, component-wise function of the linear predictor $Ax$ to predict expected value of the response.
• mean_first_derivative::Function, function for the first derivative of the mean function.
• mean_second_derivative::Function, function for the second derivative of the mean funciton.
• variance::Function, variance function that estimates the variance of each response.
• variance_first_derivative::Function, function that returns the first derivative of the variance function.
• weighted_residual::Function, computes the weighted residual of the model.
• p::T, parameter for the variance function for testing purposes
• c::T, parameter for the variance function for testing purposes
• d::T, parameter for the variance function for testing purposes

Constructors

Inner Constructors

QLLogisticCenteredLog{T, S}(meta::NLPModelMeta{T, S}, counters::Counters,
    design::Matrix{T}, response::Vector{T}, p::T, c::T, d::T)

Initializes the data structure for a quasi-likelihood estimation problem with a logistic link function and a centered shifted log variance function.

Outer Constructors

QLLogisticCenteredLog(::Type{T}; nobs::Int64 = 1000,
    nvar::Int64 = 50, p::T = T(1), c::T = T(1), d::T = T(1)) where {T}

Construct a quasi-likelihood estimation problem with a logistic link function; a centered shifted log variance function; and a randomly generated design matrix and response vector that is consistent with the quasi-likelihood model. The design matrix has a column of all ones and the rest generated from a normal distribution with its entries divided by sqrt(nvar - 1). The response vector has entries generated by mean.(design * x) + variance.(mean.(design * x)).^(.5) * ϵ, where ϵ is a noise vector generated from the Arcsine distribution with default parameters.

QLLogisticCenteredLog(design::Matrix{T}, response::Vector{T}, 
    x0::Vector{T} = zeros(T, size(design)[2]), p::T = T(1),
    c::T = T(1), d::T = T(1)) where {T}

Constructs a quasi-likelihood estimation problem with a logistic link function; a centered shifted log variance function; and user-supplied design matrix and response vector.

QLLogisticMonomial{T, S} <: AbstractDefaultQL{T, S}

Implements a quasi-likelihood objective with a logistic link function and the variance function monomial_plus_constant. If the design matrix and the responses are not supplied, they are randomly generated.

Objective Function

Let $A$ be the design matrix, and $b$ be the responses. Each row of $A$ and corresponding entry in $b$ are the predictor and observations from one unit. The statistical model for this objective function assums $b$ are between $0$ and $1$.

Let $A_i$ be row $i$ of $A$ and $b_i$ entry $i$ of $b$. Let

\[ \mu_i(x) = \mathrm{logistic}(A_i^\intercal x)\]

and

\[ v_i(\mu) = \mu^{2p} + c, \]

for $p \in \mathbb{R}$ and $c \in \mathbb{R}$.

Let $n$ be the number of rows in $A$, then the quasi-likelihood objective is

\[ F(x) = -\sum_{i=1}^n \int_0^{\mu_i(x)} \frac{b_i - m}{v_i(m)} dm.\]

Fields

• meta::NLPModelMeta{T, S}, NLPModel struct for storing meta information for the problem
• counters::Counters, NLPModel Counter struct that provides evaluations tracking.
• design::Matrix{T}, covariate matrix for the problem/experiment ($A$).
• response::Vector{T}, observations for the problem/experiment ($b$).
• mean::Function, component-wise function of the linear predictor $Ax$ to predict expected value of the response.
• mean_first_derivative::Function, function for the first derivative of the mean function.
• mean_second_derivative::Function, function for the second derivative of the mean funciton.
• variance::Function, variance function that estimates the variance of each response.
• variance_first_derivative::Function, function that returns the first derivative of the variance function.
• weighted_residual::Function, computes the weighted residual of the model.
• p::T, parameter of the variance function for testing purposes.
• c::T, parameter of the variance function for testing purposes.

Constructors

Inner Constructors

QLLogisticMonomial{T, S}(meta::NLPModelMeta{T, S}, counters::Counters,
    design::Matrix{T}, response::Vector{T}, p::T, c::T) where {T, S}

Initialize the data structure for a quasi-likelihood estimation problem with a logistic link function and a monomial plus constant variance function.

Outer Constructors

QLLogisticMonomial(::Type{T}; nobs::Int64 = 1000, nvar::Int64 = 50,
    p::T = T(1), c::T = T(1)) where {T}

Construct a quasi-likelihood estimation problem with a logistic link function; a monomial plus constant variance function; and a randomly generated design matrix and response vector that is consistent with the quasi-likelihood model. The design matrix has a column of all ones and the rest generated from a normal distribution with its entries divided by sqrt(nvar - 1). The response vector has entries generated by mean.(design * x) + variance.(mean.(design * x))^(.5) * ϵ, where ϵ is a noise vector generated from the Arcsine distribution with default parameters.

QLLogisticMonomial(design::Matrix{T}, response::Vector{T};
    x0::Vector{T} = zeros(T, size(design)[2]), p::T = T(1),
    c::T = T(1)) where {T}

Constructs a quasi-likelihood estimation problem with a logistic link function; a monomial plus constant variance function; and user-supplied design matrix and response vector.

logistic(η::T} where {T}

Implements

\[ \mathrm{logistic}(\eta) = \frac{1}{1 + \exp(-\eta)},\]

where T is a scalar type.

Arguments

• η::T, scalar. In the regression context this is the linear effect.

inverse_complimentary_log_log(η::T) where {T}

Implements the link function

\[ \mathrm{CLogLog}^{-1}(\eta) = 1 - \exp(-\exp(\eta)).\]

Arguments

• η::T, scalar. In the regression context this is the linear effect.

inverse_probit(η::T) where {T}

Implements the link function

\[ \Phi(\eta) = \frac{1}{2\pi} \exp(-.5\eta^2).\]

Arguments

• η::T, scalar. In the regression context this is the linear effect.

dlogistic(η::T) where {T}

First derivative of the logistic function. Implements

\[ \frac{d}{d\eta} \mathrm{logistic}(\eta) = \frac{\exp(-\eta)}{(1+\exp(-\eta))^2}\]

where T is a scalar type.

Arguments

• η::T, scalar. In the regression context this is the linear effect.

ddlogistic(η::T) where {T}

Double derivative of the logistic function. Implements

\[ \frac{d}{d^2\eta} \mathrm{logistic}(\eta) = \frac{2\exp(-2\eta)}{(1+\exp(-\eta))^3} - \frac{\exp(-\eta)}{(1+\exp(-\eta))^2}\]

where T is a scalar type.

Arguments

• η::T, scalar. In the regression context this is the linear effect.

monomial_plus_constant(μ::T, p::T, c::T) where {T}

Implements the variance function

\[ V(\mu) = (\mu^{2})^p + c.\]

See Quasi-likelihood Estimation for details.

Arguments

• μ::T, scalar. In the regression context, this is the estimated mean of a datapoint.
• p::T, scalar. Power applied to μ^2.
• c::T, scalar. Translate the effect on variance by c.

linear_plus_sin(μ::T) where {T}

Implements the variance function

\[ V(\mu) = 1 + \mu + \mathrm{sin}(2\pi\mu).\]

See Quasi-likelihood Estimation for details.

Arguments

• μ::T, scalar. In the regression context, this is the estimated mean of a datapoint.

centered_exp(μ::T, p::T, c::T) where {T}

Implements the variance function

\[ V(\mu) = \exp\left( -|\mu - c|^{2p} \right).\]

See Quasi-likelihood Estimation for details.

Arguments

• μ::T, scalar. In the regression context, this is the estimated mean of a datapoint.
• p::T, scalar. Power applied to (μ-c)^2.
• c::T, scalar. Center where noise level is highest.

centered_shifted_log(μ::T, p::T, c::T, d::T) where {T}

Implements the variance function

\[ V(\mu) = \log(|\mu-c|^{2p} + 1) + d.\]

See Quasi-likelihood Estimation for details.

Arguments

• μ::T, scalar. In the regression context, this is the estimated mean of a datapoint.
• p::T, scalar. Power applied to |\mu-c|^2.
• c::T, scalar. Center where noise level is lowest.
• d::T, scalar. Irreducible variance. Corresponds to the minimum variance. Must be positive.

dlinear_plus_sin(μ::T) where {T}

Compute the following function

\[ \frac{d}{d\mu} (1 + \mu + \sin(2 \pi \mu)) = 1 + 2 \pi \cos(2\pi \mu).\]

Arguments

• μ::T, point at which to compute the derivative. In the context of regression, this is the mean.

dcentered_exp(μ::T, p::T, c::T) where {T}

Compute the following function

\[ \frac{d}{d\mu} \exp\left( -|\mu - c|^{2p} \right),\]

where $c \in \mathbb{R}$ and $p \in \mathbb{R}$. See Quasi-likelihood Estimation for details.

Arguments

• μ::T, scalar. In the regression context, this is the estimated mean of a datapoint.
• p::T, scalar. Power applied to (μ-c)^2.
• c::T, scalar. Center where noise level is highest.

dcentered_shifted_log(μ::T, p::T, c::T) where {T}

Compute and returns the following function

\[ \frac{d}{d\mu} \log(|\mu-c|^{2p} + 1) = sign(\mu - c) \frac{2p|\mu - c|^{2p-1}}{|\mu-c|^{2p} + 1},\]

where $sign(x)$ returns the sign of $x$. This equality is only correct everywhere when $p > .5$.

Arguments

• μ::T, point at which to compute the derivative. In the context of regression, this is the mean.
• p::T, scalar. Power applied to $|\mu-c|^2$.
• c::T, scalar. Center where noise level is lowest.

dmonomial_plus_constant(μ::T, p::T) where {T}

Compute the following function

\[ \frac{d}{d\mu} (\mu^{2p} + c) = 2p \mu^{2p-1}\]