Notes forSTA437/1005 —Methods for MultivariateData
RadfordM. Neal, 26 November 2010
Let Xbearandom vectorwith pelements, so thatX=[X1,...,Xp]′,where ′denotes
transpose.(By convention, our vectors are column vectors unlessotherwise indicated.)
Wedenote aparticular realized value of Xbyx.
The expectation (expected value, mean) ofarandomvector Xis E(X)=Rxf(x)dx,
where f(x)is the jointprobabilitydensityfunctionfor the distributionofX.
Weoften denote E(X)byµ,with µj=E(Xj)being the expectationof the j’th element
The variance ofthe randomvariable Xjis Var(Xj)=E[(Xj−E(Xj))2], whichwesome-
The standard deviation of Xjis pVar(Xj)=σj.
Covariance and correlation:
The covariance ofXjand Xkis Cov(Xj,Xk)=E[(Xj−E(Xj))(Xk−E(Xk))], whichwe
sometimes writeasσjk.Note that Cov(Xj,Xj)is the variance of Xj,soσjj =σ2
The correlationof Xjand Xkis Cov(Xj,Xk)/(σjσk), whichwesometimes write as ρjk.
Notethatcorrelations are alwaysbetween −1and +1, and ρjj is alwaysone.
Covariance and correlation matrices:
The covariances for all pairs ofelements of X=[X1,...,Xp]′can beput in amatrix called
the covariance matrix:
σ11 σ12 · · · σ1p
σ21 σ22 · · · σ2p
σp1σp2· · · σpp
Notethatthe covariancematrix is symmetrical, with the variances of the elementson the
The covariance matrix can alsobewritten asΣ=E[(X−E(X)) (X−E(X))′].
Similarly,the correlations can beput into a a symmetrical correlation matrix, whichwill
haveones on the diagonal.
Suppose wehavenobservations, eachwith values for pvariables. Wedenote the value of
variable jin observation ibyxij,and the vector of all values for observationibyxi.
Weoften view the observed xias arandomsample ofrealizations of arandomvector X
with some (unknown) distribution.
The is potential ambiguitybetween the notation xifor observation i,and the notation xj
for arealization of the randomvariable Xj.(The textbook uses bold face forxi.)
Iwill (try to) reserveifor indexing observations, and usejand kfor indexing variables,
but the textbook somtimes uses ito index avariable.
The sample mean of variable jis ¯xj=1
The sample mean vector is ¯x=[¯x1,...,¯xp]′.
If the observations all havethe same distribution, the sample meanvector, ¯x,is an unbiased
estimate of the meanvector, µ,of the distribution from whichthese observations came.
The sample varianceof variable jis s2
If the observations all havethe same distribution, the sample variance, s2
of the variance, σ2
j,of the distribution for Xj,and will bean unbiasedestimate if the
observations are independent.
Sample covariance and correlation:
The sample covariance ofvariable jwith variable kis 1
(xij −¯xj)(xik −¯xk).
The sample covariance is denoted bysjk.Note that sjj equals s2
j,the sample variance of
The sample correlation of variable jwith variable kis sjk/(sjsk), often denoted byrjk.
Sample covariance and correlation matrices:
The sample covariances maybearranged as the sample covariance matrix:
s11 s12 · · · s1p
s21 s22 · · · s2p
sp1sp2· · · spp
The sample covariance matrix can also becomputed asS=1
Similarly,the sample correlations maybearranged as the samplecorrelation matrix, some-
times denoted R(though the textbookalso uses Rfor the population correlationmatrix).
Notes for sta 437/1005 methods for multivariate data. Let x be a random vector with p elements, so that x = [x1, . , xp] , where denotes transpose. (by convention, our vectors are column vectors unless otherwise indicated. ) We denote a particular realized value of x by x. The expectation (expected value, mean) of a random vector x is e(x) = r xf (x)dx, where f (x) is the joint probability density function for the distribution of x. We often denote e(x) by , with j = e(xj) being the expectation of the j"th element of x. The variance of the random variable xj is var(xj) = e[(xj e(xj))2], which we some- times write as 2 j . The standard deviation of xj is pvar(xj) = j. The covariance of xj and xk is cov(xj, xk) = e[(xj e(xj))(xk e(xk))], which we sometimes write as jk.