Estimation: chapter 2+3
Minimum variance unbiased estimation + the CRLB
Natasha Devroye
[email protected]http://www.ece.uic.edu/~devroye
Spring 2011
Estimation: a first example
Estimate the DC level, A, of a signal given noisy measurements x[0], x[1], ... x
[N-1] where
x[n] are samples of this!
Find a few estimators
Compare their performance
mean?
variance?
pdf?
Estimation: a first example
Estimators of the DC level, A
Estimation: definitions
Estimation: definitions
How would you pick a ``good estimator?
Vector versions....
Minimum variance unbiased estimation
Why?
So?
Minimum variance unbiased estimation
Give a counter-example! (b1pg.20)
The Cramer-Rao Lower Bound
the CRLB give a lower bound on the variance of ANY UNBIASED estimator
does NOT guarantee bound can be obtained
IF find an estimator whose variance = CRLB then its MVUE
otherwise can use Ch.5 tools (Rao-Blackwell-Lehmann-Scheffe Theorem and
Neyman-Fisher Factorization Theorem) to construct a better estimator from
any unbiased one - possibly the MVUE if conditions are met
The Cramer-Rao Lower Bound (CRLB)
Use?
Intuition?
The Cramer-Rao Lower Bound (CRLB)
CRLB examples
CRLB proof
CRLB T or F
What is I()?
Why information?
non-negative
additive for independent observations
Vector form of the CRLB
Vector form of the CRLB
Vector form of the CRLB examples
What can we conclude?
CRLB for transformations
Vector CRLB for transformations
Example of vector CRLB with transformation
CRLB for General Gaussian Case
When observations are Gaussian and one knows the dependence of the
mean and covariance matrix on the unknown parameters, we know the
closed form of the CRLB (or Fisher information matrix):
CRLB for Gaussians examples
Example: range estimation
678$9$:)('.$6;<&=)<&,($/>,?*.=
678$9$:)('.$6;<&=)<&,($/>,?*.=
-.&/0%12345607833
"+#)333333/9/:7.939;7.3#&<,($"=
-.&/0%12345607833
"+#)333333/9/:7.939;7.3#&<,($"=
>7?71;73>7@67?219/8333 "+#'( !!)
>7?71;73>7@67?219/8333 "+#'( !!)
A7&05.73-1%73B76&C8 !!
A7&05.73-1%73B76&C8 !!
!"#$%&'()*$+,-.*
Continuous time signal model!"#$%&'()*$+,-.*
& + # ) # " + # % ! ) " %+ # )
, $ # $ $ # $ "!
!
$
!#!
"
& +# ) #" +"#+*!# %) ! ! ) " %+# )
$
!!#!
"
"
" + # *! ! )
"+#)
"+#)
D&/E61%127E
FG1273H&5001&/
D&/E61%127E
FG1273H&5001&/
/%0$,1$!2"3
D4I &+#)
J3K%L
D4I &+#)
J3K%L
$"
#$45
/%0$,1$!2"3
#$45
(*
(*
"+#''( $! " )
"+#''( !!)
)'
!(%&'
, $ # $ $ # $" " ! !(%&'
)'
!"#$
!"#$
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm
Example: range estimation
/0123&45678067'1&-9:&,7210;&<'=3;
Discrete time
signal model
/0123&45678067'1&-9:&,7210;&<'=3;
#$%&'(&$)%*
#$%&'(&$)%*
"! " !"#
"! " !"#
+,-&'(&$)%*
+,-&'(&$)%*
"#.!
"#.!
*!
*!
%&'()*+,-*./+! 0+#"1!"2*3
%&'()*+,-*./+!
0+#"1!"2*3
#4$5+0+#6$
!7
#4$5+0+#6$!7
&
)"
&
#"1! #"! !"1!
#"1! #"! !"1!
)"
'4$ 5 $ & 6 $! # & ( 7 % #4$ 5
'4$ 5 $ & 6 $! # & ( 7 % #4$ 5
89+:;<=*+
89+:;<=*+
>&?22<&@+AB<2*
>&?22<&@+AB<2*
C&.+"! " !"#
C&.+"! " !"#
$ $ ED#D ! D % # #
$ $ ED#D ! D % # #
&4$F&(5G ;&2++ @B@HI*.B+2&'()*2+2=&.=<@J+&=+$(
&4$F&(5G ;&2++ @B@HI*.B+2&'()*2+2=&.=<@J+&=+$(
E + $ + $( # #
E + $ + $( # #
$( + $ + $ ( % + # #
$( + $ + $ ( % + # #
$( % + + $ + % # #
$( % + + $ + % # #
* #4$ 5
*'#4$ 5
'
'4$ 5 $'') & 6 $! # & ( 7 % #4$ 5
'4$ 5 $ )'& 6 $! # & ( 7 % #4$ 5
'(' #4$ 5
'( #4$ 5
!"#$
!"#$
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm
Example: range estimation
!"#$%&'()*+")*,#&-!./
%&'()**+,(-.)/0)10(2345(16-7+.(8&1(-9:/)+(;(<=%>
H+7:(9/I()/0(J66*(
/&/KL61&(.61M-
G)1B/F$ A -
.@
% "#
+ ,&E"D/ C (
! )) ,/ $ $ &&
'
" # ?*
.@
"$ % ! "#
" # "$
.@
"$ % ! "# +
" # "$
) ,& B# A
) ,#
*
NO*+&9.(2)+P7+7-QQQ
(
&
&
# # "$ "/ $ '
+ ,& B "$ " / $ A (
))
&&
,/ $
*
'
.@
! "# +
,& B# A
! )) ,#
" #? *
(
&
&
# # "$ '
R-6()**1&O9M).9&/>(/$'('$ "$
ST6/(0&(PT)/:6(&8(G)19)U+6-QQ
!"#$
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm
Example: range estimation
/'%0"&1+2$*'2$3%&4/5(&673%289
-../012.+03412.3+567826.2.0+449 +33,:;<2./02=>267?18,+49
*+,(-)# ' +
,&
&
# "! ' (! (& ' $
%
" %&
* !% & (& #
'# " &
"! ' (! ( & ' $
!%
&
%
" %&
& (& #
"! ' (! ( & ' $
$!
'# " &
!%
&
%
" %&
& (& #
$!
"!
*+,(-)# ' +
$!
'# " &
"
&
&
!% .&2( / ) ( ( ' %(
!
$!
0
$!
'# " &
$ ! ) ! ! & (& '%&
%
!"#$%"&'&()&*"'+,-".
0
@A2AB1:,10
C2D+,.1*+4
!10 .&2( / ) ( ( ' %(
0
&
!10 ) ( ( ' %&
&
&
D+,.1*+4
*+,! )
!10 .&2( / ) ( ( ' %(
0
&
!10 ) ( ( ' %&
&
&
*+,! 6.2EFGH2IJK22(LM'
!"#$
-2?>312:N2EHOFK
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm
Example: range estimation
&"#$%'()*+,"*+-#'.&/0'12-#*34
&"#$%'()*+,"*+-#'.&/0'12-#*34
%&'()*+,-&-*'.-/&*0-*/11'2-*/+*+,-*3456*7(*+,-*.-8/9:
%&'()*+,-&-*'.-/&*0-*/11'2-*/+*+,-*3456*7(*+,-*.-8/9:
#
2/1<#>$ ; "
<&-= ? ;
#?
&'(
!
%
!"#
2/1<#>$ ; "
<&-= ? ;
?
&'( ! %!"#
@7*)-+*+,-*3456*7(*+,-*!"#$%A*B&-*C+1/(&DE*7D*F/1G&H*1-&B8+:*
@7*)-+*+,-*3456*7(*+,-*!"#$%A*B&-*C+1/(&DE*7D*F/1G&H*1-&B8+:*
?
) *( &
$ *(+%
*(+%(> + ''
0'+,* (
?
#>$
*#) $*$( &
*(+%(> (+ '' % $$ *(+%#>$ 0'+,*
( *# $ %
J*)#$,"*?
( J*)#$,"*?
3456*'&*'(2-1&-89*F17F71+'7(/8*+7:
3456*'&*'(2-1&-89*F17F71+'7(/8*+7:
3456*'&*'(2-1&-89*F17F71+'7(/8*+7:
KK LM4*N-/&B13456*'&*'(2-1&-89*F17F71+'7(/8*+7:
LM4*N-/&B1LM4*N-/&B1KK KK4NL*6O*N-/&B1LM4*N-/&B14NL*6O*N-/&B1KK 4NL*6O*N-/&B14NL*6O*N-/&B1-
)? " I
2/1< (> ; "
<G ? ;
) ?%"?I
&'(
!
>
!"#
2/1< ( ; "
<G ? ;
?
&'( ! %!"#
L7*+,-*3456*+-88&*B&A
K 3,77&-*&')(/8*0'+,*8/1)-*%
L7*+,-*3456*+-88&*B&A !"#
K P(&B1-*+,/+*LM4*'&*8/1)K 3,77&-*&')(/8*0'+,*8/1)-*%!"#
K 6-++-1*7(*M-/1Q9"8/1)-*+/1)-+&
K P(&B1-*+,/+*LM4*'&*8/1)K O,'=,*'&*Q-++-1R**
K 6-++-1*7(*M-/1Q9"8/1)-*+/1)-+&
S T7BQ8-*+1/(&G'++-.*-(-1)9R
K O,'=,*'&*Q-++-1R**
S T7BQ8-*4NL*Q/(.0'.+,R
S T7BQ8-*+1/(&G'++-.*-(-1)9R
S T7BQ8-*4NL*Q/(.0'.+,R
!"#$
!"#$
Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm
