Entropy & – Information
Entropy & – Information
0 0
Entropy-And-Information-Part-I
Introduction to the nature of Information, and its relationship with uncertainty, probabilities and entropyOn Github SUNY-Albany-CCI-LearningEncounters / Entropy-And-Information-Part-I
Entropy &
Information
Created by Luis Ibanez
Information
A Question...
Something that you
know
Something that you
don't know
Something that you
can guess
How
Likely ?
Why ask?
if you can guess...
The Cost
of asking...
The Reward
of getting
an answer
The Value
of an answer
Let's Try...
Is it Cold
in Albany
on January ?
Is it Cold
on January ?
Is it Cold ?
Would you
Eat a Monkey ?
Probabilities
If you pick
at random
a day in the calendar...
What are the chances
that it will be a
Tuesday ?
1
----
7
What are the chances
that it will be
the first day
of the month?
12
----
365
If you pick
at random
a card from a deck...
What are the chances
that it will be
an Ace ?
4
----
52
What are the chances
that it will be
a Red card ?
26
----
52
Amount
of
Information
Of an event
with probability
p
-log(p)
-----------
log(2)
Tossing
a coin...
probability
of head ?
1
--
2
amount of
information?
-log(1/2)
-----------
log(2)
= 1
Units ?
= Bits
The answer to a
YES / NO
question
Where
YES and NO
are equally likely
Carries
an amount of
Information
= 1 bit
STOP !
Reflect
consider the
question...
Is the card
an Ace ?
The YES answer
has probability
p = 4 / 52
Its amount of
information is...
-log(4/52)
-----------
log(2)
= 3.7 bits
The NO answer
has probability
p = 48 / 52
Its amount of
information is...
-log(48/52)
-----------
log(2)
= 0.115 bits
Why is it that
the most likely
answer...
Carries
less
Information?
Is the sky blue ?
How good
is the question ?
Back
to the cards
Is the card
an Ace ?
p(Y) = 4/52
p(N) = 48/52
I(Y) = 3.700 bits
I(N) = 0.115 bits
If we ask
many times...
In average...
We will get a YES
4 out of 52 times
We will get a NO
48 out of 52 times
So...
4 out of 52 times
We get 3.7 bits
48 out of 52 times
We get 0.115 bits
In average
We get
( 4 x 3.7 bits + 48 x 0.115 bits)
------------------------------------------
52
= 0.39 bits
The Moment
of Zen
The best
YES / NO
question
that we can
possibly ask
Is the one
where
YES and NO
are equally likely
That question...
Will carry...
an expected
amount of
information...
equal to...
( 1 x 1 bits + 1 x 1 bits)
-------------------------------
2