Explaining Information Entropy to a 5 year old

Suppose there are 8 numbers. I choose one, but you don’t know which number it is. Your task is to guess which number. You can ask yes or no questions. What is the minimum number of questions you would need to ask to guess the number?


All have an equal chance of occurring (P(any number) = 1/8).

What would be your approach for asking questions?

Start at the mid-way point and asking if the number is less than 5.  Suppose the answer is yes. Now you know the number must be


Do the same thing and ask if the number is less than 3. Suppose the answer is yes again. Now you know the number must be


Now you ask whether the number is 1, and if the answer is no, then you know the number is 2. You have guessed the number in 3 questions.

3 is the minimum number of bits you would need to use to encode the probability distribution of these 8 numbers. This algorithm is at the core of how we are all able to send emails, pictures, and videos so quickly now. Before the father of information theory Claude Shannon had invented this algorithm, it was unthinkable to send a picture — even a voice message, across the globe. There was simply no efficient way to send it. Nobody knew how many bytes it would take to encode the message, or even how to encode the message.

Next question: What if the numbers did not have an equal chance of occurring?

The Basics of Fourier Transforms: a Conversation with a Random Dude

I’ve been doing a lot of signal processing lately (Python tutorial on analyzing .wav files to come) and came upon this nice metaphor for Fourier Transforms, which can help extract information from sound.

Random dude: I need to know how many 7’s there are in 49.

High school algebra student: Duh, just divide 49 by 7. There are seven 7’s in 49.

Random dude: I need to know how many x’s there are in y.

High school algebra student: Duh, just divide y by x. There are y/x x’s in y.

Random dude: I need to know how much of the frequency 23Hz there is in my signal x(t).

High school algebra student: Duh, just divide x(t) by 23Hz. There is x(t)/23Hz amount of 23Hz in your signal.

via dmorris at TechHouse

Relationship Questions

For troubleshooting.

  1. What would you tell your younger self about love?
  2. Which of your flaws do you want to be treated with more generously?
  3. Is there anything I have done in the past that may have unknowingly hurt you?
  4. What are the main stressors in your life and how can I help?
  5. When I am anxious in our relationship, I tend to …
  6. What scares you about our relationship?

Headache-Free Installation: How to set up a virtual environment

One of the most irritating things about learning a new framework or switching to a new computer is often not learning the different nuances in syntax or even debugging — is installing the darn thing. I can recount the numerous dependency and version management errors I was met with upon when forking my friend’s Django project or deciding to pick up Flask, both Python web frameworks.

When I discovered virtual environments, a lot of these headaches disappeared. Virtual environments allow you to compartmentalize your framework and package installations into the clean slate, separate from the rest of your computer.

Why a virtual environment?

When people type ‘sudo’ before attempting to install something, I fret at the amount of trouble they may be causing themselves later down the line. Installing something globally will inevitably lead to version management problems and other clashes down the line.

What to install globally

  • Python (usually comes installed)
    • Easy_install (comes with Python)
  • Pip (Python package manager)

What to NOT install globally

  • Packages
    • OK: sudo easy_install pip
    • May lead to problems: sudo pip install django

How to install a virtualenv.

Assuming you already have Python installed:

  1. Install Pip.
    sudo easy_install pip
  2. Use Pip to install virtualenv
    sudo pip install virtualenv
  3. Change directories into the folder where you want the virtual environment installed.
    cd ~/Desktop/projects/my_app
  4. Create your virtual env
    virtualenv env
  5. Activate your virtual environment
    source env/bin/activate

Now your command line will include the virtual environment:

And you can install frameworks like django and packages in this new environment as necessary.


  • Your environment will only exist when you activate it. Notice that if you open a new terminal window, it disappears.
  • You can deactivate the environment by typing
    Or closing the terminal window.

Two Inflection Points on Love


I’ve decided that if I’m going to love, I’m going to love. hard.

I’ve noticed a phenomenon where someone will have a bad break up, and then will try to prevent the pain in the future by being more guarded about who they let into their heart.

But there is a flaw in this way of being — the person is conflating emotional investment with open communication. “If I had just been less invested in the person, I could have avoided all this pain.” I think people are deluding themselves. Two reasons:

(1) It wasn’t emotional investment that lead to them being hurt — it was a lack of open communication — consistent transparency about each other’s pulse on the relationship and trusting each other enough to be vulnerable about thoughts and emotions.

If you had just openly communicated in the first place, then you would’ve been able to exchange quick feedback loops and adjust your speedometer accordingly.

It’s when these pulses are not in sync and people begin to leave things (addictingly!) ambiguous, that signals get crossed and people end up being hurt.

(2) There’s nothing wrong with being hurt. I can be less hurt by being more guarded about who I give my heart to, but what is the fun in that.

To feel emotion is to be human. And you can never fully live out what it means to be human without living intensely.


I’m going to prioritize the person over the relationship.

A lot of times, when just entering the rush of a new intimate relationship, we get worried about defining the relationship – what it should be, whether or not we’re boyfriend/girlfriend, whether or not he’s seeing someone else. These thought patterns make it difficult to live in the moment, to appreciate the person for who they are, and savor the time when you’re with them.

It’s the prioritization of the relationship over the person that creates bad outcomes. Things start to become binary — we’re either exclusive or we’re not friends at all — and power dynamics come into play — he took 5 hours to respond to my text, is the ball in his court or mine? We’re no longer happy with the time we spend with them, rather, we suddenly build up expectations and want more.

Few people are worth your EQ. When you meet someone you believe is worth emotionally investing in, instead of trying to put them into a hole and asking, “Is he the one?” why not ask, “Am I building something amazing with this person? What else do I want to build with them?”

How to Model a Neuron in Our Visual Cortex, A Layman’s Understanding (Part I)

Close your eyes and listen to the brain.

What does it sound like?

Scientists did just that — they used amplifiers to listen to how neurons in our brain sounded, so they could discover how our eyes form an image of the outside world.

Where would you start if you wanted to model this process?

One thing that is generally helpful is to first identify the different components of a system. We have:
(1) the image in front of us
(2) some representation of the neuron
(3) an output of that image we see

How should we put those pieces together?

But what is wrong with this? That it’s probably more like:

But the brain is made up of many neurons, how would we model just one?

Now what if we do this for every neuron? Then we would have a whole picture.

The next question that arises is: How does each neuron know which slice of the picture to look at?

Well, what are the different components of a picture? A picture is fundamentally made up of:
(1) colors
(2) edges

Edges seem like they would give us more information about a picture, so let’s go down that route. What is an edge and how can we break that concept down? A major component of edges is the orientation of an edge.

So maybe each neuron has a preferred orientation it favors. And all of these neurons work together to create one picture.

So we can take lots of bars with different orientations, run them through a gabor filter (a fancy term that refers to a function that is similar to those in our visual system), and see when the gabor filter produces the most excited response.

If we run these through a gabor filter, we end up with a curve looking like this.

Thus, this neuron’s preferred orientation is ~40 degrees. If we do this for every neuron, we’ll have a starting model of how our brain sees edges.

You just learned a few fundamentals about how your retina sends signals to your brain, creating a picture for the world in front of you.

To come: Code as to how you would model this.

The Cost of Making Your Work Public

The other day my co-worker Bret Victor was speaking of the requests he gets to release the source code for his prototypes on the future of creation tools.

While he’s been getting requests for years, Bret said that releasing his code to the public could potentially stifle innovation.


The release creates a standard for what these new products should look like.

When an inventor creates a tool and makes it available to the world, the public has a tendency to simply accept the tool as is, use it, and stop thinking of new ways to do things, thus halting development of the product).

Example #1: the Pie Chart

Pie charts are pervasive…



… And a poor way to represent information. (See Edward Tufte, The Worst Chart in the World, and Oracle’s Reasons Not to Use a Pie Chart)

A simple example illustrates that pie charts make it difficult to make comparisons between two quantities. See:


What if we represented the same information like this, instead? This illustration enables us to make direct comparisons between quantities.


Because pie charts cannot spatially fit all information, they also are pleasantly accompanied by a key (right-hand side), which attempts to illustrate what all the components of the pie chart designate.


Could we design another way to represent this information — one that doesn’t require you darting your eyes back and forth to understand the information?

What about this?


Okay. Maybe not as aesthetically pleasing. Perhaps a reason why people use pie charts is because all the circles look pretty. But I would argue that we should not sell ourselves short — can we not have a representation that is both intuitively functional, and aesthetically pleasing?

I re-designed a representation of the same information; illustrated below.



So what?

These are just pie charts. Okay, it’ll take a millisecond longer to process the percentages. Why does it matter?

I think we vastly underestimate how good the quality of an experience (in this case, understanding and exploring the world) could be, because we already have models in our heads of what the medium is currently like.

One step further: Our tools for everyday statistics

Does the same tragedy that happened with pie charts also extend to other branches?

Let us look at the field of statistics.

The traditional probability problem presents you with several numbers on the percentage of women who get a positive or negative result when they get a mammogram.


You are typically asked:

Assuming you have a positive test, what is the chance you have cancer?

When calculated out, the number seems much lower than expected.

Some of us simply accept that our brains do not intuitively grasp probabilities, but I’d argue that statistics is only unintuitive because we don’t have proper representations.

How would you design another representation of Bayes’ Theorem?

More to come.

Analogizing Information to Nutrition


Few would disagree that the foods they eat affect how they feel. I feel much better eating vegetables and rice for dinner than I do a cheese pizza.

I take this analogy one step further by being selective about the information I expose myself to.

Media geared for average consumption : food geared for average assumption.

Consuming popular television, movies, magazines is akin to consuming McDonalds, potato chips, and deep-fried Twinkies.

Delicious in the short run (reading about gossip and exploits, even love songs on the radio are scary) — deadly in the long run (how do they impact your values and what you think about?).

Think of it as a mental paleo diet. Tasty.