This is Part 1 of a 2 Part Series on the good side of big data. Following will be another series on the bad side of Big Data.

Part 1: You are a fountain of data…

We have all heard about Big Pharma, Big Oil, Big Sugar…Big John. However, the new kid on the big block, Big Data, is coming of age.

So what is data? Simply put, it is a fact or figure that has been recorded. Be it your age on an application form, or the type of browser you are using stored in an electronic cookie.

In the modern age, not only are we creating more data…partly because there a lot more of us, there is also computer technology which has allowed us to not only measure more of the world in more ways but as well store data longer, access it easier and ‘interpret’ it faster.

In the past, the only way to record data was to write it down. That required a person, pencil, paper and time.

Computers can do this alone and in milliseconds…and our storage capacity moves from a dozen data points on a piece a paper to billions on a memory stick.

How big is Big Data?

All the data created by humanity from the dawn of awareness to 2003…equals about 5 Exabytes, we now produce that every 2 days.

The amount of information generated in a day is seventy times more than found at the library of Congress.

The amount of information each one of us faces in a single day, is more than someone in the 15th century would face in a lifetime.

So, how is Big Data better than old data?

Let me give you a practical example.

In the old movies where you had a cop ‘tail’ a suspect. The cop would make notes every hour or so, about were the suspect was, what they were doing…This will give you an idea of what the suspect is doing, but minor details may be missed.

Now, you put a GPS tracker, wireless camera and record his electronic shadow…and you have a complete record of what the suspect is doing.

Not only that but this data is easy to access. So, in the past, if you had a hunch, you could go back “into the files” which may necessitated a trip to a warehouse, search a card catalogue, then finding the box where the report was, then reading all the reports with the hope of solving a cold case.

Big Data, not only makes it simple to search old records as in a Google search, but you can set up flags…that is if a crime had three elements in common with a cold case, you would be notified…automatically.

In another more uplifting angle, a nurse will take a patients vitals 3 or 4 times a day. It’s not practical to have a nurse constantly monitoring a pulse…but a function digital recorders can accomplish easily.

It is able to record your vitals every second and store it indefinitely.

In a recent discovery in Toronto, this constant data awareness is now being used to help save the lives of premature babies.

Well, first let me give you a quick medical primer…every day we cycle through the fight-or-flight (FoF) response and the rest-and-digest (RaD) response.

This is normal, however, doctors noticed that in premature babies, this cycling leveled off two days prior to a detectable infection. That is, an infected baby will spend more time in each state (FoF and RaD) before ‘cycling’ to the next state…thus having fewer switches between F0F and RaD per hour.

So Big Data was able to see an infection two days before the doctor could. This will likely result in saving thousands of lives.

It is now possible for us to monitor everything every day and store it forever.

It will make it possible for us to keep track of our exercise, spending…every aspect of our lives. For instance spending too much time in front of the monitor…well there are apps that detect your inertia, correlate that with the days travel pattern and remind you to go out for a walk.

It is a growing part of how we manage our days. In fact we may look back and wonder how we lived with the ‘fog & grog’ of old data.

I heard an interesting analogy.

If we got our Visa bill and all it said was how much we owed them…if there was no itemization, no one would blindly pay it!

But we do that every month when we pay the hydro bill without knowing the ‘detailed’ particulars of what we are paying for…i.e. what is the ‘charge’ or cost of toasting my bread…the load of laundry I did…how can you be a ‘green’ if you don’t know this data!

Well, there is now available a monitor that plugs into an electrical system that can ‘read’ the digital signature of all your electrical devices and discern how much electricity is being used.

Keeping with hydro, another thing that new Wi-Fi meters might help us do is to determine when we use our electricity.  You see, the cost of electricity is based on the overall current usage, so during a hot afternoon…AC running….doing your laundry may cost twice as much as if you left it for the night. There are dryers that have this auto-timing ability built in.

Of course you would need to restructure more than the meters and your dryer.

Big Data allows for better greening of our world by allowing us to see in detail not only energy usage but in helping us adapt renewable energy sources to our grid. As mentioned, one of the obstacles to solar power, for example, is that it’s only available during the day…and more energy is produced in the afternoon than other times.

If we knew in detail what appliances were the biggest users of electricity, we could use them during peak solar output…and minimize our nighttime power. The opposite of my Wi-Fi example because now we rely upon largely fossil fuel plants that produce the same amount 24/7…so cost is not based on supply, like renewables…but on demand.

One of the good things Big Data brings is the ability to do this…something impossible a generation ago.

Another area of Big Data is genomics. It is now possible for a few grand, to get our DNA sequenced. This makes it possible for us to tailor medical treatment.

There are drugs developed that are effective treatments for diseases but for 10% of the population…well, they are fatal, so the drug is shelved.

But with big data…we will have our DNA profile on record; so for those at risk, the drug will not be prescribed, while the rest will be able to access the treatment.

DNA data also has it’s controversies as well.

First, we don’t yet really understand how to interpret it. I was listening to a debate about whether doctors should tell patients who have had sequencing, the total results or should they withhold some.

The argument to withhold was based on the idea that for most ‘genetically’ linked conditions, there are a lot of false positives…as much as two thirds. Also, for conditions that are incurable, are we not subjecting the patient to unnecessary torment?

The other side, is of course it’s our data, we should know what we can…as long as the appropriate caveats are given.

One item in the news lately is the US Supreme Court decision to allow security forces to take DNA samples from suspects. Many have likened this to finger printing, and have no problem with it.

But in the Big Data world, this is a poor analogy. A fingerprint is a single point of data uniquely tied to one person. Your DNA is billions of points of data that not only are linked to you, but all of your relatives. Not only is the suspect, and at this point they are only a suspect, having their privacy breached…I mean what can be more personal than our life essence in the form of DNA…but one’s whole family’s  as well for generations.

And DNA is not only an identifier, it is a complete medical record of sorts.

In one way this is a frustrating time, because as our ability to collect data is expanding exponentially….our ability to process this information has lagged behind. But it is exciting as well because as computing power increases and the sheer number of computers increases…data processing is catching up to data collection.

We are fountains of data…with the devices we use…the data about us they can provide. Think about your location apps, and motion sensors, and the like.

Did you know that modern pacemakers record every heartbeat, every ‘assist’ you need….and it will then upload that to the manufacturer, who can use the data to tweak the software on a pacemaker…potentially inform the doctor of health issues…oddly enough though, the ones I know of, the manufacturer will not let you see the data. Although your heart generated the information, the data itself is somehow proprietary…?

Another example of where personal big data has surprising results is a sensor carpet that can predict when you will fall. It is designed for the aged and infirm.

Basically what happens is it learns the walking pattern of a person, if that pattern alters in a curtain way, it then will signal an alarm warning that the person is likely to fall soon or a fall has occurred. It can do that by detecting slight variation in you stance, gate, muscle usage…lots of pieces of data that can be mined to make predictions.

There is an app developed for diabetics. One problem patients face is that during bouts of depression, they often skip medication leading to obvious health complications. Well, this app tracks their activities and can predict two days prior the subject will be suffering from depression. A warning can then be sent to friends, family and/or caregivers or just an alert to the person themselves.

It does this by noticing the number of calls you make, your tweeting frequency, how far from home you go…correlated variations in your daily pattern…again lots of little pieces of data that is can used to predict the onset of depression.

Another way the world is being datafied is the sources and types of new information generated. IF we think about the web…we know of things called cookies and trackers and even bots…maybe.

If you are on your phone or tablet or PC…when you click on a web site not only is the root data point…the page you are going to…created but also where you are when you clicked, the time of day, the type of devise, where you came from, what page you are going to next…

The dimensions of data have increased. One expert likened it a cyclopes gaining a second or third eye. Not only are you getting more data, but new dimensions of seeing that data and how all this information is related.

In the arena of commerce…Big Data comes from big corporations. IF you think about your corner store…no I don’t mean the 7/11 but the old time mom and pop corner store. In a day they would have tens of sales…if business was good a few hundred a week…and several thousand over their life time.

Wal-Mart alone generates over a million sales transactions an hour. THAT is Big Data and they are using it.

Do you know what product becomes a big seller when a hurricane is looming?

Pop tarts…the data doesn’t say why but whenever a hurricane warning comes out, Wal-Mart brings out the pop tarts.

Another great advancement…well, depending on your perspective, is the ability for marketers and companies to target ads and promotions specifically to you that you genuinely might want (not all data mining gets this right). They will track your purchase history, cross-reference that with other purchases and can predict what you are interested in.

And this marketing has shown weird connections, like people who buy hair dye are also looking for kitty litter…insert crazy old cat lady joke now…

I offer the following funny interaction and perhaps to foreshadow of our next show about the dark side of Big Data.

One day, a Target manager got a very irate call from a father. He was upset that Target sent his teenage daughter a coupon book for expectant mothers. He assured the manager that his daughter was pure and nowhere near pregnant. The Target manager apologised profusely and said they meant no disrespect…much groveling ensued

So, what happened is Target knew from feedback that just sending out coupons for baby products had a negative effect on many women…it’s like “WHAT you’re not pregnant, what’s wrong with you” or something.

Anyway, Target wanted a better way to target its advertising. In steps Big Data. It seems that based on your purchasing habits…like deodorant, moisturizer, etc…they could target the expectant mother market exclusively.

Well, back to our father. As it turned out a few days later the father called the manager again and this time offered an apology. It turned out that after a very intense and frank conversation, the daughter in question was pregnant and had not yet told her father.

So Big Data was right in the fact, but was it wrong in using that information?

In the next part we will see how Big Data has allowed science to see the universe like never before.