Managing the Deluge of Big Data From Space – NASA Jet Propulsion Laboratory October 18, 2013Posted by aquillam in Science.
Tags: astronomy, big data, technology
This is another one of those stories that reminds me about the importance of fundamental research. You might not think that the search for distant galaxies has anything to do with your daily life, but it does.
Scientists working on NASA projects get 10s of terabytes of data every single day. That’s like getting around 500 movies on blu-ray every day. It’s not possible for them to actually look at all that data themselves. In fact, many computers can’t handle downloading that much data in a single day. So the scientists have to develop tools to store, transfer, and process huge data sets in a way that makes it manageable and useful.
Of course, software that makes it possible to accurately predict the weather on Mars or a volcanic eruption on Venus will also work here on Earth. But there are other things too. Health records for example are notoriously huge, and typically require a human to look at it to decide which parts are relevant and which parts aren’t. When you change doctors, it’s not uncommon for your new doctor to only receive the past 10 years worth of records, and that may not include something important, like a childhood disease. Having a tool that would make the records manageable, and that could search for key features without a human would be extremely valuable.
The best part is, this is NASA, so the tools developed become publicly available almost as soon as they’re finished. So those tools that make it possible for an astrophysicist to find a dwarf galaxy in the early universe today might, with minor modification, make it possible for your doctor to spot changes in your health that you haven’t even noticed.