Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Data Storage Australia Space Science IT

SKA Telescope Set To Generate More Data Than Current Net 73

Posted by timothy from the recycle-the-bits-responsibly dept.
angry tapir writes "The forthcoming $2.1 billion Square Kilometre Array (SKA) radio telescope could generate more data per day than the entire internet when it comes online in 2020, according to the director of the International Centre for Radio Astronomy Research (ICRAR), Professor Peter Quinn. SKA — which Australia with New Zealand and South Africa are competing to host, and which will help the search for Earth-like planets, alien life forms, dark matter and black holes — will be 10,000 times more powerful than any telescope currently used. Slashdot has previously discussed the proposal to use 'Skynet' — a grid-computing-based solution for processing and storage."
This discussion has been archived. No new comments can be posted.

SKA Telescope Set To Generate More Data Than Current Net More

SKA Telescope Set To Generate More Data Than Current Net

Comments Filter:

  • Uh oh. (Score:2)

    by ErikZ ( 55491 ) *

    Cue in the HAARP freaks in 3...2...1...

    • Cue confused rude boys in 3...2...1...
      Let's hope this satellite will pick it up, pick it up, pick it up [youtube.com] ...

    • How can they afford this with Congress cutting the budget for everything except military spending and they aren't yet that delusional into planning for an interstellar invasion?

      This thing will blow through their ISP's data cap in the first couple minutes, after which they'll be on the hook for more than $1/Gb... Hell, it might even increment at a rate within 1 or 2 orders of magnitude of the national debt's rate of increase.

  • finally we can retire that saying/meme about Internet is for porn....or that the mass storage market is driven by porn

    unless...I guess we happen to be able to spot the alien equivalents via this SKA.....

  • Really, how much "data" is "generated" by the internet every day?

    Sure, there's lots of traffic, but that's millions of copies of the same data.

    The new data going on to the internet probably isn't too heinous in quantity.

    And the summary blew the meme. It's not "generate more data per day than the internet", it's "generate more data per day than the earth does in a year, and conduct more internal networking traffic than the internet."

    • Depends if you're logging that traffic. I would imagine if you logged all internet traffic, which by strict definition would be considered "new data", it would dwarf whatever data this SKA can produce. Now admittedly, that logged data would be "mostly useless and never read" - but I imagine most of the data the SKA generates will be "mostly useless and never read", too.

      • Re: (Score:2)

        by blair1q ( 305137 )

        if you start logging all traffic, you'll quickly run out of space to store it.

        if on the other hand you start logging only novel content, you'll take somewhat longer to run out of space to store it.

        just ask google how their cache works.

    • Re: (Score:2)

      by sFurbo ( 1361249 )
      Sure, not that much new information is made by the net, but it sure is represented by a lot of data. Or, that is how I would use the terms, YMMV.

  • SKA and other astronomy projects (Score:3)

    by JoshuaZ ( 1134087 ) on Thursday July 07, 2011 @07:31PM (#36689074) Homepage
    Given that we just had a Slashdot article about how the space based James Webb telescope is already on the ropes with Congress, http://science.slashdot.org/story/11/07/07/0038247/Congress-Dumps-James-Webb-Space-Telescope [slashdot.org], perhaps we should be worried that the same will happen for SKA. Unlike Webb, SKA is an international project, so it won't necessarily go down the tubes if the US backs out. Moreover, the US has backed out of European lead science projects before often with very little warning. SKA is going to allow some very interesting work. Among other things, SKA might be able to detect extraterrestrial life, either through direct radio signals (from intelligent life) which would be a really big deal, or more indirectly detect non-intelligent through the analysis of extrasolar planets' atmospheres (such as the detection of large amounts of oxygen). SKA will also be used for many other astronomy and astrophysics projects, such as examination of supernovas. SKA is very good science, let's hope that the penny pinchers who repeatedly cut tiny science programs while leaving defense, social security and medicare alone will not touch it. In the long run, science helps everyone.

    • The US is not even involved in the SKA.

    • Re: (Score:2)

      by st0nes ( 1120305 )
      The SKA isn't all-or-nothing. It can start small with a few receivers (South Africa already have the MeerKAT array [wikipedia.org]) and scale when funding is available. It doesn't really matter if there are delays--good science can still be done with half an SKA.
  • Sure but think about how much more data the Internet generates today than it did 10 years ago. I'm sure we'll be able to cope in 10 years. Also, I know the LHC produces a ton of data. Does anyone know how it compares?
  • More data/traffic than the internet NOW, or then the internet will be doing in 2020? 'cause let me tell you, the last 9 years has had a pretty sharp increase...
  • This will true until someone post the data on the pirate bay.

  • NBN (Score:4, Interesting)

    by dakameleon ( 1126377 ) on Thursday July 07, 2011 @07:44PM (#36689188)

    (Warning: Australian content ahead!)

    I hope this lays down a water-tight case for the NBN going ahead - or the combination of the two being a catalyst for each other. If there's one thing this is good for demonstrating, it's that future data requirements will outgrow the current infrastructure very quickly, and a project which is as far-sighted as installing FTTH throughout the country has a justification for the unforeseen benefits it can help happen.

    (and bah humbug to anyone who thinks the SKA isn't justified to begin with!)

    • That's absurd on the face of it, you don't need FTTH for the SKA, because the data isn't going to be distributed on a large scale to homes! For the SKA, you only need one fat pipe leading from the SKA, out to the under-ocean cables.

  • I did not know that the music created by the likes of "Reel Big Fish", "The Clash" or other Ska bands could be used to find intelligent life somewhere...

    • Re: (Score:3, Funny)

      by Hsien-Ko ( 1090623 )
      Implying that there's depth in ska to use a telescope for that is. A reggae telescope would be better. Let's all get together and see deep space, mon.

      • Re: (Score:1)

        by Kyont ( 145761 )

        No, there is just too much data. They chose ska because they needed something exactly like reggae except much, much faster.

    • Calling the Clash a ska band is like calling the Beatles a ska band.

  • is this a good thing? (Score:3)

    by Charliemopps ( 1157495 ) on Thursday July 07, 2011 @07:50PM (#36689222)
    If the CIA invented a device that listened into every phone call in the entire world, real time and dumped it all as a WAV file on a storage device in the basement, would that really do them any good at all?

    • Re: (Score:1)

      by Anonymous Coward

      If the CIA invented a device that listened into every phone call in the entire world, real time and dumped it all as a WAV file on a storage device in the basement, would that really do them any good at all?

      Why would they invent that when they already have a device that listens into every phone call in the entire world, real time and dumps it all as a bunch of AMR files on a storage array in the basement?

  • ...calling your global computing initiative 'theskynet' might be detrimental to your acceptance by the technologically affluent who operate the computers you need help from.
  • Finally a new unit for large amounts of data, appropriate for this new golden age of progress and hyperbole What I want to know is how many library of congresses are in an 'internet-year'...

  • Sensationalistically inaccurate article... (Score:4, Informative)

    by Dahamma ( 304068 ) on Thursday July 07, 2011 @08:15PM (#36689424)

    The project is expected to deliver up to an exabyte a day of raw data, compressed to some 10 petabytes of data in images for storage.

    So, 10 petabytes of data - who cares about the raw source. I work for a video streaming company and we have several petabytes of H.264 video. If that were to be uncompressed into 30 FPS 1080p raw data, it would be 50-100x that, so already approaching a couple hundred petabytes. And think of all of the JPEGs out there - why don't we just uncompress all of those for the comparison as well?

    A (likely conservative) back of the hand calculation by Google estimated at least 5 exabytes accessible on the Internet (so even the wrong estimate is wrong). I'd imagine a huge percentage of that is compressed video, audio, and images. So, basically 5 exabytes vs 10 petabytes - it's off by 3 orders of magnitude.

    • Re: (Score:2, Informative)

      by Anonymous Coward

      Indeed ... while it's an impressive number, we already have experiments that generate more "raw data" per day than that: "CERN experiments generating one petabyte of data every second" http://www.v3.co.uk/v3-uk/news/2081263/cern-experiments-generating-petabyte That's 84EB per day. But "all" of it is crap, and they eventually store only about 25PB per year.

    • It's a specious analogy. Video and audio can be compressed with loss, and the algorithms make heavy use of human perceptual limitations. Scientific data produced by large instruments need to have breadth and depth; the instrument is a scarce resource and there are unlimited ways of reducing radio astronomy datasets to produce different data and different insights. Especially with radio, you're going to be collecting a ton of white noise-looking data, but you can't use a lossy compression algorithm to trim i

      • Re: (Score:2)

        by Dahamma ( 304068 )

        It's a specious analogy.

        If you RTFA, no it's not.

        The project is expected to deliver up to an exabyte a day of raw data, compressed to some 10 petabytes of data in images for storage.

        They clearly say it's compressed before it's stored. The raw data number itself was pure boast. Besides, no one said it was lossy compression, anyway. Lossless image compression can be very efficient depending on the image.

        • It's a distortion on the part of the article. Radio astronomy raw data are not images. And the computational effort to reduce the raw data into data from which you could make images is large enough that you tend to store the result of the processing alongside the original data--it takes up more space, not less.

          It still doesn't matter, because there's no way they'll be running the telescope at full throttle until several years after commissioning.

    • So, 10 petabytes of data - who cares about the raw source.

      Depends on what you're doing. If you're taking pictures of the Eiffel Tower on your vacation you're not really interested in if the CCD on your camera's LSB was a 1 or a 0. If you're taking images from a hyperspectral sensor for scientific purposes the better the accuracy the better (hopefully) your results. It depends on the type of application and how important accuracy is to you (and how accurate your sensor is).

      I work for a video streaming company and we have several petabytes of H.264 video.

      We've got several hundreds of terabytes in lossless compressed high resolution hyper spectral

      • Re: (Score:2)

        by Dahamma ( 304068 )

        So, 10 petabytes of data - who cares about the raw source.

        Depends on what you're doing. If you're taking pictures of the Eiffel Tower on your vacation you're not really interested in if the CCD on your camera's LSB was a 1 or a 0. If you're taking images from a hyperspectral sensor for scientific purposes the better the accuracy the better (hopefully) your results. It depends on the type of application and how important accuracy is to you (and how accurate your sensor is).

        I already pointed this out above, but it you RTFA they clearly say it would be compressed before it's stored. If they don't store the raw data, then, yes, who cares about it because it doesn't exist for analysis. And as I also pointed out, nowhere did it say it was lossy, just that the ~1EB of data compressed to 10PB.

    • Re: (Score:2)

      by dargaud ( 518470 )
      This kind of back of the hand calculations and the fact that hard drive capacity hasn't increased much in the last few years (from 2Tb 3 years ago to only 3Tb now) makes me wonder if there aren't much larger prototype HDs in use in large datacenters à la Google.

  • In nine years my home DVR system will generate more data than the entire internet!

  • This begs for research on how the neural system in the eye compresses raw data into information that can be transmitted through the limited bandwidth optic nerve. Collecting data is great but we can drown in data. We need information processing near to the source.

    • Re: (Score:3)

      by ceoyoyo ( 59147 )

      We know a decent amount about it. And we don't want to replicate it in our scientific instruments. The brain does all sorts of extrapolating, interpolating and other forms of making shit up. Which is great if you're a mammal who needs to see the sabre toothed tiger stalking you, but not so good if you're trying to get accurate, quantitative data out of a scientific instrument.

      • Re: (Score:1)

        by Anonymous Coward

        Also, computationally intensive to do algorithmically, free when it happens due to quantum interference or neurotransmitter binding or whatever in the optic nerve.

  • storage called Skynet... ...I'm sure this won't turn out bady.

  • I wonder if the system will run on a Perl one-liner.
  • This merely says they are unable to filter out all the noise. Producing huge numbers of bits isn't a particularly remarkable achievement. I guess there is far less than 1% actual information in that bit stream.
  • was a checkerboard pattern in black and white. When reached for comment, the leading scientist on the project had this to say,
    "Pick it up, pick it up, GO!"

Slashdot Top Deals

What is research but a blind date with knowledge? -- Will Harvey

Close