backtop

Print E-mail del.icio.us 24 comment(s) - last by KnIgHtCoM.. on Jul 13 at 3:45 AM
Researcher claims 70% of intelligence gathered falls on the floor

One of the keys to any modern war or intelligence gathering operation is information. The more information that planners have about an enemy or a situation, the more likely they are to come up with a resolution to the situation that is faster, cheaper, and saves lives.

The problem for many branches of the military is that the glut of information produced by intelligence gathering is so overwhelming that much of it is never properly analyzed. A report from December of 2008 called Data Analysis Challenges was provided recently after a Freedom of Information Act request, the report was originally withheld from the public.

The report was written by a DoD advising group known as JASON in conjunction with a non-profit company MITRE Corporation. The report says that the huge amounts of data being collected by the myriad of sensors and imagery devices the military uses today is becoming increasingly difficult to store, analyze, and integrate into defense systems.

InformationWeek reports that one DoD surveillance system called Constant Hawk produces up to 100 or more terabytes of data over a period of only a few hours. Before the information can be used, it has to be collected, analyzed, and distributed quickly. According to the JASON report, intelligence and defense communities complain that much of the intelligence data gathered isn’t made useful.

MIT research scientist Pete Rustan is cited in the report saying, "Seventy percent of the data we collect is falling on the floor."

Scientists say that the problem of information overload will only grow as sensors improve and more data volume is collected. Sensor volume is projected to potentially grow to Yottabyte levels by 2015. A Yottabyte is 10^24 bytes.

The JASON report recommends that the DoD take queues from the way Google, Facebook, and Yahoo handle the huge amount of data they process. Investment in algorithmic advances and cloud-based storage and computing are critical according to the report. InformationWeek reports that the military sees potential use for the Hive language used by Facebook for data warehousing.

Comments     Threshold


This article is over a month old, voting and posting comments is disabled
No kidding.
By MrBlastman on 7/10/2009 11:07:21 AM , Rating: 2
It is insane how much information is available at their fingertips nowadays. Whether they are using all of it or not, I'd rather have it available than not there at all. The article here sounds like they are referring to the C&C level rather than the front lines - but - even on the front lines there can be overload as well.

Think about all the things a typical fighter pilot has to take in during a single combat mission. They have to spend 30+ minutes - 2 hours planning for the mission, going over recon reports, intel, maps, etc. They have to set their flight plan + additional spotting for hostile areas, program it into their plane. Once they are in the cockpit they have to deal with AWACS, Radar, Threat displays, ground reports, C&C comms, datalink with all their wingmen in their package, outside packages they are co-ordinating with.

Once they are over FLOT they have to sometimes deal with a FAC, they have to contend with hostiles on the ground and on top of that depending on the region, have to worry about airborne threats. They have to contend with all this while also flying the jet - watching the hud, altitude, location, fuel management, weapons arming/priming/configuration, designation of targets vs. FAC targets - all in an ever changing environment. If that isn't enough information - all passed to them through the cockpit, they have to additionally keep their eyeballs outside of the cockpit on the environment around them, constantly scanning due to sensor limitations. All of this isn't even counting the additional insanity a combat helo pilot (such as an AH-64) has to contend with.

But, you know what? I'm sure our troops would rather have access to all of this data than not having it at all. It is far better to have then to be out in the field cut off from everything with no idea what is going on.

Perhaps, at the C&C level, they need to make the information more readily accessible with ease to those whom need to use it and improve the distribution scheme.




RE: No kidding.
By FITCamaro on 7/10/2009 12:10:19 PM , Rating: 2
Next generation fighters will drastically simplify the flying of a jet for a fighter pilot to where they'll only have to physically fly the jet. ;)


RE: No kidding.
By MrBlastman on 7/10/2009 12:38:46 PM , Rating: 2
While I think they'll simplify the management of the technology, I think at the same time it will provide even more information than they had before thus leading to the same problem. I doubt we'll see their workload decreasing substantially. Why? Because a good commander will look at his squadron and see - hey! They are working less here so we can push them harder in this way to increase results even further.

Well, that or either they'll kill off the manned fighter program completely (which some are trying to do in the military making everthing like a UAV). I'm old fashioned in thinking though - I feel strongly that nothing can replace a set of eyeballs physically in the pilots seat. I don't think technology is completely there yet to give a remote pilot the same amount of SA in a full blown combat environment.

Fighters have only gotten more and more complicated with every generation. My hats go off to the guys that fly them, it is a complex and demanding job that almost requies you to be a superhuman to do well.


RE: No kidding.
By majBUZZ on 7/10/2009 5:09:17 PM , Rating: 2
From a intelligence gathering point of view I believe that more field agents ( traditional spys and undercover agents ) operating overseas doing target specific intelligence gathering would eliminate some of this information overload, pictures from space and uavs can only tell so much. And the raw data still have to be analyzed by human eyes at some point in the process.


That's a Yotta Data...
By 440sixpack on 7/10/2009 3:01:02 PM , Rating: 5
Launch a few satellites, and yotta yotta yotta, information overload.

I can hear the groans now. ;-)




What kind of storage..
By Silver2k7 on 7/10/2009 1:13:57 PM , Rating: 2
100TB in a few hours.. how much data are they gathering in 24hours, 500TB or more ?

Thats almost insane :D
I wonder what kind of storage setup they are using!




RE: What kind of storage..
By breakonenine on 7/10/2009 11:15:41 PM , Rating: 2
IBM Servers the size of small cars, and all automated.

Lets see, at aprx 18 million over 5 years, they should be able to least store 80 terabytes a day for the Stimulous Looting Package Website?

They are probably paying less than that, if they are paying twenty five cents for a gigabyte of info storage medium, I don't think they are worried about how much data they collect.


A Big Deal
By InfantryRocks on 7/10/2009 2:12:28 PM , Rating: 4
This is getting to be quite an issue. When I was a rifle Platoon Leader, my guys and I would gather tons of raw intel for our intel guys. God knows how much of it they actually were able to process and analyze.

Now, in my network nerd job, "knowledge management" is getting to be a pressing concern at all levels. It's not a very sexy topic, and there's no clear way on how to do it, so it's not real popular, but we're gonna have to figure something out.




Info Overload
By breakonenine on 7/10/2009 11:08:04 PM , Rating: 2
Info over load can make you spend money on things that you don't need, if you can't handle the data, then you don't need the data. Do soldiers run out of data or bullets?

On the other hand you may want a library of information. I can't review every episode of CSPAN daily, but eventually I want to. Sometimes speed is not as important as relevance.




overloading
By djkrypplephite on 7/12/2009 11:19:50 PM , Rating: 2
Seems like the only overload we don't yet have in the military is a pay overload . . .




a yodabyte?!
By KnIgHtCoM on 7/13/2009 3:45:16 AM , Rating: 2
think of how much pr0n that could fit on a yodabyte.




Like dumb camera users
By marsbound2024 on 7/10/09, Rating: -1
RE: Like dumb camera users
By DASQ on 7/10/2009 11:18:40 AM , Rating: 1
People take the pictures at higher resolution and then shrink them. What do you think looks better, a picture at 8MP, or a 16MP photo shrunk to 8MP size?


RE: Like dumb camera users
By bhieb on 7/10/2009 11:29:10 AM , Rating: 2
They both look the same. A 16MP image has by definition 16MP, if you shrink it to 8MP with no cropping, you have cut the pixels in 1/2, and therefore cannot in no way look "better" than 8MP.


RE: Like dumb camera users
By bhieb on 7/10/2009 11:31:41 AM , Rating: 4
Just to clarify, there are uses for higher MP, but the only purpose for taking at a higher MP is for cropping, printing at large scale, and zooming. These are obviously very important on say a aerial spy photo so to the OP, no you cannot just take them at lower res if you intend on zooming in on that jeep.


RE: Like dumb camera users
By marsbound2024 on 7/10/2009 11:42:23 AM , Rating: 2
Or you can take the image and the reduce the size to what is needed and cut out the excess so that you don't have unnecessary data.


RE: Like dumb camera users
By bhieb on 7/10/2009 12:12:23 PM , Rating: 2
Ummm, thanks for saying exactly what I said????

quote:
but the only purpose for taking at a higher MP is for cropping , printing at large scale, and zooming


Is Cropping/Zooming not the same as

quote:
Or you can take the image and the reduce the size to what is needed and cut out the excess so that you don't have unnecessary data.


RE: Like dumb camera users
By bohrd on 7/10/2009 12:24:49 PM , Rating: 2
But that is the exact problem they are having. How do you know what is useful information until you analyze it? Just because the target is, for example, a house, how do you know that the car driving towards it isn't necessary? Throwing out information or not having it is much worse than having too much.


RE: Like dumb camera users
By jimhsu on 7/10/2009 11:42:49 AM , Rating: 2
Incorrect, assuming that we are using a standard digital camera with Bayer interpolation. Thus, the information density of the sensors is significantly less than the advertised spec.


RE: Like dumb camera users
By bhieb on 7/10/2009 12:09:44 PM , Rating: 2
Oh sure go all technical I see your point that the 8MP pictures is not really 8MP but the shrunk one is.

However that was not really the scope of what was said. He is trying to say that taking x and shrinking it to y is better than taking z which is the same as y (you say it is not the same as y thus making it not an apples to oranges case and no comparison can be made since fundamentally they are not the same quality).

I was merely pointing out that you cannot magically get better image quality taking a 16MP image and shrinking it to 8MP vs the same native 8MP. The pixels are not "better" just because it was 16MP originally, there are just more of them.

So taking away 1/2 via software shrink or via sensor size would result in the same image. Now that may not really be accurate either since some shrinking algorithms might compress like pixels first (ie the blue sky) and leave more detail in the persons face for example (I honestly don't know how all of them work).


RE: Like dumb camera users
By rippleyaliens on 7/10/2009 12:21:45 PM , Rating: 2
Wow, all this Nerd fighting is getting crazy.. This 8MP will look fine, with this and that,, Nerd2,, no the 16MP can crop this and that., blah blah blah..
How about the fact that Yottabyte amount of storage was mentioned....
Cheezzeee..... you nerds need to like unplug.. and get more of a life, other than hitting google once Dailytech post's a news article, and you start picking it apart, and having your nerdy wars... AND PS, Recon/Intel is not just pictures.....


RE: Like dumb camera users
By bhieb on 7/10/2009 12:34:11 PM , Rating: 2
Hey it's Friday and I'm bored :)

Agreed though what started as a simple correction has gotten a bit off topic.

No more off topic clutter from me....well except this post in response to uhhhh your off topic post about clutter.

OH THE HUMANITY!


RE: Like dumb camera users
By jimhsu on 7/10/2009 9:37:08 PM , Rating: 2
Something about Pascal and the inability of man to sit in a quiet room? I guess these so called "nerd wars" are a reflection of that.

People are bored, that's all.


RE: Like dumb camera users
By stromgald30 on 7/10/2009 11:53:37 AM , Rating: 2
It's alot different than that. I was in a class once where the professor was involved in intelligence gathering back in the 80s. Alot of the satellites take pictures before and after the intended target. You don't want to miss the target, so you have to turn it on before you get there and turn it off afterwards. Considering the speed at which the satellite is moving, there are alot of excess pictures taken that are never processed.

In fact, many of the older satellites didn't even have any control of when pictures were taken. They just took pictures until it ran out of film, then came back down in the ocean. Tons of pictures of American farmland in those days.


"There is a single light of science, and to brighten it anywhere is to brighten it everywhere." -- Isaac Asimov

DailyTech Poll
Which web browser do you use on your primary personal machine? 






44 Comments







botimage
Copyright 2009 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki