Print 56 comment(s) - last by mlembeck.. on Mar 12 at 11:38 AM

New research into "mixed reality states" promises Matrix-like "whoa"

Real-time model-based feedback is something that is far from commonplace in today's world.  The basic concept of real-time feedback is to take a physical system, modeled by mathematic equations, and then couple it with a computer as a virtual system. 

Sensor monitoring gives the processing logic an idea of how well the real world system is conforming to the theoretical model.  The results are processed and yield adjustments (feedback) to the real world system to make it in tune with the theoretical model.  The result is that the virtual and real world models converge into a single "mixed reality" system, bridging a the virtual (theoretical) and physical world.

Such an approach holds large benefits for everything from car handling and fuel economy, to better aircraft dynamics and smoother robotic control.  To accomplish such useful applications, researchers working on mixed reality had to start simple -- real simple. 

Researchers at the University of Illinois created a virtual pendulum and a real world counterpart that behaves as the world's first mixed reality system.  Bidirectional instantaneous coupling, adjustments both to the real world pendulum by motor feedback and the virtual pendulum by tweaking mathematical parameters, yielded a single system in which both systems' are synchronized.  The result is two pendulums swinging as one.

The experiment, the first fully successful one of its kind, sounds simple but raises mind-blowing questions about reality.  According to Illinois physicist Alfred Hubler, "In a mixed reality state there is no clear boundary between the real system and the virtual system.  The line blurs between what’s real and what isn’t."

Hubler describes the pendulums synchronization, stating, "[The pendulums] suddenly noticed each other, synchronized their motions, and danced together indefinitely."

Two physical mechanical systems have been previously coupled, but never before has a real world and virtual one been mixed.  Such a breakthrough was only possible thanks to ultra-fast computing, which allowed real-time processing of the pendulum data, and real-time response.  Hubler states, "Computers are now fast enough that we can detect the position of the real pendulum, compute the dynamics of the virtual pendulum, and compute appropriate feedback to the real pendulum, all in real time."

Hubler thinks that eventually coupling of the real and virtual worlds, may lead to it being hard to tell what is real and what is fake -- a topic immortalized by generations of science fiction writers.   Hubler worries people may become defensive and paranoid in the real world, based on threats in the virtual world.

The research was funded by the National Science Foundation and will be presented by Hubler at the annual American Physical Society meeting, which will be held in New Orleans, March 10-14, 2008.

Comments     Threshold

This article is over a month old, voting and posting comments is disabled

Say What
By BoxCutterLou on 3/10/2008 4:30:11 PM , Rating: 2
So the pendulem on the computer screen moved with the real one? And stoped when they were un-allinged? I'm missing something here.

RE: Say What
By DASQ on 3/10/2008 4:38:06 PM , Rating: 5
The computer pendulum constantly adjusted for the position of the real pendulum. The concept is that if they're indistinguishable in terms of what IS happening and what SHOULD be happening, how do you know which one is the 'real' one?

RE: Say What
By DASQ on 3/10/2008 4:41:16 PM , Rating: 2
And vice versa. The physical system can mimic the computer system, and the computer system can mimic the physical system.

RE: Say What
By BoxCutterLou on 3/10/2008 4:54:01 PM , Rating: 2
"The physical system can mimic the computer system"

AHH I SEE! Well after staring at this phraze for a few moments. This is incredible. Who's in control should be the question at hand, not which one is real.

RE: Say What
By DASQ on 3/10/2008 10:33:03 PM , Rating: 2
You have a point, but following that thought, it doesn't matter which is real, because whoever is controlling it can make each alternatively real or "unreal" at any given time. "He" can essentially switch the two back and forth, if the two systems will synchronize as well as it should.

If you can't tell the difference between the two, does it matter which is "real" or not? Because if the physical system is real, it's simply mimicking the 'virtual' system. If the 'virtual' system is what you are perceiving, then it's simply mimicking the physical system anyway. It doesn't matter.

The only constant is the control.

RE: Say What
By JustTom on 3/11/2008 12:39:20 AM , Rating: 5
You'd be able to tell the difference easily; stand in front of both pendulums, the one that smacks you in the nose is the real one.

RE: Say What
By paydirt on 3/11/2008 9:10:39 AM , Rating: 2
Sounds like mental masturbation to me.

RE: Say What
By bhieb on 3/10/2008 5:09:16 PM , Rating: 5
So if I unplug the computer does the real world pendulum stop? How about disappear in front of my eyes, since the virtual one has been destroyed is the physical one. If not then this is no more than a computer controlled pendulum.

RE: Say What
By bhieb on 3/10/2008 5:12:56 PM , Rating: 2
Let me clear up that last part it is either a computer controlled pendulums, or a real time pundulum monitor. Either way they did not...
"suddenly noticed each other, synchronized their motions, and danced together indefinitely."

No they did not you idiot the hours of programming is what made them interact, nothing happened suddenly or indefinately.

RE: Say What
By DASQ on 3/10/2008 11:10:16 PM , Rating: 2
Before we resort to name calling, let me try and clarify the article some:

A motor was attached to the real pendulum. Software controlled the other.

If you told the software to speed up the pendulum slowly, the motorized pendulum would follow along precisely. Obviously a pendulum does not speed up without external stimuli, so you'd notice something was wrong. But you can't tell which pendulum is real or not.

The 'hours of programming' can be thought of as the computer version of the brain keeping track of the two pendulums and updating each perfectly, synchronously. The programming is 'smart' enough to know when one has change, and will adjust the other in real time to match it. I think that's just a short sighted comment :/

RE: Say What
By DASQ on 3/10/2008 10:41:29 PM , Rating: 2
If you told the computer pendulum to stop, the real world pendulum would stop as quickly (slowly) as the electronic one would.

The "disappearing" pendulum is kind of irrelevant. The point is, as long as someone cannot tell the difference between the "computer" pendulum and the "physical" pendulum (for instance, in a room with two small boxes with viewing windows, one is a real pendulum, and one is a hyper-realistic computer monitor displaying a 'perfect' render), then they are both as real as you, standing there staring at the two pendulums. You can't tell which is real or not.

The pendulums simply represent systems in which reality or illusion are displayed. To maintain the illusion, we have to assume that if your pendulum 'disappeared' for some reason, the computer would somehow make it disappear that followed our rules of physics, so you wouldn't realize that it wasn't "real". Of course, like in the Matrix, this is sometimes impossible, resulting in the person "plugged in" rejecting the Matrix, and "waking up".

RE: Say What
By bhieb on 3/11/2008 9:34:12 AM , Rating: 2
Ok first if you attach a motor to a pendulum it is no longer a pendulum by definition a pendulum swings freely. So now you have created a computer controlled robotic arm.

Secondly if all you are talking about is if it is real visually then yes looking at two in a box, one could not tell the difference. However that is not how we judge reality, if I reach into said box and cannot touch it then it is not real.

All I am trying to say here is that this is not reality, it is just a mimic of it. Simple 2-way robotics if you will. The computer adjusts an image based on inputs from a physical device, it is a mimic not reality.

RE: Say What
By DASQ on 3/11/2008 10:40:09 AM , Rating: 2
... the fact that the physical pendulum has a motor is irrelevant. You're utterly missing the entire point of the experiment. The viewer has no idea the pendulum has a motor attached to it.

Forget it, I give up, re-read the comments in this article and the article itself, if you still don't get it, then either you're not old enough to get it, or you're just not going to get it at all.

RE: Say What
By BoxCutterLou on 3/10/2008 4:44:57 PM , Rating: 2
"how do you know which one is the 'real' one?"

Thats the part I'm not getting. Unless there creating a virtual pendulem in real life, I'd say the one in the moniters the fake one.

Thats what it is isn't it? Theres no moniter?

RE: Say What
By Cygni on 3/10/2008 5:16:42 PM , Rating: 3

RE: Say What
By Parhel on 3/10/2008 4:58:30 PM , Rating: 2
I'm not following you. The 'real' pendulum is the one that physically exists and has physical properties such as mass.

And how is this categorically different from the kind of software that already exists to monitor automobiles, production line equipment, etc.? Software adjusts the behavior of the real (physical) system to make it behave like a simulated one.

The excitable physicist reminds me of a recent trip I made to the art museum. There was a canvas painted entirely black, with a three page explanation attached to tell you why you should think that it's significant. I bet someone paid a lot of money for that painting too.

RE: Say What
By BoxCutterLou on 3/10/2008 5:10:05 PM , Rating: 2
I know this is a stretch, but you could potentially pick something up in the computer environment( say your very heavy dresser or fridge) and move it with out ever touching the real physical thing. Well thats where I percieve this kinda tech is headed. but what do i know. I still don't understand how we won't beable to tell whats real and whats not unless its used for vr. And all i want in life is a vr helmet. They have 50 yrs to make one before I'm probably dead. Get on it!

RE: Say What
By Silver2k7 on 3/10/2008 6:13:33 PM , Rating: 2
Yeah bring on the vr =)

It just need some sensor(s) to detect the real world then add some layer of vr into the real world.. only visible for the ones wearing the helmet/glasses of course.. then you can have mixed reality vr games :D

RE: Say What
By DASQ on 3/10/2008 11:04:55 PM , Rating: 2
Okay, think of it this way: (this will seem kind of ridiculous, but just read it to the end :))

Your skin is biomechanical (like the Terminator). At the same time you feel sensation (and the electricity races up to your brain through the appropriate nerve clusters), the same 'feeling' is sent to a computer. If the computer has some kind of infinite bandwidth and CPU power, it can essentially duplicate everything you see, feel, smell, taste and hear (information). Like a super realistic video game, except you 'play' it with your own body.

Following me so far?

Now, if the computer instead 'hijacks' your senses, and says, creates the image and smell of a pie (let's say blueberry) on a windowsill, how can you tell if that pie is real or not? The computer is telling your eyes there is a blueberry pie on the windowsill. You see steam rising from the little holes in the top, and you smell it's flaky delicious crust and hot filling. But as you walk forward to pick up the pie, the computer tells your hands (brain) that you have a hot pie in your hands that weighs 1.3lbs, and otherwise perfectly obeys our known laws of physics.

How can you tell if the pie is real or not?
Kind of a basic example of how the illusion can be translated into our reality.

I think the usages listed in the article are pretty basic and nowhere near as 'whoah' as the Matrix comparison.

Think of a robot arm. It welds two pieces of metal together. Simple. Now the computer program is following along, movement for movement, tracking the arm. If the computer program detects the physical robot arm has gone off by 1nm (say, a gear is getting slightly rusty, or one side is lubricated better than the other), the computer can correct it in real time and put the arm back on track. The mistake will be still be there, but it was caught almost as soon as it started (far more efficient than human error correction). The arm doesn't realize it is off by such a tiny amount because it can't actively track everything. It think it's okay. A human brain can tell if his drawing looks 'off' ever so slightly, but a computer would normally need time to analyze the differences between the source and the actual output.

It's not that it was "omg impossible" before, it was simply never accurate to this level. Meaning we can now start mimicking the experience of the Matrix.

RE: Say What
By JustTom on 3/11/2008 11:32:29 AM , Rating: 2
How can you tell if the pie is real or not?

I could tell when I bit into it and the blueberry goodness does not end up in my belly.

mmmmmmmmmmmmmmmmmmmmmm pie.....

RE: Say What
By KristopherKubicki on 3/11/2008 1:36:17 AM , Rating: 2
And how is this categorically different from the kind of software that already exists to monitor automobiles, production line equipment, etc.? Software adjusts the behavior of the real (physical) system to make it behave like a simulated one.

Software that monitors cruise control, for example, is typically just a mechanical system. If its not a mechanical system, its some sort of analog transform that just compensates for jitter.

What they're doing here is actually simulating what should occur (in this instance, where the pendulum should be without friction) and what actually does occur, and then compensating for it.

By wushuktl on 3/10/2008 4:35:05 PM , Rating: 2
maybe i'm just stupid... but i don't understand the significance of this. How is this any different than somebody modeling a joystick and somebody moving it and the computer taking in the position of the joystick and moving it around in a 3d model?

RE: uh...
By SilthDraeth on 3/10/2008 4:48:27 PM , Rating: 2
I think that the software, aka virtual pendulum syncs itself with the real one, and then they can make changes to the virtual that will change the real through some sort of advanced force feedback...

Or I am totally off base.

RE: uh...
By KristopherKubicki on 3/10/2008 5:07:05 PM , Rating: 2
When you or I move a joystick on a computer, you and I are just sending electrical signals to an interpreter that then moves your avatar around a virtual world.

What these guys have done is created a device that exists in both the virtual and physical world using real-time monitoring and feedback mechanisms. If the pendulum is stopped in the virtual world, it stops in the real world. If it stops in the real world, it stops in the virtual world. Like the article mentioned, it's pretty basic right now.

Hübler mentions that his device swings indefinitely, "frictionless" if you will. The virtual pendulum, not programmed for things like friction, just ignores the difference and compensates the "real" pendulum with feedback mechanisms instead.

This could all be done mechanically, of course, but the mechanical solutions for solving these problems get incredibly hard with additional variables.

Fly-by-wire systems, for example, are more or less a collection of various leveling sensors tied to adaptive surfaces. With mixed-reality methods, the plane would instead adjust and compensate its flying mechanisms based on simulated environments with real-time data.

RE: uh...
By Zurtex on 3/10/2008 5:12:44 PM , Rating: 2
Oh, I understand, basically having a layer of abstraction between the purpose, method or function of an object and the mechanics which make it.

That sounds quite nice and if you run its course it would lead to awesome god like powers over reality I suppose. But even in the long term (as apposed to the very very long term), wouldn't this just make mechanical devices far too complex and unworkable in real world environments?

RE: uh...
By KristopherKubicki on 3/10/2008 5:17:26 PM , Rating: 2
Yeah, I don't personally know how deep the rabbit hole goes for mechanical systems, since humans are actually pretty good at mechanical compromising systems.

But this has other uses, especially in economics, where mixed reality economies are already here (World of Warcraft, Second Life) and the principles are basically the same.

RE: uh...
By bhieb on 3/11/2008 9:39:44 AM , Rating: 2
Ah, but once you add a motor to a pendulum, it is no longer a pendulum, by definition a pendulum must swing freely. All you have is a robotic arm.

RE: uh...
By KristopherKubicki on 3/11/2008 1:38:51 PM , Rating: 2
It does still swing freely, the motor only kicks in to adjust for friction.

RE: uh...
By dever on 3/11/2008 3:30:44 PM , Rating: 2
If the feedback is two-directional, which system has precedent, the frictional reality or the frictionless virtual world?

By Chris Peredun on 3/10/2008 4:17:57 PM , Rating: 6

RE: Dude.
By Neamhtearanntacht on 3/10/2008 4:28:19 PM , Rating: 2
Whoa, this article can't be real, can it duDE!

RE: Dude.
By chrispyski on 3/10/2008 4:31:15 PM , Rating: 2
Agreed. Take THAT all those people who said DT needs to dumb down their articles!

RE: Dude.
By R Nilla on 3/10/2008 4:43:26 PM , Rating: 2
Free your mind.

And this makes sense why?
By pomaikai on 3/10/2008 5:17:01 PM , Rating: 2
"Sensor monitoring gives the processing logic an idea of how well the real world system is conforming to the theoretical model. The results are processed and yield adjustments (feedback) to the real world system to make it in tune with the theoretical model."

Here is my understanding. In the first statement above we are comparing real world to math theories. This will help us figure out if the theory is correct. What I get out of the second sentence is that if the theory is not correct lets adjust real life to make our theory correct.

RE: And this makes sense why?
By KristopherKubicki on 3/10/2008 5:20:01 PM , Rating: 2
You basically got it. Friction does not apply to the virtual pendulum, though all other forms of physics do. So the virtual pendulum just compensates for friction in the real pendulum, enough to allow it to swing "indefinitely." :)

RE: And this makes sense why?
By SlyNine on 3/10/2008 6:56:18 PM , Rating: 2
Is their a motor forcing the real pendulum to adjust to what the virtual model is doing? If so then it wont swing "indefinitely" once power runs out.

RE: And this makes sense why?
By KristopherKubicki on 3/10/2008 9:56:18 PM , Rating: 2
Yes, the motor is adjusting. And that's why everyone puts "indefinitely" in quotes -- it's indefinitely in the classical physics sense, not in the practical physics sense.

RE: And this makes sense why?
By SlyNine on 3/11/2008 1:39:54 AM , Rating: 2
Ok pessimism aside, This should help cars run more efficiently, jets super-manuvarable, stuff like that right. or am I just missing the point.

So, what exactly is the point?
By Zurtex on 3/10/2008 4:51:41 PM , Rating: 3
As a mathematician looking at this I seem to be missing something.

They've created a pendulum that can be slightly manipulated by a complex computer program to slightly get closer to a theoretical model of it. The computer program can be viewed as an abstract and is slightly correlated to being closer to the real life version. All of this in a highly controlled environment, doesn't seem that exciting.

This is also not truly bidirectional, there is still going to be a small error, it would of seemed to make more sense to create a 1-1 mapping from real life to the virtual model as to exactly measure the error of the theory with a great degree of accuracy.

Can someone explain what I'm missing?

By KristopherKubicki on 3/10/2008 5:14:14 PM , Rating: 1
The breakthrough isn't exactly that they've been able to create a mixed reality state, but that via processing and feedback the device can fix the real pendulum in real time when changes occur in in the real environment that aren't expected in the virtual environment.

RE: So, what exactly is the point?
By clovell on 3/10/2008 5:15:17 PM , Rating: 3
I'm kind of seeing the same thing. The idea, from what I can gather, is to be able to accurately, and adaptively simulate reality before it happens - like some sort of real-time bayesian analysis (which probably a better term as this idea will inevitably be probability-based).

Still, there will always be lag between the stimulus and response, however minute - if there wasn't then we wouldn't have any need of a model, as we'd simply use the instantaneous (for lack of a better word) mechanism to adjust based solely on the real-world conditions - there'd be no simulation involved. The philosophical arguement seems to be headline fodder, as it simply can't happen without holding time still.

Adaptive computing simulations are useful, and I'm pretty sure they're already used in common applications like automatic transmissions among other things. Another bottleneck for this type of thing has been measurement technology its economics (you wouldn't put a breadboard on a 4x4...)

RE: So, what exactly is the point?
By RobinBee on 3/10/2008 5:18:08 PM , Rating: 2
Mixed reality can:

- Let a robot man grab an egg with a »soft hand«.
- Let a robot car drive with a »soft foot«.
- Anything in a »soft« way. Meaning clever.

It will make machines much safer. And more dangerous.

Combat robots: A young female soldier can be far away from the battlefield, with feedback sensors on her body, feeling the fight being done with her (avatar) robot. That would make a young girl a very dangerous opponent, because of her much faster reactions compared to Ordinary Man, and even helped by mixed reality technology, a »model« that anticipates what is going on. Much better than that, is currently impossible.

By SlyNine on 3/10/2008 6:36:47 PM , Rating: 2
I know we humans tend to be very symbolic in nature, but too me this is over the top. Yes it does the same things very accurately. But it is not the same thing.

RE: Symbology
By SlyNine on 3/10/2008 6:51:40 PM , Rating: 2
That said, Doesn't mean I don't understand the potential for this type of technology. Its our Symbolic nature that allows us to make leaps and bounds in science and technologies.

RE: Symbology
By DASQ on 3/10/2008 11:16:10 PM , Rating: 2
"I think the word you're looking for is ssssyyymmmbbolism."


RE: Symbology
By wackie999 on 3/11/2008 12:44:06 PM , Rating: 2
But it is not the same thing.

Please clarify.

There is no...
By Legolias24 on 3/10/2008 4:35:15 PM , Rating: 3
There is no pendulum!

RE: There is no...
By AlexWade on 3/10/2008 4:42:33 PM , Rating: 2
I know Kung-Fu.

By Twinmeister on 3/10/2008 4:09:55 PM , Rating: 2
I have been waiting for this for a few years now. This, or something like it, is the future.

By nace186 on 3/10/2008 4:50:43 PM , Rating: 2
Which one is red, and which one is the blue?

Zion Awaits
By kyleb2112 on 3/10/2008 5:26:40 PM , Rating: 2
I've read this twice. Am I bulletproof yet?

Real vs. Virtual?
By teckytech9 on 3/10/2008 11:49:39 PM , Rating: 2
Software controls hardware and the outcome is the hardwares behavior in reality.

The frequency of an event can be logged and analyzed by software to predict the occurrence/probability of the event from happening again. Simulations, virtual reality, modeling, and expert systems yield a theoretical and probable outcome of events (777 first digital plane).

IMO, creating Pandora's box to self-replicate reality into many facets of reality poses dangers not yet addressed. As long as humans can pull the plug, then control of the synchronized/autonomous systems can be returned when a probable meltdown/malfunction is evident (Pilots fighting for control from fly-by-wire).

By DeltaZero on 3/11/2008 9:46:20 AM , Rating: 2
...raises mind-blowing questions about reality...

The real pendulum is the one with all the sensors and motors attached, and has a supercomputer in the next room. This is a good scoop, but in my opinion most people here are overreacting.
To say that we may one day arrive at the situation when there is absolutely no difference between real and virtual reality, is reminding me the legend about the cartographers who made a real-time map of the Roman Empire.

By LivingDedBoy on 3/11/2008 11:10:57 AM , Rating: 2
Both pendulums are real, as in we can see them and they exist. The physical Pendulum has a motor which governs its actions of speeding up or slowing down, the virtual pendulum is governed via software.

They are both synced to each other, as in if one speeds up or slows down the other mimics to the most exact degree as possible to a point if they set them side by side you wouldn't be able to tell which was which, as far as the mathematics were concerned. Viewing and physically touching would of course tell the difference between the two. But as far as the motion and physics were concerned they'd be identical. As they both would put out the same numbers, in Gs just one would be virtual, and the other would be physical.

Put simply, its a new physics engine which can mimic real life movements to an exact degree.

By DeadPanda on 3/12/2008 3:47:11 AM , Rating: 2
Okay, so this is the first time this has been motorized, but this paper:

Paul Fergus, David Llewellyn-Jones, Madjid Merabti, and Abdennour El Rhalibi, Briding the Gap between Networked Appliances and Virtual Worlds, 21st IEEE International Conference on Advanced Information Networking and Applications (AINA): Workshop on Telecommunication Networking, Applications and Systems (TeNAS), 2007, Niagara, Canada, IEEE Computer Society, p. 935 - 940.

Described and implemented something similar (albeit without moving parts).

Been There, Done That
By mlembeck on 3/12/2008 11:38:20 AM , Rating: 2
This is not really something all that new. We did it at JPL in 1983. Made two Galileo spacecraft actuators "interact" with a 4 body real-time dynamics model of the spacecraft. What goes 'round, comes 'round...see refs below:

Lembeck, M.F. and Pignatano, N.D., "Galileo Attitude and Articulation Control Closed Loop Test System," AIAA Computers in Aerospace IV, AIAA#83-2380, Hartford, CN, October 1983.

Lembeck, M.F. and Rasmussen, R.D., "Simulator Tests Controller Performance," NASA Tech Briefs, #NPO-15744, September/October 1986.

"A lot of people pay zero for the cellphone ... That's what it's worth." -- Apple Chief Operating Officer Timothy Cook

Latest Headlines

Most Popular ArticlesAre you ready for this ? HyperDrive Aircraft
September 24, 2016, 9:29 AM
Leaked – Samsung S8 is a Dream and a Dream 2
September 25, 2016, 8:00 AM
Inspiron Laptops & 2-in-1 PCs
September 25, 2016, 9:00 AM
Snapchat’s New Sunglasses are a Spectacle – No Pun Intended
September 24, 2016, 9:02 AM
Walmart may get "Robot Shopping Carts?"
September 17, 2016, 6:01 AM

Copyright 2016 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki