Print 34 comment(s) - last by ElFenix.. on Sep 13 at 5:29 PM

Photosynth composing a scene from many photographs

Complex algorithms analyze photos and stitch them together with amazing accuracy

Don't know what something is? Take a photo of it and have Photosynth find information for you
Turning photographs into 3D spaces and more

Imaging technology, in both physical media such as print and in digital have come a long way since the pencil and paper. Consequently, photos and online websites dedicated to hosting and displaying photos and images have also become extremely popular and in some cases made some revolutionary steps forward. Sites such as Flickr, DeviantArt and others are all contributing to moving digital imaging as well as digital photography forward. Interestingly, Microsoft also has something that's sure to stir up the hornet's nest in online and offline visual media.

Microsoft is currently hard at work on a project it calls Photosynth. Currently, the project is under the Microsoft Live development team, and no release date has been planned. However, what is known about Photosynth is that it will attempt to revolutionize the way people view their photo collections as well as change the way people take photos. Incorporating exciting new technology, Photosynth can take a collection of photos, analyze them, and model them into a 3D virtual scene, placing each photo in their perspective-correct position -- users can see where one photo was taken in relation to another photo!

The technology is indeed a big step forward over the traditional slide show. Say for example a photography enthusiast visited Las Vegas. Over the course of a few days, the user has taken several hundred photographs of the Las Vegas strip. With Photosynth, the user can catalog all the photos and build a virtual world out of them, modeling Las Vegas as he saw it. Photosynth allows users to move around in the 3D world, viewing scenes from any angle -- regardless of how the photos were taken. Photosynth's powerful algorithms do all the background work. According to Microsoft:

Each photo is processed by computer vision algorithms to extract hundreds of distinctive features, like the corner of a window frame or a doorhandle. Then, photos that share features are linked together in a web. When a feature’s found in multiple images, its 3D position can be calculated. It’s similar to depth perception—what your brain does to perceive the 3D positions of things in your field of view based on their images in both of your eyes.

Of course, Photosynth can perform traditional photo collection management duties too. Using its analysis engine, Photosynth also allows users to search their collection for "similar" photos, perhaps of a particular person or scene. The possibilities right now sound very exciting.

In February of this year, Microsoft acquired a company called Seadragon Software. Since then, Microsoft has been working Seadragon's imaging technologies into its own projects and Photosynth incorporates a good portion of them. According to Microsoft, Photosynth will allow users to zoom in, pan and fly through any photo or 3D scene without hiccups or pauses -- no matter if a photo is 1 megapixel or 1 gigapixel. Microsoft claims "scaling is near perfect and rapid for screens of any resolution."

Another very interesting and potentially powerful feature of Photosynth is its ability to "DNA" and recognize photos featuring distinct similarities. Microsoft claims that Photosynth will allow users to take a picture of an object, have Photosynth analyze it, and automatically find information about it online. For example, if a user was on a trip and took a picture of the Petronas Towers in Malaysia, they could import it and have Photosynth correctly identify the image as the Petronas Towers and automatically do an Internet search for information on the towers.

With Photosynth you can:

  • Construct a virtual 3D scene composed of several photos
  • Walk or fly through a scene to see photos from any angle
  • Seamlessly zoom in or out of a photograph whether it’s megapixels or gigapixels in size
  • See where pictures were taken in relation to one another
  • Find similar photos to the one you’re currently viewing
  • Find information on an object in a photo
  • Explore a custom tour
  • Send a collection to a friend

Clearly, Microsoft has some very large plans for Photosynth, and it's also clear that Photosynth incorporates some very interesting and powerful technologies. From the looks of it, it also appears that Microsoft is also planning to allow users to construct virtual scenes online, for other viewers to browse "fly" through and experience.

As of right now, Microsoft says that Photosynth will be arriving soon, and we hope so because we can't wait to try it.

Comments     Threshold

This article is over a month old, voting and posting comments is disabled

Using for games/movies!-
By giantpandaman2 on 9/4/2006 1:08:07 PM , Rating: 2
Now if they could use this technology to build 3d models (even textures!) for games it could make them a whole lot cheaper to develop. Obviously in its current iteration it's not quite that powerful yet, but it wouldn't surprise me if MS has another version hidden somewhere. Well, and if they don't, they damn well should!

Hmm, turn movies into 3d so we can play roving camera in a scene? Maybe "recut" a scene? Oh, the possibilities!

RE: Using for games/movies!-
By AaronAxvig on 9/4/2006 2:47:16 PM , Rating: 3
OMG, I never really thought what they could do by taking all the frames from a movie. Or how about frames from an IMAX movie (because they tend to have camera shots that cover lots of sides of an object, and are higher quality)? They need to hurry up and release this software for free. :)

RE: Using for games/movies!-
By hoppa on 9/4/2006 11:22:25 PM , Rating: 2
3D scans of models (which is basically what you suggested would be) will never really be suitable for use in gaming, as they tend to deliver their product in many many tiny triangles, rather than intelligently position polygons. Not only would these be terrible for deforming and intensive for real-time texturing, but they would simply have too many polys for use in a game.

The other thing is, 3D artists/modellers (like myself) have become incredibly efficient in modelling stuff. Even something like Notre Dame could be modelled in a couple days at quite high detail by a good artist, and would certainly give a better product than a mish-mash of photo-composition.

RE: Using for games/movies!-
By lemonadesoda on 9/5/2006 4:22:39 AM , Rating: 2
Odd thing for you to say if you are a pro 3D modeller:
they would simply have too many polys for use in a game
1./ Technology is moving toward ever more integration of APU/GPU units
2./ GPU horsepower breaks Moore's law, hence we only need to add another 2-3 years to get 10x the power, or another 5-6 years to get 100x the power
3./ Just as in any simple vector processing package, there are smoothing, or reducing, algorithms that reduce the overall vector count. Similar algorithms can be used for game manufacture.

The OP was correct in this obvious use. A game "map" can be created very very quickly, e.g. The White House, A village, A mountain range, then reduced, then edited for additional detail by hand.

It seems that the techology works well for solid masses viewed from their exterior. Any interiors (needed for 3d gaming environments) are another challenge...

RE: Using for games/movies!-
By pixelslave on 9/5/2006 10:10:05 AM , Rating: 2
We may have a GPU capable of doing that, but if we use "scanned" 3D objects in game, it will be poor usage of the GPU's power. We didn't even perfect 2D bitmap-to-vector conversion! Just try it on a medium level complex bitmap image and send it to a printer. A manually drawn vector almost always print faster. 3D object is similar -- just way more complicated.

RE: Using for games/movies!-
By JeffDM on 9/6/2006 12:50:59 AM , Rating: 2
I can see it working, but I can also see the value of manual intervention to do the final clean-up and tuning because it is a very performance-demanding market segment.

I've toyed with the idea of making software that simplifies a vector object by removing superflous objects and joining objects that are below a certain threshold. For example, the program could find two or more small line segments in a series that don't change much in angle and replace it with one larger line segment, simplifying the object. I've done this to some success with CNC engraving, but have not made a general form program that has an acceptable user interface and such.

The same concept can be applied to 3D surfaces too.

RE: Using for games/movies!-
By ElFenix on 9/13/2006 5:29:28 PM , Rating: 3
GPU power does not break moore's law. moore's law only applies to the maximum potential, not to designs that are behind the cutting edge. when nvidia was doubling and tripling power every 6 months they were not breaking moore's law because they were coming from a position of nothing. now that they've caught up to the cutting edge you've seen GPU releases slow.

RE: Using for games/movies!-
By ZeeStorm on 9/5/2006 11:30:30 AM , Rating: 1
Well, based on what it's been described to do, the model itself could be used, but it would be hollow. You would have to take pictures of all the insides for it to draw out a model inside it as well, otherwise it's just a 3D model on the outside (which I'm not saying is a bad thing, but most games with 3D models you can go inside buildings now).

Not new, but ...
By emboss on 9/4/2006 10:50:51 AM , Rating: 5
This sort of thing has been around for a while. However, it's pretty much always wrapped up with some CAD package, with a combined cost that pushes it way out of the reach of most individuals. Or exists by itself and still costs a bomb.

I'd be very interested if MS can ship out something that a) works well b) doesn't cost a fortune and most importantly c) allows data export to something portable like XML. I can think of lots of useful things to do with such software ...

RE: Not new, but ...
By penter on 9/4/2006 11:50:54 AM , Rating: 2
I hope they can deliver what they promise, but if you know of something that does something in the likes of this, go ahead and spread the word, what software can do this?

RE: Not new, but ...
By complectus on 9/4/2006 12:12:59 PM , Rating: 3
MetaCreations Canoma could turn photos into 3D models in 1999

RE: Not new, but ...
By QueBert on 9/4/2006 3:27:57 PM , Rating: 1
it's only news with Microsoft does it, and you are correct Metacreations offering did basically what MS's will. From the blurb it sounds like it will be way more advanced, but the keyword here is "sounds" With all the features they've scrapped from Vista. I wouldn't doubt by the time this photo thingy they're working on comes out. It's nothing more then a slideshow viewer :)

RE: Not new, but ...
By Homerboy on 9/6/2006 10:22:10 AM , Rating: 2
yeap I recall seeing that at a demo show in LA. VERY impressive. Played around with a *ahem* trial version. Of course their demo made it look super easy and running on Pentium Pros helped Im sure :)

Regardless, this (sort of) thing has been around for a while. With today's hardware though Im sure we can do much more with it. VERY interested in where this goes.

RE: Not new, but ...
By Saist on 9/4/2006 8:57:32 PM , Rating: 3
Not new was going to be my comment, thanks for beating me to it. I remember when Leo Laporte was talking about this type of technology on Screensavers several years back.

I also have been under the presumption that crime labs had been using this type of software for several years as well, at least since the 80's, and I'm fairly well certain movie studios have similar packages already in use.

What I think sets Microsoft's described package apart is that it is targeted at the average Limited User, and probably won't require a few weeks of classes to use for good results. Then there is the pricing, which while we can mostly agree that Microsoft has a reputation for overcharging, they are pretty cheap compared to existing software packages when they do enter some markets.

Hardly revolutionary, but then again, when someone isn't filling a market, filling it can be seen as revolutionary.

By Calin on 9/4/2006 8:34:51 AM , Rating: 3
Once in a time, Microsoft comes with something really really interesting - I hope this will be something well done.
Never used much graphic image processing programs, but it looks like this will be very nice.

RE: Revolutionary
By Laitainion on 9/4/2006 9:39:36 AM , Rating: 2
Yeah, I for one will be *really* impressed if they can deliver.

RE: Revolutionary
By therealnickdanger on 9/5/2006 8:52:22 AM , Rating: 2
I remember reading about this in Wired (or something) a couple years ago. The article had to do with Microsoft's China labs and things they were working on there. An "unannounced photo-to-3-D tool" was one of the projects alongside "top secret Xbox technology".

The 3-D conversion thing is cool, no doubt, but it's the integration with search and geography that I find most interesting. More likely than not, this technology will find itself in two locations: and part of future versions of their photo editing suite. You know all those pictures posted on Well now you can search for other people that have been to Vinny's Pizzaria in Bakersfield, TX.

I took an extensive hiking trip to The Big Island (Hawaii) and spent considerable time trying to make a "photo journey" using Google Earth to share with other peeps. I can only imagine how much cooler it would be if Microsoft's application used its GIS powers to geographically link my photos to a map, let me walk around certain spots, bring up the history of each location, and find additional photos or video of the area...

Sure beats the hell out of traditional family slide shows (you know - with actual 35mm slides and the smelly, slightly water-damaged screen).

I was hoping it would do HDR
By JonB on 9/5/2006 9:41:44 AM , Rating: 2
Perhaps it will, but they didn't mention it. The PhotoShop plugin for High Dynamic Range photo creation is expensive (on top of the cost of PhotoShop). If PhotoSynth can do this 3-D trick, them perhaps it will be able to layer multiple shots of the same subject that have been exposure bracketed to get a single HDR image. The 24bit dynamic range of standard digital cameras is just not good enough.

By zsouthboy on 9/11/2006 7:44:05 PM , Rating: 3
Photoshop CS2 already has "Merge To HDR" built in. No expensive plugin needed. It works great and AFAIK exports to OpenEXR format, too.

And saying "24bits of dynamic range" regarding current digital cameras leads me to believe you don't actually know much about the subject. (I hope I don't sound rude, sorry)

The signal at the CCD or CMOS level is converted to a 12-bit digital representation upon processing, when a picture is taken. That 12-bits is per channel, but is also before a curve (as linear photo data is useless as an end result) and gamma ramp are applied to the image. More bits doesn't automatically equal more dynamic range.
Depending on the amount of inherent noise in the capture and the contrast curve applied, dynamic range of the image may be higher or lower. Most DSLRs today come in around 7 to 10 stops of dynamic range, with a bit more squeezed out of RAW files.

... google...
By dwelve on 9/4/2006 12:14:24 PM , Rating: 2
I bet google will have a counter to this sooner or later...

By msva124 on 9/4/2006 6:07:12 PM , Rating: 2
This looks like the gimmicky kind of thing Google usually puts out. Cool sounding idea, but less useful in practice.

Sounds intriguing, but...
By veggiedude on 9/5/2006 9:00:37 PM , Rating: 2
That sure looks like a Dock below.

: )

^__^ wonderfull
By SakuraChan on 9/4/2006 3:16:22 PM , Rating: 1
I like it photoSynth :Dhehe.... kind of catchy though,
anyway is it Digital collage-ing ? sound like it hmmm...

By fxnick on 9/4/2006 10:37:21 PM , Rating: 1
Quicktime VR has been doing this for years

crap idea
By proamerica on 9/5/06, Rating: -1
Imagine what will happen !!!
By greylica on 9/4/06, Rating: -1
RE: Imagine what will happen !!!
By Ari3s on 9/4/06, Rating: -1
RE: Imagine what will happen !!!
By tacorly on 9/4/2006 11:23:52 AM , Rating: 1
ya, after all, they haven't done enough for the computer industry

I wonder if it will incorporate more tools to play around with the photos themselves, or if you will need to photoshop the images and then import to this program

RE: Imagine what will happen !!!
By AaronAxvig on 9/4/2006 12:38:21 PM , Rating: 2
Here's a small demo of the program's capabilities from their partner Washington State University:

It doesn't have all the features they talk about, but still shows you the power of what they have.

RE: Imagine what will happen !!!
By pcjjman on 9/9/2006 2:01:28 PM , Rating: 2
That's actually the University of Washington, not Wazoo. U-Dub is based out of Seattle's University District, and has an excellent Computer Science program, while Wazoo is in Pullman and is rather unremarkable in that regard. U-Dub is about 30 minutes away from Redmond (Microsoft's home), depending on how bad traffic is across the 520 bridge.

By SmartWarthog on 9/5/06, Rating: -1
RE: Vaporware?
By mforce on 9/5/06, Rating: -1
By fritzthink on 9/5/06, Rating: -1
RE: Experence
By JeffDM on 9/5/2006 4:09:53 PM , Rating: 2
Microsoft's experience with graphics programs is an expansive list including 'paint', and uh... I think that is it.

That is where you are wrong. They have a photo manipulation program available on retail shelves right now. I haven't tried it or checked reviews but my impression is that it's competitive with Photoshop Elements in capabilities.

"So, I think the same thing of the music industry. They can't say that they're losing money, you know what I'm saying. They just probably don't have the same surplus that they had." -- Wu-Tang Clan founder RZA
Related Articles

Copyright 2016 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki