DailyTech - New Method Tests for Software Bugs Quickly, Cheaply

Submit News

Software New Method Tests for Software Bugs Quickly, Cheaply
Levi Beckerson (Blog) - December 18, 2007 11:04 AM

E-mail

del.icio.us

37 comment(s) - last by TomZ.. on Dec 19 at 1:10 PM

NIST and University of Texas scientists a working on an open source project that can weed out program bugs much faster than current techniques.

Conventional programming wisdom has long said that most software failures result from a mere two variables colliding in an unforeseen way. Finding these possible bugs can be an arduous and expensive undertaking for large programs. Computer scientists and mathematicians at the National Institute of Standards and Technology (NIST) and the University of Texas, Arlington are developing a new, open source application that can test for these bugs quickly and inexpensively.

Though their research shows that most of these program bugs are caused by only two variables interacting, faults caused by the interaction of six variables or more are not unseen. The application under development by the NIST-University of Texas team takes this into consideration, using a technique more commonly used in the chemistry field known as combinatorial testing.

Trying all combinations of 10 variables in a given program would take 1,024 tests. The software, however, would be able to generate smarter tests, making the count for every combination of three of the variables interacting just 13 tests. Using the combinatorial approach, it can test 120 of the three-way combinations simultaneously. The tool has no problem generating efficient tests for combinations of six or more variables and does so quickly.

Given that many software programs -- even a simple web browser -- can have many more than 10 variables interacting in interesting ways in many different combinations, the approach would be a boon for bug testing for many programmers and software engineers. The researchers are looking to release the open source application next year.

The team is currently inviting developers to use the software in beta testing. The application could be useful for programming in everything from e-commerce to robotic assembly lines.

Comments

Threshold

Username
Password
remember me

This article is over a month old, voting and posting comments is disabled

Neat idea

By bhieb on 12/18/2007 11:43:57 AM , Rating: 5

Here I thought the cheapest way to test for software bugs was just to release the product and let your paying customers find them for you at no charge (cough EA, MS, ...)

RE: Neat idea

By omnicronx on 12/18/2007 12:07:55 PM , Rating: 5

Windows is probably tested more than anything you can think of. So many pieces of hardware and so many things can go wrong, its nearly impossible to fix everything. I find MAC OSX has just as many bugs, yet they get to chose their hardware, drivers, so its essentially a closed system.

Nobody else markets their product in the Billions so you have to cut them some slack. don't get me wrong MS is far from perfect, but they are also far from the 'Let the User fix our bugs' attitude you are presenting.

EA does just what you say though, BF2142 is a nightmare.

Parent

RE: Neat idea

By amanojaku on 12/18/07, Rating: -1

RE: Neat idea

By Oregonian2 on 12/18/2007 1:05:00 PM , Rating: 2

Yes, and that way Microsoft can move their OS sales number to match Linux too!!!!!

Parent

RE: Neat idea

By FITCamaro on 12/18/2007 1:08:38 PM , Rating: 5

The average person doesn't want to think about what their OS comes with and have to worry about downloading extra packages and installing them. They want to turn it on and use it.

Parent

RE: Neat idea

By amanojaku on 12/18/2007 1:43:51 PM , Rating: 1

True. That's why software installers come with the "include all" option. For those of us who know what we're doing it would be nice not to have to rely on nLite or vLite. I can see Microsoft one day refusing to support stripped Windows installs.

Don't get me wrong, I'm not advocating Linux; package dependencies are atrocious. But being forced to install all of creation when the basics don't work (Explorer and memory leaks) is ridiculous.

Parent

RE: Neat idea

By Inkjammer on 12/18/2007 2:04:24 PM , Rating: 2

It would be nice if Windows came with a minimal install. No IE, no Media Player, no other useless apps. It's good for the average consumer, but I use VCL Lan Player for my video and Firefox for my browser.

Alone half the other basic apps it has that I never use like Windows Movie Maker. I can go through and remove/customize things on my own, but it would be nice to have a "basic" install that just incorporates the bare necessities.

Parent

RE: Neat idea

By TomZ on 12/18/2007 2:10:30 PM , Rating: 2

quote:
But being forced to install all of creation when the basics don't work (Explorer and memory leaks) is ridiculous.

You're thinking of FireFox I believe - IE doesn't really have any serious memory problems.

Also, with as fast as Vista installs, and as cheap as HDD space is, I find it hard to get enthused about stripped down installs. That just means more different combinations of OS install variants to install, which will lead to more bugs. I therefore don't think that's a very good idea.

Parent

RE: Neat idea

By amanojaku on 12/18/2007 2:34:22 PM , Rating: 2

While I love the security of FireFox, the memory leaks (both from internal code and add-ons) is simply annoying. I connect to work through a VPN for 8 hours a day and witness physical RAM peaking at 200MB and virtual memory peaking at 350MB. Eh?!?

I meant Windows Explorer, not IE. Funny thing about Windows, the longer you leave it running the more memory Explorer uses. Even after you close apps and leave the system alone for an hour the memory doesn't free up. The same is true for rundll32 and svchost. If MS spent less time developing trivial applications in Windows it might get the core components to work properly.

As for Windows branches, I don't believe that's an issue. First of all, optional components won't create branches. And if the focus is on the Windows core there will be less bugs are more opportunities to make optional components that are stable, fast, and secure. In other words, MS should win a battle at a time in order to win the war.

Parent

RE: Neat idea

By FITCamaro on 12/18/2007 2:59:39 PM , Rating: 2

I leave my 1GB RAM MCE2005 video box on for weeks (only restart for updates that require it) at a time with no memory issues. Yes ME had massive memory issues but XP does not. Nor does Vista.

And its not like it just sits there serving video. I use Firefox and AIM on it nearly daily as well as rip and encode video on it.

Parent

RE: Neat idea

By shamgar03 on 12/18/2007 2:38:58 PM , Rating: 2

Firefox doesn't have memory leaks. It actually stores EVERY website you have been to in a cache. Everything that uses lots of memory doesn't have memory leaks. I believe you can also turn this off.

As for Vista installs....I literaly laughed out loud when you said "as fast as vista installs." Was that a joke...seriously, it took forever to install vista on my dual raptor, dvd dual, core opteron with 8800GTX. I don't know how long, but it was definitely twice or more as long as XP. Additionally your point about HDD space is useless. People who don't want the "extras" in Vista don't want to save HDD space they just don't want to have freaken services for said extras running in the background....all the time. Memory and CPU time are not as cheap has harddisk space. If Windows were (intelligently) modularly designed then cutting stuff would decrease bugs. When every part has a well defined interface then you tend to get far less bugs and far fewer security problems. Instead you have the most monolithic of monolithic OS's Vista where you can't go to the proverbial golf course without the proverbial kitchen sink or the proverbial 60 inch TV. In short I don't think you had anything good to say...I'm sorry.

Parent

RE: Neat idea

By TomZ on 12/18/07, Rating: 0

RE: Neat idea

By Alexstarfire on 12/18/2007 5:33:58 PM , Rating: 1

100-200 GB? That's it? My games take up more pace than that. And that doesn't include my movies, music, and anime. Heck, my anime takes up nearly 300 GB.

Vista really installs in under 10 minutes? I find that hard to believe, but if it's true then I'm really impressed. Even my cut down version of XP takes 15-20 minutes to install. I can say that I use compression on the stuff I put on it, which slows it down a bit, but I don't think it's by THAT much. I know that XP is pretty compressed on the CD. Perhaps it just needs more decompressing whereas Vista doesn't. IDK, but faster install times are always a good thing.

Parent

RE: Neat idea

By TomZ on 12/18/07, Rating: 0

RE: Neat idea

By FITCamaro on 12/18/2007 3:07:20 PM , Rating: 2

On my friends system Vista installed in about 5 minutes. Q6600, 4GB RAM, 250GB hard drive. Was 64-bit Ultimate.

When I installed it on P4 3.0GHz, 1GB RAM, 80GB HDD machines in Beta 2 it took about 10 minutes. Was more than reasonable considering how many GB it is. Was 32-bit Ultimate.

Yes I do agree that it would be nice if it was more modularized. As far as a lot of the background services you don't use, you can disable them from startup. You just open up msconfig and turn them off. That won't stop them from starting if something uses them, just won't be on at startup.

Honestly the biggest feature I'm not a fan of in Vista is SuperFetch. But people yelling at Microsoft for having it also need to yell at Apple then since they do the same thing.

Parent

RE: Neat idea

By imperator3733 on 12/19/2007 10:30:35 AM , Rating: 2

When Vista is installed on a new drive, it does install faster since it's image based. I believe it takes longer when you upgrade, but I haven't done that so I can't know for sure.

Parent

RE: Neat idea

By TomZ on 12/19/07, Rating: 0

RE: Neat idea

By rcc on 12/18/2007 1:59:22 PM , Rating: 2

Damn, you let the cat out of the bag. Now the EU will sue MS to force them to remove all those "optional" elements and sell them to us on separate discs.

Parent

RE: Neat idea

By bhieb on 12/18/2007 12:46:43 PM , Rating: 2

Yeah I am a MS fan boy, I just listed them right off the top of my head. Fact is everyone does it, but your right games are the worst. I cannot tell you how many times I have reinstalled BF2, SF, or BF2142. It just boggles my mind what they will release these days as a "complete" product, and charge you $50 to work the bugs out.

Oh well look on the bright side at least the patches are't 300MB and only available through 3rd party shares giving you a whole 10kbps download speed. Oh wait they are! :)

Parent

RE: Neat idea

By EndPCNoise on 12/18/2007 1:01:35 PM , Rating: 1

Yeah, just one more reason why the masses have turned to consoles for gaming.

Does anyone have a link to the actual November NPD software sales numbers?

The hottest PC title couldn't even sell 100,000 units (Crysis) pathetic.
Consoles sold titles in the millions.

Parent

RE: Neat idea

By bhieb on 12/18/2007 1:24:04 PM , Rating: 2

True if I could just learn to play with my thumbs I would probably head that route myself, but I just cannot do it. Spent 2 hours getting killed on Halo 2, and it just never felt right. Keyboard and mouse is great.

Parent

RE: Neat idea

By T4RTER S4UCE on 12/18/2007 3:59:15 PM , Rating: 2

And I'm the exact opposite.
I believe it's just what people use first. A few months back I got Halo for PC, mostly just to toy around with the player-made maps in Custom Edition. Frankly I'm getting the crap beaten out of me. Plus on consoles we dont have to worry as much about patches, botters, hackers, and (except in TF2's case) lag is less common.

Parent

RE: Neat idea

By T4RTER S4UCE on 12/18/2007 4:05:20 PM , Rating: 2

But then on PCs you guys do get more player-created content (Forge, save us!), and WAY more people per server in online games. Plus not to mention on PCs the hardware continually advances. Whereas consoles have to wait four-five years.

Parent

RE: Neat idea

By BladeVenom on 12/18/2007 2:23:20 PM , Rating: 2

quote:
The hottest PC title couldn't even sell 100,000 units (Crysis) pathetic.

Crysis isn't the hottest title on the PC, not even close. COD 4 is. Crysis was 7th on last weeks list. Besides NPD numbers are horribly inaccurate for PC games.

The masses are still playing PC games. Name something on any of the new consoles that has sold as well as the Sims 2 plus all of it's expansion packs or WoW.

Parent

RE: Neat idea

By EndPCNoise on 12/18/2007 3:10:49 PM , Rating: 2

WoW and Sims (and expansion packs) have overall success, but neither has been near the Top 10 for a some time.

The much hyped Crysis made #7 on a "PC only" sales chart. On the overall game sales charts NO PC games have made the Top 10.

UT3 PC version sold less than 70,000 units.

What were the actual sales of the PC version of CoD 4? I bet it's less than impressive.

BladeVenom, please provide for us a more respected sales tracking company than NPD.

Parent

RE: Neat idea

By micksh on 12/18/2007 4:49:40 PM , Rating: 2

quote:
NO PC games have made the Top 10.

That is not true.
http://en.wikipedia.org/wiki/List_of_best-selling_...

PC game sales are on par with current generation console game sales. Only old NES and gameboy games were sold in siginficantly bigger quantities.

Top selling PC games:
The Sims (16 million shipped)
The Sims 2 (13 million)
StarCraft (9.5 million)
World of Warcraft (9.3 million subscribers)
Half-Life (8 million)
Myst (6 million)

And, to compare, top selling Xbox games:
Xbox 360 - Halo 3 (5 million)
Xbox - Halo 2 (8 million)

Top PlayStation 2 game is Grand Theft Auto: Vice City - 15 million and that may include PC and Xbox versions

Parent

RE: Neat idea

By BladeVenom on 12/18/2007 5:02:08 PM , Rating: 2

If you're looking at Novembers top ten lists, the ones I saw said PC games hadn't even been tabulated yet.

Anyways over 9 million people just paid for another month of WoW, so it's still the number one video game in the world.

Parent

RE: Neat idea

By EndPCNoise on 12/18/2007 6:31:56 PM , Rating: 2

@micksh Good points, however, we're talking more recent monthly sales numbers here, not "lifetime" numbers tabulated since the days of Pong, Space Invaders, and Pac Man. So you can eliminate The Sims, StarCraft, Half-Life, and Myst from your list.

@BladeVenom Good arguments. Ok, so, 9 million will play WoW on PC's (yippie...), but so much more is played on consoles (CoD 4, Bioshock, Mass Effect, Assassin's Creed, Halo 3, Mario Galaxy). These few titles alone amount to millions, and yet there are still others I could add to this list.

By the way, BladeVenom, we're all still waiting for you to come up with a more industry respected source for game sales data than NPD Group.

I have no beef with PC gamers, nor do I own a console, and I do game on a PC from time to time. I'm simply trying to point out how the gaming masses are shifting away from PC gaming to console gaming.

Parent

RE: Neat idea

By FITCamaro on 12/18/2007 1:05:54 PM , Rating: 2

I never had any problems with Battlefield 2 when I played it...

I'm no fan of EA by any means either. As far as I'm concerned they're the worst thing to ever happen to gaming.

Parent

RE: Neat idea

By bhieb on 12/18/2007 1:22:00 PM , Rating: 2

BF2 itself was not too bad, but when you tried to install Special forces and the 1.1 patch it caused all kinds of problems. I would have to uninstall everything. Reinstall BF2 then SF then the patch, every time they released a patch.

Still a great game though, just poor execution.

Parent

RE: Neat idea

By Master Kenobi

(blog) on 12/18/2007 3:59:13 PM , Rating: 2

You mean the 1.3 patch that was supposed to fix 1.1 and the failed 1.2 LOL. Yea, BF2 was great, then they started patching it and everything went downhill from there.

Parent

RE: Neat idea

By Hoser McMoose on 12/18/2007 3:49:19 PM , Rating: 2

There's an old saying that goes: "Cheap, reliable or available now, choose two".

For consumer software we've voted with our wallets that Cheap is the #1 priority, available now is the next highest priority and reliable is a DISTANT 3rd. Software testing costs lots of time and money and while consumers SAY they want quality they generally aren't willing to wait and pay for it. End result, testing and bug fixing gets cut short to meet budget and timing constraints.

Parent

Self Testing?

By Mitch101 on 12/18/07, Rating: 0

RE: Self Testing?

By bfellow on 12/18/2007 12:01:26 PM , Rating: 3

Yes the program is called "Skynet"

Parent

RE: Self Testing?

By Etsp on 12/18/2007 12:08:44 PM , Rating: 2

Since this program most likely utilizes the source code to check for bugs, and is probably written in a language that needs to be compiled before executing... I'd say it shouldn't have any issue with scanning a copy of its source code for bugs...

Parent