backtop


Print 26 comment(s) - last by setzer.. on Apr 2 at 4:11 AM

We benchmark and analyze the top mobile browsers for Android

Google's Android is the world's most used mobile operating system, having recently surpassed Symbian.  In sales it's currently far ahead of would-be competitors like Apple's iOS or Research in Motion's Blackberry OS.

A major reason why Android became the market leader is because it often offers the most cutting edge advances in mobile hardware and software.  That's why we received news of two high profile browser releases -- Opera Mobile 11 and Firefox 4 -- for Android with extreme interest.

Firefox was just announced on Tuesday, exclusively for Android and Maemo (a Nokia operating system that has relatively few users).  Opera Mobile 11 aired exactly one week before on Tuesday, March 22.  It is available on both Android and Symbian.

Android users now find themselves in the fortunate position of getting to pick between four elite mobile browsers -- Android's built in Webkit 3.1 browser (aka "Chrome Mobile" or the "Internet" app on Android phones) and three major browser releases on Android Market -- Opera Mini 6, Opera Mobile 11, and Firefox 4.

Which browser reigns supreme?  Read on to find out -- or skip to the conclusions to get a brief summary to help you decide.

Note:
It's been a while since our last mobile browser benchmark, so we've changed our testing regime a bit.

All tests were performed on a Sprint HTC EVO 4G running an up-to-date Android 2.2 "Froyo" kernel.


I.  Speed

Today on PCs, people often ask whether page load speeds really matter.  While it's true "slow" browsers like Internet Explorer might take a bit longer than "fast" browsers like Google Chrome, typically the difference is only a few seconds.  For many users that is acceptable.

In a mobile setting speed is far more important.  You're pulling data off a wireless connection and you have limited processing power, so making the most of your resources is critical.

To test the base speed of the mobile browsers you need to load webpages.  In order to remove the variability of the wireless network from the equation, we hopped on a local wi-fi connection.

To test our Wi-Fi connection's speed, we ran the Consumer Broadband Test, available from the U.S. Federal Communications Commission.

Our results were:



We started with a page we enjoy that's pretty lightweight in terms of load times -- the web comic XKCD.  

XKCD.com


1. Opera Mobile 11 ("Turbo" on)   : 2.5s, 1.4s, 1.4s  AVERAGE: 1.77s
2. Opera Mini 6                               : 3.2s, 1.8s, 1.7s AVERAGE: 2.23s
3. Chrome Mobile (Webkit 3.1)       : 3.3s, 1.4s, 2.2s AVERAGE: 2.30s
4. Opera Mobile 11 ("Turbo" off)    : 2.4s, 2.7s, 2.0s AVERAGE: 2.37s
5. Firefox 4                                     : 2.7s, 2.2s, 2.3s AVERAGE: 2.40s


We next loaded a page with a "moderate" amount of content -- our own homepage.  DailyTech has some Flash ads, runs some light web scripts, and features a number of images.  It represents an average scenario in terms of browser load times.

We obtained the following results:

DailyTech.com

1. Opera Mini 6                           : 3.9s, 3.8s, 4.8s  AVERAGE: 4.17s
2. Opera Mobile 11 ("Turbo" off) : 11.5s, 4.3s, 5.3s AVERAGE: 7.03s
3. Opera Mobile 11 ("Turbo" on) : 12.0s, 5.2s, 6.0s AVERAGE: 7.73s
4. Firefox 4                                  : 8.4s, 8.8s, 7.4s   AVERAGE: 8.20s
5. Chrome Mobile (Webkit 3.1)   : 12.8s, 5.8s, 5.6s AVERAGE: 8.07s

For our final page, we picked one of the internet's top sports sites, Sports Illustrated.  A property of CNN, the site features a lot of flash, script-heavy ads, and site content.  It's one of the slowest loading, most demanding sites we typically visit.  Thus we selected it for our "heavy" load time test.

SI.com

1. Opera Mini 6                          : 15.2s, 7.3s, 8.2s     AVERAGE: 10.23s
2. Chrome Mobile (Webkit 3.1)  : 14.2s, 9.5s, 9.8s     AVERAGE: 11.17s
3. Opera Mobile 11 ("Turbo" off): 18.2s, 8.8s, 8.0s *  AVERAGE: 11.67s
4. Firefox 4                                 : 19.1s, 13.8s, 16.7s AVERAGE:16.53s
5. Opera Mobile 11 ("Turbo" on): N/A *                       AVERAGE: N/A

*When we found that Opera Mobile had a nasty habit of timing out on us, when visiting this page.  With Turbo off, sometimes the page would fully load, but with Turbo on, it sometimes got stuck in some sort of weird loading loop that would last over 3 minutes, forcing us to abort.

Summary

Lightweight Opera Mini shows its strength here, breezing through page loads.  Opera Mobile 11 stumbled somewhat, failing to load some pages with "Turbo" on.  Stranger still, Turbo actually slowed down page loads (Turbo is Opera's server-side compression technology).


II. Benchmarks

To get a more in-depth perspective on how fast the browsers are, including the JavaScript performance, we ran a number of freely available benchmarks.  Browser benchmarks are still somewhat of a new frontier, and for mobile devices even more so than for PCs.

As the ultimate goal of a smart phone like Android is to offer an easy to use touch user interface atop a full PC browser, we decided to just run some of the typical PC-side browser benchmarks we've come to know and trust.  Is that fair?  We think so.


SunSpider is a freely available JavaScript benchmark tool.  While designed by the WebKit group, it works on a variety of PC-side browsers.  The benchmark features many script tests and is an in-depth performance analysis.  A lower score in milliseconds is better.

1. Firefox 4                                   : 2890.8ms +/- 0.6% *
2. Opera Mobile 11 ("Turbo" off)  : 3767.2ms +/- 2.2%
3. Opera Mobile 11 ("Turbo" on)  : 3951.1ms +/- 3.7%
4. Chrome Mobile (Webkit 3.1)    : 4669.6ms +/- 1.4%
5. Opera Mini 6                           : Failed **

* Text with timings only briefly appeared at the end of some tests -- this could have an impact on performance.

** Displayed "0 ms" for several tests, froze on test "math-spectral-norm"


Notes
As JSBenchmark is unfortunately down to server issue, we opted to use Celtic Kane's older JavaScript benchmark.  This benchmark includes several tests and offers a good general indication of Javascript performance.  As the test is relatively short, we ran it five times and averaged the results for better statistical quality.  A lower score in milliseconds is better.

1. Opera Mobile 11 ("Turbo" off): 1062.2ms (Date Object: 44ms)
2. Opera Mobile 11 ("Turbo" on): 1096.6ms (Date Object: 48.2ms)
3. Firefox 4                                 : 1433.8ms (Date Object: 40ms)
4. Chrome Mobile (Webkit 3.1)   : 1515.4ms (Date Object: 67ms)
5. Opera Mini 6                           : Failed   (Date Object: 94.8ms) *

*Several tests ran at certain points, but most displayed zero.  The only test that was able to run consistently was the Date object test, so we compiled an average for it.
 

Notes:
PeaceKeeper is "the mother of all browser benchmarks", as we put it at one point.  It test many things overlooked in other tests, like rendering performace, video playback, CSS performance, and more.  A higher score in points is better.

1. Opera Mobile 11 ("Turbo" on)   : 534 points
2. Firefox 4                                    : 496 points *
3. Opera Mobile 11 ("Turbo" off)   : 495 points
4. Chrome Mobile (Webkit 3.1)      : 474 points
5. Opera Mini 6                            : Failed **

* Rendered strange vertical white lines during the orange/brown-colored cells test.
** Froze on "Rendering # 1/6"

Summary

Opera Mini and Opera Mobile are night and day in terms of benchmark performance.  Here we see the limitations of Mini's lightweight design really coming through.  While Opera Mobile appears to be the fastest browser of the crop, Opera Mini is the slowest.

Firefox seems to be solidly in second place, and even took the lead in SunSpider, though it's victory was marred by rendering problems that could have skewed the results.

Android's built in browser looks pretty slow, but it's at least more proficient than Opera Mini at script and other standards.


III. Web Standards/Technology

Why are web standards important?  Well, if your browser supports a standard it can view pages that use that standard.  If it can't the page may render improperly, have missing elements, or refuse to load altogether.

To that end we ran two multi-purposes tests to gauge how well the browsers supported the latest round of web standards.  

Note, the HTML 5 standard has not yet been completed, but there is a degree of agreement/cross-browser compliance on portions of it, so we included it in this section.


Notes:
HTML5 is competing with Flash to deliver next-generation website video and audio.  While the standard is still being debated, this test gives a pretty good idea of how much of the pending standard a browser has supported.

1. Firefox 4                                : 235 (+9)
2. Opera Mobile 11 ("Turbo" on): 234 (+8)
3. Opera Mobile 11 ("Turbo" off): 234 (+8)
4. Chrome Mobile (Webkit 3.1)  : 192 (+0)
5. Opera Mini 6                           : 35 (+0)


Notes:
Acid 3 tests a variety standards including CSS, SVG, and rendering.

1. Opera Mobile 11 ("Turbo" on): 100/100
2. Opera Mobile 11 ("Turbo" off): 100/100
3. Firefox 4                                 : 97/100
4. Chrome Mobile (Webkit 3.1)   : 93/100
5. Opera Mini 6                           : 97/100 *

*Did not draw properly, put the the rectangles in three diferent blocks, rather than linearly.

Summary
Firefox and Opera Mobile are roughly tied for supporting the most standards.  Chrome lags a bit behind and Opera Mini is even more distantly behind.  This means that both Chrome and Opera Mini may have problems viewing some webpages.

IV. Install Size, Memory Usage, CPU Usage

Install Size

Notes:
Storage space is somewhat of a concern on smart phones, if you take a lot of pictures, store a lot of music on your phone, and/or use a lot of apps.  Your storage is dictated by the size of your microSD card for most Android phones.  For example, the EVO we tested comes with 358 MB free of internal storage, and we used an 8 GB card.  Thus large apps are a bit of a concern, if you get enough of them.

1. Opera Mini 6                         : 1.61 MB (+0.0 MB of cache)
2. Chrome Mobile (Webkit 3.1) : 8.37 MB (+0.0 MB of cache)
3. Opera Mobile 11                   : 33.09 MB (+0.0 MB of cache)
4. Firefox 4                               : 37.11 MB (+17.93 MB of cache)

Memory and CPU Usage

Notes:
Memory and CPU usage are even bigger concerns on smart phones than storage.  Thus efficiency in these categories is very important.  For this task we opened 4 tabs in each browser and then measured the performance using "Process Monitor" app by 3c06.


Results:

Conclusions
Again, here we see a strength of Opera Mini -- it's light, it's small, and it doesn't take up much memory/CPU.


V. Stability

The least stable browser we found was actually Android's built-in (Chrome) browser.  It frequently closed/crashed, particularly when multiple tabs were open.  We found this somewhat puzzling, given that Firefox 4 used more memory.  There may be some sort of bug in the tab code for the browser.

Firefox also crashed once in our testing, but this occurred when we had tabbed out to a different app.  Android sometimes closes apps to conserve memory, so it's possible this is what happened.

Opera had its own unique issues.  As mentioned Opera Mobile in Turbo mode timed out on one webpage.  It also consistently froze when you tried to turn on the option "Mobile View" (we're not sure what this is supposed to do, we're guessing it's not "kill the browser").  Similarly, Opera Mini had a regularly occurring problem where the app stopped responding to touch.  In both the Mobile View and the Mini/Touch bug, the corresponding app had to be force killed.

We'd give Firefox a slight advantage in stability.  While very hungry, it seldom crashes and when it does, it restores all your tabs (though form data may have been lost).

Opera Mini/Mobile restore tabs, but require the extra inconvenience of force closes sometimes (Settings > Applications > Manage Running Applications > ...).  By contrast, Chrome closes itself all on its own when it crashes, but it fails to restore your tabs.


VI. Privacy

A key ability is for the browser to clear your personal data.  We broke the different browsers capabilities down as follows:

Here Chrome scores a win.  Firefox lets you clear pretty much everything, but you have to clear it all at once.  While a mass clear option in Chrome would be nice, we prefer at least having individual clearing options.  And it's a bit troubling to see no ability to clear form data in Opera Mini/Mobile.  It's unclear whether any of the clear options actually ever get rid of this data.


VII. Add-Ons/Search-Engines/etc.

Only Firefox and Opera allow you to customize your search engine.  The Chrome browser doesn't even have a search plug-in, though the one is included on the Sprint homepage that comes up when you create a new tab.

Firefox is also the only browser to give you add-ons.  Granted, Opera Mobile adds some add-on like functionality in its stock build.  For example in Opera Mobile you can set Flash to only display if you click on it, similar to NoFlash/NoScript.



Overall Firefox has a clear lead on extensibility.


VIII. Sync

For mobile browsers an increasingly important feature is the ability to sync settings with your PC-side install of the browser.  Let's be frank -- even with the most optimized mobile UIs (Windows Phone 7, iOS, and Android), it still takes longer to type, browse, menu navigate, and more.

With Sync you can save a lot of time by directly picking up your settings over the internet.

Sync capabilities by browser are as follows:

Firefox, again, comes out the winner here.  It allows you to sync tabs with your home machine, which is a brilliant touch.  Opera Mobile/Mini offer decent capabilities here, but fall short of Firefox.  And Chrome earns a fail for offering no Sync capabilities.


IX. Intangibles

What differentiates a sports player with great stats but never wins a championship (*cough* Phillip Rivers) from a player with decent stats by an incredible winning streak (e.g. Ben Roethlisberger)?  Well if you ask a sports writer, they will often say "intangibles".


Source: AP/NFL

Intangibles are qualitative measures that are hard to quantify in nice tidy statistics.  And they certain apply to browsers.

We find Opera to be a bit clunkier than the built in Chrome browser for some reason.  The UI feels a bit less responsive, even though pages are loading faster.  We can't quite quantify it, but it's definitely there.

On the other hand, Firefox feels the best of any of the browsers.  A big part of this is the inclusion of the "Awesome Bar", Mozilla's beefed up address bar.  This bar packs sophisticated auto completion features.  And it's smart enough to differentiate between pages you actually visited and searches on a term, etc.  Its ranking is often very close to how you would rank the usefulness of the links.



In terms of the experience, we would say Firefox is the easiest to use, followed by Chrome, followed by Opera Mini/Mobile.

We wanted to love Opera Mobile, given its great performance, but the UI has changed little since Opera Mini.  And given the general feel of Mini's UI, that was a definite turn off.


X. Conclusions



At the end of the day we have four browsers for Android each with unique assets.

We would highly recommend installing Opera Mini 6 as an "emergency" sort of browser.  In locations where your signal is poor, its ability to load pages faster will be a life saver.  At these times all the advantages of Firefox will likely fade from your mind.

Similarly we highly recommend Firefox 4.  It's intuitive, packed with features, compatible with the latest web standards, and relatively fast.

Opera Mobile 11 is a bit tougher sell.  It's the fastest browser in synthetic benchmarks, but we found it to be relatively unstable.  The UI is a bit clunky and it lacks the options of Firefox.

By contrast the built in Chrome/WebKit browser is relatively undesirable not because of its weaknesses, but due to its lack of strengths.  It feels comfortable enough to use, but it's not overly fast, and it lacks certain features.  About the only category it squeaked its way out of the bottom of the pack in was in Privacy. 

We look forward to testing more mobile browsers in the future, more versions, and possibly more platforms.  Mobile testing is a bit more challenging, but is very rewarding.

Updated:

As some of you pointed out, Firefox 4, for all its goodies has a very bad flaw.  It lacks Flash.  We didn't really notice that in our preliminary testing (as honestly much of the web you use Flash minimally).  But when going to YouTube later in the day, we received a rude reminder that those of you who complained were right.

This shortcoming makes the overall picture much more difficult.  Opera is fast and has Flash, but is buggy.  Chrome isn't the fastest, but it has Flash and with a nice UI reskin, like Dolphin it can really shine.

Ultimately, which you pick largely depends on your use scenario, if you're a power user.  For Flash-free sites, we'd recommend Firefox 4, still.  For sites with Flash, we'd recommend the stock browser or a reskin like Dolphin.  And for low-bandwidth situations, we'd recommend Opera Mini.


Comments     Threshold


This article is over a month old, voting and posting comments is disabled

RE: Dolphin????
By InfinityzeN on 3/31/2011 1:53:47 PM , Rating: 2
That is because Dolphin is actually not a browser, but a front end. It using the same browser rendering engine as the default.


RE: Dolphin????
By omnicronx on 3/31/2011 2:57:32 PM , Rating: 1
Of course its a browser. Web Browsers are modular, they consist of many pieces (at the very least a user interface and a rendering engine). By itself webkit is not a browser but a rendering or layout engine.

The fact that Dolphin leverages the same engine as the stock browser does not change this. Its no more of a 'front end' then the stock Android, Desktop Chrome or Safari.

That said, they are both browsers that will most likely perform very similar in benchmark tests like these as a result of using the same rendering engine.


RE: Dolphin????
By DigitalFreak on 3/31/2011 6:24:55 PM , Rating: 3
quote:
That said, they are both browsers that will most likely perform very similar in benchmark tests like these as a result of using the same rendering engine.


So then what would be the point of benchmarking it?


RE: Dolphin????
By adiposity on 3/31/2011 7:23:16 PM , Rating: 2
It would be worth it if for nothing else than to prove that Dolphin doesn't suffer performance-wise vs. stock browser.


RE: Dolphin????
By drycrust3 on 4/1/2011 3:01:06 PM , Rating: 2
Isn't that like saying you want bumper stickers manufacturers to prove they don't reduce the economy of the cars they are attached to?


"I modded down, down, down, and the flames went higher." -- Sven Olsen














botimage
Copyright 2014 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki