Test on Real Mobile Devices without Breaking the Bank
Mobile is the future of the web, so it’s time to start investing in some mobile devices. Testing on actual devices is now an absolutely essential part of web design. Stephanie Rieger explains that there are crucial aspects of a web experience you simply can’t realize by resizing your browser or using emulators: true site performance, device capabilities, form factor, pixel density and the impact of the network.
What devices should you get? How much is it going to set you back? Obviously you can’t test on every device under the sun, so here’s my recommendation:
iOS
iOS traffic accounts for more than half of mobile traffic, so if you don’t have your iOS bases covered, get on it.
iPhone (iOS5)
Cost: $199 w/ plan
This device is likely already in your pocket, but if you don’t own one you need to make the investment. Aso make sure to download and test on Opera Mini.
iPod touch (iOS4)
Cost: $65-100 (used 2nd Gen)
It’s incredibly important to test on older versions of iOS and on older hardware, as Stephanie recently stressed, especially considering the fact that the iPhone 3GS is still around and free. The iPod setup works really well for me as it’s not terribly expensive. It isn’t used as my main device so I don’t feel like I’m missing out leaving it un-updated.
Android phone(s) (2.1, 2.2, 2.3, 4.0)
Cost: $45-?
Get at least one Android device, but I would recommend getting a large smattering of Android devices. Look to get a variety of OS versions, sizes, processing power, resolutions, and manufacturers. I’ve had lots of issues with browsers on HTC devices (zoom issues, touch events, layout), and it’s definitely good to have at least one running the vanilla OS. In addition to the stock browsers, you gain access to these wonderful browsers:
Android’s growth and diversity has been tremendous, and the device capabilities range from highly capable smartphones to bargain devices. Even if you can’t afford multiple devices right now, be aware that Android isn’t just a singular device, so while your experience might test well on one Android device, you might have trouble elsewhere.
Windows Phone 7
Cost: ~$125 and up
I noticed a lot of people on Twitter post-holiday mentioning how they’ve acquired a Windows Phone to test/play with. I think the Metro UI is a solid interface and the device is a worthy investment. This year the Microsoft partnership with Nokia goes into full-swing, and could see a boost as manufacturers start experimenting with non-Android offerings.
The Mango update brought a version of Internet Explorer 9 instead of the original IE7, but the browser still don’t support javascript touch events amongst other things. It’s also worth noting that after spending some time with a real Windows Phone, web experiences that resemble iOS and Android seem oddly out of place.
WebOS device (Pre or Pixi)
Cost: $25-100
Even though WebOS’s long-term future is up in the air, the devices are dirt-cheap. I got a Palm Pixi on a daily deals site for $25 so the price is most certainly right. The smaller form factor also makes for good testing.
Blackberry <6 device (Bold or Curve)
Cost: $17-200
I think these devices are very important to own. It’s a humbling experience testing on a non-Webkit Blackberry running 4.5, 4.6, 5.0. It keeps me honest as it butchers a lot JS and CSS. There’s lots of false positives and might be my biggest reminder that “support” is in no way a binary thing. It’s also a great reminder to accommodate for non-touch interfaces in your designs.
I just picked up a Blackberry Curve for $17, so the price is definitely right. And while BB 6.0+ now sports a capable Webkit browser, the majority of Blackberry subscribers are still sub 6.0 (and yes, they do use the web on these phones).
Nokia devices
Cost: $100-?
Obviously the rest of the world is different than the US so I don’t have specific recommendations as far as Nokia devices go, but don’t forget about the mobile giant and its massive reach. I use an N-95 to test on and it serves as a great “lowest common denominator” device. It supports some Javascript, has a d-pad input method, has a screen size less than 320 pixels and is a great little device despite being 5+ years old.
Nokia isn’t all old feature phones though, their N8 and N9 devices are definitely extremely capable smartphones. There’s a massive array of Nokia devices, so I’ll defer to the Riegers and their excellent and thorough breakdown of Nokia browsers.
Kindle Fire
Cost: $199
Getting a Kindle Fire for testing is absolutely a worthwhile purchase. It’s increasingly popular and is extremely affordable. The Silk browser is similar to Android 2.3 browser and the form factor is unique as it’s a larger than a mobile phone, but smaller than an iPad.
iPad
Cost: $499
Obviously not the cheapest thing in the world, but insanely popular (and super fun to own!). This device is extremely influential and we’re already seeing 7% of holiday online sales coming from iPads, just 18 months after the original iPad launch (via lukew). Tablet computing is most certainly on the rise, and the iPad2’s browser is currently one of the most capable browsers out there.
Miscellaneous Android Tablets
Cost: $250-499
The smattering of Android tablets out there are nice to have, but I wouldn’t say required to start with. If budget allows, definitely snag Motorola Xoom and/or Galaxy Tab running Honeycomb.
Considerations
- Getting a bunch of used wi-fi enabled devices is a great way to go. Not having to deal with a ton of carrier plans is of course a good thing, and once you have the wi-fi set up you’re good to go.
- Obviously having a device or three with carrier plans help for field testing. You should always tast your experiences in the real world over real connections, including 3G, EDGE and crappy free Wi-fi spots. It can be a sobering endeavor, but this is the world we live in and we should be considerate of the on-the-go aspect of mobile context.
- Check out what devices other mobile web people are testing. Here’s what devices we have at R/GA.
- The good people at Cloudfour are in the process of putting together a very thorough device testing lab in Portland. I’m excited to see the results.
- And just because emulators aren’t the real thing, that doesn’t mean they aren’t useful. Check out Max Firtman’s very thorough rundown of mobile emulators.
- Paul Irish has a nice thread collecting some relevant links
- View some of the comments on the mobile web mailing list (which is where this post originated from).
Testing on real mobile devices is now an essential part of the web creation process. You can get a pretty representational cross-section of the mobile landcape for a couple hundred bucks. Make an investment in the future of the web. People are accessing the web through the lens of their mobile phones and we need to do our best to give them great user experiences.
29 Comments
Jim Newbery
Brad, thanks for posting this, and for responding to my query in the Mobile Web mailing list. Very helpful.
brad
Thanks for getting the ball rolling! I’ve had a lot of these conversations lately so your message was very welcome. Keep up the awesome.
Ian
Hi, good read. Though it is an expensive option for a starting designer like me. I made a mobile version for my website, and tested it with Opera Mobile emulator. Can you tell me what you think about it as an emulator? I can see why it isnt sufficient, as it does not test crappy connections and such.
Kind regards
brad
I think that the Opera Mobile emulator (and the Opera Mini simulator) are fantastic tools for mobile web designer. I encourage you still to use them, but ultimately you should invest in a few devices to truly get a feel for how your site feels on these devices. Maybe at the start of your projects you could squeeze some extra money out of your clients by promoting your “sophisticated device testing lab”. 😉
simurai
In Japan it’s quite common to rent a test lab with hundreds of devices. It’s kinda like renting studio space to record your music if you don’t have your own.
Jon Muir
Queue the founding of a device leasing company targeting all major brands and saving the average company thousands. This would also avoid the inevitability of walking around the office knee deep in devices in 2020.
James Young (@welcomebrand)
Hey Brad,
Love this article and it got me thinking about bundles for designers as Ian above mentioned, it’s still quite a lot to invest for freelancers/small agencies.
I’d love to set up something like this http://www.welcomebrand.co.uk/blog/2012/01/05/website-testing-phone-bundles/ and it would no doubt be helpful in other countries as well if you have similar second hand/reconditioned phone selling companies.
Cheers!
J.
karennjohnson
Brad, a solid list to be sure. The one element I don’t see in your list (and this seems to be true for other lists that I’ve read is) it may also depend on your user audience. This list reflects a North American slant. There are other devices – namely Nokia that dominate other markets and drop Android and even the iPhone lower on the list. I’d recommend a thorough review of any user stats so that testing adjusts based on the particular audience.
brad
Karen,
Thanks so much for the great reminder. It’s absolutely essential to look at your user audience, geographic location and project requirements when making decisions on what devices to invest in. Asking questions is a great exercise. What’s the nature of the project? Who’s our audience? What kind of phones are they using and more importantly, how are they using them? What are the technical requirements of the project?
It’s also worth noting that looking at site analytics is a great exercise, but beware as they can often misrepresent many (largely non-iOS, non-Android) devices. So definitely use analytics to your advantage, but don’t use them as an excuse to throw non-iOS, non-Android phones out the window.
joey
so basically, while even big companies stop to support IE6 because it costs too much ressources, you consider testing every crappy and old phone is an option ?
“look at your user audience”… right. Then reduce your testing set (btw you didn’t even mentionned Windows < 7).
my theory is that people owning a phone with a crappy browser give it a few tries, see that it's unusable and stop trying. you might have a 100% usable site on their phone, they won't even try to load it until they buy a new phone with a modern browser installed. they gave up already.
brad
Hi Joey,
The theory that people don’t use browsers on devices with crappy browsers is unfortunately false. The phrase “The best camera is the one that’s with you” definitely applies to the mobile web. For example, I was talking to my geologist cousin while home for the holidays. He has a Blackberry Curve that’s that’s company-issued. I asked if he uses the web on it and he said he uses it all the time. What for, I asked? Checking the weather, getting movie tickets, google searches and reading articles. He is fully aware that the phone isn’t an iPhone, but he’s fine with that as long as he’s able to get the information he needs. That’s just one example.
There are many other people, many without the disposable income to afford a high-end device, that depend on the mobile web as their lens to the internet. These people don’t just “give it a few tries, see that it’s unusable and stop trying.” It’s our responsibility to be aware of the thousands of connected devices in the world.
However, this assumption definitely brings up a very interesting chicken-and-egg question. Do mobile users with less-capable browsers use the web less because we keep giving them miserable experiences, or do we neglect these users because they’re under-represented in our stats? Whatever the cause is, these people are being denied an experience because of the device they use.
Again, I’ll make it really clear that there is a difference between support and optimization. It’s not that we have to stop the presses and “test on every crappy and old phone,” rather just be more *considerate* when creating our experiences. As the connected landscape grows more and more diverse, we need to cast aside assumptions about what devices access our experiences.
I didn’t mention Windows Phone < 7 because it doesn't hold substantial market share. http://gs.statcounter.com/#mobile_browser-ww-monthly-201012-201112
If you haven't checked out this presentation yet, you should: http://www.slideshare.net/bryanrieger/rethinking-the-mobile-web-by-yiibu
Thanks,
Brad
DevonRW
my theory is that people with a crappy phone go to a website that looks good despite the device and then continue to use the crappy phone and understand it’s a crappy phone (aside: I have a crappy phone)
Aldonio
I was going to reply directly, but I wrote pretty much that I ended up writing a blog post: http://aldo.mx/47
tl;dr version:
We are in an era of cross-browser compatibility, where standards are here to make our lives easier and browser vendors are releasing updates more often than ever in both, desktop and mobile, and specially, trying to be as close as possible to recommended & stable standards.
I believe it’s better to combine bleeding edge with old techniques, such as having a content negotiator with a modern HTML5/CSS3 version + responsive design for whitelisted browsers, and a 90’s-like version with basic HTML, no JS and the most elementary CSS such as font-family & font-size for the rest of browsers.
Brent
Thank you. Very timely. I look fwd to @grigs pdx device testing center as well.
Rachel McCollin
Interesting post Brad, if a bit daunting cost-wise – and I wonder what happens when new versions of these devices come out?
There is a bigger question here which is parallel to the debate we’ve been having about standards in the desktop world for some time and it’s over whether we should be busting a gut (and charging our clients extra) to design for multiple devices, or if we should to some extent draw a line somewhere so that the mobile browser manufacturers catch up in the same way as the desktop ones have mainly done.
The important consideration is always going to be the target audience for a project – so if it’s corporate client, for example, older blackberries will be important, but for other clients they may not.
I realise that there’s not much that can be done about the capabilities of older devices (as against browser capabilities), but I find your point interesting about Windows Phone 7. Could it end up being like IE6 where it feels like all the people using it are actually devs testing sites, not ‘real’ users?
Rachel.
brad
Hi Rachel,
It *shouldn’t* be daunting cost-wise to get started. Even if you have to pick and choose based on your needs, you can get a pretty good representation of devices without spending too much. For example, you can get a used Android, Blackberry and WebOS device for $100. I don’t think that’s not bad. Sure, as you start taking tablets and higher-end devices into account it can get pricey, but you don’t have to chase down every single top-shelf device the moment they come out.
“What happens when new versions of these devices come out?” That’s an excellent question and I think that the fact that we can’t enter into an arms race is a blessing in disguise. Creating experiences for only bleeding-edge, top-notch devices can be dangerous and risks leaving a lot of people out in the cold. Owning/testing devices that are a year or three old is a great reminder to provide support for more than just the top-shelf, state-of-the-art devices. This also reflects how people refresh their mobile devices as contracts are typically two years.
Every project timeline has a finite amount of time and resources, but it’s important to prevent drawing arbitrary lines in the sand and hope everything else will just “catch up”. There’s a difference between “supporting” a platform and bending over backwards to optimize for it. I talk about that more here: https://bradfrost.com/blog/mobile/support-vs-optimization/
Chris Bailey
I agree on needing to test on real devices. Excepting the solo developer where budget may factor in, if you run a business around mobile, then it’s a reasonable investment to make. A few other things we’ve found at HotelTonight (http://hoteltonight.com) in regards to this, as we’re testing iOS, Android, and then mobile web (covering iOS, Android, BlackBerry, and more)…
– Just buy “all” the iOS devices you support, it’s a small enough number that it’s worth it. Some will say the simulator works fine, etc., but that’s not the case. We’ve seen interesting compiler issues for older phones, and slight differences that only came up by checking on device.
– For Android, it’s impossible to buy every device, there are just way too many. Once you’re in the App Market, look at your stats, and buy at least most of the top 5 or more devices you’re showing, then buy a smattering of others.
– What’s worse on Android is that the same “exact” device can behave differently by carrier! This is due to how the carriers can modify things. We saw the same Samsung Galaxy phone use different menu implementations between Sprint and Verizon, one of which caused a bug, the other didn’t. So, when you do get bug reports, etc., make sure to find out what carrier the person is on as well, otherwise it’s possible you won’t be able to reproduce it.
– Check out Device Anywhere and Perfecto Mobile. These companies provide remote access to testing on real devices (check out some of the photos for Device Anywhere, it’s pretty interesting setup). I’ve done a trial of Device Anywhere and it works, but is slow. I’d use this more as a way to get access to a specific device we didn’t have where we weren’t able to reproduce a bug elsewhere. We’re evaluating Perfecto, don’t know on that yet. Also see this Quora thread: http://www.quora.com/What-are-some-alternatives-to-Device-Anywhere-for-mobile-device-testing
Overall, testing on multiple platforms, especially Android, is a significant time sink, but your users will of course appreciate it!
brad
Hi Chris,
Thanks for the comment! The fact is that mobile is becoming increasingly important, so essentially if you’re making websites you are by default running a “business around mobile”. Buying real devices is essential even for the solo web developers on a budget, and hopefully the post shows that even solo developers can get an Android device, Blackberry and WebOS for $100. I’d say that’s a pretty good deal.
The carrier problems and specific bugs are indeed some of the best reasons to test on real devices. That Quora link is especially helpful for finding remote mobile testing labs. Thanks for sharing your experience, it’s much appreciated!
Brett Jankord
It never hurts to run in to Best Buy, or you local phone carrier’s store and check you site on the devices they have. To me, this is the most cost effective way to test on real devices. Though if you are doing a lot of mobile sites, I can understand investing in multiple devices to test on. I often check mobile sites I’m working on my phone, co-workers phones, family members phones, and any devices I can get my hands on when I go to Best Buy.
Gray Ghost Visuals
I totally agree with Karen about the audience you are targeting. It is the first thing we must keep in mind when designing. It would certainly be absurde to just randomly go out and start buying devices just because. For small companies -like myself- a few are enough to test on and the rest are emulators. To expect a developer to buy all these handsets and plans is foolish. We can surely increase our rates to compensate for the device testing problem, but no designer/developer wants the room filled with mobile devices they have to walk over.
Modernizr has an issue thread which explains the Windows Phone returns ‘Modernizr.touch == false’ Its certainly important to test with devices for the presence of touch events as some devices like we see with the Windows phone don’t support this feature.
My Current Testing Platforms:
iPod Touch 3rd Gen
iOS5 w/Safari
T-Mobile Motorola MB611 w/Android 2.2.2
Google Browser
Firefox
Dolphin
Opera Mobile
Desktop Mac 10.7.2
Chrome, Firefox, Opera, Camino, Safari
iOS Simulator
Fennec Mobile
Android Emulator using eclipse
14″ Laptop Windows 7
IE7-9, Opera, Chrome, Safari, Firefox
Fennec Mobile
Android Emulator
Eric Sorenson
Awesome post! I’m building native apps for iOS and Android that use Bluetooth communication. The limited number of iOS devices makes testing on iOS easy, but testing on Android has been difficult and results have been inconsistent between phones.
Are there any options other than purchasing every popular Android device? Do Verizon or AT&T stores ever allow developers to install demo apps for testing?
Patrick H. Lauke
Just a tiny side note about testing on real (slow) connections rather than just on wifi: one possible solution (though it requires some initial setup, and not sure if all devices support it) would be to run a throttling proxy on your dev machine and have the devices (over wifi) connect through that…and then set it to limit bandwidth, drop the occasional packet, etc.
Bruce Bowman
Great article. I’d like to recommend Adobe Shadow as a tool that should be in your toolkit too. Shadow lets you connect devices to your computer, and keeps them in sync as you browse and reload your page. It also allows you to remotely inspect your devices, and make and see HTML and CSS changes instantly.
Check out this short video to get a better idea of what it can do for you. http://tv.adobe.com/watch/adobe-technology-sneaks-2012/adobe-shadow
Shadow is (currently) free, and available on Adobe Labs: http://adobe.com/go/shadow, the Chrome Web Store, the iTunes Store, the Google Play Market, and the Amazon Appstore for Android.
Bruce Bowman
Shadow product manager
James
I like how you say “without breaking the bank” and then you say to buy a whole lot of devices 😛
apmeyer
I assume that the carrier-locked phones on ebay can be used on WiFi without signing them up for a plan with the carrier? Any tips in that area or is it pretty easy to get these suckers connected to WiFi and bypass carrier setup processes? I have all my old iPhones layout around — and they all connect to WiFi with no issues (I’m assuming these others will as well?) Thanks!
Patrick Haney
Fantastic device breakdown, Brad. I’ve read a few blog posts from others talking about “device test labs” but none of them went into much detail. I appreciate that you took the time to talk about each platform, OS, and cost.
One thing to consider is that most of us have only a single cell carrier plan. That makes it difficult to test some phones over 3G or Edge if SIM cards aren’t compatible (or even exist with some carriers). Like you said before, it’s important to test devices on crappy connections, wifi or cellular. What I do is use my old iPhone 3GS as a wifi hotspot, allowing any wifi-enabled device in our lab to load websites over a cellular connection for a more real-world feel.
Jailbraking it was fairly simple, and the MyWi app allows tethering over a data plan without signing up for a tethering-specific plan. Of course this may be frowned upon, but it’s something we use only on occasion and I highly recommend it.
With that, I’m off to browse eBay for a used Windows Phone.
Dale Morris
Brad,
Great :Post and thread…. I have been collecting devices for sometime now just for this reason. Thanks for the concise laundry list and etc., all.
Theo
Great article! There’s nothing like testing in the “real world” however, before that i use http://lab.maltewassermann.com/viewport-resizer/ for primary testing. Would like to know what you think about this tool (as a first stage testing tool).
Yes knowing your audience is very important but i think we should support as much devices as possible as the audience and the used devices may change.
Sonia
Great post and the discussions are very informative too. I am new to mobile web testing and have been looking info difference in javascript engine that I will see in a emulator vs real device? how does jquery behave on a emulator vs real device? We are just launching a mobile version of our website which uses HTML/CSS/Javascript. If you have any insight please let me know. Thank you.
Comments are closed for this post. If you've got something to add, feel free to reach out on Twitter.