The tool developed by North Carolina State University researchers improves on previous techniques by targeting code written in the C programming language which is often used to create root exploit malware, whereas the bulk of Android applications are written in Java.
Root exploits take over the system administration functions of an operating system, such as Android. A successful Android root exploit effectively gives hackers unfettered control of a user's smartphone.
The new security tool is called Practical Root Exploit Containment (PREC).
It refines an existing technique called anomaly detection, which compares the behaviour of a downloaded smartphone application, such as Angry Birds, with a database of how the application should be expected to behave.
When deviations from normal behaviour are detected, PREC analyses them to determine if they are malware or harmless "false positives."
If PREC determines that an app is attempting root exploit, it effectively contains the malicious code and prevents it from being executed.
"Anomaly detection isn't new, and it has a problematic history of reporting a lot of false positives," said Dr Will Enck, co-author of the research paper.
"What sets our approach apart is that we are focusing solely on C code, which is what most if not all Android root exploits are written in," said Enck.
"Taking this approach has significantly driven down the number of false positives," said Dr Helen Gu, co-author of the paper.
"This reduces disturbances for users and makes anomaly detection more practical," said Gu.
Researchers are hoping to work with app vendors, such as Google Play, to establish a database of normal app behaviour.
Most app vendors screen their products for malware, but malware programmers have developed techniques for avoiding detection - hiding the malware until users have downloaded the app and run it on their smartphones.
The research team wants to take advantage of established vendor screening efforts to create a database of each app's normal behaviour. This could be done by having vendors incorporate PREC software into their app assessment processes.
The software would take the app behaviour data and create an external database, but would not otherwise affect the screening process, researchers said.