The growth of smartphone usage worldwide is nothing less than remarkable, with Android being the most popular mobile Operating System globally. According to the latest statistics, Android’s market share of smartphones worldwide was 71.4% in 2023 Q1. The numbers do not stop here; Android phone users lead the charts in most countries, including the US. The rapid increase in Android phone usage is mainly attributed to the open-source nature and affordability, making it easily accessible to people of all classes.

With the ever-growing number of Android users and applications, the concern for user security and privacy grows too. The question “How secure all these applications are?” is what we will try to answer in this blog post, through a mass scan of the top 50,000 most downloaded Android applications from the Google Play Store since the beginning.

In our effort to analyze these applications, we faced many challenges. A primary motive of this publication is to enable researchers from the community to reproduce the findings or take them to the next level. Therefore, this blog not only discusses what was done and what was found, but also how we did it. Let’s dig into it!

Objectives

In order to analyze security issues and misconfigurations in the top 50,000 applications, we had two primary prerequisites –

1. List of package names for the top 50,000 most downloaded applications from Google Play Store (Eg. com.whatsapp is the package name for WhatsApp). But, there’s no publicly and openly available ranked (based on the number of downloads/installs) list of Android applications. Moreover, Google Play Store scraping is subject to limitations since Google only lists around 500-600 applications in each Play Store category (Android and Web). This ranking is also not subject to reviews, installs, etc. Instead, it considers factors like trending, recent popularity, etc.
2. Source code of the top 50,000 most downloaded applications from Google Play Store – The only way to access the source code is to download, de-compile and reverse the APK files.

Both of the above-listed problem statements were tricky.

For the first problem, we searched for free lists of ranked applications in terms of the number of installs/downloads, but the minimum number of applications we wanted to take into consideration (50,000) was way more than what was available on the Internet. An example of this was the Android Rank list with just 500 applications. The inevitable solution to this problem was to create our own list by scraping the metadata of all the applications we knew using the Google Play Store. But, before that, we would need an extensive list of all the package names we can gather that possibly exist on the Google Play Store.

For the second, we decided to extract AndroidManifest.xml for most of our analysis. We also considered reversing the Java classes and running them through semgrep rules, which can be covered in a different post.

To proceed, from a high-level perspective, we formulated the following steps:

1. Fetch as many package names as possible.
2.  Validate these packages for presence on Google Play Store.
3.  Fetch the metadata of the validated packages (primarily the maximum number of downloads/installs).
4.  Rank the applications using the metadata and create a list of the top 50,000 most downloaded Android applications.
5.  Download the APK file for the above applications.
6.  Extract AndroidManifest.xml and import it into a database.
7.  Analyze the data and deduce results.

Approach

Step 1 – Fetching Package Names

This was the step where we played around a lot. Be it Google Play store or third-party Android app stores like APKPure, APKMirror, apk-dl, Aptoid, etc., we extensively tested all platforms to query for packages, and to the best of our interest, we found that APKPure’s sitemap.xml exposes precisely what we were looking for.

There are thousands of gunzip’ped XML files with hundreds of package names inside each XML file.

Using a simple bash script, a list of 12,102,707 package names was generated. This list represents the packages that are available on APKPure and not Google Play Store. But, since we have the package names for more than 12 million Android applications, we can use these package names and query them on Google Play Store. If the application is available on Google Play Store, it will be considered a valid application; else, we’ll remove it from the list.

Note that we also fetched the files listed in sitemap.xml on Google Play Store and apk-dl but we didn’t find any valid package name in the case of Google Play Store, and a lot of popular apps were missing from the package names retrieved from apk-dl.

Step 2 – Validating Package Names

“It’s easier said than done” is what we’ve observed throughout this journey.

Theoretically, if an HTTP request to https://play.google.com/store/apps/details?id=com.whatsapp (where com.whatsapp is the package name) returns 200 as the status code in the HTTP response, that means the package exists on Google Play Store, else, it does not. Simple, right? No, that’s where Google’s rate limiting kicks in. This is when status codes like 204 and 429 were observed.

After diving deep into the matter and several trials and errors, it seemed like a basic IP address-based rate limiting. As a workaround, we used FireProx, which uses AWS API Gateways to proxy requests to any proxied URL endpoint. With each request, a new IP from the AWS pool of public IPv4 addresses is used, which was exactly what we needed to overcome Google Play Store’s rate limiting.

This is what FireProx in action looks like:

Using FireProx and a go program, we generated a list of validated package names that were available on the Google Play Store. The result was a total of 3,560,749 applications. But, we still do not know the top 50,000 most downloaded applications from this list. For the same, we were required to fetch the metadata for more than 3 million applications from Google Play Store.

Note: AWS API Gateway is a paid service with additional charges for the VPC outbound traffic. Read more about it here.

Step 3 – Getting Metadata

With this step, we required working around Google Play Store’s rate limiting again, which we already sorted out in the previous step. In our best interest, we found an open source project called google-play-scraper written in Golang that we used to fetch the metadata for our validated package list. But, the project required some modification as the official code base uses the URL https://play.google.com/store/apps/details for scraping the metadata, which is rate limited. We modified this part of the code to include the FireProx-generated URL(s) from the previous step. Our modified version of google-play-scraper can be found here.

Using the FireProx and a go program, we fetched metadata for the list of validated packages we got from the previous step. The output data looks like the following:

The output JSON file was imported into a MongoDB instance using the following command

mongoimport -h <host> -d <db_name> -c <collection-name> -u <user> -p <password> --authenticationDatabase <auth_db_name> --file <output.json>

Step 4 – Ranking

To get the top 50,000 most downloaded Google Play Store applications, we ranked the data that was populated into MongoDB using a Jupyter Notebook available here. In the given metadata, there’s a JSON key, “InstallsMax” reflecting the total maximum number of downloads of an application since its launch. We simply sorted all the applications basis on this value in descending order and exported the top 50,000 results into a CSV file, which was later converted to a TXT file with line-separated package names in the ranked order.

Step 5 – Downloading APK(s)

In simpler terms, to download an APK using the package name, one needs to request https://d.apkpure.com/b/APK/com.whatsapp?version=latest (behind Cloudflare) which redirects the client to another URL on another domain, winudf.com (no Cloudflare protection) which looks like https://d-11.winudf.com/b/APK/Y29tLmhhcHB5bW9kLmFwa18xNjFfZTBlZDI3YjE?_fn=SGFwcHlNb2RfMi45LjJfQXBrcHVyZS5hcGs&_p=Y29tLmhhcHB5bW9kLmFwaw%3D%3D&download_id=1823403701104292&is_hot=true&k=43ef6b204bee3d411b71a337b4141824649a3f0d. Our approach was to query the first link with our list of package names and get the final download link (*.winudf.com) for each application. The final links would be queued to a CLI download utility like aria2c to download the APK files.

As simple as it might sound, this was the most resource-intensive step, both in terms of storage and Internet bandwidth. But that was not actually the primary problem statement for us. It was rather the fact that APKPure (*.apkpure.com) uses Cloudflare to

1. Detect popular bot traffic
2. Rate limit the number of requests

First, we figured out that using a proper browser User-Agent like Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0 allows us to bypass the bot detection and CAPTCHA challenges.

For the second, AWS API Gateway should’ve done the trick, right? Unfortunately not. According to our test cases and understanding, Cloudflare does not necessarily treat AWS traffic well. FireProx broke because of Cloudflare’s bot detection and CAPTCHA challenge page. We tried injecting the User-Agent into the AWS API Gateway proxy, but that didn’t work either.

As a solution, we tried other cloud services with potential IP rotation, like Cloudflare Workers, AWS Lambda, and Digital Ocean Functions. Our findings were –

1. Cloudflare Workers do not rotate IP on each outbound request. Multiple Workers were created to use the same outbound IP
2. AWS Lambda rotates IP on function update, which can be invoked with each function trigger, but, we could not bypass APKPure’s Cloudflare protection with this.
3. Digital Ocean functions do not rotate IP with each outbound request, but each function creates routes for the outbound traffic using a varied IP.

That was it. We just needed a lot of Digital Ocean Python functions under the same namespace to fetch us the *.winudf.com final download link against each package name. The function code can be found here.

Using a Python script, we downloaded the APK files for the top 50,000 applications.

Step 6 – Extracting AndroidManifest.xml

For extracting out AndroidManifest.xml, we used axmldec. The resultant XML outputs were converted to JSON using a go program.

The output JSON file (manifest.json) was imported into a MongoDB instance using the following command

mongoimport -h <host> -d <db_name> -c <collection-name> -u <user> -p <password> --authenticationDatabase <auth_db_name> --file <output.json>

Step 7 – Analysis

We used this Jupyter Notebook to deduce the statistics discussed below.

Results

Google Play services top the chart for the most number of permissions required. This is justified because it is one of Android smartphones’ core components. It is followed by a couple of applications that enable an Android user to use multiple isolated instances of the same application to use/login multiple accounts. These applications perform user land isolation and management tasks requiring a vast number of permissions. The appearance of the Galaxy Watch4 Plugin app is interesting to note as well. The tight ecosystem between the smartphone and the smartwatch can be the reason why this application requires more than 200 permissions.

Finally, we queried for the keyword key in the meta-data keys and got the result above.

Conclusion