Twitter X Data: Collect It via Authorized API Access

To collect Twitter (now X) data through authorized API access, you must apply for a developer account, create a project, and use the official X API endpoints with your unique credentials (API key and Bearer token) to programmatically fetch tweets, user profiles, and trends within the rate limits of your chosen access tier. This article explains the entire process, from setting up your developer account to writing your first API call, including the costs, limits, and common pitfalls you’ll encounter.

Understanding the X API Landscape

The X API (formerly Twitter API) is the only legitimate, authorized way to programmatically access public conversation data on the platform. Scraping the website or using third-party tools that bypass the API violates X’s Terms of Service and can lead to account suspension. The API is grouped into several product tracks: the free tier (which allows a limited number of posts per month), the Basic tier (a paid low-volume tier), and the Pro and Enterprise tiers (for higher-volume and academic research needs). Each tier has different rate limits, data access levels (e.g., recent search vs. full-archive search), and pricing structures.

Prerequisites: What You Need Before You Start

Before you can make your first API call, you must have:

An X account – The account you use to apply must have a confirmed email address and phone number.
A developer account – Apply at developer.x.com. You’ll need to describe your use case (e.g., academic research, bot development, or business analytics) and agree to the developer terms.
A Project and App – Within the developer portal, create a Project (which acts as a container) and then an App inside that project. The App generates your API keys and tokens.
App permissions – Choose the appropriate permissions: read-only is sufficient for most data collection. If you need to post tweets or read direct messages, you’ll need elevated permissions, but for data collection, read-only is standard.

Step-by-Step: Getting Your API Credentials

Apply for a developer account – Go to developer.x.com, sign in with your X account, and click “Apply for a developer account.” Choose the “Standard” or “Academic” track depending on your needs. Academic access requires a verified academic affiliation and is for non-commercial research.
Complete the application form – Describe your intended use in detail. Be specific about the data you need and how you’ll use it. Vague applications are often rejected.
Create a Project and App – Once approved, navigate to the Developer Portal, click “Projects & Apps,” then “New Project.” Name your project and select the appropriate access level (Free, Basic, Pro, etc.). Then create an App within that project.
Generate your keys and tokens – In your App’s “Keys and Tokens” tab, you’ll find your API Key and API Key Secret. You’ll also need to generate an Access Token and Secret (for user-context requests) and a Bearer Token (for app-only requests). Copy these and store them securely – you won’t see the full secrets again.
Set App permissions – In the App settings, under “User authentication settings,” set the app permissions to “Read” (if you only need public data). This is crucial for OAuth 1.0a user-context requests.

Making Your First API Call

Once you have your Bearer Token, you can make requests to the X API v2 endpoints. The most common endpoints for data collection are:

Recent Search – GET /2/tweets/search/recent – Returns tweets from the last 7 days that match a query. This is available on the Free tier, but with limited monthly quota.
Full-Archive Search – GET /2/tweets/search/all – Returns tweets from the complete history of Twitter, but this requires a paid tier (Pro or higher) or Academic access.
User Lookup – GET /2/users/by/username/{username} – Returns user profile information.
User Timeline – GET /2/users/{id}/tweets – Returns the most recent tweets from a specific user.

Here’s a simple example using cURL to search for recent tweets containing the word “sod”:

curl -X GET 'https://api.twitter.com/2/tweets/search/recent?query=sod&max_results=10' -H 'Authorization: Bearer YOUR_BEARER_TOKEN'

Replace YOUR_BEARER_TOKEN with your actual token. The response will be a JSON object containing tweet data (text, author ID, created_at, etc.).

Choosing the Right API Tier and Managing Costs

Your choice of API tier directly affects how much data you can collect and what it costs. Here’s a breakdown:

Free Tier – $0 per month. Allows 500 posts per month (as of early 2025) and access to recent search with limited fields. Good for learning and small experiments.
Basic Tier – $100 per month (or $200 with additional features). Provides 10,000 posts per month, access to recent search, and more fields. Suitable for small businesses or hobbyists.
Pro Tier – $5,000 per month. Offers 1,000,000 posts per month, full-archive search, and higher rate limits. For serious data science and commercial use.
Enterprise Tier – Custom pricing. Provides even higher volumes, real-time streaming, and dedicated support. For large-scale research and enterprise needs.

Costs can escalate quickly if you exceed your monthly quota, so monitor your usage in the developer portal. For academic researchers, the Academic Research product track offers full-archive search and higher limits at no cost, but requires a rigorous application process.

Common Pitfalls and How to Avoid Them

Here are frequent issues developers face when collecting X data:

Rate limit exceeded – Each endpoint has a per-15-minute window limit (e.g., 450 requests per 15 minutes for recent search on the Basic tier). Use the X-Rate-Limit-Remaining header to track your usage and implement exponential backoff.
Query complexity – X API queries have a limited complexity score. Using too many operators (e.g., OR, AND, has:media) can exceed the limit. Simplify your queries or split them into multiple requests.
Missing fields – By default, the API returns only basic fields. To get additional data (like public metrics or context annotations), you must specify them in the tweet.fields parameter. For example: &tweet.fields=public_metrics,created_at.
Pagination – To collect more than the maximum results per request (usually 100 for recent search), you need to handle pagination using the next_token from the response meta.
Data retention – Free and Basic tiers only allow access to the last 7 days of tweets. If you need historical data, you must use a paid tier or academic access.

Alternatives and Complementary Tools

If you find the X API too restrictive or expensive, consider these alternatives:

Academic Research Access – If you’re affiliated with a university, this gives you free access to full-archive search and higher limits.
Third-party data providers – Companies like Brandwatch or Meltwater resell X data, but they often aggregate and clean it, which can be useful for analysis but comes at a cost.
Open-source libraries – Libraries like tweepy (Python) or twitter-api-v2 (Node.js) simplify API calls, but they don’t bypass rate limits or costs.
Manual export – For small amounts of data, you can use X’s built-in export feature (Settings > Your account > Download an archive of your data), but this only includes your own tweets.

Frequently Asked Questions

Can I use the X API for free?

Yes, the Free tier is available, but it’s limited to 500 posts per month and only recent search (last 7 days). It’s enough for testing and small projects, but not for serious data collection.

What is the difference between OAuth 1.0a and OAuth 2.0 Bearer Token?

OAuth 1.0a is used for user-context requests (e.g., posting tweets on behalf of a user) and requires signing each request with your API secret and access token secret. OAuth 2.0 Bearer Token is simpler – you just include the token in the Authorization header – and is used for app-only requests like searching public tweets.

How do I get more than the default 10 results per request?

You can set the max_results parameter up to 100 for recent search (and 500 for full-archive, depending on tier). To get more than that, you must paginate using the next_token from the response.

Can I collect tweets from a specific user’s timeline?

Yes, use the GET /2/users/{id}/tweets endpoint. You’ll need the user’s ID (which you can get from their username via the user lookup endpoint). Note that this endpoint only returns the user’s own tweets, not retweets or replies.

Is it legal to collect X data?

Yes, if you use the official API and comply with the Developer Agreement and Policy. This includes respecting rate limits, not storing deleted tweets, and not using the data for surveillance or other prohibited purposes. Scraping without API access is against the terms of service and can lead to legal action.

What happens if I exceed my monthly quota?

You’ll receive a 429 error (Too Many Requests) and won’t be able to make further requests until the next billing cycle. You can upgrade your tier or wait for the reset. Monitor your usage in the developer portal to avoid surprises.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →