Text Classification using TF-IDF

In order to classify any text into categories, we require a set of pre-classified data. For example, in above classification problem, we need a few news articles with categories of each article. This data is usually prepared in the form of a csv file with first column as news article and second column as category. We call it as "Training Data".

Other examples of classification problems are mentioned below.

Lets take a classification problem and try to build a solution.

Classify a property address in indian city names.

As shown above, we want to predict city name of a property address. As discussed earlier, we require training data with "Property Address" as first column and "City Name" as second column. We need as much training data as possible. Ideally, we require all the city names, location names, road names, building names, popular landmarks of all the cities in India. The more training data we have, more accuracy we can achieve.

For the sake of understanding, lets take limited training data only sufficient to predict these 4 property addresses.

We can convert these addresses in individual keywords by splitting each address by "," (comma). A python script does the job.

Now lets try to predict which city an address belongs to. Lets take third address, "Sree Krishna Medicos, Shaheed Mangal Pandey Marg, Janak Park, Hari Enclave, Hari Nagar" and break it into 5 keywords (we call it as address-keywords),

1. Sree Krishna Medicos
2. Shaheed Mangal Pandey Marg
3. Janak Park
4. Hari Enclave
5. Hari Nagar

Now, we create a simple matrix with above address-keywords in rows and city names in columns. Each cell represents "Number of times a particular keyword is occurring in that city".

By adding all the occurrences in the city and taking city with maximum number of occurrences, we can fairly predict that third address belongs to Delhi.

A function can be built in Python to achieve this.

What if things are a little more complex. In any text classification problem, there is a possibility that 1 or more keywords can appear in multiple categories (in our case, cities). For example, "Joshi Lane" and "Pant Nagar" are appearing twice in whole training data. Also "MG Road" is appearing 5 times (once in each city).

Try to predict city name for first address using above method.

We have a tie between Mumbai and Delhi. While working with large volume of training data, this problem is bound to occur. A better way to deal with this is finding out "how important a keyword is to the city" and then predict city name based on this "Importance".

TF-IDF is used to calculate importance of a keyword.

First 2 columns is our training data. Now lets see how other columns are prepared.

Training-keyword count in city

Number of times a particular keyword is occurring in a city. Here, keywords like "Joshi Lane" or "Janak Park" or "Yellappa Garden" or "MG Road" each occur once in the city. Hence Keyword count in city is 1 for all keywords

Total training-keywords in city

Total Number of Keywords in a city. You will notice, that

• Mumbai, Bangalore and Chennai has 4 keywords,
• Pune has 5 keywords
• delhi has 7 keywords in our training data.

Total number of cities

It is the total count of cities in out training data. In total, we have training data for 5 cities.

Training-keyword count in all cities

Number of times a particular keyword is occurring in the whole training data. Keywords like "Janak Park" or "Shastri Nagar" are occurring once and "MG Road" is occurring 5 times in whole training data.

Term Frequency (TF)

Inverse Document Frequency (IDF)

Here, term represents training-keyword and document represents city.

TF represents weight of a keyword. In our example, we did not break keywords into individual words or else the word "Nagar" in delhi would have higher Term Frequency and hence higher weight.

IDF represents "Importance" of a keyword in the city. Note that "Vallabh Apartment" in Mumbai has more importance than "Joshi Lane" or "Pant Nagar" because these keywords are also present in Delhi. "Vallabh Apartment" is unique to Mumbai and hence commands higher importance. Similarly, keywords like "Craig Park Layout" in Bangalore or "Thiruvanmiyur" in Chennai has higher importance.

By now, you must have noticed that "MG Road" has least importance, 0 to be precise.

Total number of documents (cities) = 5

Total number of documents (cities) with "MG Road" = 5

So, IDF("MG Road") = ln(5/5) = ln(1) = 0

The keywords which are present in all cities have 0 importance and the keywords unique to a city has highest importance.

TF alone cannot be considered as weight, as words like "Nagar", "Road", "Park" are too common in a property address and can lead to less accuracy in prediction. On the other hand, IDF reduces importance of such keywords as these keywords are common across cities. Hence, TF and IDF together can be considered as weight for our classification problem.

Weight

It is derived by multiplying TF and IDF to normalise weights. We call it as "Weight" of a keyword.

Let us create similar matrix as earlier. Here each cell represents "Weight" of that keyword in that city.

This can be achieved by adding TF and IDF values to our dataframe and making a small change in prediction function.

Now, replace column "Keyword_Count_in_City" with "Weight_(TFxIDF)" in previous prediction function.