text data sets

Patrick Marques Ciarelli (pciarelli@lcad.inf.ufes.br)

There are four distinct single-label text data sets in this file: WebKB-4, Reuters-8, Reuters-52 and CNAE-9.

The first three data sets were obtained from [1] and the author in [1] applied a pre-processing procedure on these
data sets such that: 
a) only letters were kept, i.e., numbers, punctuation and etc. were removed;
b) all letters were turned into lowercase;
c) the title/subject of each document was added on the top of the body of the document;
d) words with length less than 3 characters were removed;
e) stopwords also were removed [2];
f) all the words were stemmed, i.e., the words were reduced to their stem by applying Porters Stemmer [3]; 

In addition to that, we represent each document as a vector, where each dimension of the vector is one stemmed word
and its value is the frequency of the word in the document. Words which do not happen in the document have frequency
zero. Besides these procedures, we reduced even more the number of words down to 3000 by using Mutual Information [4]
for the selection of the words to be left out. Further information about these data sets can be found in [1].

CNAE-9 is a data set containing 1080 documents of free text business descriptions of Brazilian companies categorized
into a subset of 9 categories cataloged in a table called National Classification of Economic Activities
(Classificao Nacional de Atividade Econmicas - CNAE) [5]. For the CNAE-9 data set the following steps were performed:
a) initially, we kept only letters and then;
b) we specifically removed the prepositions from the texts;
c) next, the words were transformed to their canonical form, that is, the words are represented in the form that they
appear in a dictionary;
d) finally, similarly to the other databases, each document was represented as a vector, where in each vectors dimension
is set the words weight, based on its frequency in the document. 


Information about data sets:

Data sets   Classes terms Documents
WebKb-4       4     3000  4199
Reuters-8     8     3000  7674
Reuters-52    52    3000  9100
CNAE-9        9     856   1080


File format:

Each file contains a data set. The data are stored in the following format. Each line corresponds to a document
and each column to a term, except the last column that is the class assigned to the document.

Below is an example of a data set with 5 documents and 6 terms. The last column is the class assigned to each
document. The first document is assigned to class 1, the second is assigned to class 2, and so on. 

0.000000 1.000000 0.000000 2.000000 0.000000 0.000000 1  
3.000000 1.000000 1.000000 0.000000 0.000000 0.000000 2  
0.000000 1.000000 0.000000 2.000000 0.000000 0.000000 3  
0.000000 0.000000 0.000000 0.000000 3.000000 1.000000 4  
0.000000 0.000000 2.000000 1.000000 0.000000 2.000000 1  



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
References:

[1] 	A. Cardoso-Cachopo,
	Datasets for single-label text categorization.
	http://web.ist.utl.pt/~acardoso/datasets/, 2007.

[2] 	R. Baeza-Yates and B. Ribeiro-Neto,
	Modern Information Retrieval,
	first edition ed. New York: Addison-Wesley, 1998.

[3] 	M. Porter,
	The Porter Stemming Algorithm. http://tartarus.org/ martin/porterstemmer/, 2006.

[4] 	K. W. Church and P. Hanks,
	Word Association Norms, Mutual Information, and Lexicography, Computational Linguistics,
	vol. 16, no. 1, pp. 22 - 29, 1990.

[5] 	CNAE.
	Classificao Nacional de Atividades Econmicas Fiscal. IBGE - Intituto Brasileiro de Geografia e Estatstica,
	Rio de Janeiro, RJ, 1.1 edition, 2003. http://www.ibge.gov.br/concla.
