CSE534 Fundamental of Network Project

Project Demo

This week we add tf-idf function module and using NLTK to help us lemmatize the text.

The following is the function that lemmatizate the text.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def tokenize(text):
    text = ' '.join(text)
    tokens = nltk.tokenize.TreebankWordTokenizer().tokenize(text)
    #print tokens
    # lemmatize words. try both noun and verb lemmatizations
    lmtzr = WordNetLemmatizer()
    for i in range(0,len(tokens)):
        #tokens[i] = tokens[i].strip("'")
            res = lmtzr.lemmatize(tokens[i])
            if res == tokens[i]:
                tokens[i] = lmtzr.lemmatize(tokens[i], 'v')
            else:
                tokens[i] = res

    # don't return any single letters
    tokens = [t for t in tokens if len(t) > 1 and not t.isdigit()]
    return tokens

We also use the NLTK DATA help us to deal with the text analysis. Here are the English names(male and female) provided by NLTK Data.

What we have done:

• Combinated the same threads(original posts and reply posts)
• We used python to remove the punctations and some unrelated words(stopwords). The origal stopwords what we are using is here.

Problems and New tasks:

The stopwords are not enough. We should also remove the people names, places names, etc. So we summary the unrelated words to the following 9 categories.

1. Spurious data.
2. Links. Then we ignored the url, website links and email links in the posts.
3. Punctuations and Numbers.
   Read on →

Today is our team regular meeting.
In this meeting, we arrage the detail tasks for this week. In this week, we will try to complete the first two steps of Data Preprocessing.

• Collate threads.
  We have contacted with the author of our reference paper, one of Phd students in our university, he shared with us some experience about this project and give us some guide, here we want to say thanks for him. We will classify these dataset at the level of threads.

• Remove spurious data and stop-words In this part, we will firstly try to learn some basic tools. What’s we need to learn: SMART information retrieval system, Stanford CoreNLP toolkit, tf-idf. After that, we need try to remove spurious data and stop-words(e.g., articles, prepositions and pronouns).

Today We submitted our proposal. But we still had some problems about this project.

Questions about the real-time

• Outage mail is sent in real-time by someone?
• If the outage is recovered right now, so what should I do for the diagnostic in this situation?
• what’s the meaning of real-time.

This project is based on this Paper Internet Outages, the Eyewitness Accounts: Analysis of the Outages Mailing List.

Today We discussed the projects provided by the professor. We went through all the topics and talked about every details for each project. Based on our interests, we finaly picked out topic 5 and topic 8.(Refer to Projects). Then we found our professor, and talked with her to know what these two projects do. At last, we all took the topic 8(mailing list outage) as our project topic.