Text Mining
(Project-Based or Problem-Based Learning)(PBL)

• Occidental Languages vs. Oriental Languages
  What are the differences between Occidental Languages and Oriental Languages?
  
• Natural Language Processing (NLP) vs. Deep Neural Network?
  
• Text Mining Applications:

  Information Retrevial:
  keyword extraction: ,
  

  ---

  Text Categorization(classification): Spam mails,
  Text Similarity (Clustering),
  Anomally and Trend Detection,
  Social Network, ...
  Text Mining with cybercrime,
  Text Streams : Events and trends in text streams,
  

• Introduction to Python for Data Science (巨匠) Python install (64bit & 3.X version)
  python package installation : pip
  
• Data Science Research Methods: Python Edition (巨匠)
• Essential Math for Machine Learning: Python Edition(巨匠)
• Principles of Machine Learning: Python Edition(巨匠)
• Natural Language Processing (NLP) 巨匠
  Natural Language Processing with Python
  (Code Examples provided by Lewis Yang (Thanks) )
  
  Question:
  

  ---

  Document vs. Document

  ---

  Bag-Of-Word (BOW) vs TF*IDF (Term Frequency * Inverse-Document_Frequency)

  ---

  (1) How to measure the similarity of DocA and DocB ?
  (2) How to rank the similarities of one DocA among a group of Docs?
  (3) How to evaluate the similarities of one Doc among several classes of Docs?
  (4) How to determine which class should one Doc belong to? (single class)
  (5) How to determine which classes can one Doc be assigned to? (Multi-class)
  (6) How to evaluate the performance or efficiency of one text classification ?
  

  ---

  ---

  How to represent "Categories" in the vector for records?
  Label Encoding vs One-Hot Encoding

  ---

  初學Python手記#3-資料前處理( Label encoding、 One hot encoding) (From Pat Huang)
  數據處理——One-Hot Encoding (From https://www.itread01.com/)
  

  ---

  ---

  Word2Vec : Find the similarityies between or among words
  [NLP] 秒懂?向量Word2vec的本? From:穆文
  

  ---

  ---

  python>pip install gensim==3.7.1
  Word2Vec :
  (Word2Vec with Gensim - Python(From YouTube :The Semicolon ))
  (產品標籤分群實作-Word2Vec(From 王選仲))
  ([NLP] 秒懂?向量Word2vec的本?(From 穆文))
  (Word2Vec 的簡介與實作(From 周煜軒的個人網站))
  

  ---

  existing word vector : GloVe: Global Vectors for Word Representation

  ---

  ---

  Word Embedding with keras

  ---

  ---

• Ethics and Law in Data and Analytic(巨匠)

• (Inside Testing)
  What is the best performance you achieve with the "task2_trainset.csv" ?
  Accuracy?
  F1 measures for each of four classes ? total F1 measure?
  
• Make a report about what you have done about this project.
  Data preprocessing?
  Classifier?
  Experimental Results?
  
• (Outside Testing)
  Upload your own prediction (task2_sample_submission.csv) to AI CUP leaderboard, get your own score (F1) before 12/29.

chi2

• sklearn.feature_selection.chi2
• Chi-Square Test for Feature Selection in Machine learning Feature selection always plays a key role in machine learning(From: sampath kumar gajawada)
Infornation Gain (IG) and Mutual Information (MI)

• Information Gain and Mutual Information for Machine Learning (by Jason Brownlee on October 16, 2019 in Probability)
• 1.13.5. Feature selection as part of a pipeline
• Classification of text documents using sparse features