Text Mining

jdwang@asia.edu.tw, Room:I517, ext:1847

Time: Wednesday, 9:10am~12:00am, Room: H213

• Text Book

  • Text Mining Application Programming

    • Manu Konchady,

    • Thomson (2006), ISBN: 1-58450-460-9

    • Install

  • Introduction to Information Retrieval

    • Christopher D. Manning et al. Cambridge(2008)ISBN:978-0-521-86571-5

    • http://www-csli.stanford.edu/~hinrich/information-retrieval-book.html

    • 投影片ppt

  • The Text Mining Handbook

    • Ronen Feldman & Hame Sanger, Cambridge(2007)ISBN:0-521-83657-3

• Reference Book

  • Web Data Mining
    Exploring Hyperlinks, Contents and Usage Data
    Bing Liu, Springer, December, 2006, ISBN:978-540-37881-5

    • Data mining resource site: KDnuggets Directory

    • MDR: Mining Data Records from Web Pages

    • (CBA - Asociation rules based classification)

    • http://www.cs.uic.edu/~liub/WebMiningBook.html

  　

  ---

• 成績分組(2009.3.4前繳交分組名單)(大學部2~4人/每組)
• 專題(最多遲交一週，逾期不收, 90%/per day)
  　
  • FTP: 210.70.82.???, login: TM, TM???,
  • 建立一個個別目錄, (學號﹍姓名) 上傳
• WampServer

  • Apache, MySQL, PHP on Windows

  • PSPad

    • PSPad - text and code editor

• How to call Perl from PHP
  • http://www.justskins.com/forums/how-to-call-perl-function-from-php-34646.html
  • http://devzone.zend.com/node/view/id/1712
  • http://www.real-blog.com/programming/358
• jdwangProgram
  • 程式範例
    • 如何擷取CNA文字資料2009.3.2
    • 如何文件向量化
  • Perl and MySQL
  • 104人力銀行 Program (perl)
    • 104Bankdownload_jdwang2009_4_14new.ZIP
• 薛傑仁 (PHP+MySQL)Demo

• Use Weka in your Java code

• Text Mining Software

  www.provalisresearch.com    Content Analysis & Text Mining Tool Download a free trial version now ! 
• VisualText
  www.textanalysis.com    Build it. NLP, IE, parsers, smart web crawlers, text data mining. 
• (1) 中文文件(35%)(2009.4.8)
  • 資料來源
    • 中央社新聞(CNA news)
      • CNA1991.ZIP
      • UltraEdit(for decoding the special characters)
    • 中華名國專利資訊檢索系統
      • 取得中華民國專利系統公開資料
      • example
    • 104人力銀行
      • 電子資訊專區
  • 資料處理
    • 資料下載
    • 特徵抽取(tf,df,idf)
    • 1-gram,2-gram,...
    • Zipf's Law(Power Law Distribution)
  • 參考
    • On-line Chinese Tools
    • Perl 中文程式設計
    • MadEdit-0.2.5-win32-bin.rar

  (2) 英文文件(35%)(2009.5.13)

  • Resources
    • PubMed
      • 資料下載 說明
    • 20 Newsgroups
    • Reuters Corpus
      • Reuters-21578,
      • Reuters-RCV1
    • TREC (Text Retrieval Conference)
  • Stop Word, Stemming
    • TextMining_StopWord_Stemming_example.rar
  • Feature extraction, Term Weighting
  • 專題作業
    • (70%)(公開組)根據 (20Newsgroup)(Reuters-21578)分兩組做

      (可參考網路資料20 Newsgroups Reuters-21578)

      • Source Parsing
      • Stop words removing
      • Stemming
      • Term weighting (tf*idf)
      • Vectorization (LIBSVM format)
      • Categorization (SVM, Decision Tree, Neuron Network, kNN, ...)
      • Classification Comparison (Weka, LIBSVM,..)
    • (30%)（自由選擇資料來源）

(3)生物資訊基因體序列(30%)(2009.6.10)

• Virus Genome Pattern Analysis, Virus Classification
  • DNA Sequence Classification
  • Demo by 黃進福