Hadoop MapReduce Programming for Big Traffic Data Management


Associate Prof. Jing-Doo Wang (王經篤)
jdwang@asia.edu.tw

TA: 王家恩(rtfgvb74125@gmail.com ) ,楊永聖(ams7758@gmail.com)

Student List
2019/7/19, 9-12: C (2019/7/19, 13-16: B?)(2019/7/23, 9-12: D?)
2019/7/23, 13-16: A
  • Introduction
  • [0_2] When and Why we adopt Hadoop(pdf)
  • [0_3]Hadoop MapReduce (pdf)
  • [1] Introduction to Hadoop(pdf)
  • [2] Windoop Download, Installation and Test(pdf)

    Download: (Hadoop 2.7.1) windoop_2.7.1_with_HBase_jre8_x64_zh_TW.7z (Thanks Windoop 林奇暻 先生提供)

    [1]:Windoop Download
    [2]:Windoop Install
    [3]:Windoop Start.bat (use "dministrator" Account)
    NameNode HDFS Web http://localhost:50070/
    Resource Manager Web http://localhost:8088
    [4]:Eclipse (use "dministrator" Account) [5]:Windoop Program Testing : PresentElection

  • The GPS locations of all eletronic gentries in Google Map
    : "03F-186.0S"(國道三號 龍井-和美)=> GantryID="03F1860S"
    高速公路計費匝道位置-Google Map 匯入教學
    The GPS locations of all eletronic gentries (國道計費門架座標及里程牌價表)
    (Download the file: "國道計費門架座標及里程牌價表1080515.CSV")
  • 交通部臺灣區國道高速公路局「交通資料庫」
    『Traffic Data Collection System,TDCS』+ Web_Robot for TDCS download(pdf)
    Web_Robot for TDCS download (TDCS_WebURLDownload_jdwang_2017_10_20.zip)
  • [3] Hadoop Java Programming Import, add Jar Library(pdf)
    [3-1]:Download
    [3-2]:Eclipse Import
    [3-3]:Eclipse Include External Jar
    (1)\windoop\hadoop\share\hadoop\common\*
    (2)\windoop\hadoop\share\hadoop\common\lib\*
    (3)\windoop\hadoop\share\hadoop\hdfs\*
    (4)\windoop\hadoop\share\hadoop\mapreduce\*
    (5)\windoop\hadoop\share\hadoop\yarn\*
    [3-4]:Eclipse Environment Set up
    HADOOP_HOME=>${eclipse_home}\..\hadoop
    PATH=> %PATH%;${eclipse_home}\..\hadoop\bin
    [3-5]: Eclipse Run
    [3-6]: Eclipse Run (input_jdwang)
  • [4]交通部臺灣區國道高速公路局「交通資料庫」
    [4-1] The format of TDCS (TDCS_M06A)(pdf)
    [4-2]&[4-3]說明
    [ Parsing Proram via MapReduce(One Hour)(TDCS_GIDSequence_MapReduceParser_jdwang2017_10_11.zip)]
    [TDCS testing data：TDCS_M06A_20161127_230000.csv(One hour)]
    [4-2] 車輛(VehicleType)統計
    [4-3] 閘道(GantryID_On)統計

  • (Optional)
    [ Parsing Proram via MapReduce (24 hours)TDCS_GIDSequence_MapReduceParser_24Hour_jdwang2017_10_13.zip)
    Example: (201701_1-1.7z)2017_1_1(24 Hours)(Asia University)
    Example: 2019_7_17 (24 hours)(Google Drive)
    [5-2] [ Parsing Proram via MapReduce(24 hours)(pdf)
    車輛(VehicleType)+24小時統計)
    閘道(GantryID_On)+24小時統計)
    特定閘道(GantryID_On)+24小時統計)
    （國道計費門架座標及里程牌價表104.09.04版.csv)
    如: "03F-186.0S"(國道三號 龍井-和美)=> GantryID="03F1860S"

  • (Optional)
    [6]專題:『交通部高速公路閘道資料』資訊擷取
    專題閘道資料處理-範例程式(24 hours)TDCS_GIDSequence_MRE_24Hour_GantryAndVihicleType_ jdwang2017_11_6.7z)
    [6-1] 高速公路計費匝道位置-Google Map 匯入教學
    [6-2] 比較(特定閘道+車輛+24小時統計)
    Example : input_TDCS_24Hour_TDCS_M06A_20171030.7z
    Experimental Result: part-r-00000_output_GantryID_24Hour_GantryOn_VehicleType_LongChing.xlsx
    [6-3] (Optional)比較(特定閘道+車輛+24小時統計)+ Hadoop Cluster
    Hadoop Cluster環境設定(Windoop 2. 0) (感謝:賴敬勳,王俊平,楊松儒 環境測試)
  • Hadoop MapReduce 課程總結與計算環境改進
  • Further Study: HBase Tutorials for Beginners (From : Guru99)
    Its goal is to provide Fun and Free Education for ALL.
    • References for Big Traffic Data Analysis :
  • A Novel Approach to Extract Significant Time Interval Patterns of Vehicles from Freeway Gantry Timestamp Sequences
    Jing-Doo Wang and Ming-Chorng Hwang
    Applied Sciences, 2017, 7(9), 878; doi:10.3390/app7090878. (SCIE, IF=1.679)
    (Special Issue "Selected Papers from IEEE ICASI 2017")
  • Analyzing the Regularities of Passengers according to Different Time Intervals via Local Electronic Bus System Data in Taiwan
    Jing-Doo Wang, Yao-Nan Lien and Shin Hung Pan* (*Corresponding author)
    The Institute for Operations Research and the Management Sciences(INFROMs 2018) June 17-20, 2018, Taipei, Taiwan
    Demo: Taichung Bus Data (2015&2016)(guest,guest)
    • [0_0](ewant) 亞洲大學 大數據資料處理 –Hadoop MapReduce 程式設計與資料視覺化 教師: 王經篤,何承遠)