Tajo Query Monitoring and Node Management Interface
This interface provides real-time monitoring and management of Tajo queries and nodes. Users can access functionalities such as getting node descriptions, table lists, query progress, and logs. It offers detailed insights into running jobs, including the execution time, used nodes, input/output metrics, and job status. With this interface, users can efficiently manage and troubleshoot their Tajo data processing activities, ensuring optimal performance and resource utilization across the data processing framework.
Tajo Query Monitoring and Node Management Interface
E N D
Presentation Transcript
Tajo master Web server rpc json Web page
1. getNodeDescs() 2. getTableList() 3. getTableDesc(tableID) 4. getNodeLog(NodeID) 5. getQueryProgress() 6. getQueryList() 7. getQuery(queryID)
(Main page) Last Job ( id : 00xx00) more Running time : 00:00:00 Node num : 00 Tajo Select * from table Where id > 00; more getQuery(lastQuery) getQueryProgress() Progress : 00/100 Job call time : 00:00:00 Job time : 00sec Used nodes : 00 Input bytes : 0000MB Output bytes : 00000MB Output path : xxxx/xxx/xx getTableList() getNodeDescs() ...
Server/servlet.json?q=getquery&&op=last Server/servelt.json?q=getprogress (Main page) Last Job ( id : 00xx00) more Running time : 00:00:00 Node num : 00 Tajo Select * from table Where id > 00; more Progress : 00/100 Job call time : 00:00:00 Job time : 00sec Used nodes : 00 Input bytes : 0000MB Output bytes : 00000MB Output path : xxxx/xxx/xx Server/servlet.json?q=tablelist Server/servlet.json?q=nodedescs ...
(Attached table page) schema Tajo Running time : 00:00:00 Node num : 00 Xxxxx : INTEGER Xxxxx : INTEGER XXXX : LONG XXXX : IPv4 Table 1 File name : xxxxxxx File path : xxxx/xxxx/xxxx/xxxxxxx Store type : CSV or RAW or etc ... ... getTable(tableId) All tables getTableList() ...
(Attached table page) schema Tajo Running time : 00:00:00 Node num : 00 Xxxxx : INTEGER Xxxxx : INTEGER XXXX : LONG XXXX : IPv4 Table 1 File name : xxxxxxx File path : xxxx/xxxx/xxxx/xxxxxxx Store type : CSV or RAW or etc ... ... Server/servelt.json?q=gettable&&op==id All tables Server/servlet.json?q=tablelist ...
(Query page) Job SubQuery Unit query @a == Select * From table , table2 Where id > 000 Order by number SAVE @a USING [STORAGE FORMAT] AS ‘files://today.data’; Select xxx From xxx,xxx,xx Where xxxxx Select *** From xxxx Where …. Order by lll Select *** From xxxx Where …. Order by lll getQuery(queryId) ... All jobs getQueryList() ...
(Query page) Job SubQuery Unit query @a == Select * From table , table2 Where id > 000 Order by number SAVE @a USING [STORAGE FORMAT] AS ‘files://today.data’; Select xxx From xxx,xxx,xx Where xxxxx Select *** From xxxx Where …. Order by lll Select *** From xxxx Where …. Order by lll Server/servlet.json?q=getquery&&op==id ... All jobs Server/servelt.json?q=guerylist ...
(slave page) Log (or console out) of slave Go to main page Aaaaaaaaaaaaaaaaaaaa Bbbbbbbbbbbbbbbbbbbbbbbb Ccccccccccccccccccccccccccccc Dddddddddddddddddddddddddddd Eeeeeeeeeeeeeeee Fffffffffffffffffffffffffffffffffffffff Gggg Hhhhhhhhhhhhhhhhhhhhhhhhhh Iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii J Kkkkkkk Ll Mmmmmm Nnnnnnn Ooooooooo Pppppppp Qqqqqqqqqqqqqqqqq Rrrr getNodeLog(nodeID) ...
(slave page) Log (or console out) of slave Go to main page Aaaaaaaaaaaaaaaaaaaa Bbbbbbbbbbbbbbbbbbbbbbbb Ccccccccccccccccccccccccccccc Dddddddddddddddddddddddddddd Eeeeeeeeeeeeeeee Fffffffffffffffffffffffffffffffffffffff Gggg Hhhhhhhhhhhhhhhhhhhhhhhhhh Iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii J Kkkkkkk Ll Mmmmmm Nnnnnnn Ooooooooo Pppppppp Qqqqqqqqqqqqqqqqq Rrrr Server/servlet.json?q=nodelog&&op=id ...
GetNodeDescs() { “nodeDescs ” : [ {“nodeType”:”master”, "nodeName":"ryuCom", "ipAddr":"127.0.1.1", "totalMemSize":16845991936, "freeMemSize":7576879104, "totalDiskSize":61877604352, "freeDiskSize":45686317056, • “nodeStartTime”:111223344 } , ... ] } GetTableList() { “tableList” : [ { “tableId”:“table1” , “path” : “/xx/xx/xx”} , { “tableId”:”table2” , “path” : “/yy/yy/yy”}, … ] } getTableDesc(tableId) {"tableId":"aaa", "uri":"a/b/c", "meta":{"classname":"nta.catalog.TableMetaImpl", "property":{"schema":{"fields":[{"name":"age","dataType":"INT"}, {"name":"height","dataType":"INT"}, {"name":"city","dataType":"STRING"}], "fieldsByName":{"height":1,"age":0,"city":2}}, "storeType":"CSV","options":{}}}}
GetQueryProgress() {“queryProgress”:45} GetNodeLog() {“nodeLog” : “aaaaaaaaaaaaaa”} GetQueryList() {“queryList” : [ {“queryId” : “asd” , “query” :”select * from xxx” , “queryStartTime”:1112233 } , {“queryId” : “bcd”, “query” : “select * from yyy” , “queryStartTime”:22233444}, ... ] } getQuery(queryID) { "query":"a", "subQuery":["aaass","ddddea"], "unitQuery":["aaeafadf","efeaef","2222"], "inputPath":["/a/b"], "outputPath":["/c/d"], "queryStartTime":0, "queryEndTime":1, “usedNodeNum”:15, "inputBytes":11111, "outputBytes":111111 }
