{"id":38,"date":"2010-07-12T14:54:37","date_gmt":"2010-07-12T14:54:37","guid":{"rendered":"http:\/\/simion314.wordpress.com\/?p=38"},"modified":"2010-07-12T14:54:37","modified_gmt":"2010-07-12T14:54:37","slug":"extracting-webpage-content-using-qt4","status":"publish","type":"post","link":"http:\/\/ploscariu.com\/wordpress\/2010\/07\/12\/extracting-webpage-content-using-qt4\/","title":{"rendered":"Extracting webpage content using Qt4"},"content":{"rendered":"

Sometimes we want to check a web page or even to extract it’s content but we do not need to get all the information in the page, the images, the format ,we want only the essential information, In this case I will present I needed a application that will check a forum for new posts and display only the important stuff to me(no adds or other stuff). This particular forum has no mechanism to notify a member if no posts appear in one topic. This program works without using an account.<\/p>\n

This can be done very easy in Qt using the class QWebView<\/em><\/strong>. An object of this class that in my code is declared <\/p>\n

QWebView<\/em><\/span> <\/em><\/span>*<\/em><\/span>view<\/em>;<\/em><\/span><\/p>\n

can display a web page. The important feature that i use is not displaying the page but executing JavaScript code on the page,so instead of using RegExp or other techniques to extract the information i use a JavaScript query<\/span><\/p>\n

<\/span>\/\/get<\/span> <\/span>the<\/span> <\/span>main<\/span> <\/span>frame<\/span><\/p>\n

<\/span>QWebFrame<\/span> <\/span>*<\/span>frame<\/span>=<\/span>view<\/span>-><\/span>page<\/span>()-><\/span>mainFrame<\/span>();<\/span><\/p>\n

<\/span>if<\/span>(<\/span>frame<\/span>==<\/span>NULL<\/span>)<\/span><\/p>\n

<\/span>return<\/span> <\/span>"NULL"<\/span>;<\/span><\/p>\n

<\/span><\/p>\n

<\/span>\/\/get<\/span> <\/span>the<\/span> <\/span>elements<\/span> <\/span>i<\/span> <\/span>need<\/span><\/p>\n

<\/span>QWebElementCollection<\/span> <\/span>elements<\/span>=<\/span>frame<\/span>-><\/span>findAllElements<\/span>(<\/span>"*.tr_list_f"<\/span>);<\/span><\/p>\n

My application formats this data and stores it and displays it,here is the code<\/p>\n

QString<\/span> <\/span>RL_HtmlParser::<\/span>onPageLoad(<\/span>bool<\/span> <\/span>ok)<\/span><\/p>\n

{<\/span><\/p>\n

<\/span>if<\/span>(!<\/span>ok<\/span>)<\/span><\/p>\n

<\/span>throw<\/span> <\/span>11<\/span>;<\/span>\/\/throw<\/span> <\/span>a<\/span> <\/span>exception<\/span> <\/span>,<\/span> <\/span>create<\/span> <\/span>a<\/span> <\/span>nice<\/span> <\/span>exception<\/span> <\/span>class<\/span> <\/span>later<\/span><\/p>\n

<\/span>\/\/get<\/span> <\/span>the<\/span> <\/span>main<\/span> <\/span>frame<\/span><\/p>\n

<\/span>QWebFrame<\/span> <\/span>*<\/span>frame<\/span>=<\/span>view<\/span>-><\/span>page<\/span>()-><\/span>mainFrame<\/span>();<\/span><\/p>\n

<\/span>if<\/span>(<\/span>frame<\/span>==<\/span>NULL<\/span>)<\/span><\/p>\n

<\/span>return<\/span> <\/span>"NULL"<\/span>;<\/span><\/p>\n

<\/span><\/p>\n

<\/span>\/\/get<\/span> <\/span>the<\/span> <\/span>elements<\/span> <\/span>i<\/span> <\/span>need<\/span><\/p>\n

<\/span>QWebElementCollection<\/span> <\/span>elements<\/span>=<\/span>frame<\/span>-><\/span>findAllElements<\/span>(<\/span>"*.tr_list_f"<\/span>);<\/span><\/p>\n

<\/span>int<\/span> <\/span>length<\/span>=<\/span>elements<\/span>.<\/span>count<\/span>();<\/span><\/p>\n

<\/span>if<\/span>(<\/span>0<\/span>==<\/span>length<\/span>)<\/span><\/p>\n

<\/span>return<\/span> <\/span>"NO<\/span> <\/span>MATCH<\/span> <\/span>FOUND"<\/span>;<\/span><\/p>\n

<\/span>QString<\/span> <\/span>sb<\/span>(<\/span>"<html><head><\/head><body><table<\/span> <\/span>border=’3′>"<\/span>);<\/span><\/p>\n

<\/span>for<\/span>(<\/span>int<\/span> <\/span>i<\/span>=<\/span>0<\/span>;<\/span>i<\/span><<\/span>length<\/span>;<\/span>i<\/span>++)<\/span><\/p>\n

<\/span>{<\/span><\/p>\n

<\/span>if<\/span>(<\/span>0<\/span>==<\/span>i<\/span>%<\/span>2<\/span>)<\/span> <\/span>continue<\/span>;<\/span><\/p>\n

<\/span>sb<\/span>+=<\/span>"<tr><td><\/span> <\/span>"<\/span>+<\/span>elements<\/span>.<\/span>at<\/span>(<\/span>i<\/span>).<\/span>toPlainText<\/span>()+<\/span>"<\/td><\/tr>"<\/span>;<\/span><\/p>\n

<\/span>}<\/span><\/p>\n

<\/span>sb<\/span>+=<\/span>"<\/table><\/body><\/html>"<\/span>;<\/span><\/p>\n

<\/span>\/\/old<\/span> <\/span>text<\/span> <\/span>contains<\/span> <\/span>the<\/span> <\/span>old<\/span> <\/span>data<\/span> <\/span>that<\/span> <\/span>was<\/span> <\/span>read<\/span><\/p>\n

<\/span>\/\/so<\/span> <\/span>if<\/span> <\/span>we<\/span> <\/span>have<\/span> <\/span>an<\/span> <\/span>update<\/span> <\/span>we<\/span> <\/span>store<\/span> <\/span>this<\/span> <\/span>data<\/span><\/p>\n

<\/span>if<\/span>(<\/span>sb<\/span>!=<\/span>oldText<\/span>)<\/span><\/p>\n

<\/span>{<\/span><\/p>\n

<\/span><\/p>\n

<\/span>oldText<\/span>=<\/span>sb<\/span>;<\/span><\/p>\n

<\/span>settings<\/span>-><\/span>setValue<\/span>(<\/span>"oldtext"<\/span>,<\/span>oldText<\/span>);<\/span><\/p>\n

<\/span>}<\/span><\/p>\n

<\/span>\/\/we<\/span> <\/span>display<\/span> <\/span>the<\/span> <\/span>current<\/span> <\/span>data<\/span><\/p>\n

<\/span>this<\/span>-><\/span>view<\/span>-><\/span>setHtml<\/span>(<\/span>sb<\/span>);<\/span><\/p>\n

<\/span>\/\/and<\/span> <\/span>return<\/span> <\/span>it<\/span> <\/span>as<\/span> <\/span>a<\/span> <\/span>string<\/span><\/p>\n

<\/span>return<\/span> <\/span>sb<\/span>;<\/span><\/p>\n

}<\/span><\/p>\n

This is how a web page look like<\/span><\/p>\n

<\/p>\n

and this is the application’<\/p>\n

<\/p>\n

It is possible to use proxy and authentication to login and get the page, i downloaded the page in a temporary file and then i loaded it up in the webview but is possible to load it directly from the web<\/p>\n","protected":false},"excerpt":{"rendered":"

Sometimes we want to check a web page or even to extract it’s content but we do not need to get all the information in the page, the images, the format ,we want only the essential information, In this case I will present I needed a application that will check a forum for new posts … Continue reading Extracting webpage content using Qt4<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[35],"class_list":["post-38","post","type-post","status-publish","format-standard","hentry","category-programming","tag-web-scrapping-qt"],"kl_preview_url":"http:\/\/ploscariu.com\/wordpress\/?p=38&preview=1&kc_p=8c2d8b7b3e","_links":{"self":[{"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/posts\/38","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/comments?post=38"}],"version-history":[{"count":0,"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/posts\/38\/revisions"}],"wp:attachment":[{"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/media?parent=38"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/categories?post=38"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/ploscariu.com\/wordpress\/wp-json\/wp\/v2\/tags?post=38"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}