mirrors/v
1
0
Fork 0
mirror of https://github.com/vlang/v.git synced 2025-09-13 14:32:26 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions 13

examples: fix examples/web_crawler/web_crawler.v to fetch from https://news.ycombinator.com/ instead of tuicool.com (which is offline) (fix #21854)

Browse source
This commit is contained in:
Delyan Angelov 2024-07-17 17:58:15 +03:00
parent e9652f86c5
commit be7b24d87b
No known key found for this signature in database
GPG key ID: 66886C0F12D595ED
2 changed files with 15 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

6
examples/web_crawler/README.md
View file

@ -1,7 +1,5 @@
# web_crawler
web_crawler is a very simple web crawler.
This web crawler fetches news from tuicool.com,
(a chinese site similar to hacker-news.firebaseio.com).
This simple web crawler fetches news from the homepage of HN (https://news.ycombinator.com/).
# Compile and Run
@ -19,4 +17,4 @@ v run web_crawler.v
This project shows how to use http.fetch() to get http.Response,
and then html.parse() to parse the returned html.
It's easy, isn't it?
It's easy, isn't it?

30
examples/web_crawler/web_crawler.v
View file

@ -2,24 +2,20 @@ import net.http
import net.html
fn main() {
// http.fetch() sends an HTTP request to the URL with the given method and configurations.
config := http.FetchConfig{
site_url := 'https://news.ycombinator.com'
resp := http.fetch(
url: site_url
user_agent: 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0'
}
resp := http.fetch(http.FetchConfig{ ...config, url: 'https://tuicool.com' }) or {
println('failed to fetch data from the server')
return
}
// html.parse() parses and returns the DOM from the given text.
)!
mut doc := html.parse(resp.body)
// html.DocumentObjectModel.get_tags_by_attribute_value() retrieves all tags in the document that have the given attribute name and value.
tags := doc.get_tags_by_attribute_value('class', 'list_article_item')
for tag in tags {
el := tag.children[1].children[0].children[0].children[0]
href := el.attributes['href'] or { panic('key not found') }
title := el.attributes['title'] or { panic('key not found') }
println('href: ${href}')
println('title: ${title}')
println('')
tags := doc.get_tags_by_attribute_value('class', 'titleline')
for i, tag in tags {
el := tag.children[0]
mut href := el.attributes['href']!
if !href.starts_with('http') {
href = '${site_url}/${href}'
}
title := el.content
println('${i + 1:2}. title: ${title:-90s} href: ${href}')
}
}