WebExtractor

Extract content from web pages

Overview

WebExtractor retrieves and extracts content from web pages.

Class Signature

from qwen_agent.tools import WebExtractor

class WebExtractor(BaseTool):
    name = 'web_extractor'
    description = 'Get content of one webpage.'
    parameters = {
        'type': 'object',
        'properties': {
            'url': {
                'description': 'The webpage url.',
                'type': 'string',
            }
        },
        'required': ['url'],
    }

Parameters

url · str · required

Webpage URL to extract

Usage Example

Basic Usage

from qwen_agent.tools import WebExtractor

tool = WebExtractor()

content = tool.call({
    'url': 'https://example.com/article'
})

print(content)
# Returns extracted text content from the page

With Agent

from qwen_agent.agents import FnCallAgent
from qwen_agent.llm.schema import Message

agent = FnCallAgent(
    function_list=['web_extractor'],
    llm={'model': 'qwen-max'}
)

messages = [
    Message(
        role='user',
        content='Summarize the content from https://example.com/news'
    )
]

for response in agent.run(messages):
    print(response[-1].content)

See Also

• Retrieval
• BaseTool