webinfo -- Extract metadata from web pages

webinfo extracts common metadata (title, description, canonical, image, etc.) from web pages and provides helpers to download images and generate thumbnails.

Design goals

• Keep metadata extraction simple and deterministic.
• Use clear precedence rules for HTML/meta parsing.
• Provide practical image utilities with minimal API surface.
• Keep context-aware network operations as the default style.

Development

Requirements

• Go 1.25.10 or later
• Task command (local tool for this repository)

Local validation

task test
task govulncheck

Run all maintenance tasks:

task

CI Workflows

• ci: lint (golangci-lint with gosec), tests, and govulncheck
• CodeQL: scheduled and push/PR static analysis

Usage

Install and import

go get github.com/goark/webinfo@latest

import "github.com/goark/webinfo"

Fetch metadata

ctx := context.Background()
info, err := webinfo.Fetch(ctx, "https://example.com", "")
if err != nil {
  return err
}
fmt.Println(info.Title, info.Description)

Download image and thumbnail

flagHTTP11 := false

imgPath, err := info.DownloadImage(ctx, flagHTTP11, "images", true)
if err != nil {
  return err
}

thumbPath, err := info.DownloadThumbnail(ctx, flagHTTP11, "thumbnails", 150, false)
if err != nil {
  return err
}

imgBytes, err := info.ImageBytes(ctx, flagHTTP11)
if err != nil {
  return err
}
fmt.Println(len(imgBytes))

Public API

• DefaultUserAgent() returns the package default User-Agent string.
• Fetch(ctx, rawURL, userAgent) extracts metadata from a page. When userAgent is empty, the default value is used.
• (*Webinfo).ImageBytes(ctx, flagHTTP11) downloads Webinfo.ImageURL into memory. The flagHTTP11 parameter controls whether to use HTTP/1.1 only.
• (*Webinfo).DownloadImage(ctx, flagHTTP11, destDir, temporary) downloads Webinfo.ImageURL. The flagHTTP11 parameter controls whether to use HTTP/1.1 only. The destDir parameter specifies the destination directory, and temporary flag indicates if the file should be created as a temporary file.
• (*Webinfo).DownloadThumbnail(ctx, flagHTTP11, destDir, width, temporary) creates a resized thumbnail. The flagHTTP11 parameter controls whether to use HTTP/1.1 only. The destDir parameter specifies the destination directory, width sets the thumbnail width, and temporary flag indicates if the file should be created as a temporary file.

Behavior notes

• Fetch uses explicit precedence for metadata extraction:
  • title: title -> twitter:title -> og:title
  • description: meta[name=description] -> twitter:description -> og:description
  • image: twitter:image -> og:image
• DownloadImage resolves extension in this order:
  1. URL path extension
  2. response Content-Type
  3. sniff first 512 bytes (http.DetectContentType)
  4. fallback .img
• DownloadThumbnail uses width 150 when width <= 0.
• ImageBytes reads the full response body into memory; very large images can increase memory usage.

Error handling

This package wraps errors with github.com/goark/errs and attaches context values such as url, path, and dir.

Modules Requirement Graph