FeedFetcher
Fetches feeds with a stateless conditional GET over a UrlFetcher. Stable Core.
What a fetch does. It sends GET url with Accept and, when the validators you pass hold them, If-None-Match and If-Modified-Since exactly as held, through UrlFetcher.fetch with your per-fetch block. A 200 or 203 is parsed and returned as FeedFetch.Fetched with the response's validators (FeedValidators.None when it carries neither; a header that appears more than once counts as absent). A 304 to a conditional request is FeedFetch.NotModified with the sent validators, each one the 304 carries replacing its counterpart. Every other status, a 304 to an unconditional request and every other 2xx included, is UpstreamError(status, headers, body). Both outcomes carry finalUrl (the URL the response came from) and permanentlyMovedTo (the target of the leading run of 301/308 hops, or null).
What the caller owns. Storing the validators after every response; dropping them when finalUrl differs from the one stored with them (a validator from one server must not reach another); deciding whether to adopt permanentlyMovedTo (a fact about the redirect chain, not an instruction: adopt it only when the chain's first request was https, or after the same value was reported on several polls, because one injected 301 over cleartext would re-point the subscription); polling and back-off; the meaning of 410; what to show on the first poll; and which entries are new (by id, never by date).
Failures are values in the Result. Every failure of the UrlFetcher passes through unchanged (RefusedByPolicy, RateLimited, BudgetExhausted, transport failures, Closed("fetcher") of a UrlFetcher the caller closed); UpstreamError for a status that is not a feed; a FeedFailure for a document that is not; DecodeFailure for a parser that threw or returned a failure that is not an HttpFailure; Closed("feed fetcher") after close.
Limits. A feed over the transport's decoded-body cap (2 MiB under Untrusted) fails with BodyTooLarge, and one with more than 1,000 entries (podcasts) with LimitExceeded(Entries); with an Untrusted policy a feed that redirects from https to http is refused (allowHttpsToHttp is off). Parses are bounded by maxConcurrentParses permits and run on parseDispatcher; a parse cannot be cancelled, so a cancelled caller returns when its parse finishes.
Lifecycle. close is non-suspending, idempotent, never throws and closes only a UrlFetcher this fetcher built (the owning factories); every member may be called concurrently; a fetch after close returns Closed("feed fetcher"). Closing a fetcher that owns its UrlFetcher closes the transport under it, which cancels every request still on the network: each such fetch returns Closed("transport"). A fetch racing close may also return the composed fetcher's Closed("fetcher"), so match HttpFailure.Closed rather than one component name. Closing a fetcher over a UrlFetcher you own leaves requests in flight alone; a parse already running is never interrupted.
Logging, under com.spartanlabs.webtools.feeds.FeedFetcher: INFO once on creation and once on close; one DEBUG line per fetch that returns a Result (none for a fetch after close, a cancelled fetch or one that throws), with the outcome, the status, the byte and entry counts and the times; a WARN for a refused document; a DEBUG line when closing an owned UrlFetcher fails. Every URL this module prints is scheme, host and port only. The UrlFetcher it composes, and the transport under it, print a URL with Url.redacted(), which hides the query and shows the path, so a secret carried in a feed URL's path appears in their lines.
toString() names the parser's kind, maxConcurrentParses, whether the fetcher owns its UrlFetcher and whether it is closed; it never prints a URL.
Heap. About 128 MB of headroom covers the default two concurrent parses of one fetcher; the figure grows with the sum of every fetcher's maxConcurrentParses in the process.
Types
Functions
As the overload with validators, with FeedValidators.None: an unconditional request, so a 304 is an UpstreamError.
Fetches url conditionally.