Kotlin / JVM · Apache-2.0 · Maven Central

Internet I/O in single-concern artifacts.

A small Kotlin/JVM collection of internet I/O helpers, published as independent, single-concern artifacts: depend only on the one you need. Every fallible operation returns kotlin.Result rather than throwing, so callers decide how to recover.

  • CI status
  • Latest version on Maven Central
  • Kotlin 2.2
  • Apache 2.0 licence
PollFeed.kt
FeedFetcher.untrusted("my-app/1.0 (+https://example.com/contact)").use { fetcher ->
    val stored: FeedValidators = loadValidators(feedId)                      // FeedValidators.None on the first poll
    fetcher.fetch(feedUrl, stored).fold(
        onSuccess = { fetch ->
            saveValidators(feedId, fetch.validators)                          // after every response
            fetch.permanentlyMovedTo?.let { moved -> considerMoving(feedId, moved) }
            when (fetch) {
                is FeedFetch.Fetched -> store(fetch.feed.entries)             // new items: entries whose id you have not seen, never by date
                is FeedFetch.NotModified -> Unit
            }
        },
        onFailure = { failure -> report(failure) },
    )
}
5independently published artifacts
11Java bytecode floor of every JVM module
0retries: a throttle, a transport failure and a 5xx are each returned once
7test-level tasks, each runnable on its own

What you get

Each artifact adds one concern, so you can take the parts you want and leave the rest.

A typed failure model

Every fallible operation returns kotlin.Result. HttpFailure carries TransportFailure (nothing usable came back), UpstreamFailure (RateLimited, UpstreamError), BudgetExhausted, DecodeFailure and Closed.

One fixed call path

UpstreamClient checks each call against the outbound-URL policy, then the response cache, then the call budget, then the transport. A hit costs nothing, and N concurrent callers of one key cost one permit and one request.

Outbound-URL policies

Trusted for an upstream you chose, Untrusted for a URL a third party supplied: lenient host forms, reserved names, denied addresses, other ports and sensitive headers are refused.

A response cache

CaffeineResponseCache: one load per key at a time, per-entry lifetimes, a weight bound, a per-call maxAge, failures never stored. ResponseCache.Disabled stores and shares nothing.

A call budget

WindowedCallBudget: epoch-aligned rate windows with a boundary guard, per-lane floors, first-come-first-served waits that fail fast or are refunded and re-planned, pauses from Retry-After, and a hard window that survives a restart through a BudgetStore.

Hardened feeds

RSS 0.90 to 2.0, RSS 1.0 and Atom 0.3 and 1.0 behind FeedParser. RomeFeedParser refuses a DOCTYPE before any parser runs and bounds depth, attributes, elements and namespaces; FeedFetcher does a stateless conditional GET and returns a 304 as a value.

UDP with a reliable channel

MultiConnectionUDPServer multiplexes every client over one socket after a NAT-traversal handshake. UdpChannel gives each connection an UNRELIABLE or a RELIABLE_ORDERED delivery mode.

Test support

webtools-http's test fixtures (Gradle testFixtures(...)): FakeHttpTransport, VirtualClock, .http fixtures, a recorder, LoopbackFetchers and FakeBudgetStore.

Architecture

Five artifacts, one concern each. Only webtools-feeds depends on another.

How the five WebTools artifacts relate webtools-feeds builds on webtools-http. webtools-udp stands alone. webtools-scraping and webtools-browser are archived. webtools-feedsFeedFetcher · RomeFeedParser · Feed api dependency webtools-httpUpstreamClient · UrlFetcher · OkHttpTransport webtools-udpMultiConnectionUDPServer · MultiConnectionUDPClient · UdpChannel webtools-scrapingConnector · archived webtools-browserWebViewer · archived
Responsibility of each artifact
ArtifactResponsibility
webtools-httpThe suspend HTTP client stack: request and response types, a typed failure model, the transport, cache, budget and outbound-URL policy ports with their supplied implementations, and the UpstreamClient and UrlFetcher facades.
webtools-feedsRSS and Atom behind a hardened parser port, the feed model with a frozen entry id, and FeedFetcher, a stateless conditional GET over UrlFetcher. Exposes webtools-http as an api dependency.
webtools-udpA multi-client UDP connection layer: one common port, a NAT-traversal handshake, framed datagrams, keepalive, a link-quality probe and a reliable-ordered channel. Depends on slf4j-api only.
webtools-scrapingConnector: open a URL and read it line by line, plus one-shot get, skrape and image download helpers. Archived.
webtools-browserWebViewer: headless-browser screenshots through Selenium Manager. Archived.

Read how each is built: the HTTP stack and feeds, the UDP components, threads and lifecycles and the UDP wire protocol.

Modules

Depend on the one artifact you need.

ArtifactCoordinatesVersionStatusAPI
webtools-udpio.github.spartanlaboratories:webtools-udp2.0.1Stable CoreAPI
webtools-httpio.github.spartanlaboratories:webtools-http1.0.0Stable CoreAPI
webtools-feedsio.github.spartanlaboratories:webtools-feeds1.0.0Stable CoreAPI
webtools-scrapingio.github.spartanlaboratories:webtools-scraping1.0.0archivedAPI
webtools-browserio.github.spartanlaboratories:webtools-browser1.0.0archivedAPI

Everything a consumer calls, passes, configures or matches is Stable Core. Implementing a seam of webtools-http or webtools-feeds (a HttpTransport, ResponseCache, CallBudget, FeedParser, ...) is Experimental and needs @OptIn(ExperimentalForInheritanceWebToolsApi::class).

Install

Published to Maven Central. Requires JDK 11 or newer; built with Kotlin 2.2.

build.gradle.kts
dependencies {
    implementation("io.github.spartanlaboratories:webtools-udp:2.0.1")
    // and/or
    implementation("io.github.spartanlaboratories:webtools-http:1.0.0")
    implementation("io.github.spartanlaboratories:webtools-feeds:1.0.0")
    // tests only (Gradle): fakes, a virtual clock, .http fixtures and a recorder
    testImplementation(testFixtures("io.github.spartanlaboratories:webtools-http:1.0.0"))
}

Maven and the -jvm artifacts

webtools-http and webtools-feeds are Kotlin Multiplatform modules that publish webtools-http and webtools-http-jvm (likewise for feeds). A Gradle build resolves the right one itself; a Maven build depends on the -jvm artifact. testFixtures(...) is Gradle-only.

Kotlin standard library

webtools-http's okhttp-jvm 5.5.0 dependency requires kotlin-stdlib 2.2.21, so a consumer's Kotlin standard library resolves to 2.2.21 or newer.

The archived modules

webtools-scraping and webtools-browser are archived. Their install lines and the Java 11 floor notes for their older releases are in the README.

Quick start

A call to an upstream, a UDP session, and a test without a network.

Upstream.kt
val client = UpstreamClient(Url.parse("https://api.example.com/v2/").getOrThrow(), "my-app/1.0 (+https://example.com/contact)") {
    headers = HttpHeaders { addSensitive("Authorization", "Bearer $token") }      // default headers; credentials are never printed
    budget(listOf(RateWindow("minute", 60, 1.minutes) { margin = 10 }))              // built and owned by the client
}

// Declare each call once: a stable id, a decoder, how long to keep the answer, which status means "nothing there".
val player = Endpoint("player", ::decodePlayer) {
    ttl = 10.minutes
    misses(404)
}

val result: Result<Lookup<Player>> = client.call(player) {
    path("players", id)              // raw segments; each is encoded, so "a/b" stays one segment
    query("region", "eu")
    maxAge = 1.minutes               // accept a cached answer no older than this
}

result.fold(
    onSuccess = { lookup -> when (lookup) { is Lookup.Found -> show(lookup.value); Lookup.Missing -> showNotFound() } },
    onFailure = { failure -> when (failure) {
        is UpstreamFailure.RateLimited -> backOff(failure.retryAfter)        // wait this long, then call
        is HttpFailure.BudgetExhausted -> backOff(failure.wait)
        else -> report(failure)                                              // keep an else: new failure types may be added
    } },
)

client.flush()   // the exact final write of a budget with a persistent store
client.close()   // closes what the client built (cache, budget, transport), never what it was handed
UdpClient.kt
val client = MultiConnectionUDPClient(serverAddress)
client.handshake("alice").getOrThrow()
val unreliable = client.channel(DeliveryMode.UNRELIABLE)
unreliable.actuate { message -> /* handle inbound data; runs on client's dispatch thread */ }
unreliable.send("hello")
// client.start { ... } / client.send("hello") - the pre-2.0-channel spelling, deprecated but
// still fully functional; channel(UNRELIABLE) is the recommended replacement (see Migrating to 2.0)
client.startKeepAlive()   // library times it; no caller timer needed
// client.sendKeepAlive() is still available for callers who want to drive the cadence themselves

// or, for a binary protocol:
unreliable.actuateBytes { bytes -> /* exact datagram body, no decode, no trim */ }
unreliable.send(byteArrayOf(0x01, 0x02, 0x03))   // any bytes, no reserved first byte needed
// ...
client.stop()
FakeTransportTest.kt
runTest {
    val clock = virtualClock()
    val fake = FakeHttpTransport(clock.timeSource).on(RequestMatch.get("/v2/players/7"), Step.respond(200, body = """{"name":"Ada"}"""))
    UpstreamClient(fake, baseUrl) { timeSource = clock.timeSource; wallClock = clock.wallClock }.use { client ->
        assertEquals(Lookup.Found(Player("Ada")), client.call(player) { path("players", "7") }.getOrThrow())
        assertEquals(1, fake.calls.size)
    }
}

Testing & building

Seven level tasks, each independently runnable so CI can gate them separately. Building needs a JDK 17 to 26; every JVM module compiles to Java 11 bytecode against the Java 11 API.

Gradle taskLevelScope
gatingTest1 - GatingFast pre-commit checks
componentTest2 - ComponentIsolated component behaviour
integrationTest3 - IntegrationReal sockets and external interfaces
deterministicTest4a - DeterministicPure input-to-output mappings
e2eTest4b - End-to-endFull-stack flows
nonfunctionalTest4c - Non-functionalRobustness and security properties
uatTest5 - AcceptanceManual acceptance (mostly @Disabled)
Terminal
./gradlew build                          # compile + full test suite, every module
./gradlew :webtools-udp:build            # one module only
./gradlew componentTest                  # just the isolated component tests
./gradlew verifyJavaFloor                # the Java 11 floor of every published module
./gradlew :build-logic:test              # build-logic's pre-commit gate
./gradlew dokkaGeneratePublicationHtml   # generate this site's API reference