Skip to content

feat(java): opt-in lenient reading of off-spec JSON value shapes - #4235

Open
pavel-ptashyts wants to merge 15 commits into
apache:mainfrom
pavel-ptashyts:feat/json-lenient-read
Open

pavel-ptashyts wants to merge 15 commits into
apache:mainfrom
pavel-ptashyts:feat/json-lenient-read

Conversation

@pavel-ptashyts

@pavel-ptashyts pavel-ptashyts commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why?

Moving a JSON read path from Jackson to Fory JSON stops at input that is slightly off-spec but that
Jackson accepts: a lone string where a list is declared, a number where a string is declared,
300.0 or " 300 " for an int, "" for an absent number or object. Third-party input (OpenRTB
bid requests in our case) contains all of these, and today each one needs a user codec on every
affected property, which also turns off the generated fast path for that property.

Today the request model of our ad server carries a trimming/coercing @JsonCodec on every String
property just to accept numbers and booleans as text, a collection codec whose only purpose is to
accept a lone string as a one-element list, and a Jackson fallback that re-parses the whole document
for the remaining shapes. The same ForyJson instance also reads DSP responses that must stay
strict, and we do not use Mixins, so the switch has to be selectable per class and per property; a
mapper-wide flag alone would not help.

Alternatives considered: mapper-wide flags only (one instance often reads both untrusted and strict
documents); a single lenient = true boolean (the features differ in risk, FLOAT_AS_INT loses data
and EMPTY_STRING_AS_NULL changes meaning, while the "all" default keeps the one-liner); attributes
on @JsonProperty (no class-level default for models with hundreds of properties); user codecs (what
we do today); and making some coercions unconditional like #3967 (would change every strict reader).
Happy to change the shape, for example separate annotations per feature or a different name.

What does this PR do?

  • API. @JsonLenient on a class, field, getter, setter or @JsonCreator parameter, with
    independent features SINGLE_VALUE_AS_ARRAY, SCALAR_AS_STRING, FLOAT_AS_INT,
    TRIM_QUOTED_SCALAR and EMPTY_STRING_AS_NULL (all of them when no argument is given), and
    ForyJsonBuilder.lenientRead(Feature...) as the default for properties without a declaration.
    Default: no feature, so reading is unchanged. A property declaration replaces the class setting,
    which replaces the builder setting; @JsonLenient({}) makes one property strict. Like
    @JsonInclude, the class annotation is not inherited, and an exact-target Mixin can supply it
    (registered in the runtime and annotation-processor Mixin lists).
  • Semantics. Features apply to the property value and to the direct boolean, numeric and
    string elements of a property-owned array or collection, only where Fory owns the
    representation. Properties with @JsonCodec, @JsonFormat or @JsonByteArray, and types with a
    registered codec, codec factory or @JsonValue, keep their representation. A property
    declaration that cannot apply, or one on a never-read, Any or unwrapped property, fails when
    metadata is built. Root values, map values and Optional contents are not affected. Writing is
    unaffected. Exact rules are in docs/json/annotations.md#jsonlenient.
  • Implementation. JsonFieldInfo / JsonCreatorFieldInfo pass their effective feature mask to
    JsonTypeResolver.withLenientRead, next to withContentNullRead. It returns the occurrence
    unchanged when nothing applies; otherwise it builds an occurrence of kind OBJECT around a
    read-only LenientReadCodec (the mechanism @JsonFormat already uses), and for built-in arrays
    and collections it rebuilds only that occurrence with lenient scalar elements, keeping its
    content null handling. The codec peeks one token and delegates every native token to the wrapped
    occurrence's current reader slot, so generated child capabilities stay in use. Token-level
    helpers (tryReadEmptyString, beginPaddedQuotedScalar/finishPaddedQuotedScalar,
    readTruncatedLong, readTruncatedBigInteger) live in JsonReader and are reached only from
    lenient codecs.
  • Untrusted input. Truncation decides the result's integral digit count from the mantissa
    digit positions and a clamped exponent before any allocation, and builds only the surviving
    digits (for BigInteger, at most the token length with the long range as the minimum), so long
    mantissas and huge exponents cannot drive allocation or work. Padding accepts spaces
    only, never escapes or raw control characters. Empty strings decoded to null are still
    rejected for non-null occurrences.
  • Strict path. Strict properties, their generated readers and all existing reader methods are
    unchanged. The generated-codec key includes the builder mask and each read property's mask.
  • Tests. JsonLenientTest runs with codegen on and off and reads every case from a Latin-1
    string, a UTF-16 string, UTF-8 bytes and a stream. It covers every feature and target type,
    element-only leniency through generated readers, empty and space-only strings, native shapes
    reading the same as strict, truncation limits (overflow, huge and tiny exponents, BigInteger
    expansion), malformed padding, property/class/builder precedence, getters, setter and creator
    parameters (named and property-list creators, static factories), Mixins, registered and module
    representations, non-null occurrences, content null handling, and invalid declarations.

Related issues

Related: #3967 (quoted scalars, kept as the native path), #4095 (read-side null handling),
#4166 (@JsonDuplicateProperties).

AI Contribution Checklist

  • Substantial AI assistance was used in this PR: yes
  • If yes, I included a completed AI Contribution Checklist in this PR description and the required AI Usage Disclosure.
  • If yes, my PR description includes the required ai_review summary and screenshot evidence or equivalent persisted links of the final clean AI review results from both fresh reviewers described in AI_POLICY.md, the Fory-guided reviewer and the independent general reviewer, on the current PR diff or current HEAD after the latest code changes.
AI Usage Disclosure
- substantial_ai_assistance: yes
- scope: design drafting, code drafting, tests, docs
- affected_files_or_subsystems: java/fory-json (annotation, builder/config, resolver, field metadata, codecs, JsonReader), java/fory-annotation-processor (Mixin annotation list), docs/json
- ai_review: line-by-line self-review completed; Fory-guided reviewer (Codex CLI) and independent general reviewer (Claude Code CLI) were run in fresh sessions after every fix until both reported no further actionable comments on the same HEAD; findings fixed along the way: non-null bypass by empty strings, control characters in padding, module/registered container and map representations, creator and setter parameter merging, BigInteger exponent expansion, whole-token parsing of long mantissas, documentation gaps; final clean run of both on head 8fb6c242
- ai_review_artifacts: see the two "AI review artifact" comments on this PR
- human_verification: from java/: mvn -pl fory-json test (1377 tests, 0 failures); mvn -pl fory-json verify -DskipTests (GraalVM feature jar verifier passes), mvn -pl fory-json,fory-annotation-processor checkstyle:check spotless:check (clean); prettier --check on changed docs
- performance_verification: strict read path unchanged by construction; JMH comparison below
- provenance_license_confirmation: Apache-2.0-compatible provenance confirmed; no third-party code introduced

Does this PR introduce any user-facing change?

Yes: a new opt-in annotation and builder option. Default behavior is unchanged.

  • Does this PR introduce any public API change?
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

JMH 1.37, JDK 25.0.2, Windows, one fork, -Xmx512m, average time; a small OpenRTB-like Imp
model (11 properties, nested object, list, map). Baseline is apache/main c8bb6101; strict cases
were run in adjacent base/current pairs. The host was shared with other heavy jobs, so the error
bars are wide; the strict code path itself is unchanged. Measured on the first commit of the
branch; later commits changed only lenient-only code.

Case Baseline (ns/op) Current (ns/op)
strict, UTF-8 bytes (run 1) 368.8 ± 131.1 274.8 ± 44.4
strict, UTF-8 bytes (run 2) 289.3 ± 61.6 236.9 ± 52.1
strict, Latin-1 string 298.6 ± 92.2 319.4 ± 91.0
@JsonLenient class, well-formed UTF-8 451.8 ± 409.4
@JsonLenient class, well-formed Latin-1 468.8 ± 274.2
@JsonLenient class, off-spec UTF-8 550.0 ± 294.6

🤖 Generated with Claude Code

pavel-ptashyts and others added 14 commits November 17, 2026 15:55
Add @JsonLenient (class, field, setter, creator parameter) and
ForyJsonBuilder.lenientRead(...) to accept common off-spec input shapes
per property: a single value as a one-element array or collection,
numbers and booleans as String text, truncated floats for integral
targets, whitespace-padded quoted scalars, and empty strings as null.

Lenient properties get a read-only coercion codec in front of their
Fory-owned codec when metadata is built; strict properties and their
generated readers are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject an empty string decoded to null for a non-null occurrence and a
null single element for a non-null array, limit lenient quote padding to
spaces, and reject @JsonLenient on a property that is never read. Add
tests for element-only leniency, single values into primitive and byte
arrays, class settings on creator parameters, and invalid placements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…leniency

Rebuild lenient array elements only when the array uses the default representation of its component, and read collection single values with the collection's own element type information, so module carriers such as unsigned arrays keep their semantics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er leniency

Skip lenient element and single-value handling for arrays and collections produced by a registered codec or codec factory; they own their element semantics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iency

Apply the registered-codec exclusion before any lenient dispatch so registered maps and other factory codecs keep their reader, and drop JsonRawValue from the excluded annotations since it only affects writing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…end lenient tests

Document getter placement, which declarations are rejected, and that explicit-null handling does not apply to empty strings; cover boolean text, separate generated codecs per instance, creator codegen, and content null handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Range-check truncated integers in JsonReader with the target's overflow message and token position, read zero with an out-of-range exponent as zero, and document that padded non-finite literals stay strict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Property-list creators, static factories, and setter parameters now merge a parameter-level @JsonLenient like other property annotations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Limit the integral digits a truncated BigInteger may expand to by its token length (at least the long range), truncate tiny numbers with out-of-range exponents to zero, treat space-only strings as empty for trimmed numeric containers, and test element leniency through generated readers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route integral tokens that may not fit a long through the decimal range check so FLOAT_AS_INT reports the target's overflow message, and document that EMPTY_STRING_AS_NULL skips JsonSubTypes bases and dynamic Object targets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Do not re-parse whole numbers beyond the long range only to report a target-specific overflow message; the integer parser's own error is sufficient.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…part

Without an exponent, parse only the integer part of a lenient truncated number after checking its digit count, document that conflicting member declarations are rejected, and test unaffected targets and one-sided padding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Report unboxed value properties separately when @JsonLenient is declared on them, document that other methods are ignored, and test non-inheritance, nested objects, getter declarations, and JsonValue types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le-token parsing

Decide the truncated integer's digit count from the mantissa digits and the clamped exponent before any allocation, and build only the surviving digits, so long mantissas with exponents no longer parse the whole token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pavel-ptashyts

Copy link
Copy Markdown
Contributor Author

AI review artifact: Fory-guided reviewer (final round)

  • Reviewer: Codex CLI codex exec, model gpt-6-astra, reasoning effort medium, read-only sandbox, fresh session
  • Instructions: Fory-guided reviewer per AI_POLICY.md, following AGENTS.md, .agents/ci-and-pr.md and .agents/languages/java.md
  • Reviewed: git diff apache/main...HEAD at head e12ea509 (base c8bb6101)
  • Round: 14; earlier rounds found the module-owned array, registered map, property-list creator, and allocation-for-error-message issues, all addressed in the branch
  • Result:
No further actionable comments.

@pavel-ptashyts

Copy link
Copy Markdown
Contributor Author

AI review artifact: independent general reviewer (final round)

  • Reviewer: Claude Code CLI claude -p, model claude-opus-5-5, effort medium, read-only tools (Read Grep Glob), no MCP, fresh session
  • Instructions: independent general reviewer per AI_POLICY.md; not pointed to .agents/ci-and-pr.md or any Fory review checklist (AGENTS.md independent-review carve-out)
  • Reviewed: git diff apache/main...HEAD at head e12ea509 (base c8bb6101)
  • Round: 9; earlier rounds found the empty-string non-null bypass, control characters in padding, BigInteger exponent expansion, whole-token parsing of long mantissas, and documentation and test gaps, all addressed in the branch
  • Result:
No further actionable comments.

JsonConfig retains the builder's lenient feature set, so the enum must be build-time initialized like the other configuration enums.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pavel-ptashyts

Copy link
Copy Markdown
Contributor Author

AI review artifact: Fory-guided reviewer (final round)

  • Reviewer: Codex CLI codex exec, model gpt-6-astra, reasoning effort medium, read-only sandbox, fresh session
  • Instructions: Fory-guided reviewer per AI_POLICY.md, following AGENTS.md, .agents/ci-and-pr.md and .agents/languages/java.md
  • Reviewed: git diff apache/main...HEAD at head 8fb6c242 (base c8bb6101)
  • Round: 15 (after the native-image configuration fix); earlier rounds found the module-owned array, registered map, property-list creator, and allocation-for-error-message issues, all addressed in the branch
  • Result:
No further actionable comments.

@pavel-ptashyts

Copy link
Copy Markdown
Contributor Author

AI review artifact: independent general reviewer (final round)

  • Reviewer: Claude Code CLI claude -p, model claude-opus-5-5, effort medium, read-only tools (Read Grep Glob), no MCP, fresh session
  • Instructions: independent general reviewer per AI_POLICY.md; not pointed to .agents/ci-and-pr.md or any Fory review checklist (AGENTS.md independent-review carve-out)
  • Reviewed: git diff apache/main...HEAD at head 8fb6c242 (base c8bb6101)
  • Round: 10 (after the native-image configuration fix); earlier rounds found the empty-string non-null bypass, control characters in padding, BigInteger exponent expansion, whole-token parsing of long mantissas, and documentation and test gaps, all addressed in the branch
  • Result:
No further actionable comments.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant