Structured Outputs: JSON Schema 'nullable' modifier ignored
Problem
Confirm this is a Node library issue and not an underlying OpenAI API issue - [X] This is an issue with the Node library Describe the bug In Zod one can make a field nullable in two ways, with A) a union (eg. `authorName1` below) or using B) the .nullable() method (eg. `authorName2` below). A: `z.union([z.string(), z.null()]` B: `z.string().nullable()` The final JSON schema will also be different: A: `{ "type": ["string", "null"] }` B: `{ "type": "string", "nullable": true }` When using the later, B, the LLM (gpt-4o-2024-08-06) always return a string and never null. The former, A, works well. I would imagine B is quite a bit more common in Zod schemas so it would make sense to support it. I see many ways to fix this, including just clarifying that "nullable" has no effect in the docs. To Reproduce Consider this Zod schema, `authorName1` and `authorName2` are both nullable strings using two different syntaxes. [code block] Passing it through `zodResponseFormat`: [code block] Output: [code block] As you can see in the generated output, responseFormat also use two different syntaxes to make the two fields nullable. The model (gpt-4o-2024-08-06) seem to ignore the later `"nullable": true`. [code block] Code snippets _No response_ OS macOS and Linux Node version Node 20 Library version openai 4.58.1
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Enhance Zod Schema Handling for Nullable Fields
The issue arises because the OpenAI API does not recognize the 'nullable' modifier in the JSON schema generated by Zod when using the .nullable() method. This leads to the API consistently returning a string instead of null for fields defined with 'nullable'. The difference in schema representation between the union method and the nullable method causes the API to misinterpret the intended behavior.
Awaiting Verification
Be the first to verify this fix
- 1
Update Documentation
Clarify in the Zod documentation that the 'nullable' modifier is currently not supported by the OpenAI API and that users should use the union method to define nullable fields.
- 2
Implement a Warning in Zod
Modify the Zod library to issue a warning when a user defines a field as nullable using the .nullable() method. This warning should inform users that this method may not function as expected with the OpenAI API.
typescriptz.string().nullable().warning('The nullable modifier may not be supported by the OpenAI API. Use z.union([z.string(), z.null()]) instead.') - 3
Create a Migration Guide
Develop a migration guide for users transitioning from using .nullable() to the union method, providing examples and best practices for defining nullable fields.
typescriptExample: Replace `z.string().nullable()` with `z.union([z.string(), z.null()])`. - 4
Test and Validate Changes
Conduct tests to ensure that the warning is displayed correctly and that the documentation reflects the changes. Validate that the union method works as expected with the OpenAI API.
typescriptconst schema = z.union([z.string(), z.null()]); // Test with OpenAI API - 5
Monitor User Feedback
After implementing the changes, monitor user feedback for any issues related to nullable fields and adjust the documentation or library as necessary based on user experiences.
Validation
Confirm that the documentation has been updated, the warning is issued when using .nullable(), and that the union method returns the expected results when tested with the OpenAI API.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep