The SchemaField {FIELD_NAME} doesn't have a default value and was not provided in the ParsedSchema

[RyanSephton]

Hey all,

I keep encountering the above error when I call InMemorySource.put([my_df]). There doesn't seem to be any documentation/examples/remedy in the codebase as to what causes this or how to set a default value to address this error. If anyone has encountered anything similar it would be great to know what caused it and how it was fixed.

Many Thanks

UPDATE 05/06/2024
The above can be fixed by using .fillna() with an arbitrary default value - a weird quirk is that the default value HAS to respect the dtype of the column i.e. you can't use None or NA in most cases.

There is a broader issue here as It looks like this error also happens for app.query() - to replicate the behaviour it seems that if you have an index built off an embedding space, but you dont want to add that space into your query, it throws this error. The current notebooks seem to work around this by including all spaces from the index in the query and then manually setting the weights to 0 and inserting placeholder elements instead. This seems to be odd behaviour as the expectation would be that if a parameter hasn't been attached to a space, nor is it referenced in the query, then there shouldn't be a need to specify it - this will be large issue when the index is connected to an external provider and persists over many sessions

The error message is also quite misleading as it seems there's no ability to set a default value in the parser, nor is a default value looked for in the online_node - unless the node is a timestamp in which case the context is injected. Ideally default values for all types should be passed into the context object OR we should be able to insert default values into the parser. These defaults should then be used if a query doesn't explicitly set a value for a field that is in the index

I hope that helps

[Dminor7]

I've been having the same issue, where I get an error message about not having a default value for a schema field that I'm not providing. In my case, I'm creating a CategoricalSimilaritySpace and the type is StringList, so to get around the error I had to initialize it with an empty list of strings ['']. Is there a better way to do this? In my use case, not all of my records have this category field populated.

[morkapronczay]

@RyanSephton thank you for your feedback! Sorry, for taking this long to answer, but here are a few points that hopefully help.

1. The issue you mention when doing source.put happens as we expect the dataset to be clean at the point of ingestion. That is something we probably should emphasise more - at least with a more informative error message at this point. That should explain why the filling value have to comply with the dtype of the column and schema_field.
2. Very spot on in terms of the query! We are in the process of changing that behaviour by making spaces in the index in general not affecting the query - unless specified by the user. So the 0 weight will not have to be specified by the user. This change will make all the odd behaviour you mention go away. It is expected to happen in the coming months - hopefully the next. :)

@Dminor7 Thank you too for chiming in! Very valid point indeed. A suppose right now the best you can do is use a value for these items like "no category" and when you set the categories argument of the CategoricalSimilaritySpace you do not list that specific value. If you keep in mind the uncategorised_as_category argument of the space you can create two plausible behaviours in terms of "no_category" items' similarity to each other. Let me know if that made sense!