Only first item of a field list is validated

[ralfeus]

Actual Behavior

When the field is a list only its first item is validated. In fact the method Field.process_formdata() seems to explicitly get only first one:

    def process_formdata(self, valuelist):
        """
        Process data received over the wire from a form.

        This will be called during form construction with data supplied
        through the `formdata` argument.

        :param valuelist: A list of strings to process.
        """
        if valuelist:
            self.data = valuelist[0]

and then Field.process():

        try:
            for filter in self.filters:
# Here I see only self.data is used where only first item was stored
                self.data = filter(self.data) 
#############################################
        except ValueError as e:
            self.process_errors.append(e.args[0])

Expected Behavior

All items of the field list are validated

[SoundsSerious]

Will add on to this, it doesn't appear that any items are validated by the type field in FieldList

import wtforms
class Test( wtforms.Form):
    l = wtforms.FieldList( wtforms.StringField())
    
t = Test(l=[])
first = t.validate()

t = Test(l=['hey'])
second = t.validate()

t = Test(l=[1.0])
third = t.validate()

t = Test(l=[True])
fourth = t.validate()

print('All Tests Validated:',all([first,second,third,fourth]))

Results in All Tests Validated: True

[SoundsSerious]

@ralfeus @azmeuk

A couple ideas on how to handle this but, I don't understand what the point of specifying the Field in FieldList is then if it isn't used?

Its unclear how this is supposed to work In the case of converting a string from post parameter is passed to field list. Is it supposed to convert the field to a python list? That would make the most sense. Maybe its just a bunch of comma-separated values, but its important to convert and sanitize this first? That would seem to go along with the intended purposes of the library.

Since there are so many ways this could possibly be used, my suggestion would be to add a new type of input converters for FieldList or FieldForm and potentially a new FieldDict (for JSON) which would convert incoming input that would be expected from a POST request (or another scenario)

I think the simplest one in the case of FieldList would be simply to split by commas, then convert to the specified type. Errors could then pinpoint where validators failed.

This option could be customized by user for population, and could open up JSON possibilities for FieldDict in support of #154

[SoundsSerious]

Hmm perhaps I spoke a bit too soon on the default validation:

If I change the item to IntegerField it seems to work.

It seems that if the type of item is convertible to the specified type the validation works.

import wtforms
class Test( wtforms.Form):
    l = wtforms.FieldList( wtforms.IntegerField())
    
t = Test(l=[])
first = t.validate()

t = Test(l=['hey'])
second = t.validate()

t = Test(l='[1.0]')
third = t.validate()

t = Test(l=[True])
fourth = t.validate()

print('All Tests Validated:',all([first,second,third,fourth]))

All Tests Validated: False < 1,3,4th OK, 2nd fails.

[SoundsSerious]

another example in the case of booleanField

import wtforms
class Test( wtforms.Form):
    l = wtforms.FieldList( wtforms.BooleanField())
    
t = Test(l=[])
first = t.validate()

t = Test(l=['hey'])
second = t.validate()

t = Test(l=[1.0])
third = t.validate()

t = Test(l=[True])
fourth = t.validate()

tests = {1:first,2:second,3:third,4:fourth}
print('All Tests Validated:',all(tests.values()))
print(tests)

Results in:

All Tests Validated: True
{1: True, 2: True, 3: True, 4: True}

[SoundsSerious]

in the case of floatField

import wtforms
class Test( wtforms.Form):
    l = wtforms.FieldList( wtforms.FloatField())
    
t = Test(l=[])
first = t.validate()

t = Test(l=['hey'])
second = t.validate()

t = Test(l=[1.0])
third = t.validate()

t = Test(l=[True])
fourth = t.validate()

tests = {1:first,2:second,3:third,4:fourth}
print('All Tests Validated:',all(tests.values()))
print(tests)

All Tests Validated: True
{1: True, 2: True, 3: True, 4: True}

[SoundsSerious]

Re-Checking the documentation I see that I am only using the data field via kwargs which may be wrong vs formdata, will have to look into that.

I see there is also the filters argument for field but I get TypeError: FieldList does not accept any filters. Instead, define them on the enclosed field. when I try on FieldList

[SoundsSerious]

Ok I have something of a working solution here, definitely a learning experience. It seems I had a few wrong ideas about wtforms, thankfully the library is flexible.

Here is my subclassing solution, that converts from a string, or a python iterable, and verifies every entry is of a certain type:

def validate_typelist_field(form,field):
    
    if not hasattr(field,'data'):
        raise wtforms.validators.ValidationError('No Data!')
    
    if not isinstance(field.data,list):
        raise wtforms.validators.ValidationError('Data Is Not List!')    

    carray = [ isinstance(it,field.item_type) for it in field.data]
    if not all(carray):
        inx = carray.index(False)
        raise wtforms.validators.ValidationError(f'Not All Items Are Of Type {field.item_type} @ {inx}')
    

class TypeListField(wtforms.Field):
    """
    A Field Which Converts & Validates A List Of Type Or A String That Should Be Converted To Type, after removing
    extra_characters from formdata
    
    """
    
    item_type: type 
    extra_characters: typing.List[str] = ['[',']','(',')','{','}']
    delimeter = ','
    
    def __init__(self, item_type, label=None, validators=None,  **kwargs):
        """:param item_type: a simple like str, int, float that can call a string input and convert data"""
        
        self.item_type = item_type
        
        validators = [validate_typelist_field] if validators is None else validators + [validate_typelist_field]
        
        super(TypeListField, self).__init__(label, validators, **kwargs)

    def _value(self):
        if self.data:
            return self.delimeter.join(self.data)
        else:
            return ''
    
    def process_formdata(self, valuelist:str):        
        if valuelist:
            self.data = self.convert_string_to_listtype(valuelist[0])
        else:
            self.data = []
            
    def process_data(self, data):
        if data and isinstance(data,str):
            self.data = self.convert_string_to_listtype(data)
        elif isinstance(data,(list,tuple)):
            self.data = data
        else:
            self.data = []
            
    def convert_string_to_listtype(self,string_canidate) -> list:
        valuelist = ''.join([ char for char in string_canidate if char not in self.extra_characters]) #comes in a list    
        return [self.item_type(x.strip()) for x in valuelist.split(self.delimeter)]        
            ```

[SoundsSerious]

Testing this looks like this:

class Frm(wtforms.Form):
    lis = TypeListField(float,'floater')

tests = {}    

t = Frm(formdata=MultiDict([('lis','[1.0,2.0,3.0,4.0]')]))
tests[1] = t.validate()

t = Frm(lis = [1.0,2.0,3.0,4.0])
tests[2] = t.validate()

t = Frm(lis = '[1.0,2.0,3.0,4.0]')
tests[3] = t.validate()

t = Frm(lis = [1.0,2.0,'wrong',4.0])
tests[4] = t.validate()

t = Frm(lis = '[1.0,2.0,wrong,4.0]')
tests[5] = t.validate()


t = Frm(lis = [1.0,2.0,True,4.0])
tests[6] = t.validate()

t = Frm(lis = '[1.0,2.0,True,4.0]')
tests[7] = t.validate()

Its expected the first 3 succeeded and last 4 fail
and results in a test output of {1: True, 2: True, 3: True, 4: False, 5: False, 6: False, 7: False}

[azmeuk]

OP refers to Field.process_formdata, that is called by Field.process:

https://github.com/wtforms/wtforms/blob/0524993ee57a957c8444df2b090c00464ef9bb2a/src/wtforms/fields/core.py#L328

However, process is overriden by FieldList, and calls Field.process for every subform, in the _add_entry method:

https://github.com/wtforms/wtforms/blob/0524993ee57a957c8444df2b090c00464ef9bb2a/src/wtforms/fields/list.py#L85-L90

https://github.com/wtforms/wtforms/blob/0524993ee57a957c8444df2b090c00464ef9bb2a/src/wtforms/fields/list.py#L171

Thus, every value for the list in formdata is processed. Depending on the field, it can raise ValidationError:

>>> import wtforms
>>> from werkzeug.datastructures import ImmutableMultiDict
>>> class F(wtforms.Form):
...     foobar = wtforms.FieldList(wtforms.FloatField())
>>> F(ImmutableMultiDict({"foobar-1": 3.})).validate()
True
>>> F(ImmutableMultiDict({"foobar-1": 3., "foobar-2": "foo"})).validate()
False

The data parameters are not all validated, but the behavior is the same without a FieldList:

>>> F(data={"foobar":[3., "foo"]}).validate()
True
>>> class F(wtforms.Form):
...     foobar = wtforms.FloatField()
>>> F(foobar="foo").validate()
True

With custom validators, every data in formdata is validated:

>>> class F(wtforms.Form):
...     foobar = wtforms.FieldList(wtforms.StringField(validators=[wtforms.validators.length(min=3)]))
>>> F(ImmutableMultiDict({"foobar-1": "foo"})).validate()
True
>>> F(ImmutableMultiDict({"foobar-1": "foo", "foobar-2": "A"})).validate()
False

And also the data values:

>>> F(data={"foobar":["foo"]}).validate()
True
>>> F(data={"foobar":["foo", "A"]}).validate()
False

The data coercion and validation is not very regular in wtforms, this topic is discussed in #154 and #662