🚨 [Idefics2] Update ignore index

[NielsRogge]

What does this PR do?

This PR proposes to set the ignore index of the loss function of Idefics2 like the value of any other LLM/multimodal in the Transformers library, which is -100 (the default ignore index of PyTorch's cross-entropy loss).

It's up to users to create the labels and set whatever they don't want to be included in the loss calculation to -100. In the case of Idefics2, this could look like so:

labels = batch["input_ids"].clone()
labels[labels == processor.tokenizer.pad_token_id] = -100
labels[labels == processor.image_processor.image_token_id] = -100

which means, it's up to the user to make sure padding and image tokens are ignored.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for adding!

Two things to update before merging:

• Docstring info to match this logic
• Add an example in on modeling pad on how to mask, as users will likely not know to mask the images

[VictorSanh]

thanks!

[amyeroberts]

Thanks for updating and providing an example!

Added a 🚨 to flag that this is a breaking change

[NielsRogge]

Thanks, btw @VictorSanh could you update the DocVQA notebook to account for this change?

[VictorSanh]

Thanks, btw @VictorSanh could you update the DocVQA notebook to account for this change?

good point! done!