Incorporating contrastive prefixes with PrefixTuning

[AvantiB]

Feature request

I was wondering if you could shed some light on how to incorporate contrastive prefixes with the PrefixTuning wrapper (https://huggingface.co/docs/peft/en/package_reference/prefix_tuning) as detailed in this paper (https://aclanthology.org/2022.findings-acl.229/).

Motivation

Contrastive Prefixes have shown to have a similar number of parameters and yet show a better alignment with the attributes being controlled for controllable text generation

Your contribution

academic

[BenjaminBossan]

If your question is how to use this method in PEFT right now, the answer is that it's not implemented. If you mean how PEFT could be extended to incorporate this method, that's a bit more difficult to answer, as I don't know the details of this method. Could you please clarify?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.