在具有一般和奖励的多主体部分可观察的序贯决策问题中,有必要考虑利己主义(个人奖励),功利主义(社会福利),和平均主义(公平)标准同时进行。然而,实现这些标准之间的平衡对当前的多主体强化学习方法提出了挑战。具体来说,完全分散的方法,没有所有代理人奖励的全局信息,观察和行动无法学习平衡的政策,而集中培训(分散执行)方法中的代理人由于担心他人利用而不愿共享私人信息。为了解决这些问题,本文提出了一种分散联邦(D&F)范式,在分散的代理人训练利己主义政策时,只利用本地信息来实现自身利益,联邦控制者主要考虑功利主义和平均主义。同时,分散策略和联合策略的参数在差异约束下相互优化,类似于服务器和客户端模式,这确保了利己主义之间的平衡,功利主义,和平等主义。此外,理论证据表明,联邦模型,以及分散的利己主义政策和联邦功利主义政策之间的差异,得到O(1/T)的收敛速度。大量的实验表明,我们的D&F方法优于多个基线,功利主义和平均主义。
In multi-agent partially observable sequential decision problems with general-sum rewards, it is necessary to account for the egoism (individual rewards),
utilitarianism (social welfare), and egalitarianism (fairness) criteria simultaneously. However, achieving a balance between these criteria poses a challenge for current multi-agent reinforcement learning methods. Specifically, fully decentralized methods without global information of all agents\' rewards, observations and actions fail to learn a balanced policy, while agents in centralized training (with decentralized execution) methods are reluctant to share private information due to concerns of exploitation by others. To address these issues, this paper proposes a Decentralized and Federated (D&F) paradigm, where decentralized agents train egoistic policies utilizing solely local information to attain self-interest, and the federation controller primarily considers
utilitarianism and egalitarianism. Meanwhile, the parameters of decentralized and federated policies are optimized with discrepancy constraints mutually, akin to a server and client pattern, which ensures the balance between egoism,
utilitarianism, and egalitarianism. Furthermore, theoretical evidence demonstrates that the federated model, as well as the discrepancy between decentralized egoistic policies and federated utilitarian policies, obtains an O(1/T) convergence rate. Extensive experiments show that our D&F approach outperforms multiple baselines, in terms of both
utilitarianism and egalitarianism.