From Local SGD to Local Fixed-Point Methods for Federated Learning - PowerPoint PPT Presentation
KAUST From Local SGD to Local Fixed-Point Methods for Federated Learning Laurent Condat King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia Grigory Dmitry Elnur Peter Malinovsky Kovalev Gasanov
KAUST From Local SGD to Local Fixed-Point Methods for Federated Learning Laurent Condat King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia Grigory Dmitry Elnur Peter Malinovsky Kovalev Gasanov Richtárik Fixed-Point Methods with Local Steps 1 / 22
KAUST Distributed Algorithms Master Node 1 Node 2 ... Node M Fixed-Point Methods with Local Steps 2 / 22
<latexit sha1_base64="wsZp0NAEfBi+CuX+vq05sOd25fY=">AFr3icpVTbtNAEJ0WDCVcmtJHXiLSAhVhJQSx+Q2sIDL4giNU2lJgTb2SRWfJMvhWDlW3iFT+IP4C84M9mUtCEQxFq2Z8/MOTsza68deW6SVqvfVlavXTdu3Fy7Vbh95+69eLG/ZMkzGJHNZzQC+NT20qU5waqkbqp06jWFm+7amPXzB/ua5ihM3DI7TUaTavtUP3J7rWCmgTnEzjMzCVsu30oFjefnxuFPf6hTLVbMqozRv1LRJj2Owo3VQ2pRl0JyKCOfFAWUwvbIogTXGdWoShGwNuXAYliu+BWNqQBuhiFCAvoEM8+ZmcaDTBnzUTYDlbxcMdglqgCToi4GDavVhJ/JsqMLtLORZNzG+Ftay0faEoDoH/jTSOX5XEtKfXomdTgoqZIEK7O0SqZdIUzL81UlUIhAsZ2F/4YtiPMaZ9Lwkmkdu6tJf7vEskozx0dm9GP2RZQZ6JqLBil5j10zaQ84FZD0A5tE5nqnetVhUFX2QrvShwC8HL6edIrGQFRMud9YW6OdQdQYpXKwk4nojb68Xdnc2tLdzeFd4kB1fy/fUN5cC5Z0OpSu5xdC0xeLqd2kb76X07GBZPJdX1aqrDickp9yYk7/Dstk3ZmsnopGp4wFb3Wlb/RXw7PWPux7IwlWr6sPNb/Yx/49gX2r4oHVxQP/ltxwnPl/Ul3syV/akgfYc/7sYacW3s8di5OqXnjpG7WnphP39bL+4f6BFujB/SQHuGU2qV9ekVH1ECmI/pMX+irUTOaxjvj/SR0dUVzNunSMNyfxTkgyA=</latexit> <latexit sha1_base64="r8K4prBOVR9ZWwMNyJc5HoJ+lzs=">AFr3icpVTbtNAEJ0GDCVcmtJHXiLSAhVlJQSx+Q2sIDL4giNU2lJgTb2SRWfJMvhWDlW3iFT+IP4C84M9mUtCEQxFq2Z8/MOTsza68deW6S1mrfVgrXrhs3bq7eKt6+c/feWmn9/kSZrGjGk7ohfGpbSXKcwPVSN3U6dRrCzf9lTHr5gf/NcxYkbBsfpKFJt3+oHbs91rBRQp7QRmZxs+Vb6cCxvPx43KlvdkqVmlmTUZ436tqokB5H4XrhkFrUpZAcysgnRQGlsD2yKMF1RnWqUQSsTmwGJYrfkVjKoKbIUohwgI6xLOP2ZlGA8xZMxG2g1U83DGYZaqCEyIuhs2rlcWfiTKji7Rz0eTcRnjbWsHmtIA6N9408hleVxLSj16JjW4qCkShKtztEomXeHMyzNVpVCIgLHdhT+G7Qhz2ueycBKpnXtrif+7RDLKc0fHZvTjD1lWkWciKqzYpefYNZP2kHMRWQ+AeXSOZ6p3LRZVR+k670IQAvh68neJ6RkCUzHlfmJtj3QGUWKW6sNOJqM32enF3Z3NrC7d3hTfJwZV8f31DOXDu2VBq6UpuMTRtsbj6XdrCm+/ldGwgmXzXl5WmKqy4nFJfcuIO/07LpJ2ZrF6KhidMRa915W/0l8Mz1n4sO2OJli8rj/X/2Ae+dYH9q+LBFcWD/1ac8Fx5f9LdbMmfGtJH2PN+rCHn1h6PnYtTat42TbrT8ynb7cr+4f6BFulB/SQHuGU2qV9ekVH1ECmI/pMX+irUTeaxjvj/S0sKI5G3RpGO5PwAEgxw=</latexit> <latexit sha1_base64="neAL1APoDnsc0+WSH4Mo0NxUsm4=">AFr3icpVTbtNAEJ0GDCXcUvrIS0QaCaHKSgpq6QNSW3jgpaJITVOpCcF2NokV3+RLIVj5Fl7hk/gD+AvOTDYlbQgEsZbt2TNzs7M2mtHnpuktdq3lcK168aNm6u3irfv3L13v7T24CQJs9hRDSf0wvjUthLluYFqpG7qdMoVpZve6pD1+yv3mu4sQNg+N0FKm2b/UDt+c6VgqoU1oPI7O40fKtdOBYXn487hxudEqVmlmTUZ436tqokB5H4VrhgFrUpZAcysgnRQGlsD2yKMF1RnWqUQSsTmwGJYrfkVjKoKbIUohwgI6xLOP2ZlGA8xZMxG2g1U83DGYZaqCEyIuhs2rlcWfiTKji7Rz0eTcRnjbWsHmtIA6N9408hleVxLSj16LjW4qCkShKtztEomXeHMyzNVpVCIgLHdhT+G7Qhz2ueycBKpnXtrif+7RDLKc0fHZvTjD1lWkWciKqzYpRfYNZN2kXMRWQ+AeXSOZ6p3LRZVR+k670IQAvh68neJ6RkCUzHlfmJtj3QGUWKW6sNOJqM32enF3Z3NrC7d3hTfJwZV8f31DOXDu2VBq6UpuMTRtsbj6HdrEm+/ldGwgmXzXl5WmKqy4nFJfcuIO/07LpO2ZrF6JhidMRYe68jf6y+EZaz+RnbFEy5eVx/p/7APfvMD+VXH/iuL+fytOeK68P+lutuRPDekj7Hk/1pBza5fH9sUpNW+cbJn1p+azt1uVvQN9gq3SQ3pEj3FK7dAevaYjaiDTEX2mL/TVqBtN453xfhJaWNGcdbo0DPcnUjAg4w=</latexit> KAUST Distributed Algorithms Master op. T 1 op. T 2 op. T M Fixed-Point Methods with Local Steps 3 / 22
<latexit sha1_base64="xlMn1nKprFMTIJxZUGkB+Sv94+A=">AFJ3iclVPbtNAEB2XACVcmoJ4QgKLKFKpqigpiFKkShXwEtFQU1bqWmjtb1JVvENX0qC5Q/gU/gOPoA3BI+8wRO/wNmJ2waicNko9syZOWdnZr1W6Ko4aTS+GHPnSucvXJy/VL585eq1hcri9d04SCNbtuzADaJ9S8TSVb5sJSpx5X4YSeFZrtyzBk91fO9YRrEK/J1kFMpDT/R81VW2SAB1Ku/Kptm2ZE/5mXydMrica8wVlnSzHbY9kfRtob3Hw7by2bes7FV+5MAO4yQw291I2Fkz7Zysx2nXidTG838iN0zOtB8aXivrkWl70xuWe5Uqo16g5c5bTQLo0rF2g4WjQ/UJocCsikljyT5lMB2SVCM3wE1qUEhsEPKgEWwFMcl5VQGN0WRIYAOsCzBy+DHSAnQH5OJtUQD5AZwdbKJsdTVtHobB2BmnQdI7ytQsDmlAf6N94J5n/yjsAmlCXHnG3CrMIGdFTsQuVlCegKzcnukqgEALTtoN4BNtm5slMTebE3Lueo+D4N87UqPbtIjel73+osoY6Y1bRig5t4ITqtI6ay6i6D8ylYzyT4oQiVpX0hqfu8Rx8DLEujwp3c8IiGRfn4vmZti3DyWtUps56ZjVJmc9e7qTtR0yt/sb1yD4nrPvqEMuJ7ZgHtxuLYImhZbuvs1WsFb/8c6z5jvMkvSVrHi+KMtKd1l3kGgrU83jUvIe8JVT7H8VxzF7d8U8aohZwh7Ol4zjd3Xa+Hp/d02thdrTfv1x+8XK1uPinu8Dzdoru0hHu6Rpv0nLaphTp/GDeN28ad0vSx9Kn0udx6pxRcG7QL6v09SfUvR6L</latexit> KAUST Distributed fixed-point problem We define the average operator M T : x 2 R d 7! 1 X T i ( x ). M i =1 Fixed-Point Methods with Local Steps 4 / 22
<latexit sha1_base64="XF2rZ5bA65PhacZ6aZ9h7KOR5k=">AE1HiclVPLbtNAFL0pBkp4pbBkY5FGQqiKkoJaukCqgAUb1IJIWqlJq7E9iUfxS/a4NJisEFu+gK+BJfwAfwB/wZkbtwSi8Jgonjvn3nN854zHSQKV6Vbra2XpnHX+wsXlS9XLV65eu15budHN4jx1ZceNgzjd0QmAxXJjlY6kPtJKkXoBHLPGT02+b1jmWYqjl7qcSL7oRhGaqBcoQEd1TZ28tQexiKwVWbr2B6oyLNXTw57mRZpT0W9UGjfcYoXk0Nv1c5y17e1L/Rrd5qtnjY80G7DOpUjt14pfKJeuRTC7lFJKkiDTigARl+B1Qm1qUAOtTASxFpDgvaUJVcHNUSVQIoCM8h1gViGPUxKifkE0N5GNUpoiNs35nFUMulhHoCfTxizU2qFQDX5QP/GO638V94BUE0DesC7VfAiYcS4pYqOTtgOrdndqWhkAzsYd8ithl5qmnNnMy3rvxUXD+G1ca1Kzdsjan73/osoE+M1Yxih49xAk1aQs9V9G1DygYzx1eUIpq0p6xa6H7EMEXoHcgJ0y+xkDkbw252K4Bd7rQ8moNBY6nbHarNeL3Z3trc/cwW+8aQ+K+/35DRXAjWcj3ovHvaXQdDgyu9+kNczmP9V5wvyAWZKele/YKc/IrIzuXfZAsFbIb52UX/kQ+NoZ9r+KU57i+TXflCnqoOYE8Xx+wjd3y4yNs3s6H3TXm+17zfvP1+vbj8o7vEy36DbdwT3dpG16SrvUQZ8f6CN9pi9W13pjvbXeTUuXKiXnJv0yrPc/APS/AG4=</latexit> <latexit sha1_base64="aMiFpDzQdxv8lXmwkSntVdGhmrw=">AE6niclVPLbtNAFL0pBkp4NIUlG4soUqmqKCmI0kWlCliwAFGkpq3UhGpsTxIrfmGPS4OVn2CH2PIF/AI/AUtYwV9w5npaAlF4TGL7zrn3HN8543GSwM9Uq/W5snDOn/h4uKl6uUrV68t1Zav72Vxnrqy48ZBnB4IpOBH8mO8lUgD5JUitAJ5L4zeqjz+8cyzfw42lXjRPZCMYj8vu8KBeio9qRatW38u4c+FEhX+acWJ1oLBRq6Iqg2J2snLzoZkqkt+0t24RNXSEjb4pzVKu3mi0e9mzQNkGdzNiJlysfqUsexeRSTiFJikghDkhQht8htalFCbAeFcBSRD7nJU2oCm6OKokKAXSE+wCzAnGMmhj1E7KpgXyMyhSxVrY5n7OKRufrCPSk+xj6RitEKiIdC/8U4r/5V3CFRn+7zan14kTCiXGNSs4O6M7tqVUpKCTAdOwhnyJ2mXnqc2cjNeufRSc/8aVGtVz19Tm9P0PXTbQZ8YqWtGjLexQkzbRcxVdD4EFdIy7MjuUsqkV+x6yD5E4BXI9dkpvZ4xEMlzvS+aW+C9QyhplcZcpzNWm/Z6vrvTvfWY2/+NV/bgc78/v6ECuPZsxGvxuLcUmg5HevUbtIanvkqdR8wPmCXpqXnHM7NHeqZ1V9kDwVohv3VivIB8LUz7H8VS57Pz9d8UkrUQc0J4tl8eXI39bh3dk5ng731ZvtO8+7z9fr2A3OGF+km3aIVnNMN2qbHtEMd9PmBPtEX+moF1hvrfWuLF2oGM4N+mVY738AgsUHA=</latexit> <latexit sha1_base64="xlMn1nKprFMTIJxZUGkB+Sv94+A=">AFJ3iclVPbtNAEB2XACVcmoJ4QgKLKFKpqigpiFKkShXwEtFQU1bqWmjtb1JVvENX0qC5Q/gU/gOPoA3BI+8wRO/wNmJ2waicNko9syZOWdnZr1W6Ko4aTS+GHPnSucvXJy/VL585eq1hcri9d04SCNbtuzADaJ9S8TSVb5sJSpx5X4YSeFZrtyzBk91fO9YRrEK/J1kFMpDT/R81VW2SAB1Ku/Kptm2ZE/5mXydMrica8wVlnSzHbY9kfRtob3Hw7by2bes7FV+5MAO4yQw291I2Fkz7Zysx2nXidTG838iN0zOtB8aXivrkWl70xuWe5Uqo16g5c5bTQLo0rF2g4WjQ/UJocCsikljyT5lMB2SVCM3wE1qUEhsEPKgEWwFMcl5VQGN0WRIYAOsCzBy+DHSAnQH5OJtUQD5AZwdbKJsdTVtHobB2BmnQdI7ytQsDmlAf6N94J5n/yjsAmlCXHnG3CrMIGdFTsQuVlCegKzcnukqgEALTtoN4BNtm5slMTebE3Lueo+D4N87UqPbtIjel73+osoY6Y1bRig5t4ITqtI6ay6i6D8ylYzyT4oQiVpX0hqfu8Rx8DLEujwp3c8IiGRfn4vmZti3DyWtUps56ZjVJmc9e7qTtR0yt/sb1yD4nrPvqEMuJ7ZgHtxuLYImhZbuvs1WsFb/8c6z5jvMkvSVrHi+KMtKd1l3kGgrU83jUvIe8JVT7H8VxzF7d8U8aohZwh7Ol4zjd3Xa+Hp/d02thdrTfv1x+8XK1uPinu8Dzdoru0hHu6Rpv0nLaphTp/GDeN28ad0vSx9Kn0udx6pxRcG7QL6v09SfUvR6L</latexit> KAUST Distributed fixed-point problem We define the average operator M T : x 2 R d 7! 1 X T i ( x ). M i =1 Our goal is to find x ? ∈ R d such that T ( x ? ) = x ? . Fixed-Point Methods with Local Steps 4 / 22
Recommend
More recommend
Explore More Topics
Stay informed with curated content and fresh updates.