Generalized Data Structure Synthesis Calvin Loncaric Michael D. - - PowerPoint PPT Presentation

▶
generalized data structure synthesis
SMART_READER_LITE
LIVE PREVIEW

Generalized Data Structure Synthesis Calvin Loncaric Michael D. - - PowerPoint PPT Presentation

Mar 21, 2024 570 likes •851 views

Generalized Data Structure Synthesis Calvin Loncaric Michael D. Ernst Emina Torlak 1 Stateful modules are much more complicated than their specifications Specification Implementation 1992 Lines of code 1383 292 157 3668 41 42

slide-1
SLIDE 1

Generalized Data Structure Synthesis

Calvin Loncaric Michael D. Ernst Emina Torlak

  • 1
slide-2
SLIDE 2

!2

ZTopo Sat4j

Lines of code 42 292 41 1383

Openfire Lucene

157 3668 1992

Stateful modules are much more complicated than their specifications

Implementation Specification

slide-3
SLIDE 3

User Visibility in Openfire

Alice Bob

Can Alice see whether Bob is online? Yes, if:

Alice “subscribed” to Bob AND Bob approved it Bob is a member of a group g that is visible to everyone Alice and Bob are in a group together Alice is in group g1, Bob is in group g2, and g2 is a “child” of g1

!3

slide-4
SLIDE 4

void groupUserAdded(Group group, …, JID addedUser) { return rosterItems.get(user.toBareJID()) != null || implicitFrom.get(user.toBareJID()) != null;

User visibility in code

boolean isRosterItem(JID user) {

Only correct when these maps are in the correct state!

!4

// Get the roster of the added user. Roster addedUserRoster = null; if (server.isLocal(addedUser)) { addedUserRoster = rosterCache.get(addedUser.getNode()); } // Iterate on all the affected users and update their rosters for (JID userToUpdate : users) { if (!addedUser.equals(userToUpdate)) { // Get the roster to update Roster roster = null; if (server.isLocal(userToUpdate)) { // Check that the user exists, if not then continue with the next user try { UserManager.getInstance().getUser(userToUpdate.getNode()); } catch (UserNotFoundException e) { continue; } roster = rosterCache.get(userToUpdate.getNode()); } // Only update rosters in memory if (roster != null) { roster.addSharedUser(group, addedUser); } // Check if the roster is still not in memory if (addedUserRoster == null && server.isLocal(addedUser)) { addedUserRoster = rosterCache.get(addedUser.getNode()); } // Update the roster of the newly added group user. if (addedUserRoster != null) { Collection<Group> groups = GroupManager.getInstance().getGroups(userToUpdate); addedUserRoster.addSharedUser(userToUpdate, groups, group); } if (!server.isLocal(addedUser)) { // Susbcribe to the presence of the remote user. This is only necessary for // remote users and may only work with remote users that **automatically** // accept presence subscription requests sendSubscribeRequest(userToUpdate, addedUser, true); } if (!server.isLocal(userToUpdate)) { // Susbcribe to the presence of the remote user. This is only necessary for // remote users and may only work with remote users that **automatically** // accept presence subscription requests sendSubscribeRequest(addedUser, userToUpdate, true); void addSharedUser(Group group, JID addedUser) { boolean newItem = false; RosterItem item = null; try { // Get the RosterItem for the *local* user to add item = getRosterItem(addedUser); // Do nothing if the item already includes the shared group if (item.getSharedGroups().contains(group)) { return; } newItem = false; } catch (UserNotFoundException e) { try { // Create a new RosterItem for this new user String nickname = UserNameManager.getUserName(addedUser); item = new RosterItem(addedUser, RosterItem.SUB_BOTH, RosterItem.ASK_NONE, RosterItem.RECV_NONE, nickname, null); // Add the new item to the list of items rosterItems.put(item.getJid().toBareJID(), item); newItem = true; } catch (UserNotFoundException ex) { Log.error("Group (" + group.getName() + ") includes non-existent username (" + addedUser + ")"); } } // If an item already exists then take note of the old subscription status RosterItem.SubType prevSubscription = null; if (!newItem) { prevSubscription = item.getSubStatus(); } // Update the subscription of the item **based on the item groups** Collection<Group> userGroups = GroupManager.getInstance().getGroups(getUserJID()); Collection<Group> sharedGroups = new ArrayList<>(); sharedGroups.addAll(item.getSharedGroups()); // Add the new group to the list of groups to check sharedGroups.add(group); // Set subscription type to BOTH if the roster user belongs to a shared group // that is mutually visible with a shared group of the new roster item if (rosterManager.hasMutualVisibility(getUsername(), userGroups, addedUser, sharedGroups)) { item.setSubStatus(RosterItem.SUB_BOTH); void addSharedUser(JID addedUser, Collection<Group> g boolean newItem = false; RosterItem item = null; try { // Get the RosterItem for the *local* user to add item = getRosterItem(addedUser); newItem = false; } catch (UserNotFoundException e) { try { // Create a new RosterItem for this new user String nickname = UserNameManager item = new RosterItem(addedUser, RosterItem.SUB_BOTH, R RosterItem.RECV_NONE, nic // Add the new item to the list of items rosterItems.put(item.getJid().toBareJID(), item); newItem = true; } catch (UserNotFoundException ex) { Log.error("Couldn't find a user with user } } // Update the subscription of the item **based on the item g Collection<Group> userGroups = GroupManag // Set subscription type to BOTH if the roster user belongs to a shared g // that is mutually visible with a shared g if (rosterManager.hasMutualVisibility(getUser item.setSubStatus(RosterItem.SUB_BOTH); for (Group group : groups) { if (rosterManager.isGroupVisible(group, g // Add the shared group to the list of shared g item.addSharedGroup(group); } } // Add to the item the groups of this user that g // Note: This FROM subscription is over // fact there is a TO-FROM relation betw // BOTH subscription for (Group group : userGroups) { if (!group.isUser(addedUser) && rosterManag // Add the shared group to the list of invisible shared g item.addInvisibleSharedGroup(group);

slide-5
SLIDE 5

void groupUserAdded(Group group, …, JID addedUser) { return rosterItems.get(user.toBareJID()) != null || implicitFrom.get(user.toBareJID()) != null; boolean isRosterItem(JID user) {

User visibility in code

!5

// Get the roster of the added user. Roster addedUserRoster = null; if (server.isLocal(addedUser)) { addedUserRoster = rosterCache.get(addedUser.getNode()); } // Iterate on all the affected users and update their rosters for (JID userToUpdate : users) { if (!addedUser.equals(userToUpdate)) { // Get the roster to update Roster roster = null; if (server.isLocal(userToUpdate)) { // Check that the user exists, if not then continue with the next user try { UserManager.getInstance().getUser(userToUpdate.getNode()); } catch (UserNotFoundException e) { continue; } roster = rosterCache.get(userToUpdate.getNode()); } // Only update rosters in memory if (roster != null) { roster.addSharedUser(group, addedUser); } // Check if the roster is still not in memory if (addedUserRoster == null && server.isLocal(addedUser)) { addedUserRoster = rosterCache.get(addedUser.getNode()); } // Update the roster of the newly added group user. if (addedUserRoster != null) { Collection<Group> groups = GroupManager.getInstance().getGroups(userToUpdate); addedUserRoster.addSharedUser(userToUpdate, groups, group); } if (!server.isLocal(addedUser)) { // Susbcribe to the presence of the remote user. This is only necessary for // remote users and may only work with remote users that **automatically** // accept presence subscription requests sendSubscribeRequest(userToUpdate, addedUser, true); } if (!server.isLocal(userToUpdate)) { // Susbcribe to the presence of the remote user. This is only necessary for // remote users and may only work with remote users that **automatically** // accept presence subscription requests sendSubscribeRequest(addedUser, userToUpdate, true); void addSharedUser(Group group, JID addedUser) { boolean newItem = false; RosterItem item = null; try { // Get the RosterItem for the *local* user to add item = getRosterItem(addedUser); // Do nothing if the item already includes the shared group if (item.getSharedGroups().contains(group)) { return; } newItem = false; } catch (UserNotFoundException e) { try { // Create a new RosterItem for this new user String nickname = UserNameManager.getUserName(addedUser); item = new RosterItem(addedUser, RosterItem.SUB_BOTH, RosterItem.ASK_NONE, RosterItem.RECV_NONE, nickname, null); // Add the new item to the list of items rosterItems.put(item.getJid().toBareJID(), item); newItem = true; } catch (UserNotFoundException ex) { Log.error("Group (" + group.getName() + ") includes non-existent username (" + addedUser + ")"); } } // If an item already exists then take note of the old subscription status RosterItem.SubType prevSubscription = null; if (!newItem) { prevSubscription = item.getSubStatus(); } // Update the subscription of the item **based on the item groups** Collection<Group> userGroups = GroupManager.getInstance().getGroups(getUserJID()); Collection<Group> sharedGroups = new ArrayList<>(); sharedGroups.addAll(item.getSharedGroups()); // Add the new group to the list of groups to check sharedGroups.add(group); // Set subscription type to BOTH if the roster user belongs to a shared group // that is mutually visible with a shared group of the new roster item if (rosterManager.hasMutualVisibility(getUsername(), userGroups, addedUser, sharedGroups)) { item.setSubStatus(RosterItem.SUB_BOTH); void addSharedUser(JID addedUser, Collection<Group> g boolean newItem = false; RosterItem item = null; try { // Get the RosterItem for the *local* user to add item = getRosterItem(addedUser); newItem = false; } catch (UserNotFoundException e) { try { // Create a new RosterItem for this new user String nickname = UserNameManager item = new RosterItem(addedUser, RosterItem.SUB_BOTH, R RosterItem.RECV_NONE, nic // Add the new item to the list of items rosterItems.put(item.getJid().toBareJID(), item); newItem = true; } catch (UserNotFoundException ex) { Log.error("Couldn't find a user with user } } // Update the subscription of the item **based on the item g Collection<Group> userGroups = GroupManag // Set subscription type to BOTH if the roster user belongs to a shared g // that is mutually visible with a shared g if (rosterManager.hasMutualVisibility(getUser item.setSubStatus(RosterItem.SUB_BOTH); for (Group group : groups) { if (rosterManager.isGroupVisible(group, g // Add the shared group to the list of shared g item.addSharedGroup(group); } } // Add to the item the groups of this user that g // Note: This FROM subscription is over // fact there is a TO-FROM relation betw // BOTH subscription for (Group group : userGroups) { if (!group.isUser(addedUser) && rosterManag // Add the shared group to the list of invisible shared g item.addInvisibleSharedGroup(group);

  • p addMember(g : Group, u : User)

groupMembers.add((g, u));

query hasSubscriptionTo(u1 : User, u2 : User) exists [ i | i <- rosterItems, u1.val.username == i.val.user, jidToUsername(i.val.target) == u2.val.username ]

  • r exists [ g | g <- groups,

(g, u2) in groupMembers and (g.val.rosterMode == EVERYBODY or (g.val.rosterMode == ONLY_GROUP and (g, u1) in groupMembers) or (g.val.rosterMode == ONLY_GROUP and exists [ subg | subg <- groups, (u1, subg) in groupMembers, (g, subg) in childGroups ])) ]

Yes, if:

Alice “subscribed” to Bob AND Bob approved it Bob is a member of a group g that is visible to everyone Alice and Bob are in a group together Alice is in group g1, Bob is in group g2, and g2 is a “child” of g1

slide-6
SLIDE 6

Cozy

Spec

  • short
  • self-documenting
  • inefficient

!6

.java

  • verbose
  • invariants
  • efficient
slide-7
SLIDE 7

Cozy

Abstraction relation: m = min(ints)

!7

state m : Int

  • p add(i : Int)

m = min(m, i) query findmin() return m state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return (min ints)

slide-8
SLIDE 8

Cozy

!8

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return (min ints)

slide-9
SLIDE 9

Cozy

!9

Spec

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return (min ints)

slide-10
SLIDE 10

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return (min ints)

Cozy

!10

Spec Synthesize a better representation Cost model

state m : Int m

slide-11
SLIDE 11

Cozy

!11

Spec Synthesize a better representation Maintain the new representation Cost model

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return (min ints) m = min(m, i) m state m : Int

slide-12
SLIDE 12

state m : Int

  • p add(i : Int)

m = min(m, i) query findmin() return m

Cozy

!12

Spec Implementation Synthesize a better representation Maintain the new representation Cost model

slide-13
SLIDE 13

Query Synthesis

query findmin() return (min ints)

min read(ints) read(min ints)

unpacking

return m m = min ints

!13

Abstraction relation

slide-14
SLIDE 14

Smart Brute Force Search

for size in [1, 2, …]: for exp in all_expressions(size): if correct(exp): return exp

memoize skip semantically- identical exps “improves cost” is part

  • f correctness

bounded verification yield and keep searching bias

!14

slide-15
SLIDE 15

Cozy is biased toward common patterns

x + 1 MakeMap( user_ids, λu . the {x| x ∈ users, x.id=u} )[user_id]

AST size 3 Discovered when size = 3 AST size 13 Discovered when size = 2

!15

for size in [1, 2, …]:

slide-16
SLIDE 16

Spec Implementation

Iterative Discovery

Synthesize a better implementation with a new representation Figure out how to maintain the new representation Cost model

!16

slide-17
SLIDE 17

Cost Optimization

Cost model

Expression + Invariants =, <, >, or ?

!17

Expression + Invariants

Challenge: Cardinality Estimation

slide-18
SLIDE 18

Spec Implementation

Iterative Discovery

Synthesize a better implementation with a new representation Figure out how to maintain the new representation Cost model

!18

slide-19
SLIDE 19

!19

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return m m = state m : Int

State Maintenance

???

Idea: leverage our incredible query synthesizer

slide-20
SLIDE 20

!20

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return m m = state m : Int

State Maintenance

Idea: leverage our incredible query synthesizer

new_min(i) query new_min(i) return (min (ints U {i}))

Leave optimization to later iterations

= min ints

slide-21
SLIDE 21

!21

state ints : Bag<Int>

  • p add(i : Int)

ints.add(i) query findmin() return m m = state m : Int

State Maintenance

Idea: leverage our incredible query synthesizer

new_min(i) query new_min(i) return (min (ints U {i}))

slide-22
SLIDE 22

Case Studies

ZTopo Sat4j Openfire Lucene

!22

Map tile cache Internal metadata Visibility enforcement Streaming document statistics Goals Less effort Same performance No new bugs

slide-23
SLIDE 23

Effort

Implementation Specification

!23

ZTopo Sat4j

Lines of code 42 292 41 1383

Openfire Lucene

157 3668 1992

slide-24
SLIDE 24

Performance

Speedup

1x

Win Loss

!24

ZTopo Sat4j Openfire Lucene

+0%

  • 15%

+6% +0%

slide-25
SLIDE 25

Correctness

ZTopo Sat4j Openfire Lucene ? 7 25 1 ?

!25

Handwritten Cozy

slide-26
SLIDE 26

Related Work

Iterator Inversion (1975)


Rewrite rules

SETL (1975—)


Manual separation of algorithm and data arrangement

Programming by Refinement (1990—)


Manual iterative transformation

Representation Synthesis (2011)


Limited specification language, special-purpose techniques

Cozy 1.0 (2016)


Limited specification language, special-purpose techniques

!26

slide-27
SLIDE 27

Data Structure Synthesis

Acknowledgements Professors Mike and Emina Students Daniel, David, and Haoming

Spec Implementation

Synthesize Maintain

Cost model

!27

Lower effort Equal performance Fewer bugs

https://cozy.uwplse.org