redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext formatter with the formatter2 API
- ir. Titouan Vervack
EclipseCon France
2017-06-21
Xtending our VHDL Xtext formatter with the formatter2 API ir. - - PowerPoint PPT Presentation
EclipseCon France 2017-06-21 Xtending our VHDL Xtext formatter with the formatter2 API ir. Titouan Vervack redefine.digital.design: Helping you deal with complexity in VHDL and Verilog. EclipseCon France, Toulouse 2017-06-21 Xtending our VHDL
redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext formatter with the formatter2 API
EclipseCon France
2017-06-21
redefine.digital.design: Helping you deal with complexity in VHDL and Verilog.
Xtending our VHDL Xtext formatter with the formatter2 API
EclipseCon France, Toulouse
2017-06-21
Ghent
2
What do we do? VHDL & SV IDE
3
What do we do? Advanced analysis and linting
4
What do we do? Visualisation
5
What do we do? Visualisation
6
What do we do? Eat cake
7
What do we do? And cookies...
8
What do we do? And occasionally… formatting
9
What do we do? And occasionally… formatting
9
Our formatter problems
10
Our formatter problems
○
~1.2k LoC ○ ~200 rules
10
Our formatter problems
○
~1.2k LoC ○ ~200 rules
10
Our formatter problems
○
~1.2k LoC ○ ~200 rules
10
Formatter 1.0 vs Formatter 2.0
11
Formatter 1.0 Formatter 2.0
○ Node model ○ AST ○ Text
○ Node model ○ AST ○ Grammar
○ Can’t adjust to the user
○ Conditional formatting ○ Table formatting
Formatter 2.0 region example
variable _ foo _ : _ bit _ ; _-- bar\n
variable foo : bit; -- bar
= ISemanticRegion = IHiddenRegion
_
\n
= IHiddenRegionPart
12
https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { } 13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { } } 13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] } } 13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] } } 13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] fd.regionFor.keyword(“;”) } } 13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] fd.regionFor.keyword(“;”) .prepend[noSpace highPriority] } } 13
Formatting 2.0 usage example https://de.slideshare.net/meysholdt/xtexts-new-formatter-api
class AwesomO4000 extends AbstractFormatter2 { def dispatch void format(FunctionDecl fd, extension IFormattableDocument d) { interior(fd.regionFor.keyword(“{”), fd.regionFor.keyword(“}”))[indent] fd.declarations.forEach[format] fd.regionFor.keyword(“;”) .prepend[noSpace highPriority] .append[setNewlines(1, 1, 2)] } } 13
Time for a plan
14
Time for a plan 1. Make tests compatible
14
Time for a plan 1. Make tests compatible 2. Start out easy: only indentation
14
Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one
14
Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs
14
Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 5. Do not introduce new bugs
14
Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 5. Do not introduce new bugs 6. Feature parity & extend with new features
14
Time for a plan 1. Make tests compatible 2. Start out easy: only indentation 3. Fix tests one by one 4. Fix old bugs 5. Do not introduce new bugs 6. Feature parity & extend with new features 7. Performance tweaking to same or better level than old
14
Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
15
Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
15
Actual timeline
formatter.format(node, node.offset, node.length).formattedText
1. Make tests compatible
val request = new FormatterRequest() val parsed = new XtextResource() parsed.contents.add(parseResult.rootASTElement) parsed.setParseResult(parseResult) val access = builder.forNodeModel(resource).create() request.setTextRegionAccess(access) val replacements = formatter.format(request) access.rewriter.renderToString(replacements) 15
interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline 2. Start out easy: only indentation
16
interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline 2. Start out easy: only indentation
16
interior(entity.regionFor.keyword(“is”), entity.regionFor.keyword(“end”))[indent]
Actual timeline 2. Start out easy: only indentation
16
Actual timeline 3. Fix tests one by one 4. Fix old bugs
17
Actual timeline 3. Fix tests one by one 4. Fix old bugs
17
Actual timeline 5. Do not introduce new bugs
18
Actual timeline 5. Do not introduce new bugs
18
Actual timeline 6. Feature parity & extend formatter with new features
19
Actual timeline 6. Feature parity & extend formatter with new features
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,...
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline
○ Correct indentation
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline
○ Correct indentation ○ Formatter tags
19
Actual timeline 6. Feature parity & extend formatter with new features
○ Keyword casing ○ Comment alignment to columns ○ Vertical alignment on keywords “:”, “:=”,... ○ Preserve newline
○ Correct indentation ○ Formatter tags Easy once we have alignment
19
Alignment?
20
Alignment?
20
Alignment?
21
Alignment?
21
Alignment?
21
Alignment!
22
Alignment!
22
Alignment!
22
Alignment!
22
Alignment!
start end
22
Alignment!
start end
22
Alignment!
Find smallest encapsulating object that formats region between start and end
start end
22
Alignment!
Find smallest encapsulating object that formats region between start and end
start end
22
Alignment!
clk : in std_logic := ‘X’;
23
Alignment!
clk : in std_logic := ‘X’; start end
23
PortDeclaration
Alignment!
clk : in std_logic := ‘X’; start end name direction type value end.getSemanticElement() PortDeclaration PortList
23
PortDeclaration
Alignment!
clk : in std_logic := ‘X’; start end name direction type value
x + 4 end.getSemanticElement() PortDeclaration PortList
23
val rule = portDecl.getParserRule() val oldAccess = createTextRegionAccess(rule, portDecl.text) val formatted = portDecl.format()
Alignment!
Format smallest encapsulating object
24
val rule = portDecl.getParserRule() val oldAccess = createTextRegionAccess(rule, portDecl.text) val formatted = portDecl.format()
Alignment!
val newAccess = createTextRegionAccess(rule, formatted.text) val oldIndices = oldAccess.getIndices(start, end) val length = newAccess.countBetween(oldIndices)
Format smallest encapsulating object Map unformatted string to formatted string
24
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); →return foo( →→(5, →→·6), →→3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); →return foo( →→(5, →→·6), →→3));
25
Align on first parameter?
→return ( →→foo((5, →→·····6), →→····3)); →return (5, →········foo((5, →·············6), →············3)); →return foo((5, →············6), →···········3)); →return foo( →→(5, →→·6), →→3));
Find largest encapsulating element on same line
25
Actual timeline 6. Feature parity & extend formatter with new features Correct indentation
26
Actual timeline 6. Feature parity & extend formatter with new features Correct indentation
val replacements = formatter.format(request)
26
Actual timeline 6. Feature parity & extend formatter with new features Correct indentation
val replacements = formatter.format(request) replacements .filter[ r| ]
26
Actual timeline 6. Feature parity & extend formatter with new features Correct indentation
val replacements = formatter.format(request) replacements .filter[ r| r.lineCount > 1 ]
26
Actual timeline 6. Feature parity & extend formatter with new features Correct indentation
val replacements = formatter.format(request) replacements .filter[ r| r.lineCount > 1 && r.replacementText.newlines + 1 == r.lineCount ]
26
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
27
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
27
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
val originalRegions = request.getRegions() ?: document.region
27
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on)
27
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF))
27
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF)) val newRegions = originalRegions.removeRegions(betweenRegions)
27
Actual timeline 6. Feature parity & extend formatter with new features Formatter tags
val originalRegions = request.getRegions() ?: document.region val betweenRegions = document.findRegionsBetween(off, on) .addAll(document.findRegionsBetween(off, EOF)) val newRegions = originalRegions.removeRegions(betweenRegions) request.setRegions(newRegions)
27
Performance 7. Performance tweaking to same or better level than old
28
Performance 7. Performance tweaking to same or better level than old
28
Performance 7. Performance tweaking to same or better level than old
28
Performance 7. Performance tweaking to same or better level than old
28
Performance 7. Performance tweaking to same or better level than old
emacs/xtend > 25m
29
Performance 7. Performance tweaking to same or better level than old
emacs/xtend > 25m
30
Biggest problems in hindsight
31
Biggest problems in hindsight
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
30 31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
30 31
Biggest problems in hindsight
○ Nope, old testset isn’t nearly enough
spaces, fake empty lines
comments
30 31
Warning to Xtext users Default ExceptionHandler logs DSL source code to stderr
32
Sigasi future work
33
Sigasi future work
33
Sigasi future work
33
Sigasi future work
33
Sigasi future work
33
Sigasi future work
33
Sigasi future work
○ Formatter tags
33
Sigasi future work
○ Formatter tags ○ Correct indentation
33
Sigasi future work
○ Formatter tags ○ Correct indentation ○ Performance optimizations
33
Xtext wish list
34
Xtext wish list
34
Xtext wish list
34
Xtext wish list
34
Xtext wish list
34
Xtext wish list
34
Conclusion
35
Conclusion
35
Conclusion
35
Conclusion
35
Conclusion
35
Conclusion
35
Conclusion
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
35
Conclusion
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
35
Conclusion
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
○ New features
35
Conclusion
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
○ New features ○ Very readable
35
Conclusion
○ Done in 2 months (base in 2 weeks) while first task and not used to Xtend
○ New features ○ Very readable ○ Conditional formatting
35