{"id":2868,"date":"2021-07-09T15:57:51","date_gmt":"2021-07-09T19:57:51","guid":{"rendered":"http:\/\/codinggorilla.com\/?p=2868"},"modified":"2021-12-13T11:21:13","modified_gmt":"2021-12-13T11:21:13","slug":"writing-a-universal-programming-language-extension","status":"publish","type":"post","link":"http:\/\/165.227.223.229\/index.php\/2021\/07\/09\/writing-a-universal-programming-language-extension\/","title":{"rendered":"Writing a universal programming language extension for VSCode"},"content":{"rendered":"\n

There was an interesting question in Stackoverflow<\/a> today: Is it possible to write an extension for VSCode that can accept any grammar and support semantic highlighting, defs\/refs, and other language features?<\/p>\n\n\n\n

I’ve always toyed with the idea of having a \u00e2\u20ac\u009cuniversal language server\u00e2\u20ac\u009d with the server dynamically reading a grammar then parsing the input.<\/p>\n\n\n\n

So, I decided to write such an extension<\/a>. It took me about four hours to get something working, but I would assume some could write something faster and cleaner.<\/p>\n\n\n\n

Is such an extension possible? The answer is yes, but writing one depends on several factors: Have you written a VSCode extension before? Are you starting from an existing implementation? Are you a language tool expert? If your answer \u00e2\u20ac\u009cno\u00e2\u20ac\u009d to either of these questions, it could take a bit of time to write.<\/p>\n\n\n\n

I started from the AntlrVSIX<\/a> extension for VSCode that I already wrote a year or two ago. The server is in C# and client in Typescript.<\/p>\n\n\n\n

The server code originally required a month or two to write because there weren\u00e2\u20ac\u2122t any good examples to model. In addition, a bit of infrastructure need to be written because the server-side APIs for LSP did not implement version 3.16 of Language Server Protocol, which includes \u00e2\u20ac\u009csemantic highlighting\u00e2\u20ac\u009d.<\/p>\n\n\n\n

For the grammar specification, one has to put a stake in the ground and decide what parser generator to use. There are many generators, but I used Antlr4<\/a> because that is what I am familiar with. And, while Antlr4 can target C#, the parser it outputs requires a driver, and the code compiled. Therefore, I assumed this would be done in a step prior to running VSCode. \u00e2\u20ac\u009cUsers\u00e2\u20ac\u009d can write a new grammar, generate a parser, build, then tell the extension where the parser is located.<\/p>\n\n\n\n

I\u00e2\u20ac\u2122ve noted that other parser implementations exist. One \u00e2\u20ac\u009cfast\u00e2\u20ac\u009d implementation in JavaScript is Chevrotain<\/a>. Unfortunately, the rules are specified in JavaScript, so while it would be easy to convert an EBNF-based grammar like Antlr or BNFC<\/a> to Chevrotain, scraping the code to extract the grammar from Chevrotain syntax would be difficult. That\u00e2\u20ac\u2122s because one can modify the rules on the fly.<\/p>\n\n\n\n

The output of a parse is a parse tree. But a grammar in itself is insufficient to classify tokens in some file.<\/p>\n\n\n\n

In addition to the grammar, you will need to declare classes of symbols. Here, I used XPath<\/a> expressions, which is an expression to match a particular parse tree node. The reason for XPath expression is because we want the context of a token to be used for classification. For example, this grammar-based editor<\/a> selects tokens from the lexer for a classification. But, this ignores the fact that in many languages, tokens have different meaning because they occur in different contexts in the parse. If you ignore the parse tree context that the token occurs within, you may as well just use Textmate<\/a>.<\/p>\n\n\n\n

Note, a Textmate<\/a> language spec is an alternative solution, but Textmate grammars are difficult to implement. A Textmate pattern is either a single regular expression, or a \u00e2\u20ac\u009cdual-level\u00e2\u20ac\u009d regular expression pattern. As far as I know, there are no tools to convert a context-free grammar with class patterns into a Textmate specification. My gut feeling is that an XPath expression into a \u00e2\u20ac\u009cdual-level\u00e2\u20ac\u009d pattern, but not always.<\/p>\n\n\n\n

Putting it all together: Java as an example<\/h2>\n\n\n\n

Let’s start with the Java grammar<\/a>. To get this extension to work, you will need to clone the grammars-v4 repo, and build a parser for the Java grammar.<\/p>\n\n\n\n

git clone https:\/\/github.com\/antlr\/grammars-v4\ncd grammars-v4\/java\/java\ntrgen\ndotnet build\ncd ..\/..<\/code><\/pre>\n\n\n\n
Next, we need to set up the three ~\/.grammar-*<\/code> files. In ~\/.grammar-location<\/code>:<\/p>\n\n\n\n
c:\/users\/foobar\/documents\/grammars-v4\/java\/java<\/code><\/pre>\n\n\n\n
In ~\/.grammar-classes<\/code>:<\/p>\n\n\n\n
class\nproperty\nvariable\nmethod\nkeyword\nstring\n<\/code><\/pre>\n\n\n\n
Next, we need to clone the extension code, build it, then run VSCode.<\/p>\n\n\n\n
In ~\/.grammar-classifiers<\/code>:<\/p>\n\n\n\n
\/\/classDeclaration\/IDENTIFIER\n\/\/fieldDeclaration\/variableDeclarators\/variableDeclarator\/variableDeclaratorId\/IDENTIFIER\n\/\/variableDeclarator\/variableDeclaratorId\/IDENTIFIER\n\/\/methodDeclaration\/IDENTIFIER\n\/\/(ABSTRACT | ASSERT | BOOLEAN | BREAK | BYTE | CASE | CHAR | CLASS | CONST | CONTINUE | DEFAULT | DO | DOUBLE | ELSE | ENUM | EXTENDS | FINAL | FINALLY | FLOAT | FOR | IF | GOTO | IMPLEMENTS | IMPORT | INSTANCEOF | INT | INTERFACE | LONG | NATIVE | NEW | PACKAGE | PRIVATE | PROTECTED | PUBLIC | SHORT | STATIC | STRICTFP | SUPER | SWITCH | SYNCHRONIZED | THIS | THROW | THROWS | TRANSIENT | TRY | VOID | VOLATILE | WHILE)\n\/\/(DECIMAL_LITERAL | HEX_LITERAL | OCT_LITERAL | BINARY_LITERAL | HEX_FLOAT_LITERAL | BOOL_LITERAL | CHAR_LITERAL | STRING_LITERAL | NULL_LITERAL)\n<\/code><\/pre>\n\n\n\n
Finally, we need to clone the extension code, build it, then run VSCode.<\/p>\n\n\n\n
git clone https:\/\/github.com\/kaby76\/uni-vscode.git\ncd uni-vscode\ndotnet build\ncd VsCode\nbash install.sh\ncode .<\/code><\/pre>\n\n\n\n
Semantic highlighting<\/h2>\n\n\n\n
I tried it out for Java, and have five classes of symbols defined. Here’s how the editor displays a Java source file with the standard Java extension.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
This is how the editor displays the same Java source file with the “universal language server”.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Hover, Select, Defs\/Refs<\/h2>\n\n\n\n
This implementation does support mouse hover pop-ups. The information displayed is just the classes that the symbol belongs to.<\/p>\n\n\n\n
Select (aka “TextDocument\/DocumentHighlight”) only selects one symbol because there is no symbol table implemented. Similarly, the Defs\/Refs of a symbol only fetch the one symbol selected.<\/p>\n\n\n\n
Implementation problems<\/h2>\n\n\n\n