{"id":2527,"date":"2020-01-30T09:02:39","date_gmt":"2020-01-30T14:02:39","guid":{"rendered":"http:\/\/codinggorilla.com\/?p=2527"},"modified":"2020-02-07T05:50:47","modified_gmt":"2020-02-07T10:50:47","slug":"textmate-regular-and-cfgs","status":"publish","type":"post","link":"http:\/\/165.227.223.229\/index.php\/2020\/01\/30\/textmate-regular-and-cfgs\/","title":{"rendered":"Tagging, TextMate, regular and context-free grammars"},"content":{"rendered":"\n<p>This is a question I recently asked on a Gitter conversation about LSP and colorization. It&#8217;s worth copying it here because there are a number of problems and solutions.<\/p>\n\n\n\n<p><!--more--><\/p>\n\n\n\n<p><em>Hi Folks, I have an LSP client\/server for Antlr for VS2019, but it does not perform &#8220;colorization&#8221; because&nbsp;[<a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.VisualStudio.LanguageServer.Protocol\/\" class=\"ek-link\">server<\/a>] and [<a class=\"ek-link\" href=\"https:\/\/www.nuget.org\/packages\/Microsoft.VisualStudio.LanguageServer.Client\/\">client<\/a>] do not implement version LSP 3.15 that specifies colorization. In fact, colorization of symbols has been in the LSP spec for a couple of years&#8211;since 3.6&#8211;yet it hasn&#8217;t been implemented, so I doubt it will ever be implemented. The 2-year old documentation ]<a class=\"ek-link\" href=\"https:\/\/docs.microsoft.com\/en-us\/visualstudio\/extensibility\/adding-an-lsp-extension?view=vs-2019\">here<\/a>] gives a glossy hint to use TextMate. However, TextMate tmlanguage files encode the grammar of a language using an (unreadable) XML format, whereas the grammar of the language is already encoded (in a clean, human-readable grammar using Antlr) in the LSP server. That&#8217;s the whole point of LSP. And all I really want is to register with VS a &lt;symbol type, color&gt; list. Does anyone have a simple example of an LSP client for VS 2019 with a minimal TextMate spec that specifies just the colorization? The alternative seems to be to create an MEF classification tagger that communicates directly with my LSP server. Thanks for the info. &#8211; Ken<\/em><\/p>\n\n\n\n<p>Later on, I note this:<\/p>\n\n\n\n<p><em>I think I figured out how to add colorized tagging to my LSP client extension. Using TextMate for tagging is probably the wrong idea&#8211;all syntax is encoded in the server, and duplication of the grammar (somehow) in a TextMate .tmLanguage file is hard. I tried two existing Antlr .tmLanuguage files ([<a href=\"https:\/\/github.com\/madskristensen\/TextmateBundleInstaller\/blob\/master\/src\/Bundles\/antlr\/syntaxes\/Antlr.tmLanguage\" class=\"ek-link\">here<\/a>] and [<a href=\"https:\/\/github.com\/divanburger\/st2-antlr4\/blob\/master\/Antlr4.tmLanguage\" class=\"ek-link\">here<\/a>]), and while colorized tagging does occur, they both do a poor job&#8211;ANTLRv4Lexer.g4 is all green; in ANTLRv4Parser.g4, some nonterminals are the wrong color. While it is possible to approximate the CFG rules for symbols with a regular grammar, it&#8217;s not easy. That is the point of LSP. Indeed, the better solution is to write another ITaggerProvider\/ITagger. If I can find an existing tagger in buffer.Properties[] or something in the LSP client API that gives the symbols in the file, that would be great. Otherwise, I will have to directly query the LSP server the symbols for the file.<\/em><\/p>\n\n\n\n<p>There is another possibility that could work, but I just don&#8217;t have the time for: write a tool to derive the TextMate rules automatically from the Antlr grammar.<\/p>\n\n\n\n<p><strong><em>Update Feb 1, 2020: The documentation describes a &#8220;<a href=\"https:\/\/github.com\/MicrosoftDocs\/visualstudio-docs\/blob\/master\/docs\/extensibility\/adding-an-lsp-extension.md#middle-layer\" class=\"ek-link\">Middlelayer<\/a>&#8221; that can be used to intercept messages between client and server. It seems possible to write a classifier using the information captured from the server. It turns out that after many attempts to get an ILanguageClientCustomMessage class to work, I just happened to use instead  ILanguageClientCustomMessage2, a completely undocumented interface type, and for which there are no examples in Github that use it. But, it works!<\/em><\/strong><\/p>\n\n\n\n<p><strong><em>Update Feb 3, 2020: I tried ILanguageClientCustomMessage2 with a &#8220;Middle Layer&#8221;, and that only sporadically calls the CanHandle() listener, and of course, never for &#8220;TextDocumentDocumentSymbol&#8221;. As a result, I opened an issue on the Github pages for the VS documentation (<a href=\"https:\/\/github.com\/MicrosoftDocs\/visualstudio-docs\/issues\/4735\" class=\"ek-link\">4735<\/a>). (I don&#8217;t think it&#8217;ll ever be fixed b\/c it&#8217;s &#8220;Pri3&#8221;. The developers will probably sit on it, low priority.) I&#8217;ll next try to create a custom message to send to the server to get symbols. But, this is getting a little ridiculous.<\/em><\/strong><\/p>\n\n\n\n<p><em><strong>Update Feb 5, 2020: I think I have a solution. Using the ILanguageClientCustomMessage2 interface, I can send a custom message to my server using the supplied JsonRpc. This method would duplicate the TextDocumentDocumentSymbol method. I can then implement GetTags() of a ITagger to colorize the text. The client API seems to work in shunting the results correctly back to my procedure call. I am now in position to implement the tagger. It is important to make sure when implementing the tagger to make sure the JsonRpc has been suppolied due to the non-deterministic order of tagger vs. LSP client creation.<\/strong><\/em><\/p>\n\n\n\n<p><em><strong>Update Feb 7, 2020: I now have colorized tagging working.<\/strong><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a question I recently asked on a Gitter conversation about LSP and colorization. It&#8217;s worth copying it here because there are a number of problems and solutions.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[],"tags":[],"_links":{"self":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/posts\/2527"}],"collection":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/comments?post=2527"}],"version-history":[{"count":0,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/posts\/2527\/revisions"}],"wp:attachment":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/media?parent=2527"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/categories?post=2527"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/tags?post=2527"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}