{"id":1609,"date":"2018-11-12T08:38:59","date_gmt":"2018-11-12T13:38:59","guid":{"rendered":"http:\/\/codinggorilla.domemtech.com\/?p=1609"},"modified":"2018-11-26T13:09:11","modified_gmt":"2018-11-26T18:09:11","slug":"re-inventing-the-p-invoke-generator","status":"publish","type":"post","link":"http:\/\/165.227.223.229\/index.php\/2018\/11\/12\/re-inventing-the-p-invoke-generator\/","title":{"rendered":"Re-inventing the p\/invoke generator"},"content":{"rendered":"<p>If you&#8217;ve been programming in C# for a while, at some point you found yourself needing to call C libraries. It isn&#8217;t often, but when you have to do it, it&#8217;s like pulling teeth. One option is to set up a C++\/CLI interface; the other is a p\/invoke interface to a DLL containing the C code. It&#8217;s relatively easy to set up a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Platform_Invocation_Services\">p\/invoke interface<\/a> in your C# code for the C code, which you export with a DLL&#8211;if you only need to call a few C functions. But, if the API is large, you stare at the code for a while,\u00c2\u00a0deciding whether it is really worth writing out all the declarations you need to make the\u00c2\u00a0calls. Many people throw caution to the wind, write packages for large, popular C APIs so you don&#8217;t have to, which you can find on the Nuget website. One example is <a href=\"https:\/\/github.com\/kunzmi\/managedCuda\">ManagedCuda<\/a>, an API for CUDA programming from C#. Unfortunately, people get tired of trying to keep these packages up to date, and so these packages become obsolete. Another approach is through automatic means, whereby a tool reads the C++ headers (or DLL) and output the decls you call. A <a href=\"https:\/\/en.wikipedia.org\/wiki\/Platform_Invocation_Services#Tools\">p\/invoke generator<\/a>\u00c2\u00a0reads C header files and outputs C# code with the p\/invoke declarations that you can include in your code. These tools sometimes work, but often they don&#8217;t.<\/p>\n<p>This blog entry is a &#8220;heads up&#8221; note about my thoughts for a new type of p\/invoke generator.<!--more--><\/p>\n<h2>Existing p\/invoke generators<\/h2>\n<p><a href=\"http:\/\/swig.org\/\">SWIG<\/a> is one of several p\/invoke generators. It reads C header files\u00c2\u00a0and generates code that you can add to your program. To use SWIG, you need to specify what functions to\u00c2\u00a0generate, and how to map data types between C# and C. Unfortunately,\u00c2\u00a0it doesn&#8217;t help much if the API you are trying to call changes often, forcing you to re-learn how to use SWIG&#8217;s\u00c2\u00a0byzantine\u00c2\u00a0rules. Alternatives to SWIG are <a href=\"https:\/\/github.com\/mono\/CppSharp\">CppSharp<\/a> and <a href=\"https:\/\/github.com\/Microsoft\/ClangSharp\">ClangSharp<\/a>. These tools use <a href=\"http:\/\/clang.llvm.org\/\">Clang<\/a> and hardcode <a href=\"https:\/\/clang.llvm.org\/docs\/RAVFrontendAction.html\">AST visitor code<\/a> to output finely tuned C# code. But, they aren&#8217;t extensible, and they also often get confused whether a function parameter is &#8220;in&#8221; or &#8220;out&#8221; or &#8220;ref&#8221;. Some of these tools are no longer available, adding to the &#8220;fun&#8221;.<\/p>\n<h4>SWIG<\/h4>\n<ul>\n<li><a href=\"http:\/\/www.swig.org\/\">http:\/\/www.swig.org\/<\/a><\/li>\n<li>Uses a custom compiler, which is often incompatible with newer code.<\/li>\n<li>Very general, but hard to use rules, often mysterious.<\/li>\n<li>Does not seem to handle Clang installed headers on Windows. After using\u00c2\u00a0-DCINDEX_LINKAGE -DCINDEX_DEPRECATED, I was able to get it to work.<\/li>\n<\/ul>\n<h4>CppSharp<\/h4>\n<ul>\n<li><a href=\"https:\/\/github.com\/mono\/CppSharp\">https:\/\/github.com\/mono\/CppSharp<\/a><\/li>\n<li>Written in C++.<\/li>\n<li>Uses clang to build AST.<\/li>\n<li>Uses Clang Visitors to analyze ASTs and output C# type declarations.<\/li>\n<li>Has some bells and whistles, but does not have type mapping.<\/li>\n<\/ul>\n<h4>ClangSharp<\/h4>\n<ul>\n<li><a href=\"https:\/\/github.com\/Microsoft\/ClangSharp\">https:\/\/github.com\/Microsoft\/ClangSharp<\/a><\/li>\n<li>Uses Clang to construct AST.<\/li>\n<li>Uses Clang Visitors to analyze ASTs and output C# type declarations.<\/li>\n<li>LlvmSharp contains a copy of ClangSharp which is different from ClangSharp&#8217;s official code.<\/li>\n<li>Hasn&#8217;t been maintained for a while, and requires LLVM 6.0.1 (the current version is 7.0.0).<\/li>\n<\/ul>\n<h4>P\/Invoke Interop Assistant (unavailable)<\/h4>\n<ul>\n<li>Mentioned several times in Google searches, but it appears to be no longer available.<\/li>\n<li><a href=\"https:\/\/archive.codeplex.com\/?p=clrinterop\">https:\/\/archive.codeplex.com\/?p=clrinterop<\/a><\/li>\n<li><a href=\"https:\/\/documentation.help\/SigGen\/documentation.pdf\">https:\/\/documentation.help\/SigGen\/documentation.pdf<\/a><\/li>\n<\/ul>\n<h4>Pinvoker\u00c2\u00a0(unavailable)<\/h4>\n<ul>\n<li><a href=\"http:\/\/www.pinvoker.com\/Default.aspx\">http:\/\/www.pinvoker.com\/Default.aspx<\/a><\/li>\n<li>A commercial product, but links to purchasing and downloading do not work.<\/li>\n<li>It does not support VS 2017.<\/li>\n<\/ul>\n<h4><a href=\"https:\/\/www.xinterop.com\/\" rel=\"nofollow\">xInterop C++.Net Bridge<\/a>\u00c2\u00a0(unavailable)<\/h4>\n<ul>\n<li>Written by Shawn Liu.<\/li>\n<li>Commercial product, no longer available.<\/li>\n<\/ul>\n<h3>Samples from current p\/invoke generators<\/h3>\n<h4>Generated Output of Clang&#8217;s Index.h<\/h4>\n<pre>        \/\/ Input clang-c\/Index.h\r\n        CINDEX_LINKAGE unsigned clang_defaultSaveOptions(CXTranslationUnit TU);\r\n\r\n        \/\/ CppSharp\r\n        \/\/ CS file output\r\n        public static uint ClangDefaultSaveOptions(global::clang-c.CXTranslationUnitImpl TU)\r\n        {\r\n            var __arg0 = ReferenceEquals(TU, null) ? global::System.IntPtr.Zero : TU.__Instance;\r\n            var __ret = __Internal.ClangDefaultSaveOptions(__arg0);\r\n            return __ret;\r\n        }\r\n        \/\/ ...\r\n        [DllImport(libraryPath, EntryPoint = \"clang_defaultSaveOptions\", CallingConvention = CallingConvention.Cdecl)]\r\n        public static extern uint defaultSaveOptions(CXTranslationUnit @TU);\r\n\r\n        \/\/ SWIG\r\n        \/\/ SWIG C#\r\n        public static uint clang_defaultSaveOptions(SWIGTYPE_p_CXTranslationUnitImpl TU) {\r\n           uint ret = ClangPINVOKE.clang_defaultSaveOptions(SWIGTYPE_p_CXTranslationUnitImpl.getCPtr(TU));\r\n           return ret;\/\/unsigned int\r\n        }\r\n        \/\/ SWIG C# Pinvoke\r\n        [global::System.Runtime.InteropServices.DllImport(\"gen\", EntryPoint=\"CSharp_Clang_clang_defaultSaveOptions\")]\r\n        public static extern uint clang_defaultSaveOptions(global::System.Runtime.InteropServices.HandleRef jarg1);\r\n        \/\/ SWIG C.\r\n        SWIGEXPORT unsigned int SWIGSTDCALL CSharp_Clang_clang_defaultSaveOptions(void * jarg1) {\r\n           unsigned int jresult ;\r\n           CXTranslationUnit arg1 = (CXTranslationUnit) 0 ;\r\n           unsigned int result;\r\n           arg1 = (CXTranslationUnit)jarg1; \r\n           result = (unsigned int)clang_defaultSaveOptions(arg1);\r\n           jresult = result; \r\n           return jresult;\r\n        }\r\n\r\n<\/pre>\n<h2><em>Isn&#8217;t there a better way to do this<\/em>?<\/h2>\n<p>After almost 15+ years since p\/invoke was devised, it&#8217;s surprising that there still isn&#8217;t a good solution. <a href=\"https:\/\/github.com\/kaby76\/Piggy\">Piggy<\/a> is a tool I am writing to generate p\/invoke decls using <a href=\"http:\/\/clang.llvm.org\/\">Clang<\/a> and <a href=\"http:\/\/clang.llvm.org\/docs\/IntroductionToTheClangAST.html\">AST<\/a>\u00c2\u00a0pattern matching to specify the source-to-source transformations. Like CppSharp and ClangSharp, it uses Clang&#8217;s AST to extra the data types to convert to C#. However, Piggy uses templates for AST pattern matching and output, not hardcoded C#\/C++ code.<\/p>\n<p>Source-to-source transformations with Clang is <a href=\"https:\/\/llvm.org\/devmtg\/2013-04\/krzikalla-slides.pdf\">nothing new<\/a>\u00c2\u00a0(and many more <a href=\"https:\/\/scholar.google.com\/scholar?hl=en&amp;as_sdt=0%2C22&amp;q=source+to+source+transformations+clang&amp;btnG=\">here<\/a>), but I don&#8217;t think any tool has been written that uses templates on Clang ASTs for the p\/invoke problem.<\/p>\n<h2>Clang-query<\/h2>\n<p>As pointed out by others (<a href=\"https:\/\/blogs.msdn.microsoft.com\/vcblog\/2018\/10\/23\/exploring-clang-tooling-part-2-examining-the-clang-ast-with-clang-query\/\">VSC++ blog<\/a>; <a href=\"https:\/\/eli.thegreenplace.net\/2014\/07\/29\/ast-matchers-and-clang-refactoring-tools\">Bendersky&#8217;s blog<\/a>), <a href=\"https:\/\/github.com\/llvm-mirror\/clang-tools-extra\/tree\/master\/clang-query\">Clang-query<\/a> is a developer tool that implements an AST pattern matching language and automaton to find specific nodes in the AST.<\/p>\n<p>To view a Clang AST, run<\/p>\n<pre>clang -cc1 -ast-dump test.h<\/pre>\n<p>or<\/p>\n<pre>clang -Xclang -ast-dump test.h<\/pre>\n<p>To view the reconstructed text from the AST,<\/p>\n<pre>clang -cc1 -ast-print test.h<\/pre>\n<p>While Clang-query is OK for some tree pattern matching, I found it to be limited in what I can express.<\/p>\n<ul>\n<li>How do I express a pattern to find an enumDecl() that has an enumContantDecl() with a specific integer value? There isn&#8217;t a clear way of constructing the pattern given an AST dump.<\/li>\n<li>There is no clear identification of a node&#8217;s attributes. For example, |-LinkageSpecDecl 0x2c1fb29f918 &lt;C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.15.26726\\include\\sal.h:2361:1, line:2967:1&gt; line:2361:8 C. What is &#8220;C&#8221; referred to as in Clang-query? <em>Linkage<\/em> or <em>Name<\/em> or something else? Going to the AST Matcher specification, you ask &#8220;How do I find linkageSpecDecl&#8217;s of extern &#8216;C&#8217;?&#8221; You can only find nodes with linkageSpecDecl, but cannot make sure it is of that type, e.g., &#8220;C&#8221;, and not &#8220;C++&#8221;&#8211;which is also acceptable!<\/li>\n<\/ul>\n<p>Fundamentally, the AST Matcher language has shortcomings that cannot be overcome in the near term. The only solution is to rewrite the entire output and query of ASTs.<\/p>\n<p>The first part of the problem is to come up with a new serialized Clang AST.<\/p>\n<h2>Serializing a Clang AST<\/h2>\n<p>Piggy serializes a Clang AST from code derived from <a href=\"https:\/\/github.com\/llvm-mirror\/clang\/blob\/master\/lib\/AST\/ASTDumper.cpp\">ASTDumper.cpp<\/a> in libclang. Trees are in the form of a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tree_structure#Nested_parentheses\">parenthesized expression<\/a>. Attributes of the code are\u00c2\u00a0of the form <em>Attribute=Value<\/em>. After serialization, the tree is parsed to convert it into a normalized form that regular expressions can operate on.<\/p>\n<p>So, for this .h file, we get the following AST.<\/p>\n\n\n<h2>Piggy Templates<\/h2>\n<p>The key to writing p\/invoke declarations are AST tree templates. These templates encode the tree match and the substituting expression output. Let&#8217;s see how this might work.<\/p>\n\n<p>Matching and translation occur in a DFS post-order traversal. So, in this regard, it&#8217;s like the ClangSharp and CppSharp. Templates are matched in the order they appear. In addition, when the node is matched, the corresponding output is cached, that way results are composed bottom up. In the above example, ParmDecls are rewritten, with an exception first for <em>const wchar_t*<\/em>.<\/p>\n<p>As I said before, there&#8217;s\u00c2\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Source-to-source_compiler\">a large body of research on similar problems<\/a>\u00c2\u00a0in source-to-source transformations. <a href=\"http:\/\/rosecompiler.org\/\">Rose<\/a> is a compiler that rewrites ASTs, then outputs the source for the new AST. But, with Rose at least, there are no output templates&#8211;a translator requires one to rewrite to a new AST, then\u00c2\u00a0<a href=\"https:\/\/github.com\/rose-compiler\/rose\/tree\/master\/projects\/Jovial_to_C\">the translated source code from an AST traversal<\/a>. However, we know this problem works in ClangSharp and CppSharp with an AST visitor pattern, which is a DFS visit, and without any AST rewriting.<\/p>\n<h2>Thoughts?<\/h2>\n<p>If any of you have some thoughts on this problem, let me know. I am interested. As I said, this problem has been going on for far too long\u00c2\u00a0and should be solved right. The code is still in the beginnings with an AST serializer, AST parser, template parser, and tree regular expression matcher.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>Update &#8212; November 26, 2018: After working on this a bit more, I&#8217;ve updated the grammar and templates slightly. The code templates use three variables: <em>tree<\/em> is used to access the <span style=\"background-color: #f6d5d9;\">AST\u00c2\u00a0<\/span>nodes corresponding to the matching template pattern; <em>vars<\/em> is an associative array (a <em>Dictionary&lt;&gt;<\/em>) to get values from one code template match to another; <em>result<\/em> is a <em>StringBuffer<\/em> to append output to. An example of the use is in the source code for Piggy.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you&#8217;ve been programming in C# for a while, at some point you found yourself needing to call C libraries. It isn&#8217;t often, but when you have to do it, it&#8217;s like pulling teeth. One option is to set up a C++\/CLI interface; the other is a p\/invoke interface to a DLL containing the C &hellip; <\/p>\n<p class=\"link-more\"><a href=\"http:\/\/165.227.223.229\/index.php\/2018\/11\/12\/re-inventing-the-p-invoke-generator\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Re-inventing the p\/invoke generator&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[],"tags":[],"_links":{"self":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/posts\/1609"}],"collection":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/comments?post=1609"}],"version-history":[{"count":0,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/posts\/1609\/revisions"}],"wp:attachment":[{"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/media?parent=1609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/categories?post=1609"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/165.227.223.229\/index.php\/wp-json\/wp\/v2\/tags?post=1609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}