This little piggy is not at the market yet

With a bit of hacking for the last month or two, and I can finally see that I am making progress on Piggy, a new kind of p/invoke generator. Some might say “Why in the world are you wasting time writing a p/invoke generator? Aren’t there tools already that do this?” Well, yeah, there are other generators, but they all…how should I say…suck! I need a p/invoke generator for Campy, a compiler and runtime for C# for GPUs, which I am still working on, but had to place on the back burner to work on this. Campy uses LLVM and CUDA. Because these libraries are large and constantly changing, I have to have an automated way of handling new releases.

Continue reading “This little piggy is not at the market yet”

Re-inventing the p/invoke generator

If you’ve been programming in C# for a while, at some point you found yourself needing to call C libraries. It isn’t often, but when you have to do it, it’s like pulling teeth. One option is to set up a C++/CLI interface; the other is a p/invoke interface to a DLL containing the C code. It’s relatively easy to set up a p/invoke interface in your C# code for the C code, which you export with a DLL–if you only need to call a few C functions. But, if the API is large, you stare at the code for a while, deciding whether it is really worth writing out all the declarations you need to make the calls. Many people throw caution to the wind, write packages for large, popular C APIs so you don’t have to, which you can find on the Nuget website. One example is ManagedCuda, an API for CUDA programming from C#. Unfortunately, people get tired of trying to keep these packages up to date, and so these packages become obsolete. Another approach is through automatic means, whereby a tool reads the C++ headers (or DLL) and output the decls you call. A p/invoke generator reads C header files and outputs C# code with the p/invoke declarations that you can include in your code. These tools sometimes work, but often they don’t.

This blog entry is a “heads up” note about my thoughts for a new type of p/invoke generator. Continue reading “Re-inventing the p/invoke generator”

Porting Re2/j to C#

For these last few weeks, I’ve been trying to grapple with the problem of p/invoke–the nasty but must-use feature in C#. While one could write these declarations out by hand, some libraries are too large, and change too often, so people use p/invoke generators, like SWIG. However, the is no generator that is easy to use or generates 100% correct C# declarations. So, as every software developer does, so I go to re-invent the wheel.

Continue reading “Porting Re2/j to C#”

Useful websites for algorithm designers and programmers

I’ve been lax the last six months on my blog, working instead on Campy (a C#/GPU programming language extension). Now that that is slightly under control, time to get back to the blog. And, by the way, the whole reason for Campy is to implement popular algorithms to run on a GPU, I thought I’d take some time to review what information is available on the internet on algorithms. The following is a list I’ve been working on for a few months. It is by no means a complete list. But, I hope it covers some of the more popular sites. The entries are not in any particular order. Note, this list does not include parallel algorithms, which will be a post unto itself, nor the seemingly required AI algorithms you must have nowadays.

Continue reading “Useful websites for algorithm designers and programmers”

Windows recover tools for an idiot

I’m going to tell a story that probably many of you can relate to.

Last week, I experienced a slow motion catastrophic failure in Windows. It surprise me because ever since Windows 7, I’ve had pretty good luck in rolling forward with the updates and upgrades of the OS. But, that changed when I moved to the Windows 10 Creator Update.

As Microsoft posted in its website, “Why wait? Download Creator’s Update now.” So I did.

Unfortunately, it started with problems with an old NVIDIA Ti 470 graphics card. More importantly, I couldn’t run a Samsung Android virtual machine with VirtualBox due to a problem in a VirtualBox network adapter driver.

I tried the usual: reinstalling VBox, some registry edits of some VBox settings, and sanity checks on the OS and disk drives. But, no matter what I tried, I couldn’t get VBox working.

Biting the bullet, I decided on increasingly drastic measures: a roll back to a previous restore point; a Windows 10 Reset (keeping user files). While the VBox problem was fixed, Windows broke in the process: I couldn’t use Windows 10 Start and type in “cmd” to run a shell.

In desperation, I decided to try a full Windows 10 Reset (removing all user files), but stopped immediately when Windows prompted whether I wanted to delete all data from all drivers in the machine! Really?? Being old school, I decided just to do a format/install fresh copy of the OS instead. I know, so “old school”.

In the end, I was able to get the PC back up and restore my files. But, I learned a few things about the tools MS offers to fix a Windows.

System File Integrity

SFC — System File Checker

Verifies the validity of Windows system files.



Basic File System Integrity


Chkdsk — Checks a disk and displays a status report


This tool checks the validity of a file system. It used to check for bad sectors in a drive, but that is no longer performed by the tool.



Bad Disk Drive Checks


There are a number of free tools which check for bad sectors on a hard disk drive (HDD). You should not use these tools on solid state drives (SSD) as these checks shorten the life of the SSD.


Many of the tools are manufacturer-specific programs. For example, Seatools is specific for Seagate drives. Sandisk SSD Dashboard is specific for Sandisk. Macrorit Disk Scanner seems pretty good tool with a colorful GUI to boot, and is not manufacturer specific.


Windows trouble shooter


Windows provides some trouble shooting programs to help detect and fix problems. You can operate them from the GUI, or the command line.



 


DISM – Deployment Image Servicing and Management tool


DISM enumerates, installs, uninstalls, configures, and updates features

 and packages in Windows images.



Additional examples:


dism /online /cleanup-image /checkhealth
dism /online /cleanup-image /scanhealth




Windows Driver Verifier


Verifier.exe is a tool to check drivers installed on your system. It is a Desktop GUI application. Be very careful with this program: you can easily trash your system using the tool–from the voice of experience! Make sure to create a backup before proceeding. A good intro is here.


Recreate a Profile


Sometimes recreating a profile may solve your problems. See https://community.spiceworks.com/how_to/121165-re-create-user-profile-windows


How to Start Windows in Safe Mode with Command Prompt


Once in a while, you may need to go into “Safe Mode” when booting Windows. If the PC can boot, try this at the login screen: hold down the Shift key and click on the power button and then click on Restart. If the PC cannot boot, you must boot from a recovery disk. Make sure you do that before trouble hits! Plug in a flash drive, open Control Panel’s Recovery tool, then click Create a recovery drive.


 

	


	





	

		
Windows backups for an idiot
	


	
	

		
Today, I’m writing this note about backup procedures for Windows. Up to now, I’ve been backing up my Windows machines (operating currently five) in an ad hoc manner. Cringe you may, but I’d only made a backup once a month or so. Well, for the many decades I’ve been working with computers, I’ve never lost that much, through disk failures, unexpected file deletions, and upgrades. If a machine got trashed, I could mount the driver in another machine, boot a good OS, copy the files I’m interested in, and start afresh with a reformatted/reinstalled OS. But, my luck changed unexpectedly last week.


The recent Windows 10 Creator’s Update (which is actually an upgrade) bit me big time. And, it took me a week to recover from the mess MS dealt me.


It seems that many updates from Microsoft these days are poorly tested. Updates to a box are shoved down your throat whether you like it or not. The “new” Microsoft is repeating the lessons it should have learned with Windows Vista.


Thus, backups are more important now than ever!


Backups in Windows 10


Microsoft says that Windows 10 provides a “backup” through “file history.” But this is NOT a backup. A backup should include system files and partitions, not just your personal files. Windows 10 also provides the “Windows 7 backup”, but I really do not understand why it’s called that, and not “Windows 10 backup.” Is it going to be supported in the future? Maybe not!




Online Backups


An online backup is a good alternative. There are plenty of commercial products available, some of which were reviewed just a few days ago (http://www.pcmag.com/article2/0,2817,2288745,00.asp).  However, online backups have several problems: the services require a yearly subscription; there are limits on the size or duration of the backups saved; transferring data to and from the online service places demands on the internet link; online backup providers may not be reliable in the long run.


Backup to a spare PC on a LAN


Because I have a spare box, several large disks installed in it, with the machine attached to a gigabit LAN, I went with Acronis True Image 2017 to the spare box. The cost of Acronis was $60, which isn’t too bad. On each machine, a script is run to wake up the spare server, run Acronis True Image, then shutdown the server at the end of the backup.


The script is executed by the Windows Task Scheduler, which has the ability to wake up the computer from sleep mode, or start the process when the computer is first turned on. This script is written in Powershell. I would have preferred to write it in Bash, but WSL Bash does not work with the Windows Task Scheduler.



A simple WOL program in Net Core


To wake up the backup spare box, I wrote a “WOL” program in C# Net Core. It sends out a UDP “magic packet” to wake up the backup server when needed, which must be configured in both the BIOS and the network controller driver to accept WOL packets.



 


Enjoy.

	


	





	

		
Hello World from NET Core and NET Standard
	


	
	

		
What is this Net Core everyone is talking about? Net Core is a cross-platform application framework for C#. With Visual Studio or the Net Core tool set, it’s not too hard to build server applications on any platform that can be deployed to any platform. This is remarkable considering the source code for C#/NET was proprietary code until Nov. 2014 when Net Core and parts of Net Framework were open sourced by Microsoft.


But, doesn’t Mono/Xamarin and Unity all offer a cross-platform application framework for C# as well?? While it’s true that Mono/Xamarin work cross platform, it was written from scratch. Mono was then extended to include bindings for Android, iOS, and other runtimes, and an API for UI common across selected platforms. Unity, as well, is a fork of Mono, with an API for graphics added. It too has differences with Net Framework that have emerged over time. It is expected that Net Standard, described below, will eventually supplant much of the functionality of these different cross-platform application frameworks.


Net Core has changed quite a bit in just the last few months let alone years. The file project.json, which specified the project dependencies, frameworks, targets, and runtimes, is now deprecated. Answers to questions about setting runtimes be found in Stackoverflow.com, but are confusing since they refer to the old project.json file.


First, a small side issue: WSL (Windows for Subsystem Linux)


If you are not interested in installing Visual Studio 2017 or Net Core on Windows, you can use WSL (Windows for Subsystem Linux) to easily develop a Linux from Windows 10.


After enabling developer mode for your Windows desktop, use “lxrun.exe /uninstall” and “lxrun.exe /install” to get a clean, fresh, up-to-date version of Ubuntu running on Windows. To run the shell, start up Powershell or Cmd, then type bash.


It’s also now possible to run X Windows applications in WSL with an X server installed on your Windows 10 desktop (https://sourceforge.net/projects/xming/). See Instructions1 and Instructions2. Some useful X Windows programs would be XTerm (sudo apt install xterm), gFTP (sudo apt install gftp)–just the tip of an iceberg of wonderful software.


Installing Net Core


For Ubuntu, see the instructions at https://www.microsoft.com/net/core#linuxubuntu to install Net Core in WSL.


You can install Visual Studio 2017 with the Net Core option, and develop Net Core applications from Visual Studio if you like.


Note, I’ve had problems getting the command line tool set working sometimes. This was because of multiple Net Core installations. Unfortunately, the newest Net Core package does not uninstall the older versions, at least on Windows. Beyond that, there shouldn’t be anything more to do to install. It all should find the dotnet executable, whichever platform you use.


Writing a Hello World program


To build a “Hello World” application, type the following in a WSL bash shell:


dotnet new console
dotnet restore
dotnet run



The first line creates a “Hello World” program. This may take a while while. The second command line performs essentially a “NuGet”, downloading and installing any additional packages. The third command line compiles and runs the program. Note, the product is a dotnet-dependent CIL/MSIL executable, and requires the “dotnet” executable to run, much like how Mono works.


To build a native OS executable for specific OS target, modify the CSPROJ file with a RuntimeIdentifiers tag.


<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
   <OutputType>Exe</OutputType>
   <TargetFramework>netcoreapp1.1</TargetFramework>
   <RuntimeIdentifiers>ubuntu.16.04-x64;ubuntu.14.04-x64;win10-x64;osx.10.11-x64</RuntimeIdentifiers>
  </PropertyGroup>
</Project>



Then, use dotnet restore/build with the -r option to build the programs.


dotnet restore -r ubuntu-14.04-x64
dotnet build -r ubuntu-14.04-x64
dotnet restore -r ubuntu-16.04-x64
dotnet build -r ubuntu-16.04-x64
etc.



In the directory bin/Debug/netcoreapp1.1/ubuntu.16.04-x64/ for example, you will find the executable dn. However, the executable is dependent on the dotnet framework being installed on the deployed machine.


To build a self-containing app with the framework contained with the executable, type:


dotnet publish -r ubuntu-14.04.x64
dotnet publish -r ubuntu-16.04.x64
etc.



In the directory bin/Debug/netcoreapp1.1/ubuntu.16.04-x64/publish you will find the executable dn with all the files needed to deploy the application.


Each time you change the target runtime list, use dotnet restore to install dependencies.


Relationship with Net Standard and other frameworks


Over the years, there have been many different implementations of the “NET Framework,” which resulted in different APIs. For example, in Mono/Xamarin/Unity, the Reflection API has several differences with the NET Framework Reflection API. Net Standard is a new base class layer that provides cross compatibility of different frameworks by providing a “shim” (or in my day, a “thunk”) that maps the functionality over the frameworks. The goal is to allow libraries written in one framework to be used in another framework.


Old:




New:




 


Compatibility






.NET Standard
1
1.1
1.2
1.3
1.4
1.5
1.6
2




.NET Core
1
1
1
1
1
1
1
2




.NET Framework (with tooling 1.0)
4.5
4.5
4.5.1
4.6
4.6.1
4.6.2
vNext
4.6.1




.NET Framework (with tooling 2.0)
4.5
4.5
4.5.1
4.6
4.6.1
4.6.1
4.6.1
4.6.1




Mono
4.6
4.6
4.6
4.6
4.6
4.6
4.6
vNext




Xamarin.iOS
10
10
10
10
10
10
10
vNext




Xamarin.Android
7
7
7
7
7
7
7
vNext




Universal Windows Platform
10
10
10
10
10
vNext
vNext
vNext




Windows
8
8
8.1









Windows Phone
8.1
8.1
8.1









Windows Phone Silverlight
8













Undoubtedly, you have probably seen the above figure showing the compatibility of the various NET Frameworks. But, this illustration simplifies what is actually required. To create a Net Standard library, you must compile for the appropriate Net Standard framework and provide a a list of dependencies for compatibility. See NetStandardLib and NetFrameworkApp in the example I provide.


I constructed a set of examples that check the cross compatibility of Net Core and Net Framework APIs with libraries in Net Core, Framework, and Standard. You can modify it to test out different scenarios.


Similarly, strict Net Standard 1.4 libraries can be called from Xamarin Forms libraries (Profile259), but no higher version number of Net Standard.


It is possible to compile your library to Net Framework, Xamarin, etc. while forcing your code to be Net Standard compliant by adding a <TargetFrameworks> tag to your .csproj file. For example, a <TargetFrameworks>netstandard1.4;net461</TargetFrameworks> tag will generate a Net Standard and Net Framework targets for your library. You could use a higher version number for Net Standard, but you would have to provide an additional target (<TargetFrameworks>) in the CSPROJ file to target the PCL (e.g., http://codinggorilla.domemtech.com/?p=1521), which may lead to other problems.


Examples can be found at https://github.com/kaby76/NetStandardCoreFramworkExamples


Additional Information


https://www.bartwolff.com/Blog/2017/03/20/targeting-multiple-net-platforms-in-a-single-nuget-package-w


http://stackoverflow.com/questions/43109292/reference-a-net-standard-1-6-library-from-a-net-4-6-2-library/43109941


http://www.ben-morris.com/sharing-libraries-between-net-core-and-net-framework-applications/


https://andrewlock.net/understanding-net-core-netstandard-and-asp-net-core/


gigi.nullneuron.net/gigilabs/multi-targeting-net-standard-class-libraries/