Get the default include paths from Clang

When porting our code to build for Win32 with Clang, on macOS, I got a bunch of intrinsics errors:

/usr/local/opt/llvm/bin/lld-link: error: dlib.lib(image_2.o): undefined symbol: __cpuid

/usr/local/opt/llvm/bin/lld-link: error: dlib.lib(image_2.o): undefined symbol: _mm_setr_epi16

/usr/local/opt/llvm/bin/lld-link: error: dlib.lib(image_2.o): undefined symbol: _mm_set1_epi32

/usr/local/opt/llvm/bin/lld-link: error: dlib.lib(image_2.o): undefined symbol: _mm_load_si128

This error is a bit unusual (for me at least) so it threw me off, since I had obviously linked against the dlib.lib before, only difference, it was using CL.exe at that time.

This isn't due to a missing library, but in fact a missing include.
And as it turns out, the order of includes.

The compiler checks the include file(s) to see what symbols to put in there, and if it recognises it, it outputs the correct intrinsic function in the code. Otherwise, it's unresolved, and will later become undefined.

And, in order to make the code a bit more resilient towards compiler version updates, I wanted to grep the include paths in a simple fashion on both OSX/Linux:

macOS:

$ /usr/local/opt/llvm/bin/clang++ -Wp,-v -x c++ - -fsyntax-only < /dev/null 2>&1 | grep -e /clang

 /usr/local/Cellar/llvm/6.0.1/lib/clang/6.0.1/include

Linux:

$ clang++ -Wp,-v -x c++ - -fsyntax-only < /dev/null 2>&1 | grep -e /clang

 /usr/local/lib/clang/6.0.0/include

Compile cctools for Windows

I wanted to try building cctools for Windows,

I first tried (and failed) with cctools-port
Next, I tried (and succeeded) with osxcross and Cygwin (32 and 64 bit).

Also note that I needed no iOS/OSX SDK for this!

Here's a brief recap on what I had to do...

Memory allocation debugging

Sometimes it's beneficial to see the actual memory allocations done by an app, but you don't want to code a full memory allocation system. Perhaps you cannot even rebuild the app.

Then a good option is to use a dynamic library to override all the allocation functions.

The idea is simple, make sure the application finds your library before any other, and let it use the custom malloc-functions. Here is the source

On MacOS, you'll use

    $ DYLD_INSERT_LIBRARIES=libmemprofile.dylib ./a.out

and on Linux:

    $ LD_PRELOAD=libmemprofile.so ./a.out

The result will look like this:

$ DYLD_INSERT_LIBRARIES=libmemprofile.dylib ./a.out

Memory used: 

   Total: 512 bytes in 2 allocation(s)

   At exit: 0 bytes in 0 allocation(s)

Arduino + NodeMCU / ESP8266

To program for the ESP8266, you need to install some packages for the Arduino program/computer.

To add the ESP boards to the board list in Arduino, you

• Add the line http://arduino.esp8266.com/stable/package_esp8266com_index.json into the preferences pane, in the "Additional Boards Manager URLs"
• Then open Tools -> Board -> Boards Manager and install ESP8266

You can now select your proper board from the list.

Next, you need to make sure you can select the proper port it's connected to. To do that, you install the driver from here

Then, you choose the port named "/dev/cu.SLAB_USBtoUART"

Now you're ready to build and upload as usual for your NodeMCU board!

Sources:
https://github.com/esp8266/Arduino
http://www.instructables.com/id/Quick-Start-to-Nodemcu-ESP8266-on-Arduino-IDE/
https://roboindia.com/tutorials/nodemcu-amica-esp8266-board-installation

Debugging your arduino

While I'm waiting for my first OLED screen to arrive, I decided to learn more about the debugging options available for the Arduino.

I found 4 alternatives:

• Hardware
• Emulators
• Print
• Drivers

In the end the option most attractive to me at this point is to add debugging drivers to my code.
But I'll list the other options first...

Hardware:

There are a few variants out there, costing ctom $10-$100+:
Atmega328P Xplained Mini, AVR Dragon, JTAGICE2, AVR One, JTAGICE3

The cheapest one is the Atmega328P Xplained Mini at ~$10. Unfortunately, it seems very tied to the Atmel Studio which is only supported on Windows.

Emulators:

I found 2 emulators, each with similar feature sets, but one seems a bit legacy at this point:

• Simulavr - git (legacy) - howto
• Simavr - git

Both emulators have the option to sample the output pins, and log the values into a .vcd file, which can be displayed with gtkwave.

Print:

It's of course important to mention that you can do print debugging too when you need it

Drivers:

After stumbling across this post by Jan Dolinay, I wanted to see it it worked for my cheap arduino.
I turned out it was farily easy to build it. There was a few caveats (mentioned by Jan in his post) regarding the serial interface, but in essence, you cannot use the Serial class since the debugger uses the serial port. But there is a debug_message(const char* msg) that can be used instead.

Build notes:

• Add the avr8-stub.c to your build, add avr8-stub.h to your main.cpp
• Overwrite the WInterrupts.c in your core library
• Call debug_init() in your setup() function
• [Optional] Set programmatic breakpoints using breakpoint()
• Add flags "-Og" and "-gdwarf-2"

Debugging:

Once you've built the .hex file and uploaded it, it's time to run avr-gdb:

• Upload the myprog.hex file with avrdude
• Run debugger with, connecting to your serial port:
• avr-gdb -ex "target remote /dev/cu.wchusbserial1410" -b 115200 myprog.elf

And, that's it! Now you can debug using breakpoints as usual.

Happy Debugging!

Sublime Text 3 custom build

Adding a custom build step to Sublime Text 3 is easy.

Simply choose

"Tools -> Build System -> New Build System"

and it will create a new .sublime-build file that you can configure to your liking.

How to configure it, can be found in the Configuration manual.

In my case, I wanted to invoke the build.sh file in the project root directory:

// The initial version

{
    "cmd": ["sh", "build.sh"],
    "working_dir": "${project_path:${folder}}"
}

Build output

The final output

Next, I needed to capture the output from the compiler, and in this case, it's from the arduino compilers. Unfortunately, the ansi output they produce couldn't be handled by either Sublime Text 3 or the plugin ANSIescape. The only solution I managed to conjure up was to install ansifilter and use that.

$ brew install ansifilter

And, since the "cmd" tag doesn't support more than one command, and because "shell_cmd" doesn't support "working_dir", I ended up with this:

// Final version
{
    "shell_cmd": "cd ${project_path:${folder}} && ./build.sh 2>&1 | ansifilter",
    "file_regex": "^(..[^:]*):([0-9]+):?([0-9]+)?:? (.*)$",
    "syntax": "Packages/Makefile/Make Output.sublime-syntax"
}

Note that when using the "shell_cmd", it create a shell for each build. So if you do something time consuming things in your bash startup scripts, it will take extra long time to build in Sublime as well.



You can read more on the subject here

Command line Arduino builds

Now that I have the Arduino up and running, I knew I wanted to use my editor of choice and command line build tools.

There is of course the alternative of using the Arduino IDE as a command line tool, but it still felt too clunky for my needs.

Secondly, there is the Arduino Makefile project, which would be a better fit, was it not... you know... make. And I saw a few results for CMake when googling, don't get me started.

So, again, I set out to create my first built scripts.
I looked at these instructions and changed it slightly for my needs. I also looked here.

$ brew tap osx-cross/avr
$ brew install avr-libc avr-gdb
$ brew install avrdude --with-usb

After that, I looked at the verbose output from the Arduino IDE to figure out most settings. And I also compared them to other packages (Arduino-make etc).

In the end, I got this build.sh

Monitor

To communicate with the Arduino (e.g. displaying the debug output), I use "screen" on osx (On windows there's PuTTY).

You quit "screen" with ^A+K:

$ screen /dev/cu.wchusbserial1410 9600

Foot note:

I compared the turnaround time between my script and Arduino-make:

./build.sh:

$ time ./build.sh
real 0m3.264s
user 0m0.185s
sys 0m0.100s

Arduino-make:

$ time (make -f ../Makefile && make -f ../Makefile verify_size reset do_upload)
real 0m6.275s
user 0m1.224s
sys 0m1.120s

That's 3 seconds longer. That's going to get annoying real fast! Of course you get a ton of extra features, but I just don't need them.

How to map your Parallels Desktop folder into OSX

A simple reminder for me how to do this:

$ ln -s /Users/mawe/Documents/Parallels/Windows\ 10.pvm/Windows\ Disks/C/msys64/home/mawe/work /Users/mawe/work/win10

My first Arduino

I recently got hooked by the game Shenzhen I/O which I've played and enjoyed a lot with my friends. And, after a while, I got curious about learning some (really) simple electronics. So I picked up a subscription to TronClub. But, after a little while, my coding brain wanted to skip to the fun stuff: Creating something that was really working and useful in very few bytes, and also to get to do some soldering!

So, I got onto www.aliexpress.com and ordered some components, first of which arrived today was my new Arduino Uno (or rather a $3 knock off from China). Installing the Arduino IDE was simple enough, but getting it to talk to my arduino was not so simple.

I'm using a mac, and those security settings can be a little bit in the way sometimes. In this case, it seems the USB protocols didn't support the china boards usb chips. Luckily, after much hassle (I won't mention those details, to not confuse the instructions for you), I found this driver that worked for me, by just installing:

https://blog.sengotta.net/signed-mac-os-driver-for-winchiphead-ch340-serial-bridge/

After installation, I rebooted and restarted the Arduino IDE, and now, voilà, the "Tools -> Ports -> Serial Ports" lists "/dev/cu.wchusbserial1420" as a proper port. And now I can upload new sketches.

My first hashtable, part 2

Since the implementation of my first version of my hash table, I've discovered a few things:

Key Size - The implementation was really good for small keys (e.g. uint64_t), but much worse for larger ones.

Api - The "empty key" api didn't really sit well with me. Even though it was only exposed to the user in the class declaration.

So, I began reiterating my implementations, testing and benchmarking (which is a lot of fun), and after a while I came up with a result I'm quite satisfied with. Not only is the API nicer, but the performance is better; much more uniform, and in most cases faster than my previous implementation! And at ~270 lines of code, I think it's a pretty good alternative to the more common implementations out there :)

Code: https://github.com/JCash/containers

Here are some samples from the benchmarks:

My first hash table

Recently I realized I needed a hash table where I had complete control of the memory ownership.
Previously, as a placeholder, I just used a regular std::unordered_map, but now I had come to a point where it just needed to be replaced. I wanted to be able to reload my engine, while keeping the data intact in memory (Handmade Hero style).

Since I had never written one myself, I though this might be a fun exercise/challenge.
Even though the wheel is already invented, doesn't mean you can't have fun building a new one, learn something from it, or even try to improve on it. Right?

One common way to implement a hash table is to use "chaining", which means that you use a table to do the initial lookup, and then use a linked list to resolve hash collisions. Another way is to use "open addressing", where you use a "probing sequence" to find an empty slot, in case of hash collisions.

TL;DR: The code is available on GitHub

An example of the benchmarks found at the GitHub repo

OSX + Parallels Desktop + Windows keyboard

I just bought a Windows keyboard to help with my coding(and gaming) but as you can imagine, not everything went without a hitch. In particular, I had troubles with the Command and Option keys due to my muscle memory of working with a mac for so long :)

Using the OSX default keymapper simply didn't work, since there you could only map Command (both) to Option and vice versa. And if you did that, the AltGr inside Parallels Desktoip Windows stopped working (and although you could map them back in Parallels Desktop, it was the same issue there, you mapped both left and right keys as one!)

The solution was to use Karabiner, formerly known as KeyRemap4MacBook. In there you could distinguish between left and right keys!

So I mapped Command Left to Options Left and vice versa, but now I had the issue of it not working in the Windows VM. Then I installed SharpKeys, which could do the same, map the keys back!

So now I have a working setup for both OS'es, and if I find a new issue, I'm pretty confident these two programs can solve that.

Install SSH server on Windows 7

Since I don't want to maintain several Parallels Desktop VM's on each of my machines (due to disc space and maintenance), I need to be able to ssh into them, but also use the same shared folder where I have my code.

Bitwise SSH Server

When googling for windows ssh servers, the majority of the answers were pretty old, but I didn't really mind that as long as they worked ok. I tried installing OpenSSH via cygwin (which was easy), but that gave me a cygwin prompt which didn't have all my user's environment variables and it also behaved erratically (and occasionally hung). I also looked into using the freeSSHD, and that had a nice UI to change settings and add users. But it just never let me log on (Access Denied), despite trying all the tricks google gave me.

In the end I limited my searches for the latest year, and at the top it suggested the Bitwise SSH Server. It's free for personal & non commercial use and it was really easy to setup. Run the installer, and then add a user in the UI settings. Et voila! Everything worked out of the box!

Shared Network Folders

The setup I have is that I have my code on my MBP, the Windows VM on an iMac. So I want to login in to the VM, and there I want to compile my code using MSVS.

Parallels Desktop is good at sharing Mac & Samba folders with the VM, but my problem is still with the ssh login: It cannot access the mapped drives!

If I run $> net use the drives are there, but I can't cd into them! They're listed as "Network: Parallels Desktop Folders" and their "status" fields are empty.

However, when I run $> net use L: \\10.0.1.42\shared /user:MACBOOKPRO-AB12\mawe the drive mounts nicely (after password confirmation). And fortunately, the Bitwise ssh server remembers mapped shares by default (if they're mapped as Microsoft Windows Network) so the next time you log in, the drive is still there.

Yay!

I've heard about the SSH server issues on windows before, but with such a great tool, I definitely think those days are gone. I am now a happy multi platform coder again!

xxHash64 - a fast 64 bit hash function

In my work I often use various hash functions as hashes or checksums, and it's always beneficial to know about different qualities and speed of the functions available.

Yann Collet (known for the LZ4 compression algorithm) has now updated his xxHash algorithm with a 64 bit version, and it's speed is quite excellent. On my machine, a 64bit MBP @ 3GHz:
xxHash64            - ~13.5 GB/s
CityHash128         - ~12.5 GB/s
CityHash64          - ~11.5 GB/s
MurmurHash3_x64_128 - ~5.5 GB/s

The function has been tested against the smhasher suite, and passing all tests. Although the tests are more geared towards 32 bit hashes, the well known MurmurHash3 function is also tested against this suite.

The XXH64 is implemented mainly for 64 bit platforms, so it's performance is not as good on 32 bit systems. But then you probably would use XXH32 anyways.

I'm sure more tests will reveal more of the hash quality in the future.

For code and speed comparisons, check it out at https://github.com/Cyan4973/xxHash

Installing Parallels Tools on Debian 7

This entry is about remembering how I solved this issue (as well as boost some traffix for the blogger that solved it)

Parallels Tools is a package designed to make various host/guest tools work: Clipboard, shared folders etc.

For some reason though, the Parallels' way of installing it fails. And it turns out to be a mount + security problem.

First off, the iso wouldn't mount when choosing the "Virtual Machine -> Install Parallels Tools". So I had to manually find it using the "Devices -> Cd/Dvd -> Connect Image..." and browsing to "/Applications/Parallels\ Desktop.app/Contents/Resources/Tools/prl-tools-lin.iso"

But, it still wouldn't work. So, after some googling I followed this guide: Install Parallels Tools on a Debian Virtual Machine

In short these commands (on the guest) solved the issue once I got the iso mounted:

Eclipse + SSH

I work a lot across multiple systems of various OS'es and configurations, and usually I do it via ssh. However, almost everytime, the tasks get much slower than they ought to be, due to the fact that the remote system doesn't have your local tools and configs.

Since I mostly use Eclipse on my local system, and since it's plugin based, I thought of searching for such plugins, and imagine my surprise that it actually exists!

It's built in, and it's called Remote System Explorer.

C++ / Python / C# speed

The beginning

Even though there are many language tests out there on the internet, I wanted to have a go at it myself, to find out the cases I wanted to know about and to have control over the test. Also, I wanted to do it because I thought it might be fun to write a little testing framework and reporting tools! (And it would make some nice weekend programming)

But even though this might be a bit whimsical, one real question I've had for some time is "how fast/slow is Python compared to C#" (Or C++ even) ? So I went ahead and tested the languages I come across most often: C++, Python & C#.

If you want to skip to the results, here they are:

http://jcash.github.io/languagetests/

And here is the code:

https://github.com/JCash/languagetests

Compacting Virtual Box Drives

Too many times I've forgotten the exact commands on how to compact the virtual drives. Especially for windows, where the tool used, used to have a different name of the argument passed to it. Compacting the drive can save you lots of disc space (last time my drive went from 12GB to 5GB), and for a smaller lap top drive, that makes a big difference.

Windows Guest:
Download SDelete from SysInternals and then run

Linux Guest:
Here, we create a file that fills the entire empty space, and then deleting it:
Host:
Finally, you let VirtualBox compact the drive

That's it really! I hope I'll remember it a little better this time.

Further info:
Use “sdelete -z” when Shrinking a Windows Guest’s Virtual Hard Drive
How to Compact a VirtualBox Ubuntu Guest’s VDI file

Qt Plugin Cache

When upgrading an application to use a newer Qt version (4.8), I stumbled across a weird issue with the sql plugin not loading correctly. Even though I triple checked the paths, it didn't work.

This was surprising since if I used the previous Qt version (4.7), my app worked fine!

After much trouble shooting and testing and googling, it turns out that there is a thing called the "Qt Plugin Cache"!

git (msys) + merge + araxis

I struggled with setting up the git environment for my windows environment.
And since it's easy to forget these things, I'll just add the command lines for it here: