Well to answer your first question about firmware, i would recomend marlin, although for a plotter you can get away with Grbl as well
As for the conexion issue with pronterface + marlin, make sure the com port is correctly set and the baud rate, thats most important, also, on my knock-off chinese mega 2560, i had to set the baud rate to 115200 in marlin instead of 250000 , for some reason it wont work with 250000 (probably due to the ch3xx serial usb controler), so try using 115200 in marlin and pronterface.
If on windows , you can also check in device manager if your mega actually supports high baud rate, chances are if it aint there as an option, it wont support it. Mine only shows max 115200