WSPR CSV Processing using Pandas

4 posts / 0 new
Last post
Last seen: 6 months 2 days ago
Joined: 2012/10/17 - 03:21
WSPR CSV Processing using Pandas
Image icon wsprcsv conversion results40.76 KB

Hello All,

I hope everyone had a nice holiday break and all sorts of goodies found their way under the tree(s) :-)

Some time back, Gian Piero I2GPG wrote a Python script to convert WSPR spot epoch-timestamps to Date | Time columns within the CSV files listed on the Downloads Page. I further added a to that effort (briefly) with a few Scripts to facilitate downloading files and producing a couple plots using Pavel Demin's R-Scripts.

Current Effort
As the number of spots and CSV file sizes have expanded greatly (exploded more like), I've started looking at alternative methods for processing these monster-files while chiseling out the data I am most interested in. I created a small Python Package to restructure a raw CSV file using Pandas Data-Frames. I've made a pip install-able package if anyone is interested in testing it out. All feedback is welcome, however, I am most interested in trying to improve the performance of initial Data-Frame creation and producing the processed CSV file its working now, I just want to improve the speeds). If anyone has input on that aspect it would be greatly appreciated.

I plan to add a number of additional features (downloading, plots with Seaborn, DB Utilities ==> re-working wspr-ana basically) and will be pushing the package to PyPi for easy installation across platforms if anyone is interested. For now, it (wsprcsv) can be easily installed with one pip command; See Docs.

If there is anything folks need | want that I can add to the package, let me know and I'll see if I can integrate them. Adding specific Data-Frame sets would be a logical first step as the current product is a full 15|17 field Data-Frame, inclusive of "all rows".

I've attached a sample output converting an older CSV file.

Greg, KI7MT