API to calculate the number of locks between locations

[Greg]

Good Afternoon,

As a user for route planning, I find the resource as it is, extremely useful.

However, I have a slightly odd request. I am working on a home project (running onboard on a Raspberry Pi) that tracks us travelling around the UK in our Narrowboat, taking photos, temperature and GPS location.

What I would like to do, is store the number of locks between 2 locations. Naturally I could store this by some sort of push button, or text input, but as CanalPlan has such a comprehensive database, I was hoping to utilise this already brilliant resource.

I have found that I can post Long/Lat to gazetteer, for example:
http://canalplan.org.uk/cgi-bin/gazetteer.cgi?where=52.442799333+-1.077916

This returns Google Suggestions, in my testing first result is always correct, so that is perfect for my use, and can automate the interrogate of the HTML to get the actual place name...

What I then want to be able to do is post to named locations to canal.cgi, to give me the route plan, which includes the number of locks.

However I am struggling to post the info to canal.cgi, in the way that it will plan the route.

What I would ideally like to be able to do is post:
http://canalplan.org.uk/cgi-bin/canal.cgi?pl_start=North Kilworth Narrows&pl_end=Husband's Bosworth Tunnel (Northeast end)

And the route be planned.

I can then read off the lock value from the planned route.

Any pointers you could give me would be brilliant.

Many Thanks
Greg

[Stephen Atty]

How much of the route planning functionality do you need? If you are just going to need the distance and number of locks between two places and you need the place names close to a lat / long then the data set used by the Canalplan plugin for Wordpress may suffice as it holds the lat and long of all canalplan places and all the links between places (which includes distance and number of locks)

If you store it in a MySQL database then you can easily do geographical queries on it.

Steve

[Administrator]

CanalPlan has a bit more of an API than is obvious, it's just entirely undocumented and only used internally!

So, for example, to get a place from coordinates you can use:
http://canalplan.org.uk/cgi-bin/api.cgi?mode=nearest_place&lat=52.442799333&lng=-1.077916

This returns a block of data about the place in JSON format.

That should be easier to parse than page scraping and less load at our end!

Note that this returns a different place than the top Google suggestion, but looking at the database I think it's a better match to the actual lat/lng than the one you are getting.

There actually is a way to calculate the route - your guess wasn't very far off - using the following:
http://canalplan.org.uk/cgi-bin/canal.cgi?quickroute=yes&where=north%20kilworth%20narrows,husband's%20Bosworth%20Tunnel%20(Northeast%20end)

- this is how the search box on the left-hand menu works.   For programmatic use you can improve this by using the canalplan IDs (returned from the api call - prefix them by a $ to distinguish from names):
http://canalplan.org.uk/cgi-bin/canal.cgi?quickroute=yes&where=$4283,$jjkq

But that still involves quite a lot of HTML scraping when you get the results page.

There isn't yet a \"show the route between two places\" option to the api but I can add it quite easily.  I suggest just:
http://canalplan.org.uk/cgi-bin/api.cgi?mode=route&start=4283&end=jjkq

If this works for you I'll add it in the next couple of days.  A JSON block with a list of steps and the totals already worked out should be pretty easy.

[Greg]

JSON!!!!! :-o absolutely perfect. If you were able to implement it easily that would be fabulous (either way you will have to let me have a paypal address to send some beer money to)... but I should now be able to get what I am after via HTML.

I attach a basic code sample below, which may help others around the getting location with the JSON API (C# using Mono on Linux).

Thanks so much,
Greg

        public String GetCanalPlanLocation(String strLongitude, String strLatitude)
        {
            String strRetValue = \"Unknown\";
            String strURL = null;
            try
            {
                strURL = \"http://canalplan.org.uk/cgi-bin/api.cgi?mode=nearest_place&lat=\" + strLatitude + \"&lng=\" + strLongitude;

                Console.WriteLine(\"GetCanalPlanLocation: \" + strURL);

                String result = new System.Net.WebClient().DownloadString(strURL);
                System.Web.Script.Serialization.JavaScriptSerializer jss = new System.Web.Script.Serialization.JavaScriptSerializer();

                CanalPlan.Response dCPResponse = jss.Deserialize(result);
                if (dCPResponse != null)
                {
                    if (dCPResponse.mode == \"success\")
                    {
                        if((dCPResponse.name != null) && (dCPResponse.name != \"\")){ strRetValue = dCPResponse.name; }
                    }
                    else
                    {
                        Console.WriteLine(\"CanalPlan Status: \" + dCPResponse.mode);
                    }
                }

                dCPResponse = null;
            }
            catch (Exception ex)
            {
                Console.WriteLine(\"GetCanalPlanLocation: \" + ex.Message);
            }
            return strRetValue;
        }
    }

    public class CanalPlan
    {
        public class Response
        {
            public String name { get; set; }
            public String mode { get; set; }
            public String canal { get; set; }
        }
    }

[Administrator]

Yeah, JSON is great.  Will work up that route calculation API for you soonest.

Paypal on home page is beer money address!

[Administrator]

It took about 10 minutes, so I pushed it live.

The URL you need is:
http://canalplan.org.uk/cgi-bin/api.cgi?mode=plan&start=4283&end=jjkq

Those are the IDs you get back from the nearest_place call.

There are three items in the top level JSON: \"links\" is a list of each leg of the route, \"totals\" is a very high level summary and \"places\" is all the detail to map an ID into a place (name, size, attributes, coordinates etc) - this is to save you needing API calls to translate each of the links into English.

For sheer curiosity value, this is the code that generates it:

define function Plan_Route(cg as lookup)
  database input system.config.main_data
  variable toret as lookup = {links="", totals={distance=0, locks=0},places={}}
  for i=route "'$'+cg.start,'$'+cg.end",'dot'
    toret.links += i
    toret.totals.distance += i.distance
    toret.totals.locks += i.locks
    for j=each "'place1','place2'"
      if typeof(toret['places'][i[j]]) = 'string'
        toret['places'][i[j]] = placeinfo('$'+i[j])
      end if
    end for j
  end for i
  return toret
end define
   
If you want any other items in the totals I can easily accumulate them.

Note that all distances are in metres.  \"gcdist\" is the great-circle distance between the two places.  Any queries on any of the other fields just ask!

[Greg]

I can't thank you enough - beer money incoming.

Using your API accumulating a day trip at a time, I am pleased to report we have bashed our way through 267 locks in the past 60 days!

Logic below on the off chance it helps others in the future.

Once again, many thanks for all your help!
Greg

        public Int32 GetCanalPlanLocksBetween(String strStartLongitude, String strStartLatitude, String strEndLongitude, String strEndLatitude)
        {
            Int32 strRetValue = 0;
            String strURL = null;
            try
            {
                String strStartID = GetCanalPlanLocationID(strStartLongitude, strStartLatitude);
                String strEndID = GetCanalPlanLocationID(strEndLongitude, strEndLatitude);

                if ((strStartID != null) && (strEndID != null))
                {
                    strURL = \"http://canalplan.org.uk/cgi-bin/api.cgi?mode=plan&start=\" + strStartID + \"&end=\" + strEndID;

                    String result = new System.Net.WebClient().DownloadString(strURL);
                    System.Web.Script.Serialization.JavaScriptSerializer jss = new System.Web.Script.Serialization.JavaScriptSerializer();

                    CanalPlan.Response dCPResponse = jss.Deserialize(result);
                    if (dCPResponse != null)
                    {
                        if (dCPResponse.totals != null)
                        {
                            strRetValue = dCPResponse.totals.locks;
                        }
                    }

                    dCPResponse = null;
                }
            }
            catch (Exception ex)
            {
                WriteLog(\"GetCanalPlanLocation: \" + strURL + \": \" + ex.Message);
            }
            return strRetValue;
        }

        public String GetCanalPlanLocationID(String strLongitude, String strLatitude)
        {
            String strRetValue = null;
            try
            {
                String strURL = \"http://canalplan.org.uk/cgi-bin/api.cgi?mode=nearest_place&lat=\" + strLatitude + \"&lng=\" + strLongitude;

                String result = new System.Net.WebClient().DownloadString(strURL);
                System.Web.Script.Serialization.JavaScriptSerializer jss = new System.Web.Script.Serialization.JavaScriptSerializer();

                CanalPlan.Response dCPResponse = jss.Deserialize(result);
                if (dCPResponse != null)
                {
                    if (dCPResponse.mode == \"success\")
                    {
                        if ((dCPResponse.id != null) && (dCPResponse.id != \"\"))
                        {
                            strRetValue = dCPResponse.id;
                        }
                    }
                }

                dCPResponse = null;
            }
            catch (Exception ex)
            {
                WriteLog(\"GetCanalPlanLocationID: \" + ex.Message);
            }
            return strRetValue;
        }
    }

    public class CanalPlan
    {
        public class Response
        {
            public String name { get; set; }
            public String mode { get; set; }
            public String id { get; set; }
            public String canal { get; set; }

            public Totals totals { get; set; }
        }

        public class Totals
        {
            public Int32 locks { get; set; }
            public Int32 distance { get; set; }
        }
    }

A work in progress...

[Stephen Atty]

So how often do you grab the GPS and call Canalplan? Just wondering because if you have too many places between the start end end points of the plan route then you could find there's more than one possible route.

I do like the screen grab - that's a LOT of photos!

Steve

[Administrator]

Yes, the code as provided will always provide the \"shortest\" route - this (goes off and checks the source) - just counts 4 locks as one mile and then does the shortest path.

[Administrator]

This reminds me of something I tried to do in the early days of CanalPlan using a laptop, a USB connected GPS and a webcam.  In that case I was trying to establish the locations of bridges etc by looking at the snapshots and matching to the grid references.  I could never really get it to work: running both over the serial port was more than the laptop could cope with.   Also GPS was a lot worse than today's devices, even stuck on the roof it kept losing track.

I played with the idea of using some sort of AI to detect locks: there's probably a fairly distinct pattern of stopping, a few feet back and forth, then going, particularly when accompanied by some change in altitude.  But as I never really got the basic tracking working I never took this any further.

[Greg]

So it grabs the GPS location every 20 seconds when the boat is moving more than 1mph or every 15 mins when idle (and takes a photo for a pretty sped up stop capture for each day)... saves the data locally to MySQL on the Pi.

Runs 24/7 as is very low voltage.

Then at the end of each day (or when internet is available via 3G dongle - often on a garden cane stuck in a log on the roof), I run a function that draws the map for the day and geo tags the location of each pic (using google api), then send just the start and end location to Canal Plan (so 1 query a day moving forward) for the lock count.

The only slight issue I noticed (which is a rare occurrence only when we have had visitors), is if we repeat a lock by turning around, sending start and end doesn't factor this in, as I haven't told Canal Plan what we did in between...

Or for example you get to the end of the Oxford Canal, and go down Isis Lock, turn around and come back up in the same trip - I have just told Canal Plan my start and end location was before the lock, so it doesn't count it - but that is a very rare occurrence, and can just overtype when this happens.

Attached was my prototype before we set off, its in a nice box now... switch and LEDs wired to the back of the boat so I can see its working. The switch (ON) off (ON), more of a button, for taking a photo/video on demand (and photos in tunnels where GPS drops off and doesn't know if we are moving or not).

[Stephen Atty]

To get round that you'd need to send start, turn round and end point I guess.


One thing I have noticed (when playing with tracking - which Canalplan does )  was that if the GPS signal goes on a phone then it uses the mast locations so you tend up end up with some very odd bits in the route when you plot it unless you try to do some clever \"this point is too far from the last point so ignore it\" sort of logic

[Aegidian]

It took about 10 minutes, so I pushed it live.

The URL you need is:
http://canalplan.org.uk/cgi-bin/api.cgi?mode=plan&start=4283&end=jjkq

This and the rest of the JSON based API has proven to be really useful to me.

I'm in the process of beginning a boat moving business and being able to offer a quote from a web page based on lock miles is a great boon.

I had to shunt the calls to canalplan through my own server's python based CGI to get around CORS security issues, but that was the only really tricky part.

My tool: http://yourhelmsman.aegidian.org/trip.html

Please let me know how best to attribute the back-end to Canalplan, and the beer is already waiting.

[Stephen Atty]

I suspect a \"Data provided by Canalplan\" with a link back to canalplan.org.uk  would be enough.....

[Aegidian]

Easily done. Again, thank you.