Hello..
Yes I think you are correct in your recollection. I seem to remember this building as well. I wasn't taught in it but I seem to recall having to take some books or something to a teacher there.
It was away from the main school. I think it was at the top of the boys playground up near the clinic. Try putting your question on the main messageboard on this site, it may get further confirmation

Are you thinking about Victoria Hall? When I was at the school we used to to go Victoria Hall not too far from Ruislip Manor Station and I remember Bunces sweet shop nearby in particular. We went there in year l of the junior school. This Hall was demolished about a year ago.

By the boys playground were "The Huts", which when I was there still had the area where the coal was stored by them. I think they were built from Asbestos. There is still an annex there, but more like a portacabin.