Re: So where/how do you draw the line re safety?
Gloves are often dictated in suburban caching due to the possible presence of both broken glass and needles. I wear leather "Riggers" gloves. They are around $10.00 per pair and can be purchased from suppliers of industrial workwear and safety equipment. Hardware stores might have them but they tend to stock brand name gloves in about 40 different varieties
and charge like wounded bulls for them.
"Poke it with a stick" is a maxim that applies to many geocaches throughout the world. Dangerous (to humans) wildlife may be avoided by doing so, likewise moving glass or rusty cans out of your way. Prodding with a stick into say a pile of mulch or sweeping a stick horizontally and listening for the tell-tale sound of either hollow plastic or tin (ammo can) can also be an effective way of locating a cache. When prodding, please do so carefully. We don't want to puncture the lid of a previously sound container.
As others have suggested, a mirror on a telescopic rod can also be useful. Such mirrors can be found at metal working suppliers for example https://www.machineryhouse.com.au/Search?S=mirror&Filters=&AddFilter=2&FilterID=Inspection+Mirror
I bought mine in a small kit along with a telescopic magnetic pickup for around $12.00 on sale.
Given your experience I suggest that you think of caching in the bush as a hike but one with a particular goal. Wear the same clothes and footwear and carry the same gear that you normally take on a hike which will be dependent on the duration, time of year and prevailing weather and then add to that, a few bits and pieces that you'll need for geocaching. You'll probably find that you already carry most, if not all of what you'll need for caching.
Here in Victoria, bush caches tend to be difficult to get to, but easy to find. The CO's have in mind that it is truly about the journey and so that part is made challenging but then if you've proven yourself and made it to GZ, your reward comes in the form of an easy to find geocache.